ARCSProjects

ARCS Projects

Migration of HPC Clusters to OpenHPC

As part of the Tartan HPC Initiative NJIT supports two HPC clusters. Both HPC clusters are scheduled to migrate to the OprnHPC model]. OpenHPC is expected to be implemented on the smaller of the HPC clusters in Spring 2019, and on the entire (unified) HPC cluster by the wnd of August 2019.

SLURM Scheduler

The OpenHPC scheduler/workload manager is SLURM. The SLURM project is very well-supported and active. SLURM will replace the Grid Engine scheduler/workload manager currently in use on both HPC clusters.

SLURM supports "elastic computing": automatic access to authorized off-premise compute resources - currently Amazon Web Services and Google Cloud Platform - when local resources are inadequate.

SLURM user documemntation for NJIT users is in preparation.

PoC Cluster shaar.njit.edu

For OpenHPC migration, a development proof-of-concept (PoC) cluster has been set up for testing the installation and deployment of the components of the OpenHPC software stack.

This cluster, shaar.njit.edu, joined the Eastern Research Network (ERN) on 1/4/2019.

ERN AuriStorFS PoC Cell

NJIT will set up ern.njit.edu PoC cell, using AuriStorFS

• Use AuriStorFS free trial. This trial is extendable as long as testing progresses
• ERN will have access to all of the AuriStorFS enhancements in security, performance, scalability, capacities, access control lists, administration, and client software, relative to OpenAFS
• Initially, this cell will be accessible only by Rutgers, MSU, and NJIT
• When kinks have been worked out, all ERN members will have access to this cell

New Kong.njit.edu purchases

• Gennady Gor, Chemical and Materials Engineering, in progress
• Beth Nowadnik, Physics, in progress
• Alex Zhou, BME, preliminary stage
• Huiran Jin, ET, preliminary stage
• Data Ccience GPUs, preliminary stage
• Pier Champagne, pre-preliminary stage
• Sharon Loverde, discussion

Update Base HPC Resource

Configure and price base HPC resource including high-speed node interconnect and parallel file system, to largely replace the current Kong and Stheno clusters. HPC bsaeline resource

GITC 4320 Researcher Pods

GITC 4320 provides pods for individual research groups to install special-purpose hardware that cannot be accommodated in an HPC cluster or by use of a virtual machine.

There are currently 6 research groups each occupying a pod. There is space in GITC 4320 for about 8 additional pods.

Support for the machines in GITC 4320 ranges from none (researcher completely installs and manages) to full (ARCS staff installs and manages - this is the usual).

• Big Data Lab Hadoop/SPARK Installation (Chase Wu, CS)

Hadoop and SPARK are both big data frameworks. Each has advantages over the other in some big data analytics tasks.

• Work is underway on the setup of a 3-server Hadoop/SPARK cluster with (initially) 12 virtual machine nodes, for use by the Big Data Lab (Dr. Chase Wu, CS). The servers are installed in the GITC 4320 research pod.
• Big Data Lab support also includes the installation and management of 8 workstations in GITC 4111 used by graduate students.
• Four-server installation, Iulian Neamtiu, CS
• Hundreds of TBs shared storage

Support for CEE Transportation Group

• Install and manage three Oracle database servers
• Work with the Transportation group to set up a very large scale Oarcle server, either in AWS or locally

Re-architect MySQL Services

• Collapse the current two academic MySQL servers into one. Update provisioning.
• Upgrade MySQL from 5.5 to current, 5.7 on academic and research servers

MongoDB provisioning

• Set up framework for provisioning of MongoDB accounts, as is now done for MySQL and Oracle

Patching and rebooting (~250 hosts; 90 VM, 160 physical)

• Currently COR-12189