HPCAndBDResources

From NJIT-ARCS HPC Wiki
Jump to: navigation, search

This document outlines high performance computing (HPC) and big data (BD) resources available to NJIT reserachers


High performance computing (HPC) computational resources

NJIT IST manages two HPC clusters:

  • Kong
    • Shared resource, open to all NJIT researchers
    • Researchers needing dedicated resources can purchase their own nodes on Kong
  • Stheno
    • Restricted to Dept. of Mathematical Sciences

Both Kong and Stheno contain both CPU and GPU (graphical processing unit) cores.

In addition to the two clusters, the shared-memory machines Gorgon (restricted-access) and Phi (general-access) are available to researchers.

This table gives specifications for the above resources.

Storage characteristics on HPC clusters are given in this table.

HPC software

A wide spectrum of scientific and engineering software, compilers, parallel programming libraries, and utilities are available for the HPC hardware infrastructure.

IST Academic and Research Computing Systems (ARCS) provides software support in the following areas

  • Installation of compilers, applications, libraries, and utilities requested by users
  • Customized scripts to aid users in their use of HPC resources
  • Assistance in debugging and optimizing code
  • Assistance in getting applications to run
  • Assistance in running parallel code
  • Assistance in working with and managing big data

This table gives the list of available "modules"; the presence of a module means that the corresponding software is available.

Much of the software listed is stored in the AFS distributed file system, which means that that software is available on Linux AFS clients in addition to the HPC clusters.

Big data (BD) resources

The shared hardware resource Horton supports both Hadoop and Spark environments for working with BD. See Hadoop_Overview.

HTCondor

HTCondor is workload management software that uses spare cycles on Linux computers to perform computationally intensive work. See HTCondor

Base resources

All NJIT researchers are allocated certain computational and storage base resources, listed in this table.

Resources beyond the base allocations may be purchased, in which case the resources are dedicated to the purchaser.

Off-premise resources

There may be instances in which the computational and storage resources that NJIT can provide are not adequate for researchers' needs. In such cases, researchers may need to arrange to have those resources provided by off-premise providers, such as the XSEDE science gateway, or computational and storage cloud resources provided by various vendors. For advice in this area, please contact IST ARCS at hpc@njit.edu.

Researcher-managed servers

In general, IST staff manage the servers located in the IST datacenters. This staff is is equipped with the professional skills needed to competently and securely manage these computers.

There are cases in which faculty researchers have legitimate reasons for partial or complete management - i.e., granted root privileges - of servers located in IST datacenters. These servers may be physical machines, or virtual machines (VMs) that are part of the IST infrastructure.

Root privileges are enabled by use of the "sudo" program. The extent of sudo privileges is determined on a case-by-case basis, and is based on the faculty member's experience and expertise, and the need for these privileges such that the faculty member can efficiently use the server.

Researchers' needs must be balanced against the need for network and system security, the availability of VMs with the requested configuration, and the level of IST support needed for the server.

Researchers needing computational and storage resources sometimes purchase this hardware and locate it laboratories and offices. The systems are usually managed by students. This practice is strongly discouraged by IST : in general, such systems are not provided the proper HVAC and power environment; are not physically secure; are not competently managed, leading to security problems; are not backed up; and are subject to abandonment when the students managing them leave NJIT.</p>

Researchers needing computational and storage resources are encouraged to consult with ARCS at hpc@njit.edu.

Networking

NJIT provides a robust wired and wireless high-speed data networking environment. The typical wired building network is supported by one or more 10Gb/s uplinks providing 1Gb/s connectivity to most desktops. The wireless network has more than 2,000 wireless access points blanketing the university's indoor and outdoor areas. The university provides network connectivity to the Internet and Internet2. The university network has a redundant high speed direct connection to AWS (Amazon Web Services) to support academic, research and administrative servers hosted in the AWS cloud.

The network is secured with external firewalls. The typical intranet servers and workstations are provided first level incoming firewall security and are provided full outgoing connectivity. Secure remote access is supplemented with software and appliance-based VPN support.

Internet Access to NJIT Servers

NJIT recognizes that academic and research applications may require flexibility, and access to the Internet beyond normal needs. Servers and applications requiring incoming Internet connectivity are provided that connectivity - consistent with the need to maintain network security - after passing a security scan.

GIT server

IST ARCS manages a GIT server for source code development and management. This server is open to all NJIT researchers.

Consultation

Consultation on HPC and BD matters is available by contacting hpc@njit.edu. See Consultation