-

This site is deprecated and will be decommissioned shortly. For current information regarding HPC visit our new site: hpc.njit.edu

Difference between pages "WorkingHPCBaseline" and "GITC4320"

From NJIT-ARCS HPC Wiki
(Difference between pages)
Jump to: navigation, search
(Importing text file)
 
(Importing text file)
 
Line 1: Line 1:
−
== Proposed High Performance Computing (HPC) Baseline Resource (HPC-BR) ==
 
−
=== Purpose of the HPC-BR ===
 
  
−
<ul>
+
==GITC 4320 Data Center==
−
<li>Provide resources, including compute power, accommodation of big
+
−
  data (BD), software infrastructure, and high-capacity internal and external
+
−
  networking; provide support in running applications efficiently</li>
+
  
−
<li>Provide a base level of infrastructure so that junior faculty can
+
===Purpose===
−
    establish a record of scholarship, leading to external funding</li>
+
The purpose of the GITC 4320 data center is to host physical machines purchased by
−
<ul>
+
researchers that meet certain criteria, and are managed by the researcher.
−
<li>The resources now available are barely adequate for current researchers and coursework</li>
+
−
</ul>
+
  
−
<li>Provide the capability of supporting leading edge research</li>
+
===Self-managed Machines===
 +
The owners of self-managed machines are responsible for <strong>all</strong> aspects of the
 +
management of those machines.
  
−
<li>Provide support in establishing collaborative research efforts
+
===Justification===
−
    with other institutions</li>
+
There are cases where NJIT's HPC or virtual infrastructure is not suitable for the
 +
computational needs of researchers, for a variety of reasons. In such cases, "bare metal"
 +
hardware, or "physical machine" is needed.
  
−
<li>Maintain the HPC infrastructure at a level consistent with the needs of
+
The data center provides:
−
    researchers and the maintenance of a competitive stance relative to NJIT's peers</li>
+
<ul>
 +
<li>Power, including UPS</li>
 +
<li>data center-grade networking</li>
 +
<li>HVAC</li>
 +
<li>Rack/pod</li>
 +
<li>Enterprise-level backups</li>
 +
<li>Physical and network security</li>
 +
<li>Self-service on-demand access
 
<ul>
 
<ul>
−
<li>The resources now available put NJIT at a disadvantage in attracting promising researchers</li>
+
<li>Researchers can install computational resources and bring them on-line without
−
 
+
coordinating with CST or waiting for DNS assignment</li>
 
</ul>
 
</ul>
−
<li>Provide an infrastructure suitable for educational purposes</li>
 
−
 
−
<li>Provide researchers a more desirable option than self-provisioning HPC
 
−
    equipment</li>
 
 
</ul>
 
</ul>
  
−
The HPC-BR would be one part of the strategy for providing HPC and BD resources:
+
===Racks/Pods===
−
<ol>
+
The basic storage unit is a pod.
−
<li>HPC-BR</li>
+
<ul>
−
<li>As needed, the capability to expand into:
+
<li>Pod dimensions: 39"H x 24"W x 43"D</li>
 +
<li>2 pods per 78"H rack</li>
 +
<li>Each pod has its own power, power distribution unit (PDU) and networking</li>
 +
<li>Lockable</li>
 
<ul>
 
<ul>
−
<li>Regional and national HPC networks</li>
+
<li>Provides physical security for sensitive data (e.g., medical, HIPAA)</li>
−
<li>Cloud HPC providers</li>
+
<li>Such security is not available in the GITC 5302 data center</li>
 
</ul>
 
</ul>
−
</ol>
+
<li>Dedicated to a single researcher, or shared by a group of researchers</li>
−
 
+
<li>Room security and locking racks suitable for research involving sensitive data</li>
−
== HPC Usage Statistics ==
+
<li>Accessible from front and back</li>
−
These charts show monthly queue (a set of nodes allocated for a specific purpose) wait times over a period of a year,
+
−
July 2017 to August 2018. They demonstrate consistent over-subscription - i.e., jobs on nodes waiting until other jobs finish
+
−
or give up enough resources so that the waiting job can be run.
+
−
 
+
−
<strong>All queues</strong>
+
−
<ul>
+
−
<li>Public, reservers, and GPU queues</li>
+
−
<li>Large spike in May 2018 hides other meaningful spikes in the following charts</li
+
 
</ul>
 
</ul>
−
[https://wiki.hpc.arcs.njit.edu/external/hpcusage/waits.all.jpg Waits, all queues]
 
−
<br />
 
  
−
<strong>Public queue</strong>
+
Research groups are allocated one or more pods for their exclusive use.
−
<ul>
+
−
<li>Used mostly by students for course work, and researchers who do not have access to a reserverd queue</li>
+
−
<li>The May 2018 spike is most likely related to end-of-semester course work</li>
+
−
</ul>
+
−
[https://wiki.hpc.arcs.njit.edu/external/hpcusage/waits.public.jpg Waits, public queue]
+
−
<br />
+
  
−
<strong>GPU queues</strong>
+
Wherever possible, researcher equipment that located in pods should be <strong>rack-mountable</strong>.
−
<ul>
+
−
<li>This chart includes the single public GPU node</li>
+
−
<li>Used mostly by researchers, but also some AI and machine learning courses</li>
+
−
<li>Frequently loaned to researchers lacking GPU or SMP resources</li>
+
−
<li>Differently-timed spikes suggest peak utilization unrelated to coursework</li>
+
−
 
+
−
<li>These nodes are used for both GPU and SMP jobs because they have the
+
−
    most CPU cores.  
+
−
<ul>
+
−
<li>If an SMP job needs all cores then GPU jobs must wait</li>
+
−
<li>If one GPU job is running and an SMP job wants all cores,
+
−
the SMP job must wait</li>
+
−
    <li>This makes it difficult to reliably use accounting data to assess GPU or SMP utilization</li>
+
−
</ul>
+
−
</ul>
+
−
[https://wiki.hpc.arcs.njit.edu/external/hpcusage/waits.gpu.jpg Waits, GPU queues]
+
−
<br />
+
  
−
<strong>Reserved non-GPU queues</strong>
+
===Physical Access===
 
<ul>
 
<ul>
−
<li>Used by researchers who own the queues</li>
+
<li>GITC 4320 is locked and alarmed</li>
−
<li>Occasionally loaned to reserachers that do not own the queue</li>
+
<li>Faculty and staff members of research groups are given card access to GITC
−
<li>Demonstrates a third pattern of peak usage</li>
+
4320</li>
 +
<li>Student members of research groups are <strong>not</strong> given card access</li>
 +
<li>Students in research groups allowed entry into GITC 4320 by faculty/staff
 +
must be accompanied at <strong>all times</strong> by that faculty/staff person</li>
 +
<li>ARCS staff are generally <strong>not</strong> available to accompany students
 +
when faculty/staff are not available. For <strong>emergency</strong> cases, contact
 +
arcs@njit.edu to see what, if any, arrangements can be made</li>
 
</ul>
 
</ul>
−
[https://wiki.hpc.arcs.njit.edu/external/hpcusage/waits.reserved.jpg Waits, Reserved non-GPU queues]
 
  
−
The HPC-BR proposal is similar to the existing HPC infrastructure in terms of number of
 
−
cores, but with more RAM, much faster filesystem and much faster node
 
−
interconnect. This will increase throughput; however the increased
 
−
throughput might be offset by increasing HPC demand. Moving to OHPC with
 
−
the SLURM scheduler will allow accurate usage metrics to be obtained.
 
−
These usage metrics will be evaluated regularly to determine demand and
 
−
the appropriate methods to address possible increases in demand.
 
  
−
SLURM will also allow unfunded researchers to automatically use reserved nodes when those
+
===Network Access===
−
nodes are not in use by their owners.
+
All machines are accessible from the NJIT network (includes VPN).
  
−
== Spring 2018 HPC and BD Survey Extracts ==
+
<strong>No</strong> machines are accessible from outside the NJIT network. This means,
−
<strong>Serial processing</strong> Of the 32 Kong respondents, 20 stated that a large or moderate increase in
+
for example, that machines in GITC 4320 <strong>cannot</strong> act as a web server
−
CPU speed is needed; 13 stated that a large or moderate increase in RAM is needed. Both are addressed by the HPC-BR.
+
that is accessible from the Internet.
−
[https://wiki.hpc.arcs.njit.edu/external/hpcusage/ExtractSerialProcessing.pdf Serial Processing]
+
−
<br />
+
  
−
<strong>Parallel processing</strong> Of the 35 Kong respondents, 21 stated that that a large or moderate increase in
+
If it is desirable to have a web server open to the Internet access data produced by
−
CPU speed is needed; 18 stated that a a large or moderate increase in RAM is needed; 16 stated that a large or
+
a machine in GITC 4320, that data can be stored in AFS, where it is accessible from
−
moderate increase in node interconnect speed is needed. All three are addressed by the HPC-BR.
+
web server running on a virtual machine in the GITC 5302 data center.
−
[https://wiki.hpc.arcs.njit.edu/external/hpcusage/ExtractParallelProcessing.pdf Parallel Processing]
+
−
<br />
+
  
−
<strong>Parallel file system</strong> Of the 45 respondents, 14 rated a PFS as very or moderately importatnt. Hovever.
+
===Agreement to Conditions===
−
the backgrround information for PFS stated that it was used for temporary storage, when in fact it is now used for all
+
Researchers wishing to locate their machines in the GITC 4320 data center are required
−
types of storage, due to recent innovations in Spectrum Scale. This information would likely have casued more respondents
+
to accept the conditions stated on this page, via email to arcs@njit.edu.
−
to rank a PFS as very or moderately importaant, since lack of staorage is often cited as a major deficiency. The 1-PB PFS in the HPC-BR adreeses
+
−
this deficiency.
+
−
[https://wiki.hpc.arcs.njit.edu/external/hpcusage/ExtractParallelFileSystem.pdf Parallel File System]
+
−
<br />
+
−
 
+
−
<strong>Non-NJIT Resources</strong> The following deficiences are addressed by the HPC-BR: CPU cores too slow, Not enough RAM per core. Node
+
−
interconnect speed too slow, Not enough GPU cores. Read/write of temporay files too slow/Storage space inadwquate.
+
−
[https://wiki.hpc.arcs.njit.edu/external/hpcusage/ExtractNonNJITResourcesMinimal.pdf Non-NJIT Resources]
+
−
<br />
+
−
 
+
−
<strong>HPC comments</strong>[https://wiki.hpc.arcs.njit.edu/external/hpcusage/ExtractHPCComments.pdf HPC comments]
+
−
<br />
+
−
 
+
−
== High-speed node interconnect and parallel file system ==
+
−
[[ IBandPFS | Roles of inetrnal network and PFS ]]
+
−
 
+
−
== Configuration of HPC-BR ==
+
−
The Configuration of HPC-BR addresses all of the followung areas:
+
−
<ul>
+
−
<li>Historical use of NJIT HPC resources</li>
+
−
<li>The need for a high speed node interconnect and a parallel file system.
+
−
<li>The need for:
+
−
<ul>
+
−
<li>High speed node interconnect. High speed node interconnect is available on the
+
−
DMS cluster, Stheno, but on the much larger general-access cluster, Kong, it is
+
−
available only to researhers who purchase it for their own use.</li>
+
−
<li>Parallel file system (PFS). The current HPC resources have no PFS.</li>
+
−
<li>Greatly expanded GPU resources</li>
+
−
<li>Minimizing space, power, and HVAC requirements</li>
+
−
</ul>
+
−
</ul>
+
−
 
+
−
<table width="80%" style="border:1px solid black; border-collapse:collapse;">
+
−
 
+
−
<caption>HPC Current vs. HPC Proposed Baseline Resource (HPC-BR)</caption>
+
−
<tr bgcolor="#dddddd">
+
−
  <th style="border:1px solid black;">Category</th>
+
−
  <th style="border:1px solid black;">Sub-category</th>
+
−
  <th style="border:1px solid black;">HPC Current - General Access</th>
+
−
  <th style="border:1px solid black;">HPC Current - Private Access<br />(Kong dedicated nodes + Stheno)</th>
+
−
  <th style="border:1px solid black;">HPC Proposed Baseline Resource<br />(HPC-BR)</th>
+
−
  <th style="border:1px solid black;">Notes</th>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;">CPU </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"></td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;"></td>
+
−
<td style="border:1px solid black;"> Nodes </td>
+
−
<td style="border:1px solid black;"> 240 </td>
+
−
<td style="border:1px solid black;"> 54 </td>
+
−
<td style="border:1px solid black;"> 20 </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;"></td>
+
−
<td style="border:1px solid black;"> Cores </td>
+
−
<td style="border:1px solid black;"> 1,896 </td>
+
−
<td style="border:1px solid black;"> 816 </td>
+
−
<td style="border:1px solid black;"> 1,760 </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;"></td>
+
−
<td style="border:1px solid black;"> RAM, TB </td>
+
−
<td style="border:1px solid black;"> 10.5 </td>
+
−
<td style="border:1px solid black;"> 8.6 </td>
+
−
<td style="border:1px solid black;"> 15 </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;">CPU with GPU </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"></td>
+
−
<td style="border:1px solid black;"></td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;"></td>
+
−
<td style="border:1px solid black;"> Nodes </td>
+
−
<td style="border:1px solid black;"> 2 </td>
+
−
<td style="border:1px solid black;"> 14 </td>
+
−
<td style="border:1px solid black;"> 5 </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;"></td>
+
−
<td style="border:1px solid black;"> GPU cores </td>
+
−
<td style="border:1px solid black;"> 9,984 </td>
+
−
<td style="border:1px solid black;"> 72,320 </td>
+
−
<td style="border:1px solid black;"> 204,800 </td>
+
−
<td style="border:1px solid black;">  About 1000 Kong cores are permanently out of service due to hardware failure </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;"></td>
+
−
<td style="border:1px solid black;"> CPU cores </td>
+
−
<td style="border:1px solid black;"> 40 </td>
+
−
<td style="border:1px solid black;"> 204 </td>
+
−
<td style="border:1px solid black;"> 240 </td>
+
−
<td style="border:1px solid black;"> GPU nodes are often used for CPU SMP jobs due to the large number of cores on the node </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;"></td>
+
−
<td style="border:1px solid black;"> RAM, TB </td>
+
−
<td style="border:1px solid black;"> 0.25 </td>
+
−
<td style="border:1px solid black;"> 2.7 </td>
+
−
<td style="border:1px solid black;">  3.8 </td>
+
−
<td style="border:1px solid black;">  </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;">CPU TFLOPS </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> 15 </td>
+
−
<td style="border:1px solid black;"> 8.5 </td>
+
−
<td style="border:1px solid black;"> 15 </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;">Node interconnect </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> 1 Gb/sec </td>
+
−
<td style="border:1px solid black;"> Kong: 13 of 10Gb/sec, 12 of 56 Gb/sec, remainder 1Gb/sec; <br />Stheno: 40 Gb/sec </td>
+
−
<td style="border:1px solid black;"> 100 Gb/sec </td>
+
−
<td style="border:1px solid black;"></td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;">Parallel file system </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> None </td>
+
−
<td style="border:1px solid black;"> None </td>
+
−
<td style="border:1px solid black;"> 1,000 TB </td>
+
−
<td style="border:1px solid black;"> Traditionally used for temporary files, PFS are now used for all kinds of storage. Researcher requests for storage are routinely 10 to 20TB, compared to 50 to 100GB two to three years ago </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;">Racks </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> 9 </td>
+
−
<td style="border:1px solid black;"> 3 </td>
+
−
<td style="border:1px solid black;"> 6 </td>
+
−
<td style="border:1px solid black;">  </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;">Power requirement, KWatts </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> 83 </td>
+
−
<td style="border:1px solid black;"> 85 </td>
+
−
<td style="border:1px solid black;"> 147 </td>
+
−
<td style="border:1px solid black;"> HPC-BR spec is for BeeGFS; spec for IBM Spectrum Scale is not known </td>
+
−
</tr>
+
−
<tr>
+
−
<td style="border:1px solid black;">Cost </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
<td style="border:1px solid black;"> - </td>
+
−
<td style="border:1px solid black;"> = </td>
+
−
<td style="border:1px solid black;"> $1.6M </td>
+
−
<td style="border:1px solid black;"> </td>
+
−
</tr>
+
−
</table>
+
−
== Cost comparisons ==
+
−
<ul>
+
−
<li>[[HPCBaselineAWS | Cost of AWS HPC-BR comparable resource]]</li>
+
−
 
+
−
<li>A purpose of the HPC-BR is to provide adequate resources for courses using HPC. In particular,
+
−
the HPC-BR is configured to adequately support the needs of the CS Data Science Master's program, which
+
−
uses GPUs extensively.
+
−
<p>
+
−
GPU cost analysis using AWS for Deep Learning course (U. Roshan, CS):
+
−
[https://docs.google.com/document/d/1drbDU5zEoC4ai2VIjqayc2ZnUxLvp7PJ__U9tXCe0_Y/edit?ts=5bf23ab4 GPU Cost Analysis]
+
−
</p>
+
−
<p>
+
−
This analysis shows an AWS cost of $90,688/year for the current MS enrollment of 77. By contrast, the GPU enhancement
+
−
proposed for the Data Science program in July 2018 would have cost about $90K, and would have had
+
−
a useful life of at least three years.
+
−
</p>
+
−
</li>
+
−
</ul>
+
−
+
−
<p></p>
+
−
----
+

Revision as of 16:59, 19 December 2020

GITC 4320 Data Center

Purpose

The purpose of the GITC 4320 data center is to host physical machines purchased by researchers that meet certain criteria, and are managed by the researcher.

Self-managed Machines

The owners of self-managed machines are responsible for all aspects of the management of those machines.

Justification

There are cases where NJIT's HPC or virtual infrastructure is not suitable for the computational needs of researchers, for a variety of reasons. In such cases, "bare metal" hardware, or "physical machine" is needed.

The data center provides:

  • Power, including UPS
  • data center-grade networking
  • HVAC
  • Rack/pod
  • Enterprise-level backups
  • Physical and network security
  • Self-service on-demand access
    • Researchers can install computational resources and bring them on-line without coordinating with CST or waiting for DNS assignment

Racks/Pods

The basic storage unit is a pod.

  • Pod dimensions: 39"H x 24"W x 43"D
  • 2 pods per 78"H rack
  • Each pod has its own power, power distribution unit (PDU) and networking
  • Lockable
    • Provides physical security for sensitive data (e.g., medical, HIPAA)
    • Such security is not available in the GITC 5302 data center
  • Dedicated to a single researcher, or shared by a group of researchers
  • Room security and locking racks suitable for research involving sensitive data
  • Accessible from front and back

Research groups are allocated one or more pods for their exclusive use.

Wherever possible, researcher equipment that located in pods should be rack-mountable.

Physical Access

  • GITC 4320 is locked and alarmed
  • Faculty and staff members of research groups are given card access to GITC 4320
  • Student members of research groups are not given card access
  • Students in research groups allowed entry into GITC 4320 by faculty/staff must be accompanied at all times by that faculty/staff person
  • ARCS staff are generally not available to accompany students when faculty/staff are not available. For emergency cases, contact arcs@njit.edu to see what, if any, arrangements can be made


Network Access

All machines are accessible from the NJIT network (includes VPN).

No machines are accessible from outside the NJIT network. This means, for example, that machines in GITC 4320 cannot act as a web server that is accessible from the Internet.

If it is desirable to have a web server open to the Internet access data produced by a machine in GITC 4320, that data can be stored in AFS, where it is accessible from web server running on a virtual machine in the GITC 5302 data center.

Agreement to Conditions

Researchers wishing to locate their machines in the GITC 4320 data center are required to accept the conditions stated on this page, via email to arcs@njit.edu.

Retrieved from "https://wiki.hpc.arcs.njit.edu/index.php?title=WorkingHPCBaseline&oldid=240"