Difference between pages "ForPHPCMajorExpansion" and "UserAccess"

Revision as of 18:05, 13 September 2021 (view source) Hpcwiki1 dept.admin (Talk | contribs) (Importing text file)		Revision as of 20:40, 25 October 2022 (view source) Hpcwiki1 dept.admin (Talk | contribs) (→‎Windows)
Line 1:		Line 1:
−	<div class="noautonum">__TOC__</div>	+	This page provides information on methods of connecting to NJIT's HPC hosts from Linux, MacOS, and Windows platforms.
−	==ForPHPCMajorExpansion 13Sep21-12:34==	+	==Getting a Login==
		+	Faculty can obtain a login to NJIT's HPC & BD systems by sending an email to ARCS@njit.edu.  Students can obtain a login either by taking a class that
		+	uses one of the sytems or by asking their faculty adviser email ARCS@njit.edu on their behalf.  Your login and password is the same as for any NJIT AFS
		+	system.
−	== 1. Preamble ==	+	==Secure Shell (SSH)==
−	There will be a major, multi-million dollar expansion to NJIT's high performance	+
−	computing (HPC) infrastructure, scheduled to be on-line in Spring 2022.	+
−	This expansion will include:	+	You must be on the NJIT network (this includes VPN) to ssh directly into the headnode of an HPC cluster.
−	<ul>	+
−	<li>A significant increase in the number of public-access CPU cores</li>	+
−	<li>A significant increase in the number of public-access GPU cores</li>	+
−	<li>High-speed interconnects (InfiniBand) for all new nodes</li>	+
−	<li>A parallel file system with a capacity of at least a petabyte</li>	+
−	<li>Cluster management software</li>	+
−	<li>Support for the SLURM scheduler/resource manager</li>	+
−	</ul>	+
−	The purpose of this assessment is to obtain information from researchers that will be used to	+	If you are outside of the NJIT network, you cam ssh into afsconnect1.njit.edu or afsconnect2.njit.edu,
−	determine the hardware specifications of this expansion.	+	and from there ssh into the headnode of an HPC cluster.
−	By providing input, you will influence the final specifications for this expansion.	+	===Linux and MacOS===
−	Please be as informative as possible in your written responses.	+	====Linux====
−	Please complete this assessment by <strong> Wednesday 22 September 2021</strong> - assessments	+	There are different methods to open a terminal window on the various Linux distributions.  Search the web for instructions specific to the distribution
−	submitted after that date will <strong>not</strong> be included in the results.	+	you are using. Once a terminal window is opened, enter :
−	=== Defs ===	+	<pre code> localhost> ssh -X -Y ucid@HPC_HOST.njit.edu </pre>
−	<strong>Definitions</strong>	+
−	<ul>	+
−	<li>The current IST-managed high performance computing (HPC) clusters referred to in this assessment are:</li>	+
−	<ul>	+
−	<li><em>Lochness</em></li>	+
−	<ul>	+
−	<li>Public-access and privately-owned nodes, both CPU and GPU</li>	+
−	</ul>	+
−		+
−	<li><em>Stheno</em></li>	+
−	<ul>	+
−	<li>CPU and GPU nodes, owned by the Dept. of Mathematical Sciences</li>	+
−	</ul>	+
−	</ul>	+
−	<li>The expansion will include a <em>parallel file system (PFS)</em>; currently NJIT's HPC infrastructure does not have a PFS.</li>	+	You will be prompted for your password. Enter your <b>AFS</b> password.  You can omit the -X -Y if you are not using a graphic interface. Substitute "Kong" or "Stheno" for "HPC_HOST".
−	<ul>	+
−	<li>A PFS provides cluster nodes shared access to data in parallel.	+
−	It enables concurrent access to storage by multiple tasks of a parallel application, to facilitate	+
−	high-performance through simultaneous, coordinated input/output operations	+
−	between compute nodes and storage.</li>	+
−	</ul>	+
−	</ul>	+
−	== 2. Demographics ==	+	====MacOS====
−	Your NJIT position and computational research areas.	+
−	2.1 What is your NJIT position? {button}	+	Open a Terminal window. Applications -> Utilities -> Terminal
−	<ul>	+
−	<li>Faculty</li>	+
−	<ul>	+
−	<li>Tenured</li>	+
−	<li>Tenure-track</li>	+
−	<li>Non-tenure-track</li>	+
−	</ul>	+
−	<li>Academic research staff {text box}</li>	+
−	<li>Postdoc</li>	+
−	</ul>	+
−	2.1.1 What is your department {dd menu}	+	At the terminal prompt enter :
−	Newark College of Engineering	+	<pre code> localhost> ssh -X -Y ucid@HPC_HOST.njit.edu </pre>
−	<ul>	+
−	<li>Biomedical Engineering</li>	+
−	<li>Biological and Pharmaceutical Engineering</li>	+
−	<li>Civil and Environmental Engineering</li>	+
−	<li>Electrical and Computer Engineering</li>	+
−	<li>Engineering Technology</li>	+
−	<li>Mechanical and Industrial Engineering </li>	+
−	<li>Other {text box}</li>	+
−	</ul>	+
−	College of Science and Liberal Arts	+	You will be prompted for your password. Enter your <b>AFS</b> password.  You can omit the -X -Y if you are not using a graphic interface. Substitute "Kong" or "Stheno" for "HPC_HOST".
−	<ul>	+
−	<li>Aerospace Studies (AFROTC)</li>	+
−	<li>Chemistry and Environmental Science</li>	+
−	<li>Humanities</li>	+
−	<li>Mathematical Sciences</li>	+
−	<li>Physics</li>	+
−	<li>Federated Department of Biological Sciences</li>	+
−	<li>Federated Department of History</li>	+
−	<li>Rutgers/NJIT Theatre Arts Program</li>	+
−	<li>Other {text box}</li>	+
−	</ul>	+
−	Ying Wu College of Computing	+	===Windows===
−	<ul>	+
−	<li>Computer Science</li>	+
−	<li>Informatics</li>	+
−	<li>Other {text box}</li>	+
−	</ul>	+
−	Martin Tuchman School of Management	+	A free Secure Shell client can be obtained from the [http://mobaxterm.mobatek.net MobaXterm Site]. In addition to SSH, MobaXterm provides an X windows
		+	server, VNC client and other useful utilities. Download, install, and open MobaXterm.
−	College of Architecture and Design	+	[[File:MobaXterm.open.png|800px]]
−	<ul>	+
−	<li>NJ School of Architecture</li>	+
−	<li>School of Art and Design</li>	+
−	<li>Other {text box}</li>	+
−	</ul>	+
−	2.2 For approximately how long have you and/or your research group been using IST-managed high performance computing (HPC) resources? {dd menu}	+	At the terminal prompt enter :
−	<ul>	+
−	<li>Less than 6 months</li>	+
−	<li>6+ to 12 months</li>	+
−	<li>1+ to 2 years</li>	+
−	<li>2+ to 5 years</li>	+
−	<li>5+ years</li>	+
−	<li>Don't know</li>	+
−	</ul>	+
−	2.3 What is the <strong>general</strong> classification of computations for which you and/or your research	+	<pre code> localhost> ssh -X -Y ucid@HPC_HOST.njit.edu </pre>
−	group use IST-managed HPC {check all that apply}	+
−	<ul>	+
−	<li>Bioinfomatics </li>	+
−	<li>Bioinformatics </li>	+
−	<li>Biophysics </li>	+
−	<li>Computational PDE </li>	+
−	<li>Computational biophysics </li>	+
−	<li>Computational chemistry </li>	+
−	<li>Computational fluid dynamics </li>	+
−	<li>Computational physics and chemistry </li>	+
−	<li>Condensed matter physics </li>	+
−	<li>Electromagnetism, Wave propagation </li>	+
−	<li>Granular science </li>	+
−	<li>Image forensics </li>	+
−	<li>Materials research </li>	+
−	<li>Monte Carlo </li>	+
−	<li>Neural networks, genetic algorithms </li>	+
−	<li>Software verification, static analysis </li>	+
−	<li>Statistical analysis </li>	+
−	<li>Steganalysis and image forensics </li>	+
−	<li>Transportation data analysis </li>	+
−	<li>Other {text box} </li>	+
−	</ul>	+
−	2.4 Please provide a <strong>brief, specific</strong> description(s) of the computational work for which you and/or your research group use IST-managed HPC {text box} (goes in 2.3)	+	[[File:MobaXterm.ssh.initial.png|800px]]
−	== 3. Main ==	+	You will be prompted your your password. Enter your '''UCID''' password.  You can omit the -X -Y if not using a graphical interface on Lochness. Substitute "lochness" or "Stheno" for "HPC_HOST".
−	3.1 What applications, including your own code, do you run on the Lochness and/or Stheno clusters	+	==Virtual Network Computing (VNC)==
−	3.1.1 Specify an application that you run on the clusters. If the application is your own ....	+	VNC software creates an image of a remote host's desktop on the user's local workstation or laptop, permitting access to the remote host with a graphical
		+	user interface (GUI).
−	3.1.2 Importance of application	+	Using VNC, it is possible to use a GUI environment from a remote host as if you were at the console of that host. In addition, it is also possible to
−	<ul>	+	start long-running jobs, and then disconnect without your jobs terminating.
−	<li>Minimally</li>	+
−	<li>Slightly</li>	+
−	<li>Moderately</li>	+
−	<li>Very</li>	+
−	<li>Extremely</li>	+
−	</ul>	+
−	3.2 How often do you submit jobs to be run on the Lochness and/or Stheno clusters	+	===Establish a VNC Server Session on an HPC host===
−	<ul>	+
−	<li>Several times a day</li>	+
−	<li>Once daily</li>	+
−	<li>Every few days</li>	+
−	<li>Weekly</li>	+
−	<li>Monthly</li>	+
−	</ul>	+
−	3.3 Do you compile, or re-compile, your applications prior to processing	+	*Using the SSH client on your local workstation or laptop, log onto an HPC host.
−	<ul>	+	*Run ''krenew'' with the arguments below to maintain your Kerberos and AFS credentials
−	<li>Yes</li>	+	*Establish the VNC environment and set a VNC session password.
−	<ul>	+	**Enter ''vncserver''.  You will be prompted for a password, and then a session will be started.
−	<li>What compilers do you use</li>	+
−	</ul>	+
−	<li>No</li>	+
−	</ul>	+
−	3.4 What is the maximum amount of resources you typically request to run your applications	+	<pre class=code>
−	<ul>	+	hpc-host-NM ucid>: krenew -t -b -K 60
−	<li>Number of cores</li>	+	hpc-host-NM ucid>: vncserver
−	<li>Memory, GB</li>	+	You will require a password to access your desktops.
−	<li>Storage, GB</li>	+	Password:
−	</ul>	+	Verify:
−	3.5 Do you require multiple nodes to run your application(s)	+	New 'hpc-host:8 (username)' desktop is hpc-host:8
−	<ul>	+
−	<li>Yes</li>	+
−	<li>No</li>	+
−	</ul>	+
−	3.6 Do your applications mainly depend on	+	Creating default startup script /home/ucid/.vnc/xstartup
−	<ul>	+	Starting applications specified in /home/ucid/.vnc/xstartup
−	<li>Number of cores available</li>	+	Log file is /home/ucid/.vnc/hpc-host:8.log
−	<li>Amount of RAM available</li>	+	</pre>
−	<li>Mixed - depends on the application</li>	+
−	<li>Don't know</li>	+
−	</ul>	+
−	3.7 Do your application(s) make use of GPUs	+	*Note and record your assigned VNC Server desktop number.
−	<ul>	+	*This VNC session will continue to run unless the process is terminated either by the user terminating the process, or by a host reboot or other abnormal
−	<li>Yes</li>	+	operation. '''Do not start more than one VNC Server session'''.
−	<ul>	+	*Note that if the system is rebooted or your VNC session is otherwise terminated you will need to establish a new session. VNC sessions are stateless,
−	<li>What is the maximum number of GPUs your application(s) use in a single job</li>	+	which means that desktop numbers are not preserved across logins. You will get a new new desktop number at the next login.
−	</ul>	+
−	<li>No</li>	+
−	</ul>	+
−	3.8 Do your application(s) make use of parallel processing	+
−	<ul>	+
−	<li>Yes</li>	+
−	<li>No</li>	+
−	</ul>	+
−	3.9 Do your application(s) require a parallel fIle system (PFS) for optimal processing	+	*Unless you alter the files created by the VNC server, you will not be asked to set a password again.
−	<ul>	+
−	<li>Yes</li>	+
−	<li>No</li>	+
−	<li>Don't know</li>	+
−	</ul>	+
−	3.10 Do your application(s) require a high-speed, low-latency compute node interconnect (e.g., InfiniBand)	+	=== Connecting to an established VNC Server Session on an HPC host ===
−	<ul>	+
−	<li>Yes</li>	+
−	<li>No</li>	+
−	<li>Don't know</li>	+
−	</ul>	+
−	3.11 What is the typical maximum time for your most compute-intensive runs to complete	+	Independent of the operating system on your workstation or laptop, the best way to connect to an established VNC Server
−	<ul>	+	session is through an SSH "tunnel". Connecting to an established VNC server session will require both an SSH client and a VNC client.
−	<li>Several minutes</li>	+
−	<li>Several hours</li>	+
−	<li>About a day</li>	+
−	<li>Several days</li>	+
−	<li>About a week</li>	+
−	<li>Several weeks</li>	+
−	<li>Other - specify</li>	+
−	</ul>	+
−	<ul>	+
−	<li>Is the maximum time to completion that you specified above acceptable</li>	+
−	<ul>	+
−	<li>Yes</li>	+
−	<li>No</li>	+
−	<ul>	+
−	<li>What is the maximum time to completion that would be acceptable</li>	+
−	</ul>	+
−	</ul>	+
−	</ul>	+
−	3.12 What is the maximum amount of data you need to store per run, or series of runs, for post-processing	+	====Linux and MacOS====
−	<ul>	+
−	<li>Less than a few GBs</li>	+
−	<li>Between a few GBs and a TB</li>	+
−	<li>Between a TB and a PB</li>	+
−	<li>More than a PB</li>	+
−	</ul>	+
−	3.13 What type(s) of data do you need to store - choose all that apply	+	Using the SSH client on the local computer create an SSH tunnel between the local computer and the hpc-host. The
−	<ul>	+	command is
−	<li>Numerical</li>	+
−	<li>Text</li>	+
−	<li>Images</li>	+
−	<li>Video</li>	+
−	<li>Other - specify</li>	+
−	</ul>	+
−	3.14 How frequently does the data that you store need to be accessed	+	ssh -L <<i>local-port</i>>:localhost:<<i>remote-port</i>> <i>ucid</i>@hpc-host.njit.edu
−	<ul>	+
−	<li>Several times a day</li>	+
−	<li>Once a day</li>	+
−	<li>Every few days</li>	+
−	<li>Once a week</li>	+
−	<li>Once every few weeks</li>	+
−	<li>Once a month</li>	+
−	<li>Every few months</li>	+
−	<li>About once a year</li>	+
−	<li>Other - specify</li>	+
−	</ul>	+
−	3.15 How long does this data need to be retained	+	The local port should be set to 5900 and the remote port is 5900 + (the desktop number assigned when the VNC session was established.)
−	<ul>	+
−	<li>A few days</li>	+
−	<li>A few weeks</li>	+
−	<li>A few months</li>	+
−	<li>A year</li>	+
−	<li>Several years</li>	+
−	<li>Other - specify</li>	+
−	</ul>	+
−	3.16 Other than yourself, how many individuals require access to this data	+	<pre code>
		+	localhost> ssh -L 5900:localhost:5901 ucid@hpc-host.
		+	</pre>
−	<ul>	+	=====Linux=====
−	<li>None</li>	+
−	<li>Between 1 and 5</li>	+
−	<li>Between 6 and 20</li>	+
−	<li>Other - specify</li>	+
−	</ul>	+
−	3.17 Please provide comments on how this major HPC expansion is likely to affect your research.	+	After the SSH tunnel is created, open a terminal and enter ''vncviewer localhost''. A window will appear and you will be prompted for a password. Enter
		+	the password created when the VNC session was established.
		+
		+	<pre code>localhost> vncviewer localhost
		+	</pre>
		+
		+	[[File:VNC_Authentication_VncAuth_.png]]
		+
		+	You are now connected to your VNC session.
		+
		+	=====MacOS=====
		+
		+	There are several VNC clients available for Mac. This example will use
		+	[http://sourceforge.net/apps/mediawiki/tigervnc/index.php?title=Welcome_to_TigerVNC TigerVNC].
		+	[http://sourceforge.net/projects/tigervnc/files/ Download] and install TigerVNC according to the instructions.
		+
		+	After the SSH tunnel is created, open TigerVNC.
		+
		+	Application -> TigerVNC Viewer 1.3.0
		+
		+	Enter "localhost" for the VNC viewer.
		+
		+	[[File:VNC Viewer Connection Details.png]]
		+
		+	Click "Connect". A new window will appear prompting for a password. Enter the password created when the VNC
		+	session was established. Click 'OK.'
		+
		+	[[File:VNC_authentication.png]]
		+
		+	You are now connected to your VNC session.
		+
		+	====Windows====
		+
		+	[http://mobaxterm.mobatek.net MobXterm] can be used as both an SSH and VNC client.
		+
		+	Select 'New Session.'
		+	[[File:MobaXterm.clk.nw.sess.png|100px]]
		+
		+	Choose "VNC" and click "OK"
		+
		+	[[File:MobaXterm.choose.vnc.png|100px]]
		+
		+	Enter the hostname that the VNC server is established on. The Port number is : 5900 + (the desktop number obtained when the VNC session was established).
		+	Select "Connect through SSH gateway (jump host)". Enter the hostname that the VNC server is established on for the "Gateway SSH server", 22 for the
		+	"Port", and your UCID for "User". Click "OK".
		+
		+	[[File:MobaXterm.vnc.sess.setup.1.png|100px]]
		+
		+	The first password prompt will be for the SSH tunnel. Enter your AFS password.
		+
		+	[[File:MobaXterm.vnc.sess.pwd.1.png|100px]]
		+
		+	The second password will be for "vncuser on localhost". Enter the password created when the VNC session was established.
		+
		+	[[File:MobaXterm.vnc.sess.pwd.2.png|100px]]
		+
		+	You are now connected to you VNC session.
		+
		+	[[File:MobaXterm.vnc.session.png|100px]]
		+
		+	==Ending a VNC Server session==
		+
		+	*Using ssh from your local workstation or laptop, log on to the HPC host where the vncserver is running.
		+
		+	*Enter ''vncserver -kill <desktop number>'' Where ''<desktop number>'' is the desktop number assigned when the VNC Server Session was established.
		+
		+	<pre class=Code>
		+	hpc-host-NM ucid>: vncserver -kill :8
		+	Killing Xvnc process ID 31460
		+	</pre>
		+
		+	[[Category:HPC and BD Wiki Users]]

---

Revision as of 20:40, 25 October 2022

This page provides information on methods of connecting to NJIT's HPC hosts from Linux, MacOS, and Windows platforms.

Getting a Login

Faculty can obtain a login to NJIT's HPC & BD systems by sending an email to ARCS@njit.edu. Students can obtain a login either by taking a class that uses one of the sytems or by asking their faculty adviser email ARCS@njit.edu on their behalf. Your login and password is the same as for any NJIT AFS system.

Secure Shell (SSH)

You must be on the NJIT network (this includes VPN) to ssh directly into the headnode of an HPC cluster.

If you are outside of the NJIT network, you cam ssh into afsconnect1.njit.edu or afsconnect2.njit.edu, and from there ssh into the headnode of an HPC cluster.

Linux and MacOS

Linux

There are different methods to open a terminal window on the various Linux distributions. Search the web for instructions specific to the distribution you are using. Once a terminal window is opened, enter :

 localhost> ssh -X -Y ucid@HPC_HOST.njit.edu 

You will be prompted for your password. Enter your AFS password. You can omit the -X -Y if you are not using a graphic interface. Substitute "Kong" or "Stheno" for "HPC_HOST".

MacOS

Open a Terminal window. Applications -> Utilities -> Terminal

At the terminal prompt enter :

 localhost> ssh -X -Y ucid@HPC_HOST.njit.edu 

You will be prompted for your password. Enter your AFS password. You can omit the -X -Y if you are not using a graphic interface. Substitute "Kong" or "Stheno" for "HPC_HOST".

Windows

A free Secure Shell client can be obtained from the MobaXterm Site. In addition to SSH, MobaXterm provides an X windows server, VNC client and other useful utilities. Download, install, and open MobaXterm.

At the terminal prompt enter :

 localhost> ssh -X -Y ucid@HPC_HOST.njit.edu 

You will be prompted your your password. Enter your UCID password. You can omit the -X -Y if not using a graphical interface on Lochness. Substitute "lochness" or "Stheno" for "HPC_HOST".

Virtual Network Computing (VNC)

VNC software creates an image of a remote host's desktop on the user's local workstation or laptop, permitting access to the remote host with a graphical user interface (GUI).

Using VNC, it is possible to use a GUI environment from a remote host as if you were at the console of that host. In addition, it is also possible to start long-running jobs, and then disconnect without your jobs terminating.

Establish a VNC Server Session on an HPC host

• Using the SSH client on your local workstation or laptop, log onto an HPC host.
• Run krenew with the arguments below to maintain your Kerberos and AFS credentials
• Establish the VNC environment and set a VNC session password.
  • Enter vncserver. You will be prompted for a password, and then a session will be started.

hpc-host-NM ucid>: krenew -t -b -K 60
hpc-host-NM ucid>: vncserver
You will require a password to access your desktops.
Password:
Verify:

New 'hpc-host:8 (username)' desktop is hpc-host:8

Creating default startup script /home/ucid/.vnc/xstartup
Starting applications specified in /home/ucid/.vnc/xstartup
Log file is /home/ucid/.vnc/hpc-host:8.log

• Note and record your assigned VNC Server desktop number.
• This VNC session will continue to run unless the process is terminated either by the user terminating the process, or by a host reboot or other abnormal

operation. Do not start more than one VNC Server session.

• Note that if the system is rebooted or your VNC session is otherwise terminated you will need to establish a new session. VNC sessions are stateless,

which means that desktop numbers are not preserved across logins. You will get a new new desktop number at the next login.

• Unless you alter the files created by the VNC server, you will not be asked to set a password again.

Connecting to an established VNC Server Session on an HPC host

Independent of the operating system on your workstation or laptop, the best way to connect to an established VNC Server session is through an SSH "tunnel". Connecting to an established VNC server session will require both an SSH client and a VNC client.

Linux and MacOS

Using the SSH client on the local computer create an SSH tunnel between the local computer and the hpc-host. The command is

ssh -L <local-port>:localhost:<remote-port> ucid@hpc-host.njit.edu

The local port should be set to 5900 and the remote port is 5900 + (the desktop number assigned when the VNC session was established.)

localhost> ssh -L 5900:localhost:5901 ucid@hpc-host.

Linux

After the SSH tunnel is created, open a terminal and enter vncviewer localhost. A window will appear and you will be prompted for a password. Enter the password created when the VNC session was established.

localhost> vncviewer localhost 

You are now connected to your VNC session.

MacOS

There are several VNC clients available for Mac. This example will use TigerVNC. Download and install TigerVNC according to the instructions.

After the SSH tunnel is created, open TigerVNC.

Application -> TigerVNC Viewer 1.3.0

Enter "localhost" for the VNC viewer.

Click "Connect". A new window will appear prompting for a password. Enter the password created when the VNC session was established. Click 'OK.'

File:VNC authentication.png

You are now connected to your VNC session.

Windows

MobXterm can be used as both an SSH and VNC client.

Select 'New Session.' 100px

Choose "VNC" and click "OK"

100px

Enter the hostname that the VNC server is established on. The Port number is : 5900 + (the desktop number obtained when the VNC session was established). Select "Connect through SSH gateway (jump host)". Enter the hostname that the VNC server is established on for the "Gateway SSH server", 22 for the "Port", and your UCID for "User". Click "OK".

100px

The first password prompt will be for the SSH tunnel. Enter your AFS password.

100px

The second password will be for "vncuser on localhost". Enter the password created when the VNC session was established.

100px

You are now connected to you VNC session.

100px

Ending a VNC Server session

• Using ssh from your local workstation or laptop, log on to the HPC host where the vncserver is running.

• Enter vncserver -kill <desktop number> Where <desktop number> is the desktop number assigned when the VNC Server Session was established.

hpc-host-NM ucid>: vncserver -kill :8
Killing Xvnc process ID 31460