Difference between pages "Helppage" and "MinicondaUserMaintainedEnvs"

From NJIT-ARCS HPC Wiki
(Difference between pages)
Jump to: navigation, search
(Importing text file)
 
(Importing text file)
 
Line 1: Line 1:
Return to: [[Main Page]]
+
Miniconda is an easy to install, minimal python distribution. Users can use miniconda to create virtual python environments to manage python modules. The instructions that follow are for linux.
  
<big>'''Getting Started:'''</big>
+
==Installation==
 +
Download miniconda
 +
<pre=code>
 +
login-1-95 ~ >: wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
 +
--2020-07-29 16:24:59--  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
 +
Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.131.3, 104.16.130.3, 2606:4700::6810:8203, ...
 +
Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.131.3|:443... connected.
 +
HTTP request sent, awaiting response... 200 OK
 +
Length: 93052469 (89M) [application/x-sh]
 +
Saving to: ‘Miniconda3-latest-Linux-x86_64.sh’
  
Welcome to editing the IMAG wiki. This guide provides introductory information on the screen layout of this wiki,
+
100%[========================================================================================>] 93,052,469  31.6MB/s  in 2.8s
how to edit existing pages, create new pages, format text, upload files and images, and create links. It also
+
provides references to editing guides for more advanced users.
+
  
''Note:  to prevent spam, all users are required to log-in to the IMAG wiki in order to make edits.  To obtain
+
2020-07-29 16:25:03 (31.6 MB/s) - ‘Miniconda3-latest-Linux-x86_64.sh’ saved [93052469/93052469]
log-in information, send an email to [mailto:sabourinsm@mail.nih.gov Stephanie Sabourin]along with the following
+
</pre>
information: Your name, organization, and a short description of your relationship to IMAG/MSM Consortium.''
+
Run the installation
 +
<pre=code>
 +
login-1-96 ~ >: chmod +x Miniconda3-latest-Linux-x86_64.sh
 +
login-1-96 ~ >: ./Miniconda3-latest-Linux-x86_64.sh
 +
</pre>
  
 +
Accept the license and the default location. After python and some packages are installed you will be prompted to run conda init. Enter 'yes' at the prompt.
  
 +
When the installation is complete the following appears:
  
<big>'''What is on the screen:'''</big>
+
<pre=code>
 +
> For changes to take effect, close and re-open your current shell. <
  
If you have not changed the skin of this wiki (which can be done using the preferences section), take a few minutes
+
If you'd prefer that conda's base environment not be activated on startup,
to get oriented with what is on your screen:
+
  set the auto_activate_base parameter to false:
  
:'''Left Column:''' In the left column of this screen is a search box, a navigation box, a support box, and a
+
conda config --set auto_activate_base false
toolbox.
+
::*The '''search box''' searches for matching text on any of the pages of this wiki
+
::*The '''navigation box''' contains links for content pages to learn about upcoming events, funding announcements,
+
current research, and more.
+
::*The '''support box''' contains information to help you edit the wiki.
+
::*The '''toolbox''' contains links to special pages and allows you to upload files.
+
  
:'''Tabs Row:''' At the top of each page, there is a row of tabs of which you should become familiar wit
+
Thank you for installing Miniconda3!
 +
</pre>
  
:'''Tabs Row:''' At the top of each page, there is a row of tabs of which you should become familiar with at least
+
Since you will likely be maintaining your own virtual environments, it is recommended not to activate the base environment on startup.
the first four : article, discussion, edit, history.
+
::*The '''article tab''' is the default setting for reading a page.
+
::*The '''discussion tab''' allows for a free form commentary of the page.
+
::*The '''edit tab''' allows you to edit, preview, and save changes to a page.
+
::*The '''history tab''' will show you a history of all the changes made to the current page.
+
  
 +
<pre=code>
 +
login-1-101 ~ >: conda config --set auto_activate_base false
 +
</pre>
  
 +
Log off and log in again.
  
<big>'''Login, preferences links:'''</big>
+
==Create and Activate a Conda Virtual Environment==
  
At the top right of the screen, there are links specific to your account, including: login id, my talk, preferences
+
The following example will create a new conda environment based on python 3.7 and install tensorflow in the environment.
etc.
+
  
After obtaining your log-in ID from the system administrator. Please visit your profile page to ensure that the
+
<pre=code>
information is accurate.
+
login-1-105 ~ >: conda create --name tf python=3.7
 +
Collecting package metadata (current_repodata.json): done
 +
Solving environment: done
  
<big>'''Searching the IMAG Wiki:'''</big>
+
## Package Plan ##
  
'''A user can search the IMAG Wiki in two different ways:'''
+
  environment location: /home/g/guest24/miniconda3/envs/tf
  
:'''1.)  Search with the Search Tool'''
+
  added / updated specs:
 +
    - python=3.7
  
::The search tool is located under the "support" menu to the left hand side of the screen.
 
  
::Simply type the term you are looking to find in the search box and click "go".  You will be redirected to a page
+
The following packages will be downloaded:
showing the results of your search as well as with a bar indicating in which namespaces you have searched.
+
  
:'''2.) Search Through Categories'''
+
  <output snipped>
  
::A wiki "category" is a software feature of MediaWiki that provides automatic indexes that are useful in searching
+
Proceed ([y]/n)?y
and navigating pages.
+
  
::Clicking on the "Categories" page, found in the navigation menu located to the left hand side of the page, the
+
<output snipped>
user can see all of the existing categories currently associated with the wiki.
+
#
 +
# To activate this environment, use
 +
#
 +
#    $ conda activate tf
 +
#
 +
# To deactivate an active environment, use
 +
#
 +
#    $ conda deactivate
 +
</pre>
  
::Clicking on an existing category, the user will be presented with the titles of the pages associated with a given
+
Activate the new 'tf' environment
category.
+
  
 +
<pre=code>
 +
login-1-106 ~ >: conda activate tf
 +
(tf) login-1-107 ~ >:
 +
</pre>
 +
Install tensorflow-gpu
 +
<pre=code>
 +
(tf) login-1-107 ~ >: conda install -c anaconda tensorflow-gpu
 +
Collecting package metadata (current_repodata.json): done
 +
Solving environment: done
  
<big>'''Additional Reading Material on Wikis'''</big>
+
## Package Plan ##
  
:*General MediaWiki Handbook: http://meta.wikimedia.org/wiki/Help:Contents
+
  environment location: /home/g/guest24/miniconda3/envs/tf
  
:*Editing Information: http://meta.wikimedia.org/wiki/Help:Advanced_editing
+
  added / updated specs:
 +
    - tensorflow-gpu
  
:*Editing Examples: http://meta.wikimedia.org/wiki/Help:Wikitext_examples
+
<output snipped>
  
 +
The following packages will be SUPERSEDED by a higher-priority channel:
  
<big>'''Credits'''</big>
+
  ca-certificates                                pkgs/main --> anaconda
 +
  certifi                                        pkgs/main --> anaconda
 +
  openssl                                        pkgs/main --> anaconda
  
The style of this introductory mediawiki editing guide is adapted from the one available at
 
http://www.na-mic.org/Wiki/index.php/Editing_Guide:Getting_Started
 
  
[[Category:IMAG Wiki Help]]
+
Proceed ([y]/n)?y
 +
 
 +
<output snipped>
 +
 
 +
mkl_fft-1.1.0        | 143 KB    | ####################################################################################### | 100%
 +
urllib3-1.25.9      | 98 KB    | ####################################################################################### | 100%
 +
cudatoolkit-10.1.243 | 513.2 MB  | ####################################################################################### | 100%
 +
protobuf-3.12.3      | 711 KB    | ####################################################################################### | 100%
 +
blinker-1.4          | 21 KB    | ####################################################################################### | 100%
 +
requests-2.24.0      | 54 KB    | ####################################################################################### | 100%
 +
werkzeug-1.0.1      | 243 KB    | ####################################################################################### | 100%
 +
Preparing transaction: done
 +
Verifying transaction: done
 +
Executing transaction: done
 +
 
 +
</pre>
 +
 
 +
Check to see if tensorflow can be loaded
 +
 
 +
<pre=code>
 +
(tf) login-1-108 ~ >: python
 +
Python 3.7.7 (default, May  7 2020, 21:25:33)
 +
[GCC 7.3.0] :: Anaconda, Inc. on linux
 +
Type "help", "copyright", "credits" or "license" for more information.
 +
>>> import tensorflow as tf
 +
>>>
 +
</pre>
 +
 
 +
Simple tensorflow test program to make sure the virtual env can access a gpu. Program is called "tf.gpu.test.py."
 +
 
 +
<pre=code>
 +
import tensorflow as tf
 +
 
 +
if tf.test.gpu_device_name():
 +
 
 +
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))
 +
 
 +
else:
 +
 
 +
  print("Please install GPU version of TF")
 +
</pre>
 +
 
 +
Slurm script to submit the job
 +
 
 +
<pre=code>
 +
#!/bin/bash -l
 +
#SBATCH --job-name=tf_test
 +
#SBATCH --output=%x.%j.out # %x.%j expands to JobName.JobID
 +
#SBATCH --nodes=1
 +
#SBATCH --tasks-per-node=1
 +
#SBATCH --partition=datasci
 +
#SBATCH --gres=gpu:1
 +
#SBATCH --mem=4G
 +
 
 +
# Purge any module loaded by default
 +
module purge > /dev/null 2>&1
 +
conda activate tf
 +
srun python tf.gpu.test.py
 +
</pre>
 +
 
 +
Result:
 +
 
 +
<pre=code>
 +
Starting /home/g/guest24/.bash_profile ... standard AFS bash profile
 +
 
 +
Home directory : /home/g/guest24 is not in AFS -- skipping quota check
 +
 
 +
On host node430 :
 +
        17:14:13 up 1 day,  1:17,  0 users,  load average: 0.01, 0.07, 0.06
 +
 
 +
      Your Kerberos ticket and AFS token status
 +
klist: No credentials cache found (filename: /tmp/krb5cc_22967_HvCVvuvMMX)
 +
Kerberos :
 +
AFS      :
 +
 
 +
Loading default modules ...
 +
Create file : "/home/g/guest24/.modules" to customize.
 +
 
 +
No modules loaded
 +
2020-07-29 17:14:19.047276: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
 +
2020-07-29 17:14:19.059941: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2200070000 Hz
 +
2020-07-29 17:14:19.060093: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55ea8ebfdb90 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
 +
2020-07-29 17:14:19.060136: I tensorflow/compiler/xla/service/service.cc:176]  StreamExecutor device (0): Host, Default Version
 +
2020-07-29 17:14:19.061484: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
 +
 
 +
<ouput snipped>
 +
 
 +
2020-07-29 17:14:19.817386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
 +
2020-07-29 17:14:19.817392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0
 +
2020-07-29 17:14:19.817397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:  N
 +
2020-07-29 17:14:19.819082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/device:GPU:0 with 15064 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:02:00.0, compute capability: 6.0)
 +
Default GPU Device: /device:GPU:0
 +
 
 +
</pre>
 +
 
 +
GPU was recognized.

Revision as of 16:34, 5 October 2020

Miniconda is an easy to install, minimal python distribution. Users can use miniconda to create virtual python environments to manage python modules. The instructions that follow are for linux.

Installation

Download miniconda <pre=code> login-1-95 ~ >: wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh --2020-07-29 16:24:59-- https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.131.3, 104.16.130.3, 2606:4700::6810:8203, ... Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.131.3|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 93052469 (89M) [application/x-sh] Saving to: ‘Miniconda3-latest-Linux-x86_64.sh’

100%[========================================================================================>] 93,052,469 31.6MB/s in 2.8s

2020-07-29 16:25:03 (31.6 MB/s) - ‘Miniconda3-latest-Linux-x86_64.sh’ saved [93052469/93052469] </pre> Run the installation <pre=code> login-1-96 ~ >: chmod +x Miniconda3-latest-Linux-x86_64.sh login-1-96 ~ >: ./Miniconda3-latest-Linux-x86_64.sh </pre>

Accept the license and the default location. After python and some packages are installed you will be prompted to run conda init. Enter 'yes' at the prompt.

When the installation is complete the following appears:

<pre=code> > For changes to take effect, close and re-open your current shell. <

If you'd prefer that conda's base environment not be activated on startup,

  set the auto_activate_base parameter to false:

conda config --set auto_activate_base false

Thank you for installing Miniconda3! </pre>

Since you will likely be maintaining your own virtual environments, it is recommended not to activate the base environment on startup.

<pre=code> login-1-101 ~ >: conda config --set auto_activate_base false </pre>

Log off and log in again.

Create and Activate a Conda Virtual Environment

The following example will create a new conda environment based on python 3.7 and install tensorflow in the environment.

<pre=code> login-1-105 ~ >: conda create --name tf python=3.7 Collecting package metadata (current_repodata.json): done Solving environment: done

    1. Package Plan ##
 environment location: /home/g/guest24/miniconda3/envs/tf
 added / updated specs:
   - python=3.7


The following packages will be downloaded:

<output snipped>

Proceed ([y]/n)?y

<output snipped>
  1. To activate this environment, use
  2. $ conda activate tf
  3. To deactivate an active environment, use
  4. $ conda deactivate

</pre>

Activate the new 'tf' environment

<pre=code> login-1-106 ~ >: conda activate tf (tf) login-1-107 ~ >: </pre> Install tensorflow-gpu <pre=code> (tf) login-1-107 ~ >: conda install -c anaconda tensorflow-gpu Collecting package metadata (current_repodata.json): done Solving environment: done

    1. Package Plan ##
 environment location: /home/g/guest24/miniconda3/envs/tf
 added / updated specs:
   - tensorflow-gpu

<output snipped>

The following packages will be SUPERSEDED by a higher-priority channel:

 ca-certificates                                 pkgs/main --> anaconda
 certifi                                         pkgs/main --> anaconda
 openssl                                         pkgs/main --> anaconda


Proceed ([y]/n)?y

<output snipped>

mkl_fft-1.1.0 | 143 KB | ####################################################################################### | 100% urllib3-1.25.9 | 98 KB | ####################################################################################### | 100% cudatoolkit-10.1.243 | 513.2 MB | ####################################################################################### | 100% protobuf-3.12.3 | 711 KB | ####################################################################################### | 100% blinker-1.4 | 21 KB | ####################################################################################### | 100% requests-2.24.0 | 54 KB | ####################################################################################### | 100% werkzeug-1.0.1 | 243 KB | ####################################################################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done

</pre>

Check to see if tensorflow can be loaded

<pre=code> (tf) login-1-108 ~ >: python Python 3.7.7 (default, May 7 2020, 21:25:33) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow as tf >>> </pre>

Simple tensorflow test program to make sure the virtual env can access a gpu. Program is called "tf.gpu.test.py."

<pre=code> import tensorflow as tf

if tf.test.gpu_device_name():

   print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))

else:

  print("Please install GPU version of TF")

</pre>

Slurm script to submit the job

<pre=code>

  1. !/bin/bash -l
  2. SBATCH --job-name=tf_test
  3. SBATCH --output=%x.%j.out # %x.%j expands to JobName.JobID
  4. SBATCH --nodes=1
  5. SBATCH --tasks-per-node=1
  6. SBATCH --partition=datasci
  7. SBATCH --gres=gpu:1
  8. SBATCH --mem=4G
  1. Purge any module loaded by default

module purge > /dev/null 2>&1 conda activate tf srun python tf.gpu.test.py </pre>

Result:

<pre=code> Starting /home/g/guest24/.bash_profile ... standard AFS bash profile

Home directory : /home/g/guest24 is not in AFS -- skipping quota check

On host node430 :

        17:14:13 up 1 day,  1:17,  0 users,  load average: 0.01, 0.07, 0.06
     Your Kerberos ticket and AFS token status 

klist: No credentials cache found (filename: /tmp/krb5cc_22967_HvCVvuvMMX) Kerberos : AFS  :

Loading default modules ... Create file : "/home/g/guest24/.modules" to customize.

No modules loaded 2020-07-29 17:14:19.047276: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2020-07-29 17:14:19.059941: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2200070000 Hz 2020-07-29 17:14:19.060093: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55ea8ebfdb90 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-07-29 17:14:19.060136: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-07-29 17:14:19.061484: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1

<ouput snipped>

2020-07-29 17:14:19.817386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-07-29 17:14:19.817392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2020-07-29 17:14:19.817397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2020-07-29 17:14:19.819082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/device:GPU:0 with 15064 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:02:00.0, compute capability: 6.0) Default GPU Device: /device:GPU:0

</pre>

GPU was recognized.