Jump To:

• Command Equivalents (Slurm<->LSF)

• Slurm Documentation

• Examples

Attention:

As of Dec. 17, 2014 the new Tufts cluster uses slurm for job scheduling. User job types such as serial single processor, MPI multiprocessor, GUI-based interactive applications, SMP threaded applications and batch jobs may be run on compute nodes using slurm.  This means that there is no LSF bsub command on the cluster to submit jobs to queues as before.  Also there are no queues. Note,  slurm tastes different.

Overview of slurm Concepts

Slurm is an open-source workload manager designed for Linux clusters of all sizes. It provides three key functions. First it allocates exclusive and/or non-exclusive access to resources (computer nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work on a set of allocated nodes. Finally, it arbitrates contention for resources by managing an internal queue of pending work. This is done with slurm's fair-share algorithm framework, similar to LSF, but we are not using job preemption or suspension. 

Partitions vs. queues

LSF has the concept of queues.  LSF queues on the old cluster differentiates service and job types. Slurm does not have queues and instead has the concept of a partition. The new cluster, login.cluster.tufts.edu,  has five partitions: batch, interactive, gpu, largemem and mpi.  When a compute  job is submitted with slurm, it must be placed on a partition.  The batch partition is the default partition.  Each partition has default settings of various kinds, such as cores, memory,  duration, priority, threads, etc...  Explicit slurm requests is required to alter defaults.  For example, the batch partition has a default allocation of one cpu core and a maximum of 2gig.  of ram memory.     This could be used for a serial job or shell script.  Partition settings may be viewed  using the squeue command. 

salloc vs. bsub options

Another concept is that of resource allocation.  The salloc command allows one to request resources and a partition.  salloc  is available to fine tune resources for one or more submitted jobs.  Cavalier use of salloc will likely waste resources  unless it is well thought out.  Try to assess your needs carefully and not over allocate.  It is likely that the bigger the requested allocation, the longer the wait for the resources to be granted.   Associated with an allocation via the salloc command is the notion of job steps.  These steps are nested within a granted allocation.   An allocation of resources is relinquished when you are finish with your job steps(by typing exit when used interactively) or when the allocation  times out or tasks are exhausted or the use of exit in a script is executed.  Note, for most simple and straight forward slurm needs, most users will not need to use salloc.  Commands sbatch and srun can host all the typical resource requests for most jobs.  

slurm Qos  vs. queues

The old cluster used several LSF queues to provide a quality of service distinction by mechanisms such as relative job priority, hardware separation, fairshare policy rules  and preemption. This approach supported the needs of faculty contributed hardware owners.  An equivalent solution in slurm is the concept of slurm bank account and  qos.  This is optionally applied to a partition and job submission.   Most cluster users are not affected by this and can ignore it, resulting in the profile of  normal qos. Nothing special needs to be done by cluster users. The normal qos is specified as an option to  srun and sbatch commands as follows:

--account=normal   --qos=normal

Those users that are affected, and require access as define by previous LSF  queues may use the following as appropriate:

NOTE:

These account and qos pairs have elevated slurm placement privileges.

Finally, an additional special purpose QoS,  dregs, is provided to address low priority long running jobs that can withstand pre-emption and cancellation as a possible outcome.   This is only suitable for a special class of jobs.

--account=dregs  --qos=dregs

 Note: dregs is configured to have the lowest job  priority among all slurm qos/account options.  Jobs submitted with a dregs specification may be preempted by other jobs during periods of heavy cluster use.  If slurm decides to  preempt  dregs job(s), the job(s) are re-queued for submission and subject to available cluster resources.  In effect the job starts over!    Not all work loads are appropriate for this qos treatment.  Single core/low memory jobs are more likely to be suitable for dregs access and successful use.

Partition runtime duration limit:

IMPORTANT:   Your job duration estimate

Use of slurm requires a job duration estimate. A default job time limit of 15 minutes has been set since May 26, 2016. Do not confuse this setting with a slurm partition maximum duration time limit, which already exists.  Each submitted job to all partitions(except Interactive),  needs to have a time estimate of your job's duration. For example, if your job is expected to run for 1 day and 2 hours, specify that in the sbatch or srun options:

NOTE:  Failure to not include the --time specification will result in the job's termination after the default time limit of 15 minutes(plus some overhead time).  And if your job exceeds your time estimate, slurm will then kill the job.  It is important for you to figure this out. This will require some experimentation on your part. Experienced users will have some knowledge of this. However, new job initial conditions may not be well understood. If you think your job may be a long duration job, set the --time option to something close to the partition maximum time and note the resulting duration. Subsequent job submissions may then be specified with better accuracy. Note, the exact time is not needed!

Note: The gpu partition contains model K20 Nvidia gpus on Cisco nodes. The following nodes have older gpus, but cuda support is not enabled.

Unless there is a good reason to use these, we suggest using the gpu partition.

Compute nodes:

The new cluster compute nodes follow a different naming scheme from the old cluster.  This allows and simplifies cluster support logistics.  It is good practice to not hard-code any compute node names.   Compute node redundancy and support logistics benefit from node name abstraction.  This allows slurm to place jobs independent of node names.

Some command equivalents: 

Additional detail for some command equivalents:

Getting Started

Slurm documentation section and examples can be found on this page.  If you are just migrating to slurm, a one page quick overview orientation is available, Q uickStartSlurm.  Otherwise please view recent introductory slurm Brown Bag Powerpoint session files below.

slurm Documentation

• QuickStartSlurm overview
• slurm documentation is available on the slurm site.
• Useful slurm FAQs
• Man pages are available as well on the login node.
   > man -k slurm
• A comparison matrix  of  slurm, lsf and other popular scheduler commands can be found here
• Online tutorials can be found at:  http://slurm.schedmd.com/tutorials.html
• slurm mpi docs: mpi_guide
• Introduction To Slurm Brown Bag Powerpoint session file (PDF)

FairShare scheduling

Modules

Modules are required  on the cluster. These are needed to access an application and it's software environment.  Since there are often several versions of a particular software item, always check what is available.  The default module for a particular item may change.  To view what is available:

Note, please do not run application software on the login/headnode of the cluster. Instances will be terminated and a reminder sent to you.  Submit compute jobs to compute nodes with slurm.

Overview of typical use case:

• The salloc command will address your partition and hardware environment requests.
• Next,  modules corresponding to software of interest should be invoked on the command line or within batch scripts.
• Next step, submit a job to the reserved allocation.

slurm Examples:

How to view slurm partitions and additional info:

> sinfo

  For additional info:

What jobs do I have running and provide information on the following resources requested:

I have a lot of jobs pending, when will they start? 

The scontrol command is useful for many tasks.  Note, there is a difference between slurm Running and Pending jobs when it comes to controlling their state.

How may I obtain information about my job so that I can verify what I requested is indeed granted?

Is there a graphical client to view job states instead of command line tools?
> sview

How to cancel  an allocation:

Create a request for one node with one cpu.

> salloc -N1 -n1

salloc: Granted job allocation 322

> squeue

             JOBID PARTITION  NAME     USER      ST      TIME  NODES NODELIST(REASON)

               322    batch          bash        your_utln  R       0:05     1           alpha021

> scancel 322

salloc: Job allocation 322 has been revoked.

Or type:

> exit

Other scancel options for many jobs:

How to obtain a one job step xterm session in the default batch partition for 2 hours of work:

> srun --x11=first --pty  --time=0-2:00:00  xterm

or in the interactive partition:
> srun --x11=first --pty -p interactive xterm

How to send a simple command to a node:

>  srun --pty -p interactive hostname

alpha021

>

How to allocate 1 compute node with 1 core  and with an xterm window accessing the batch partition for 2 hours:

When finished type exit to release the allocation.

How to allocate 1 compute node with 4 cores in the interactive partition with 4 xterm  shells:

Note: the four xterms may be placed in a stacked fashion. This behavior is xserver defined.

Or obtain  a bash shell:

After the exit, you are back on the login node.
> hostname
login.cluster.tufts.edu
>

Comments on potential errors with  threaded parallelism

Sometimes you might see errors involving an application where the errors are related to threads, sockets, and other odd messages.  Often  threaded applications are just fine if there is one thread per cpu.  The slurm cpu definition counts enabled hyperthreads as cores/cpus.   The slurm batch partition has compute nodes with and without hyperthreads enabled. The following list of nodes are not hyperthread enabled.  To include this list in your sbatch options, your job will land on a node in the list.  This approach often cures this problem.  There are other approaches if you need to use Cisco nodes.

Workflow/job pipeline dependent  jobs

This is a simple example of dependent jobs.   Suppose you want to set off 12 slurm jobs each with 10 tasks(matlab, fortran, C++) using a slurm job array in your script so that they are placed nearly all at once onto compute nodes.   And you want  each of the remaining  11 jobs to depend  sequentially on the one before it.

See the manual page for sbatch --dependency=  options which takes several arguments.  Several types of job conditions/options may be specified to fit your needs.

Interactive and gui applications

The srun command without a preceding salloc command is a one step default allocation.  After commands are issued, typing exit will release the allocation.  The default allocation may or may not meet your needs.

To run a bash shell in the batch partition without X window forwarding support:

> srun --pty -p batch bash -i

To run a bash shell in the interactive partition:

> srun --pty  -p interactive bash -i

How to specify a qos request:

> srun --pty   -p batch --account=normal   --qos=normal    bash

Abaqus:

Note that there are several versions of Abaqus installed and the executable may or  may not have the .exe extension. 

The OpenGL -mesa switch/option may or may not be needed depending on your xserver installed on your machine. 

Note: for larger abaqus model mesh nodes and the potential to use more memory, you should adjust access by unlimiting your shell's limits:

> ulimit –s unlimited

Do this prior  to running abaqus.

Ansys/Fluent:

Note that you must load Ansys since Fluent products are part of Ansys.

Matlab:

How to run a compiled Matlab program, that you compiled using Matlab's compiler,  interactively using 4 threads:

To run a matlab gui session in the interactive partition and request a ten thread allocation:

Comments on Matlab Parallel options:

Single node Matlab MPI jobs

The previous examples use matlab threads.  To use mpi parallelism, you must create a parallel profile in Matlab. Those settings are stored in your home directory  in a subdirectory  called

.matlab/R2015b/parallel.settings

You can create several named  profiles which will be stored in parallel.settings.  Consider creating profiles for 12,16, 20 and 32 cores.  Name them separately.  Within Matlab, look for the Parallel option on the HOME tab in the Environment section.  Choose Manage Cluster Profiles and fill accordingly and save. For example, you create a profile with name,  MATLAB-12, to define 12 cores.  Make one the default.  Close Matlab. 

If you have a matlab code using matlab mpi parallel language directives, such as parfor, then you may run matlab  on 1 node with 12 cores in the following manner:

Note, it is also possible to use the batch partition instead of interactive, but placement may take longer. 

Distributed MPI Matlab jobs

Matlab's  MDCS  software is needed to allow for distributed mpi jobs.  This is already setup on compute nodes,  but  requires you to copy a small directory into your home directory.  

Follow the examples in the Parallel Computing Documentation for related matlab pool operations. 

Note:  It may be useful to open the Manage Cluster Profiles  tool and Validate a parallel profile as a test.

Maple:

To run the graphical interface to Maple using 4 threads:

Ansys:

Ansys has several gui interfaces, the launcher145 is just one.

SAS:

NOTE: Interactive use of SAS may require that you close your desktop browser(s) in order to display SAS HTML and ODS graphics. These options are altered in the SAS Preferences pull-down menu system.

Mothur:

Interactive use of mothur should be scaled to reflect your in-memory data requirements. As an example, the following will place your job on one of the interactive nodes:

The exit is needed to release the allocation and to get back to the login node of the cluster.

Stata:

Stata is best run from a bash shell.

Rstudio:

This is an interactive session using the these versions of R and Rstudio.

Note: R and RStudio does not by default make use of threads.  Support for thread parallelism  is in the R   parallel package.  Your explicit loading of it is required along with the necessary coding changes supporting  your tasks.  If you include  -c slurm option requesting cores, your use of the parallel package options must match the cores requested.

VMD:

Suppose you would like to work out of a bash shell and you have x11 graphics support:

Batch examples:

COMSOL in batch mode using the  sbatch option --wrap:

This large memory example requests from both slurm and comsol 8 threads.  The number that works best is application and solver dependent. 

Make sure both thread arguments, 4, matches  and set the expected duration time of your job.

ABAQUS in batch mode:

Create a text file script with content such as the following.  Note that the counter-intuitive option "interactive" is an abaqus option. Check the docs to decide if you need this option. The input is in the .imp  file format containing data, model, materials,etc...

-bash-4.1$ cat my_abaqus_batch.sh

Note: some documentation suggests that your slurm request for memory should be double what is asked for in an Abaqus run.  Under some conditions this can matter.


Now, to run it:
> sbatch my_abaqus_batch.sh

But what if I need to use a fortran subroutine called,  my_test.f  with this job. How do I do that?

ANSYS in batch mode:

  Assuming you have a model that is saved as a .dat ansys file, you can use it as input.  The file below was taken from the vendor set of examples.

>  find  /opt/shared/ansys_inc/v150/v150/ansys/    -name vm233.dat   -type f -print

/opt/shared/ansys_inc/v150/v150/ansys/data/verif/vm233.dat

>  cp   /opt/shared/ansys_inc/v150/v150/ansys/data/verif/vm233.dat   vm233test.dat

Create a sbatch script file called  ansys_batch.sh with a text editor(nano, nedit, vi, vim...) and use the following content.

Note: this example constrains the job to run on M3 nodes. These nodes have hyperthreads turned off; unlike the Cisco nodes.

This job is then submitted to slurm:

> chmod 755 ansys_batch.sh

> ./ansys_batch.sh

If successful, your directory should contain the following files:

Maple as batch example

Suppose you have a simple maple code file as a text file named maple_test3:

To run this file as a slurm batch job:

Note:  your actual needs for cores and memory and time need to be specified. 

The output stored in maple_tes3.output is:

SAS in "interactive-batch" mode example without a GUI:

If successful you should have a .log and .lst file with contents.

Stata in batch mode using 4 threads(large memory example requesting 16 Gig):

Simple R example for running many R programs as a slurm array

Suppose you have a directory named  serial_R_example,  with five R scripts(could be 500) and you wish to run them somewhat in "parallel" but as separate jobs so that you don't have to wait for them to finish serially.  Each file may have different input conditions, or even different goals.   Use the slurm array feature.  Create an sbatch script, say my_big_job.sh, that has the following tasks:

The above represents a request for minimal   resource needs, but that can be changed to reflect your needs.   Then to submit this to the cluster:

> sbatch  my_big_job.sh

Each of the five jobs will be launched on different nodes and your R output will be written to the same directory as the R scripts.  There are many slurm approaches  to do the same thing.

Note: there is a tool called mktemp that assists in the temporary creation of files following a template/naming convention.  This could be useful for simulation runs involving many files. For details see:

> man  mktemp

Note: Other programs such as  comsol,  abaqus, mathematica, VMD, etc..., are addressed similarly for interactive or batch use.  Executable names may be different from marketing names, as is the case on the old cluster. 

More Script examples

Example 1:  Suppose a text file named test_batch.sh  contains the following three lines:

This will produce output files with the name of the compute node that the hostname command was executed on.  srun output file out has the jobnumber and (task/step/rank) number in the file name.

To submit it to run on 160 cores:

out_327      out_327_121  out_327_146  out_327_27  out_327_51  out_327_76

out_327_0    out_327_122  out_327_147  out_327_28  out_327_52  out_327_77

out_327_1    out_327_123  out_327_148  out_327_29  out_327_53  out_327_78

................... snip .......

Example 1a:

Suppose you wish to submit a single Matlab batch job all on one command line where you need access to 16gig of ram and 8 threads:

The key is the --wrap option that bundles all arguments for the sbatch invocation.

Example2: 

Suppose you have a text file called slurm-numbers.sh   with the following lines:

Note:  --time=  is an option for setting the max job duration of 2 minutes.  If this is the case, then slurm will use that info to help expedite job placement.  It is important to guess the max time accurately because if the job exceeds that limit it will be killed.

To run this as a batch job use sbatch command.

Make the file executable and submit:

When the batch job finishes three resulting files will be written into your current directory.

File RandomNumbers.txt will have 100 sorted intergers. The other two files will have  nodename and jobID appended to the root name provided.

Example 3:

Simple Matlab script and batch example

1) Create a text file, example.m,  with matlab commands:

2) Create a shell script, matlab_script.sh ,  with the following and make it executable with chmod:

3) submit as a batch job requesting 8 threads:

> sbatch  ./matlab_batch.sh

Example 4: Simple job array script and batch example

Suppose you wish to run several instances(say 4) of a program so that they run on different nodes.  The following sbatch script will provide this allocation.

Create a file: myprogram_array.sh  with the following content. Change mem and time as needed:

Note:  --time=5  is an option for setting the max job duration of 5 minutes.  If this is the case, then slurm will use that info to help expedite job placement by not assuming the job will take the partition time duration limit.  It is important to guess the max time accurately because if the job exceeds that limit it will be killed.


-bash-4.1$ sbatch  myprogram_array.sh
Submitted batch job 3324389

-bash-4.1$ squeue -u your_utln
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
         3324389_1     batch   myprogram_array   utln  R       0:09      1 m3n01
         3324389_2     batch  myprogram_array   utln  R       0:09      1 m3n02
         3324389_3     batch myprogram_array     utln   R       0:09      1 m3n03
         3324389_4     batch  myprogram_array    utln   R       0:09      1 m3n05

More scripting examples can be found on the slurm website and elsewhere.

Example 5:

Batch script passing variables to Matlab program with an expected runtime of 4 hours.

Create a text file,  matlab_parametric_input.sh with the following contents.  The goal is to submit 12 matlab jobs with 12 different pairs of inputs to a matlab program.

The simple matlab program being submitted is named  m_param.m  with content:

Example not using slurm array features: 
 

 Suppose you wish to run many files, perhaps  R scripts,  that are located in some directory, perhaps  myscripts/ .   Create a text file,  runBatchR.sh with the following commands. Package up your sbatch options in the variable, opts.  Some options like --mail are optional.  Make an accurate time assessment!  And make sure your R scripts end in a number, for example  your_file_5.R, etc...  Create a file list  text file, myfilelist.txt,  with the names of the R files.

To  change the permission and run the script:

> chmod 755 runBatchR.sh
>  cat myfilelist.txt  | sh runBatchR.sh