Page Comparison

Q. I want to run a large number of R scripts submitted to the cluster, but there are too many to submit individually.

A.  Several ways to do this. If they all use similar resources ( memory, cores, time) and aren't dependant on each other, a bash script to submit jobs works great. Here's an example. Note that this will work for other applications ( Matlab) that can process scripts. Just make appropriate changes. 

...

opts="-p batch -c 8 --mem=10000 --time=10:00:00 --mail-type=ALL --mail-user=$USER"
while read filenm; do
      outs="--output=$filenm.out --error=$filenm.err --mail-type=ALL --mail-user=$USER"
      echo "sbatch $opts $outs --wrap='R --no-save < $filenm'"
      sleep 1
done

...

sbatch -p batch -c 8 --mem=10000 --time=10:00:00 --output=RT_SET_1.R.out --error=RT_SET_1.R.err --mail-type=ALL --mail-user=dlapoi01 --wrap='R --no-save < RT_SET_1.R'
sbatch -p batch -c 8 --mem=10000 --time=10:00:00 --output=RT_SET_2.R.out --error=RT_SET_2.R.err --mail-type=ALL --mail-user=dlapoi01 --wrap='R --no-save < RT_SET_2.R'
sbatch -p batch -c 8 --mem=10000 --time=10:00:00 --output=RT_SET_3.R.out --error=RT_SET_3.R.err --mail-type=ALL --mail-user=dlapoi01 --wrap='R --no-save < RT_SET_3.R'
sbatch -p batch -c 8 --mem=10000 --time=10:00:00 --output=RT_SET_4.R.out --error=RT_SET_4.R.err --mail-type=ALL --mail-user=dlapoi01 --wrap='R --no-save < RT_SET_4.R'

Q. How do I install R packages locally?

A. If you want to load a package locally, R provides a simple way to do that.  Make sure that you have enough room in your home directory to hold the package software. This simple method is good if you don't plan on loading a large number of packages or do not need versioning within a subrelease ( R/3.2.2 vs R/3.2.5).

If you don't have a directory named R in your home directory, this method will create one and place libraries in it down a long path.  This is not  a problem since R will use this path not you.

From within R you can do:
  install.packages('package_name', repos="http://cran.r-project.org")

or 

  source("http://bioconductor.org/biocLite.R")
  biocLite('package_name') 

R will inform you that you can't write to the global directory, and ask if you want to use a personal library.  Say yes.  It will then give you a long path based off /home/username/R/long-arch-string/version and ask if you want to use this.  Say yes.  It will install the library and you're done!

If you do plan to load a large number of packages, or packages which require larger amounts of storage, you can use research storage on the cluster if you have space there ( request storage if you need it).  Create a directory in your research storage, e.g. R-local , then link it  from your home directory like this:    

ln -s /cluster/tufts/<yoursharename>/R-local ~/R

If you have already loaded R packages in your home directory, Create a directory in your research storage, e.g. R-local , then copy the directory tree to R-local.

rsync -av ~/R/  /cluster/tufts/<yoursharename>/R-local    # make sure there is a / after the source directory
rm -rf ~/R        # remove the directory and contents careful!
ln -s /cluster/tufts/<yoursharename>/R-local ~/R 

Q. How do I install python modules locally?

A.  First, load the module for the python version ( e.g. 2.7) that you intend to use. That way the system managed module are loaded.

      module load python/2.7.6

Then create a local directory tree to store the modules ( ~/lib/python2.7/site-packages).  Append a modified PYTHONPATH to your .bash_profile and source it. This is only needed for the first time that this is done. 

    echo export PYTHONPATH="$PYTHONPATH:~/lib/python2.7/site-packages/" >> ~/.bash_profile
    source ~/.bash_profile 

Now you can use pip or setup.py to load python modules locally.

    pip install --user <PACKAGE>

If there is a requirements file

    pip install --user -r requirements.txt

or after a python package is dowloaded and unpacked. Be sure to read the instructions for installing.

    python setup.py install --prefix=~