Previous: 3. Opportunistic computation Index Next: 5. The docker universe

4. The vanilla universe

Go to:

An example with a wrapper Top

I copy here an example where the executable is loaded by a wrapper that sets some environment variables before running the main program. After the main program execution, another script runs to analyze some output (produce plots or whatever). This code shows you some useful instructions that you may adapt to your case.

Code of the my_wrapper: 

#!/bin/bash
source .bashrc
export PYTHONPATH=.:$PYTHONPATH
analyzerIni=$1
shift
echo "running on `hostname`, in `pwd`:"
echo python myExecutable.py $@
echo " "
ulimit -s unlimited
nice -n 19 python myExecutable.py $@
err=$?
echo "main program ended with code $err"
if[ $err -eq 0 ]
then
	echo "analyzing output"
	python Analyzer.py $analyzerIni
fi
exit $err

Submit file: 

jobname=wrapper_vanilla
Universe   = vanilla

Executable = my_wrapper
Arguments  = analyzer.ini input1 arg2
transfer_input_files=myExecutable.py,Analyzer.py,analyzer.ini,.bashrc,data/input1,python/libs/someLib/

when_to_transfer_output = ON_EXIT_OR_EVICT

on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)
next_job_start_delay=60

initialdir=/home/gariazzo/
Log        = logs/$(JOBNAME).$(Cluster).l
Output     = logs/$(JOBNAME).$(Cluster).o
Error      = logs/$(JOBNAME).$(Cluster).e
Queue

An example with Conda Top

A convenient way to install and load the required software, without modifying the system available in the machines (it is a quite tedious job), is to use a virtual environment or a container. Anaconda is a very useful tool in this sense, as it allows to create virtual environments and install python packages, compilers, libraries and many other things. If you want to run a program within a conda setup, your executable should be something like this:

#!/bin/bash

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/data/user/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/data/user/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/data/user/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/data/user/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

conda activate envname

analyzerIni=$1
shift
echo "running with conda 'envname' on `hostname`, in `pwd`:"
echo python myExecutable.py $@
echo " "
ulimit -s unlimited
nice -n 19 python myExecutable.py $@
err=$?
echo "main program ended with code $err"

exit $err

Notice that the part between # <<< conda initialize <<< and # <<< conda initialize <<< should match the code written by conda itself during the installation. It is probably present at the end of the ~/.bashrc file in your home directory.

Requirements, resources Top

HTCondor submission rules allow to ask for particular resources for a job.

There are two possibilities: use some specific command to require cpus, disk, memory or define the general requirements. This is a list of possible instructions that you can add to the submit file:



Previous: 3. Opportunistic computation Index Next: 5. The docker universe