NERSC Script

To submit a single job that runs multiple simulations, you'll start the actual simulation job, but send it to the background using the ampersand.

The critical piece to getting this to work is to put a "wait" command at the end, otherwise the job controller finishes and gets to the end of the script, and thinks that the job is done.

Some useful links to the Hopper page (where I'm running these jobs):

Batch Queues and Policies: http://www.nersc.gov/users/computational-systems/hopper/running-jobs/queues-and-policies/
- page containing a list of all queues
- includes walltime limits, number of jobs allowed, priorities, etc.
Example Batch Scripts: http://www.nersc.gov/users/computational-systems/hopper/running-jobs/example-batch-scripts/
- Helped me figure out how to write the below script
Job Launch Command (aprun): http://www.nersc.gov/users/computational-systems/hopper/running-jobs/aprun/
- also helped me figure out how to write the below script

Here is a script illustrating the technique:

#PBS -q regular
#PBS -l walltime=36:00:00
#PBS -l mppwidth=26112
#PBS -N "PB"
#PBS -V



##################
# Common info
export SCRATCH="/scratch/scratchdirs/creid/validation/new_plackett-burman"
export MAIN="${HOME}/validation/plackett-burman"
export TABLE_FILE="${HOME}/tables/brown1.mix"

#########
# CaseA
export DUMP="${SCRATCH}/caseA"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseA.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseA_restart.${PBS_JOBID} &

#########
# CaseB
export DUMP="${SCRATCH}/caseB"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseB.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseB_restart.${PBS_JOBID} &

#########
# CaseC
export DUMP="${SCRATCH}/caseC"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseC.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseC_restart.${PBS_JOBID} &

#########
# CaseD
export DUMP="${SCRATCH}/caseD"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseD.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseD_restart.${PBS_JOBID} &

#########
# CaseE
export DUMP="${SCRATCH}/caseE"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseE.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseE_restart.${PBS_JOBID} &

#########
# CaseF
export DUMP="${SCRATCH}/caseF"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseF.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseF_restart.${PBS_JOBID} &

#########
# CaseG
export DUMP="${SCRATCH}/caseG"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseG.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseG_restart.${PBS_JOBID} &

#########
# CaseH
export DUMP="${SCRATCH}/caseH"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseH.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseH_restart.${PBS_JOBID} &

#########
# CaseZ
export DUMP="${SCRATCH}/caseZ"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseZ.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseZ_restart.${PBS_JOBID} &



cd ~/validation/factorial

##################
# Common info
export SCRATCH="/scratch/scratchdirs/creid/validation/factorial"
export MAIN="${HOME}/validation/factorial"
export TABLE_FILE="${HOME}/tables/brown1.mix"

#########
# CaseJ
export DUMP="${SCRATCH}/caseJ"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseJ.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseJ_restart.${PBS_JOBID} &

#########
# CaseK
export DUMP="${SCRATCH}/caseK"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseK.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseK_restart.${PBS_JOBID} &

#########
# CaseL
export DUMP="${SCRATCH}/caseL"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseL.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseL_restart.${PBS_JOBID} &

#########
# CaseM
export DUMP="${SCRATCH}/caseM"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseM.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseM_restart.${PBS_JOBID} &

#########
# CaseN
export DUMP="${SCRATCH}/caseN"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseN.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseN_restart.${PBS_JOBID} &

#########
# CaseP
export DUMP="${SCRATCH}/caseP"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseP.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseP_restart.${PBS_JOBID} &

#########
# CaseQ
export DUMP="${SCRATCH}/caseQ"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseQ.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseQ_restart.${PBS_JOBID} &

#########
# CaseR
export DUMP="${SCRATCH}/caseR"
cd ${DUMP}
export UDA="`readlink Brown_Validation_CaseR.uda`"
aprun -n 1536 ${HOME}/codes/uintah/hopper64opt_NewParticle/StandAlone/sus -mpi -restart ${UDA} > output.caseR_restart.${PBS_JOBID} &

wait

NERSC Script

From charlesreid1