PyLHC Submitter
Parametrize and submit studies to HTCondor
Felix Soubelet
2021.06.25 ABP Computing Meeting
Today
- Purpose of PyLHC Submitter
- Workflow & Demo
- Tips, Caveats, Questions
PyLHC Submitter is Two-Fold
PyLHC Submitter is a tool for us to parametrize and submit reproducible studies through the HTCondor service.
Parametrization
Allows definition of a study's parameter space
Submission
Handles directories, subfiles and HTCondor jobs submission
$ pip install --upgrade pylhc_submitterBased on Template Scripts
- The template is a baseline version of your executable script (Python, MAD-X, anything else) to be ran for a parameter space.
- All variables of the parameter space are set with the %(PARAM_NAME)s format in the template.
- The submitter will replace these values accordingly to the defined parameter space when preparing jobs.
Based on Template Scripts
! ----- Create Symlinks to Resources ----- !
option, warn, info;
system, "mkdir Outputdata";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/runII/2018 optics2018";
!option, -echo, warn, -info;
! ----- Make macros available ----- !
call, file="optics2018/toolkit/macro.madx";
! ----- Beam Options ----- !
qx=%(TUNEX)s; ! Will be replaced by values given to the submitter
qy=%(TUNEY)s; ! Will be replaced by values given to the submitter
emittance=3.75e-06;
n_part=1.0e10;
! ----- Set up Lattice ----- !
call, file="optics2018/lhc_as-built.seq"; ! LHC machine definition
! ----- Definine the optics ----- !
call, file="optics2018/PROTON/opticsfile.22_ctpps2"; ! Optics to Round 30cm collision optics
! ----- Create Beams ----- !
beam, sequence=lhcb1, bv= 1, energy=NRJ, particle=proton, npart=n_part, kbunch=1, ex=emittance, ey=emittance;
beam, sequence=lhcb2, bv=-1, energy=NRJ, particle=proton, npart=n_part, kbunch=1, ex=emittance, ey=emittance;
! ----- Tune Matching ----- !
use, sequence=lhcb%(BEAM)s;
match, chrom;
global, q1=qx, q2=qy;
vary, name=dQx.b%(BEAM)s, step=1.0E-7; ! Will be replaced by values given to the submitter
vary, name=dQy.b%(BEAM)s, step=1.0E-7; ! Will be replaced by values given to the submitter
lmdif, calls=100, tolerance=1.0E-21;
endmatch;
! ----- Output Twiss ----- !
select, flag=twiss, clear;
select, flag=twiss, pattern="BPM", column=name,s,x,y,betx,bety,alfx,alfy,dx,dpx,mux,muy;
select, flag=twiss, pattern="M", column=name,s,x,y,betx,bety,alfx,alfy,dx,dpx,mux,muy;
select, flag=twiss, pattern="IP", column=name,s,x,y,betx,bety,alfx,alfy,dx,dpx,mux,muy;
twiss, chrom, file="Outputdata/b%(BEAM)s.twiss.tfs"; ! Will be replaced by values given to the submitter
! ----- Cleanup Symlinks ----- !
system, "unlink optics2018";
Parametrize and Submit
- With a Python script.
- With a config file.
- Directly at the command line.
Submitting from Python
import numpy as np
from pylhc_submitter.job_submitter import main as htcondor_submit
if __name__ == "__main__":
htcondor_submit(
executable="madx", # points to the latest MAD-X on afs by default
mask="path/to/my_madx.mask", # template to fill and execute MAD-X on
replace_dict=dict( # parameters replaced in the template file
BEAM=[1, 2],
TUNEX=np.linspace(62.3, 62.32, 11).tolist(),
TUNEY=np.linspace(60.31, 60.33, 11).tolist(),
),
working_directory="/afs/cern.ch/work/u/username/study.tune_sweep", # outputs
jobflavour="workday", # htcondor flavour
# jobid_mask="b%(BEAM)d.qx%(TUNEX)s.qy%(TUNEY)s", # jobfiles naming
# htc_arguments=dict(
# RequestCpus=8,
# )
)
Submitting from Python
import numpy as np
from pylhc_submitter.job_submitter import main as htcondor_submit
if __name__ == "__main__":
htcondor_submit(
executable="path/to/env/bin/python", # your Python executable
mask="path/to/script.py", # template to fill and call Python on
replace_dict=dict( # parameters replaced in the template file
BEAM=[1, 2],
TUNEX=np.linspace(62.3, 62.32, 11),
TUNEY=np.linspace(60.31, 60.33, 11),
),
working_directory="/afs/cern.ch/work/u/username/study.python_run", # outputs
jobflavour="workday", # htcondor flavour
# jobid_mask="b%(BEAM)d.qx%(TUNEX)s.qy%(TUNEY)s", # jobfiles naming
# htc_arguments=dict(
# RequestCpus=8,
# )
)
When Submitting
- The parameter space is determined as the inner product of the replace_dict elements: one job for each possible combination.
- A reusable config.ini file with the submission parameters is created.
- Directories, and filled in scripts and shell scripts calling them are created for each determined job.
- A Jobs.tfs file is created as a global summary.
- A queuehtc.sub submission file is created, which tells HTCondor the paths to all the shell scripts to run.
- The created jobs are submitted to HTCondor.
After Submitting
After Running
The config.ini File
- Produced when submitting, can easily reproduce the study by calling the submitter on it with --entry_cfg
[DEFAULT]
append_jobs = False
executable = "madx"
htc_arguments = {}
job_output_dir = "Outputdata"
jobflavour = "workday"
mask = "/afs/cern.ch/work/f/fesoubel/my_madx.mask"
replace_dict = {'BEAM': [1, 2], 'TUNEX': [62.3, 62.302, 62.303999999999995, 62.306, 62.308, 62.31, 62.312, 62.314, 62.316, 62.318, 62.32], 'TUNEY': [60.31, 60.312000000000005, 60.314, 60.316, 60.318, 60.32, 60.322, 60.324, 60.326, 60.327999999999996, 60.33]}
resume_jobs = False
run_local = False
script_arguments = {}
working_directory = "/afs/cern.ch/work/f/fesoubel/htcondor_results/study.tune_sweep"$ python -m pylhc_submitter.job_submitter --entry_cfg config.iniSome Extras
- Possible to do all of this in one call at the command line (/!\), see the OMC website.
- Many extra options are available (command as a mask, rerun failed jobs, run locally...), see the API docs.
- The package also has an autosix submitter to handle SixDesk studies in a similar way.
Some Advice
- Use absolute paths to locations when submitting.
- Check you parameter space! The inner product of the replace_dict can sum up to a lot of jobs.
- You can use the dryrun option if you are unsure, then submit from the created config file with --entry_cfg.
- Make sure your scripts output to the job_output_dir (which defaults to HTCondor's default Outputdata).