Guide to Submitting HTCondor jobs

Cluster image

High Throughput Computing (HTC)

First a note: HTC is meant for a large number of small to medium size jobs. HTC is different than HPC (High Performance Computing) that would run large size jobs that require a high memory (RAM) footprint, and/or extremely large data files.

Biochemistry Department personnel can obtain access to the in-door cluster  (Biochemistry Compute Cluster, see page BCC.) For larger projects the Center for High Throughput Computing (CHTC) may be helpful and offers free access to all UW-Madison researchers. Jobs that require even more computation may be helped by the Open Science Grid (OSG.)

In all cases, computation is managed by submitting “jobs” that are managed by a “scheduler” called HTCondor.

HTCondor

A tutorial specific to BCC is available on a separate page: tutorials-htcondor.

Below are videos providing a general introduction to HTcondor and offers example on how to submit jobs.

HTC Job Execution with HTCondor (48:23)

This was the second public lecture during OSG Virtual School 2021.

Materials and slides are available at  OSG Virtual School Materials

A User’s Guide to Submitting HTCondor jobs (54:54)

Similar material is also available on this alternate version of the video from 2020.

HTCondor manual

The latest HTCondor manual is online at: htcondor.readthedocs.io

Most useful: a QuickStart guide is available: quick-start-guide


Image credit: Arek Socha – Stockholm/Sweden, aka qimono (pixabay.com)