Configuring multiple Schedds in Condor

1. Description

This document describes how to configure multiple Schedds in Condor.
Using more than a single schedd on a given node can help you scale without adding new hardware.

2. Required software

A reasonably recent Linux OS (SL4 used at press time).
The Condor distribution.
If you use Quill, also a PostgreSQL server.

3. Installation instructions

The installation will assume you have installed Condor v6.9.2.
The install directory is /opt/glidecondor, the working directory is /opt/glidecondor/condor_local and the machine name is mymachine.fnal.gov.
If you want to use a different setup, make the necessary changes.

It also assumes you have configured Quill; you can use multiple Schedds without using Quill, but this is not described here, although it should be obvious what you can skip in that case.

Unless explicity mentioned, all operations are to be done as root.

3.1 Create setup files

In order to easily manage a secondary schedd, create /opt/glidecondor/new_schedd_setup.sh, containing:
if [ $# -ne 1 ]
then
 echo "Schedd name expected!"
 return 1
fi

LD=`condor_config_val LOCAL_DIR`
QDB=`condor_config_val QUILL_DB_NAME`

export _CONDOR_SCHEDD_NAME=schedd_$1
export _CONDOR_MASTER_NAME=${_CONDOR_SCHEDD_NAME}
# SCHEDD and MASTER names MUST be the same (Condor requirement)
export _CONDOR_QUILL_NAME=quill_$1@`uname -n`
export _CONDOR_QUILL_DB_NAME=${QDB}_$1
export _CONDOR_DAEMON_LIST="MASTER, SCHEDD,QUILL"
export _CONDOR_LOCAL_DIR=$LD/$_CONDOR_SCHEDD_NAME
export _CONDOR_LOCK=$_CONDOR_LOCAL_DIR/lock

unset LD
unset QDB

The same file can be downloaded from example-config/multi_schedd/new_schedd_setup.sh.

To create the necessary directories and files, create /opt/glidecondor/init_schedd.sh:
#!/bin/sh
source /opt/glidecondor/new_schedd_setup.sh $1
# add whatever other config you need
# create needed directories
/opt/glidecondor/sbin/condor_init
# copy Quill writer passwd
cp -p $_CONDOR_LOCAL_DIR/../spool/.quillwritepassword $_CONDOR_LOCAL_DIR/spool/
This one you want to make executable:
chmod a+x /opt/glidecondor/init_schedd.sh

The same file can be downloaded from example-config/multi_schedd/init_schedd.sh.

Similarly, you will want to have a way to easily start a secondary schedd, so create /opt/glidecondor/start_master_schedd.sh:
#!/bin/sh
source /opt/glidecondor/new_schedd_setup.sh $1
# add whatever other config you need
/opt/glidecondor/sbin/condor_master
This one you want to make executable:
chmod a+x /opt/glidecondor/start_master_schedd.sh

The same file can be downloaded from example-config/multi_schedd/start_master_schedd.sh.

3.2 Initialize schedds

To initialize a secondary schedd, use /opt/glidecondor/init_schedd.sh created above.
If you came here from another document, make sure you configure the schedds specified there.

For example, supposing you want to create schedds named schedd_jobs1, schedd_jobs2 and schedd_glideins1, you would run:
/opt/glidecondor/init_schedd.sh jobs1
/opt/glidecondor/init_schedd.sh jobs2
/opt/glidecondor/init_schedd.sh glideins1

4. Starting up schedds

If you came to this document as part of another installation, go back and follow those instructions.

Else, when you are ready, you can start the schedd by running /opt/glidecondor/start_master_schedd.sh created above.
For example, supposing you want to start schedds named schedd_jobs1, schedd_jobs2 and schedd_glideins1, you would run:
/opt/glidecondor/start_master_schedd.sh jobs1
/opt/glidecondor/start_master_schedd.sh jobs2
/opt/glidecondor/start_master_schedd.sh glideins1

Note: Always start them after you have started the Collector.

5. Submission and monitoring

The secondary schedds can be seen by issuing
condor_status -schedd

To submit or query a secondary schedd, you need to use the -name options, like:
condor_submit -name schedd_jobs1@ job.jdl
condor_q -name scjedd_jobs1@