GlideinWMS The Glidein-based Workflow Management System

Components

HTCondor Configuration Attributes

Jump to:

Reference Guide to HTCondor Configuration Attributes

The tables below acts as a reference guide for condor_config attributes for GlideinWMS.

GlideinWMS
Service
Attribute Description
ALL CONDOR_IDS = 500.500 HTCondor user (uid.gid)
CONDOR_ADMIN = user at domain Contact, via email, when problems occur
UID_DOMAIN=$(FULL_HOSTNAME)
FILESYSTEM_DOMAIN=$(FULL_HOSTNAME) With glideins, there is nothing shared
LOCK = $(LOG) HTCondor lock files to synchronize access to various log files.
They use the log directory so they are collocated.
DAEMON_LIST = MASTER, COLLECTOR, NEGOTIATOR, SCHEDD HTCondor daemons.
The WMS Collector services will have the list shown above.
The User Collector service uses all but the SCHEDD.
The Submit service uses only the SCHEDD.
The Frontend service is a client only instance and will have a 'DAEMON_SHUTDOWN = True' to insure no daemons are started accidently.
SEC_DAEMON_SESSION_DURATION = 50000 Limit session caching to 12 hours
SEC_DEFAULT_AUTHENTICATION = REQUIRED
SEC_DEFAULT_AUTHENTICATION_METHODS = FS,GSI
SEC_READ_AUTHENTICATION = OPTIONAL
SEC_CLIENT_AUTHENTICATION = OPTIONAL
DENY_WRITE = anonymous@*
DENY_ADMINISTRATOR = anonymous@*
DENY_DAEMON = anonymous@*
DENY_NEGOTIATOR = anonymous@*
DENY_CLIENT = anonymous@*
Authentication settings
SEC_DEFAULT_ENCRYPTION = OPTIONAL
SEC_DEFAULT_INTEGRITY = REQUIRED
SEC_READ_INTEGRITY = OPTIONAL
SEC_CLIENT_INTEGRITY = OPTIONAL
SEC_READ_ENCRYPTION = OPTIONAL
SEC_CLIENT_ENCRYPTION = OPTIONAL
Privacy settings
GSI_DAEMON_TRUSTED_CA_DIR = full_path_to_CA_directory Certificate directory
GSI_DAEMON_CERT = full_path_to_your_hostcert.pem
GSI_DAEMON_KEY = full_path_to_the_hostkey.pem
Credentials
GSI_DAEMON_DIRECTORY = full_path_to_your_condor_mapfile_directory Set daemon cert location
CERTIFICATE_MAPFILE = full_path_to_the_condor_mapfile HTCondor mapfile
HOSTALLOW_WRITE = *
ALLOW_WRITE = $(HOSTALLOW_WRITE)
With strong security, do not use IP based controls
GSI_DAEMON_NAME= Identity of the GlideinWMS services used to authenenticate when communicating between serrvices. This is a comma separated list of their identities (DNs).
WMS Collector
User Collector
NEGOTIATOR_POST_JOB_RANK = MY.LastHeardFrom Prefer newer claims as they are more likely to be alive
NEGOTIATOR_INTERVAL = 60
NEGOTIATOR_MAX_TIME_PER_SUBMITTER=60
NEGOTIATOR_MAX_TIME_PER_PIESPIN=20
Increase negotiation frequency, as new glideins do not trigger a reschedule
PREEMPTION_REQUIREMENTS = False Prevent preemption
NEGOTIATOR_INFORM_STARTD = False Negotiator/GCB optimization
NEGOTIATOR.USE_VOMS_ATTRIBUTES = False Disable checking for VOMS extended attributes (FQAN)
NEGOTIATOR_CONSIDER_PREEMPTION = False Causes Negotiator to run faster. PREEMPTION_REQUIREMENTS and all condor_startd rank expressions must be False for NEGOTIATOR_CONSIDER_PREEMPTION to be False
COLLECTOR_NAME = wms_service
COLLECTOR_HOST = $(CONDOR_HOST):9618
COLLECTOR.USE_VOMS_ATTRIBUTES = False Disable checking for VOMS extended attributes (FQAN)
##EVENT_LOG=$(LOG)/EventLog
##EVENT_LOG_JOB_AD_INFORMATION_ATTRS=Owner,CurrentHosts,x509userproxysubject,
            AccountingGroup,GlobalJobId,QDate,JobStartDate,JobCurrentStartDate,
            JobFinishedHookDone,MATCH_EXP_JOBGLIDEIN_Site,RemoteHost
##EVENT_LOG_MAX_SIZE = 100000000
Event logging, if desired
##COLLECTOR_MAX_FILE_DESCRIPTORS = 20000 Allow more file descriptors (only works if HTCondor is started as root)
COLLECTOR0 = $(COLLECTOR)
COLLECTOR0_ENVIRONMENT = "_CONDOR_COLLECTOR_LOG=$(LOG)/Collector0Log"
COLLECTOR0_ARGS = -f -p 9641
DAEMON_LIST = $(DAEMON_LIST), COLLECTOR0
For each secondary HTCondor collector desired, the attributes specified must be set.

For more detailed information on this, refer to the Advanced HTCondor Configuration - Mulitple Collectors document
WMS Collector
Submit
MAX_JOBS_RUNNING = 6000 Allow up to 6K concurrent running jobs.
JOB_START_DELAY = 2
JOB_START_COUNT = 50
Start max of 50 jobs every 2 seconds
JOB_STOP_DELAY = 1
JOB_STOP_COUNT = 30
Stop 30 jobs every seconds
This is needed to prevent glexec overload, when used.
MAX_CONCURRENT_UPLOADS = 100
MAX_CONCURRENT_DOWNLOADS = 100
Raise file transfer limits.
No upload limits, since JOB_START_DELAY limits that. But do limit downloads, as they are asyncronous
APPEND_REQ_VANILLA = (Memory>=1) && (Disk>=1)
JOB_DEFAULT_REQUESTMEMORY=1
JOB_DEFAULT_REQUESTDISK=1
Prevent checking on ImageSize
With 7.8.+, the APPEND_REQ_VANILLA is replaced by the JOB_DEFAULT_REQUESTMEMORY/DISK attributes.
Having all 3 attributes defined does not present a problem.
MAXJOBRETIREMENTTIME = $(HOUR) * 24 * 7 Prevent preemption
SCHEDD_SEND_VACATE_VIA_TCP = True
STARTD_SENDS_ALIVES = True
GCB optimization
ENABLE_USERLOG_FSYNC = False Reduce disk IO - paranoid fsyncs are usully not needed
SHADOW.GLEXEC_STARTER = True
SHADOW.GLEXEC = /bin/false
Prepare the Shadow for use with glexec-enabled glideins
MAX_SHADOW_LOG = 1000000000 Limits size of shadow logs
SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE Submit service only and only if you have enabled the USE_MATCH_AUTH attribute in the Factory and Frontend configuration files.
SHADOW_WORKLIFE = 0 This is a workaround to a HTCondor problem if SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE and HTCondor version <= 7.5.3
See Condor ticket 1481
SHADOW.USE_SHARED_PORT = True
SCHEDD.USE_SHARED_PORT = True
SHARED_PORT_MAX_WORKERS = 1000
SCHEDD.SHARED_PORT_ARGS = -p 9618
DAEMON_LIST = $(DAEMON_LIST), SHARED_PORT
SCHEDD Shared Ports


If you install the user schedd on a separate host, incoming TCP port 9618 remains to be open (it was 9615 for versions prior to GlideinWMS 3.4.1).For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds using condor_shared_port feature
WMS Collector
SCHEDDGLIDEINS2 = $(SCHEDD)
SCHEDDGLIDEINS2_ARGS = -local-name scheddglideins2
SCHEDDGLIDEINS2.SCHEDD_NAME = schedd_glideins2
SCHEDDGLIDEINS2.SCHEDD_LOG = $(LOG)/SchedLog.$(SCHEDDGLIDEINS2.SCHEDD_NAME)
SCHEDDGLIDEINS2.LOCAL_DIR_ALT = $(LOCAL_DIR)/$(SCHEDDGLIDEINS2.SCHEDD_NAME)
SCHEDDGLIDEINS2.EXECUTE = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/execute
SCHEDDGLIDEINS2.LOCK = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/lock
SCHEDDGLIDEINS2.PROCD_ADDRESS = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/procd_pipe
SCHEDDGLIDEINS2.SPOOL = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/spool
SCHEDDGLIDEINS2.JOB_QUEUE_LOG = $(SCHEDDGLIDEINS2.SPOOL)/job_queue.log
SCHEDDGLIDEINS2.SCHEDD_ADDRESS_FILE = $(SCHEDDGLIDEINS2.SPOOL)/.schedd_address
SCHEDDGLIDEINS2.SCHEDD_DAEMON_AD_FILE = $(SCHEDDGLIDEINS2.SPOOL)/.schedd_classad
SCHEDDGLIDEINS2_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridManagerLog.$(SCHEDDGLIDEINS2.SCHEDD_NAME).$(USERNAME)"

DAEMON_LIST = $(DAEMON_LIST), SCHEDDGLIDEINS2
DC_DAEMON_LIST = + SCHEDDGLIDEINS2
Atttributes for defining multiple schedds.


When multiple schedds are used, each additional schedd must specify the unique location for the log file using JOB_QUEUE_LOG.

The execute, lock and spool directories for each schedd must be created manually.

For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds document

SCHEDDGLIDEINS2.SCHEDD_EXPRS = SPOOL_DIR_STRING
Allows the Factory to query the schedds directly bypassing the collector and thus improving performance.
Submit
SCHEDDJOBS2 = $(SCHEDD)
SCHEDDJOBS2_ARGS = -local-name scheddjobs2
SCHEDDJOBS2.SCHEDD_NAME = schedd_jobs2
SCHEDDJOBS2.SCHEDD_LOG = $(LOG)/SchedLog.$(SCHEDDJOBS2.SCHEDD_NAME)
SCHEDDJOBS2.LOCAL_DIR_ALT = $(LOCAL_DIR)/$(SCHEDDJOBS2.SCHEDD_NAME)
SCHEDDJOBS2.EXECUTE = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/execute
SCHEDDJOBS2.LOCK = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/lock
SCHEDDJOBS2.PROCD_ADDRESS = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/procd_pipe
SCHEDDJOBS2.SPOOL = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/spool
SCHEDDJOBS2.JOB_QUEUE_LOG = $(SCHEDDJOBS2.SPOOL)/job_queue.log
SCHEDDJOBS2.SCHEDD_ADDRESS_FILE = $(SCHEDDJOBS2.SPOOL)/.schedd_address
SCHEDDJOBS2.SCHEDD_DAEMON_AD_FILE = $(SCHEDDJOBS2.SPOOL)/.schedd_classad

DAEMON_LIST = $(DAEMON_LIST), SCHEDDJOBS2
DC_DAEMON_LIST = + SCHEDDJOBS2
Atttributes for defining multiple schedds.


Effective with Condor 7.7.5+, a JOB_QUEUE_LOG attribute was added. When multiple schedds are used, each additional schedd must specify the unique location for that file.

The execute, lock and spool directories for each schedd must be created manually.

For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds document

SCHEDDJOBS2.SCHEDD_EXPRS = SPOOL_DIR_STRING
Allows for queries against the schedds directly bypassing the collector and thus improving performance.
Nothing has been implemented at this time to utilize this. Only Factory/WMS Collector communication uses this.
WMS Collector GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE=5000
GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=5000
GRIDMANAGER_MAX_PENDING_REQUESTS=500
GRIDMANAGER_PROXY_REFRESH_TIME=604800
SCHEDD_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridmanagerLog.$(USERNAME)"
Condor-G attributes

The GRIDMANAGER_PROXY_REFRESH_TIME attribute forces Condor-G to re-delegate the proxy as soon as the Frontend provides one, defaulting to 1 week, since we do not expect proxies with longer lifetimes.
QUEUE_SUPER_USERS = $(QUEUE_SUPER_USERS), factory_user Only required when privilege separation is in effect. Allows the Factory user to bypass privilege separation making administration simpler without compromising security.
Submit JOB_Site = "$$(GLIDEIN_Site:Unknown)"
JOB_GLIDEIN_Entry_Name = "$$(GLIDEIN_Entry_Name:Unknown)
JOB_GLIDEIN_Name = "$$(GLIDEIN_Name:Unknown)
JOB_GLIDEIN_Factory = "$$(GLIDEIN_Factory:Unknown)
JOB_GLIDEIN_Schedd = "$$(GLIDEIN_Schedd:Unknown)
JOB_GLIDEIN_ClusterId = "$$(GLIDEIN_ClusterId:Unknown)
JOB_GLIDEIN_ProcId = "$$(GLIDEIN_ProcId:Unknown)
JOB_GLIDEIN_Site = "$$(GLIDEIN_Site:Unknown)

SUBMIT_EXPRS = $(SUBMIT_EXPRS) JOB_Site JOB_GLIDEIN_Entry_Name JOB_GLIDEIN_Name JOB_GLIDEIN_Factory JOB_GLIDEIN_Schedd JOB_GLIDEIN_Schedd JOB_GLIDEIN_ClusterId JOB_GLIDEIN_ProcId JOB_GLIDEIN_Site
User Job Class Ad Attributes