GlideinWMS The Glidein-based Workflow Management System

Components

HTCondor Configuration Attributes

Jump to:

    Reference Guide to HTCondor Configuration Attributes

    The tables below acts as a reference guide for condor_config attributes for GlideinWMS.

    GlideinWMS
    Service
    Attribute Description
    ALL CONDOR_IDS = 500.500 HTCondor user (uid.gid)
    CONDOR_ADMIN = user at domain Contact, via email, when problems occur
    UID_DOMAIN=$(FULL_HOSTNAME)
    FILESYSTEM_DOMAIN=$(FULL_HOSTNAME) With glideins, there is nothing shared
    LOCK = $(LOG) HTCondor lock files to synchronize access to various log files.
    They use the log directory so they are collocated.
    DAEMON_LIST = MASTER, COLLECTOR, NEGOTIATOR, SCHEDD HTCondor daemons.
    The WMS Collector services will have the list shown above.
    The User Collector service uses all but the SCHEDD.
    The Submit service uses only the SCHEDD.
    The Frontend service is a client only instance and will have a 'DAEMON_SHUTDOWN = True' to insure no daemons are started accidently.
    SEC_DAEMON_SESSION_DURATION = 50000 Limit session caching to 12 hours
    SEC_DEFAULT_AUTHENTICATION = REQUIRED
    SEC_DEFAULT_AUTHENTICATION_METHODS = FS,GSI
    SEC_READ_AUTHENTICATION = OPTIONAL
    SEC_CLIENT_AUTHENTICATION = OPTIONAL
    DENY_WRITE = anonymous@*
    DENY_ADMINISTRATOR = anonymous@*
    DENY_DAEMON = anonymous@*
    DENY_NEGOTIATOR = anonymous@*
    DENY_CLIENT = anonymous@*
    Authentication settings
    SEC_DEFAULT_ENCRYPTION = OPTIONAL
    SEC_DEFAULT_INTEGRITY = REQUIRED
    SEC_READ_INTEGRITY = OPTIONAL
    SEC_CLIENT_INTEGRITY = OPTIONAL
    SEC_READ_ENCRYPTION = OPTIONAL
    SEC_CLIENT_ENCRYPTION = OPTIONAL
    Privacy settings
    GSI_DAEMON_TRUSTED_CA_DIR = full_path_to_CA_directory Certificate directory
    GSI_DAEMON_CERT = full_path_to_your_hostcert.pem
    GSI_DAEMON_KEY = full_path_to_the_hostkey.pem
    Credentials
    GSI_DAEMON_DIRECTORY = full_path_to_your_condor_mapfile_directory Set daemon cert location
    CERTIFICATE_MAPFILE = full_path_to_the_condor_mapfile HTCondor mapfile
    HOSTALLOW_WRITE = *
    ALLOW_WRITE = $(HOSTALLOW_WRITE)
    With strong security, do not use IP based controls
    GSI_DAEMON_NAME= Identity of the GlideinWMS services used to authenenticate when communicating between serrvices. This is a comma separated list of their identities (DNs).
    WMS Collector
    User Collector
    NEGOTIATOR_POST_JOB_RANK = MY.LastHeardFrom Prefer newer claims as they are more likely to be alive
    NEGOTIATOR_INTERVAL = 60
    NEGOTIATOR_MAX_TIME_PER_SUBMITTER=60
    NEGOTIATOR_MAX_TIME_PER_PIESPIN=20
    Increase negotiation frequency, as new glideins do not trigger a reschedule
    PREEMPTION_REQUIREMENTS = False Prevent preemption
    NEGOTIATOR_INFORM_STARTD = False Negotiator/GCB optimization
    NEGOTIATOR.USE_VOMS_ATTRIBUTES = False Disable checking for VOMS extended attributes (FQAN)
    NEGOTIATOR_CONSIDER_PREEMPTION = False Causes Negotiator to run faster. PREEMPTION_REQUIREMENTS and all condor_startd rank expressions must be False for NEGOTIATOR_CONSIDER_PREEMPTION to be False
    COLLECTOR_NAME = wms_service
    COLLECTOR_HOST = $(CONDOR_HOST):9618
    COLLECTOR.USE_VOMS_ATTRIBUTES = False Disable checking for VOMS extended attributes (FQAN)
    ##EVENT_LOG=$(LOG)/EventLog
    ##EVENT_LOG_JOB_AD_INFORMATION_ATTRS=Owner,CurrentHosts,x509userproxysubject,
                AccountingGroup,GlobalJobId,QDate,JobStartDate,JobCurrentStartDate,
                JobFinishedHookDone,MATCH_EXP_JOBGLIDEIN_Site,RemoteHost
    ##EVENT_LOG_MAX_SIZE = 100000000
    Event logging, if desired
    ##COLLECTOR_MAX_FILE_DESCRIPTORS = 20000 Allow more file descriptors (only works if HTCondor is started as root)
    COLLECTOR0 = $(COLLECTOR)
    COLLECTOR0_ENVIRONMENT = "_CONDOR_COLLECTOR_LOG=$(LOG)/Collector0Log"
    COLLECTOR0_ARGS = -f -p 9641
    DAEMON_LIST = $(DAEMON_LIST), COLLECTOR0
    For each secondary HTCondor collector desired, the attributes specified must be set.

    For more detailed information on this, refer to the Advanced HTCondor Configuration - Mulitple Collectors document
    WMS Collector
    Submit
    MAX_JOBS_RUNNING = 6000 Allow up to 6K concurrent running jobs.
    JOB_START_DELAY = 2
    JOB_START_COUNT = 50
    Start max of 50 jobs every 2 seconds
    MAX_CONCURRENT_UPLOADS = 100
    MAX_CONCURRENT_DOWNLOADS = 100
    Raise file transfer limits.
    No upload limits, since JOB_START_DELAY limits that. But do limit downloads, as they are asyncronous
    APPEND_REQ_VANILLA = (Memory>=1) && (Disk>=1)
    JOB_DEFAULT_REQUESTMEMORY=1
    JOB_DEFAULT_REQUESTDISK=1
    Prevent checking on ImageSize
    With 7.8.+, the APPEND_REQ_VANILLA is replaced by the JOB_DEFAULT_REQUESTMEMORY/DISK attributes.
    Having all 3 attributes defined does not present a problem.
    MAXJOBRETIREMENTTIME = $(HOUR) * 24 * 7 Prevent preemption
    SCHEDD_SEND_VACATE_VIA_TCP = True
    STARTD_SENDS_ALIVES = True
    GCB optimization
    ENABLE_USERLOG_FSYNC = False Reduce disk IO - paranoid fsyncs are usully not needed
    MAX_SHADOW_LOG = 1000000000 Limits size of shadow logs
    SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE Submit service only and only if you have enabled the USE_MATCH_AUTH attribute in the Factory and Frontend configuration files.
    SHADOW_WORKLIFE = 0 This is a workaround to a HTCondor problem if SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE and HTCondor version <= 7.5.3
    See Condor ticket 1481
    SHADOW.USE_SHARED_PORT = True
    SCHEDD.USE_SHARED_PORT = True
    SHARED_PORT_MAX_WORKERS = 1000
    SCHEDD.SHARED_PORT_ARGS = -p 9618
    DAEMON_LIST = $(DAEMON_LIST), SHARED_PORT
    SCHEDD Shared Ports


    If you install the user schedd on a separate host, incoming TCP port 9618 remains to be open (it was 9615 for versions prior to GlideinWMS 3.4.1).For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds using condor_shared_port feature
    WMS Collector
    SCHEDDGLIDEINS2 = $(SCHEDD)
    SCHEDDGLIDEINS2_ARGS = -local-name scheddglideins2
    SCHEDDGLIDEINS2.SCHEDD_NAME = schedd_glideins2
    SCHEDDGLIDEINS2.SCHEDD_LOG = $(LOG)/SchedLog.$(SCHEDDGLIDEINS2.SCHEDD_NAME)
    SCHEDDGLIDEINS2.LOCAL_DIR_ALT = $(LOCAL_DIR)/$(SCHEDDGLIDEINS2.SCHEDD_NAME)
    SCHEDDGLIDEINS2.EXECUTE = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/execute
    SCHEDDGLIDEINS2.LOCK = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/lock
    SCHEDDGLIDEINS2.PROCD_ADDRESS = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/procd_pipe
    SCHEDDGLIDEINS2.SPOOL = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/spool
    SCHEDDGLIDEINS2.JOB_QUEUE_LOG = $(SCHEDDGLIDEINS2.SPOOL)/job_queue.log
    SCHEDDGLIDEINS2.SCHEDD_ADDRESS_FILE = $(SCHEDDGLIDEINS2.SPOOL)/.schedd_address
    SCHEDDGLIDEINS2.SCHEDD_DAEMON_AD_FILE = $(SCHEDDGLIDEINS2.SPOOL)/.schedd_classad
    SCHEDDGLIDEINS2_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridManagerLog.$(SCHEDDGLIDEINS2.SCHEDD_NAME).$(USERNAME)"

    DAEMON_LIST = $(DAEMON_LIST), SCHEDDGLIDEINS2
    DC_DAEMON_LIST = + SCHEDDGLIDEINS2
    Atttributes for defining multiple schedds.


    When multiple schedds are used, each additional schedd must specify the unique location for the log file using JOB_QUEUE_LOG.

    The execute, lock and spool directories for each schedd must be created manually.

    For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds document

    SCHEDDGLIDEINS2.SCHEDD_EXPRS = SPOOL_DIR_STRING
    Allows the Factory to query the schedds directly bypassing the collector and thus improving performance.
    Submit
    SCHEDDJOBS2 = $(SCHEDD)
    SCHEDDJOBS2_ARGS = -local-name scheddjobs2
    SCHEDDJOBS2.SCHEDD_NAME = schedd_jobs2
    SCHEDDJOBS2.SCHEDD_LOG = $(LOG)/SchedLog.$(SCHEDDJOBS2.SCHEDD_NAME)
    SCHEDDJOBS2.LOCAL_DIR_ALT = $(LOCAL_DIR)/$(SCHEDDJOBS2.SCHEDD_NAME)
    SCHEDDJOBS2.EXECUTE = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/execute
    SCHEDDJOBS2.LOCK = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/lock
    SCHEDDJOBS2.PROCD_ADDRESS = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/procd_pipe
    SCHEDDJOBS2.SPOOL = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/spool
    SCHEDDJOBS2.JOB_QUEUE_LOG = $(SCHEDDJOBS2.SPOOL)/job_queue.log
    SCHEDDJOBS2.SCHEDD_ADDRESS_FILE = $(SCHEDDJOBS2.SPOOL)/.schedd_address
    SCHEDDJOBS2.SCHEDD_DAEMON_AD_FILE = $(SCHEDDJOBS2.SPOOL)/.schedd_classad

    DAEMON_LIST = $(DAEMON_LIST), SCHEDDJOBS2
    DC_DAEMON_LIST = + SCHEDDJOBS2
    Atttributes for defining multiple schedds.


    Effective with Condor 7.7.5+, a JOB_QUEUE_LOG attribute was added. When multiple schedds are used, each additional schedd must specify the unique location for that file.

    The execute, lock and spool directories for each schedd must be created manually.

    For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds document

    SCHEDDJOBS2.SCHEDD_EXPRS = SPOOL_DIR_STRING
    Allows for queries against the schedds directly bypassing the collector and thus improving performance.
    Nothing has been implemented at this time to utilize this. Only Factory/WMS Collector communication uses this.
    WMS Collector GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE=5000
    GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=5000
    GRIDMANAGER_MAX_PENDING_REQUESTS=500
    GRIDMANAGER_PROXY_REFRESH_TIME=604800
    SCHEDD_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridmanagerLog.$(USERNAME)"
    Condor-G attributes

    The GRIDMANAGER_PROXY_REFRESH_TIME attribute forces Condor-G to re-delegate the proxy as soon as the Frontend provides one, defaulting to 1 week, since we do not expect proxies with longer lifetimes.
    Submit JOB_Site = "$$(GLIDEIN_Site:Unknown)"
    JOB_GLIDEIN_Entry_Name = "$$(GLIDEIN_Entry_Name:Unknown)
    JOB_GLIDEIN_Name = "$$(GLIDEIN_Name:Unknown)
    JOB_GLIDEIN_Factory = "$$(GLIDEIN_Factory:Unknown)
    JOB_GLIDEIN_Schedd = "$$(GLIDEIN_Schedd:Unknown)
    JOB_GLIDEIN_ClusterId = "$$(GLIDEIN_ClusterId:Unknown)
    JOB_GLIDEIN_ProcId = "$$(GLIDEIN_ProcId:Unknown)
    JOB_GLIDEIN_Site = "$$(GLIDEIN_Site:Unknown)

    SUBMIT_EXPRS = $(SUBMIT_EXPRS) JOB_Site JOB_GLIDEIN_Entry_Name JOB_GLIDEIN_Name JOB_GLIDEIN_Factory JOB_GLIDEIN_Schedd JOB_GLIDEIN_Schedd JOB_GLIDEIN_ClusterId JOB_GLIDEIN_ProcId JOB_GLIDEIN_Site
    User Job Class Ad Attributes