Components
- Overview
- Prerequisites
- User Pool
- User Schedd
- GSI
- HTCondor
HTCondor Configuration Attributes
Jump to:
Reference Guide to HTCondor Configuration Attributes
The tables below acts as a reference guide for condor_config attributes for GlideinWMS.
| GlideinWMS Service |
Attribute | Description |
| ALL | CONDOR_IDS = 500.500 | HTCondor user (uid.gid) |
| CONDOR_ADMIN = user at domain | Contact, via email, when problems occur | |
| UID_DOMAIN=$(FULL_HOSTNAME) | ||
| FILESYSTEM_DOMAIN=$(FULL_HOSTNAME) | With glideins, there is nothing shared | |
| LOCK = $(LOG) | HTCondor lock files to synchronize access to various log files. They use the log directory so they are collocated. |
|
| DAEMON_LIST = MASTER, COLLECTOR, NEGOTIATOR, SCHEDD | HTCondor daemons. The WMS Collector services will have the list shown above. The User Collector service uses all but the SCHEDD. The Submit service uses only the SCHEDD. The Frontend service is a client only instance and will have a 'DAEMON_SHUTDOWN = True' to insure no daemons are started accidently. |
|
| SEC_DAEMON_SESSION_DURATION = 50000 | Limit session caching to 12 hours | |
| SEC_DEFAULT_AUTHENTICATION = REQUIRED SEC_DEFAULT_AUTHENTICATION_METHODS = FS,GSI SEC_READ_AUTHENTICATION = OPTIONAL SEC_CLIENT_AUTHENTICATION = OPTIONAL DENY_WRITE = anonymous@* DENY_ADMINISTRATOR = anonymous@* DENY_DAEMON = anonymous@* DENY_NEGOTIATOR = anonymous@* DENY_CLIENT = anonymous@* |
Authentication settings | |
| SEC_DEFAULT_ENCRYPTION = OPTIONAL SEC_DEFAULT_INTEGRITY = REQUIRED SEC_READ_INTEGRITY = OPTIONAL SEC_CLIENT_INTEGRITY = OPTIONAL SEC_READ_ENCRYPTION = OPTIONAL SEC_CLIENT_ENCRYPTION = OPTIONAL |
Privacy settings | |
| GSI_DAEMON_TRUSTED_CA_DIR = full_path_to_CA_directory | Certificate directory | |
| GSI_DAEMON_CERT = full_path_to_your_hostcert.pem GSI_DAEMON_KEY = full_path_to_the_hostkey.pem |
Credentials | |
| GSI_DAEMON_DIRECTORY = full_path_to_your_condor_mapfile_directory | Set daemon cert location | |
| CERTIFICATE_MAPFILE = full_path_to_the_condor_mapfile | HTCondor mapfile | |
| HOSTALLOW_WRITE = * ALLOW_WRITE = $(HOSTALLOW_WRITE) |
With strong security, do not use IP based controls |
|
| GSI_DAEMON_NAME= | Identity of the GlideinWMS services used to authenenticate when communicating between serrvices. This is a comma separated list of their identities (DNs). | |
|
WMS Collector User Collector |
NEGOTIATOR_POST_JOB_RANK = MY.LastHeardFrom | Prefer newer claims as they are more likely to be alive |
| NEGOTIATOR_INTERVAL = 60 NEGOTIATOR_MAX_TIME_PER_SUBMITTER=60 NEGOTIATOR_MAX_TIME_PER_PIESPIN=20 |
Increase negotiation frequency, as new glideins do not trigger a reschedule | |
| PREEMPTION_REQUIREMENTS = False | Prevent preemption | |
| NEGOTIATOR_INFORM_STARTD = False | Negotiator/GCB optimization | |
| NEGOTIATOR.USE_VOMS_ATTRIBUTES = False | Disable checking for VOMS extended attributes (FQAN) | |
| NEGOTIATOR_CONSIDER_PREEMPTION = False | Causes Negotiator to run faster. PREEMPTION_REQUIREMENTS and all condor_startd rank expressions must be False for NEGOTIATOR_CONSIDER_PREEMPTION to be False | |
| COLLECTOR_NAME = wms_service | ||
| COLLECTOR_HOST = $(CONDOR_HOST):9618 | ||
| COLLECTOR.USE_VOMS_ATTRIBUTES = False | Disable checking for VOMS extended attributes (FQAN) | |
| ##EVENT_LOG=$(LOG)/EventLog ##EVENT_LOG_JOB_AD_INFORMATION_ATTRS=Owner,CurrentHosts,x509userproxysubject, AccountingGroup,GlobalJobId,QDate,JobStartDate,JobCurrentStartDate, JobFinishedHookDone,MATCH_EXP_JOBGLIDEIN_Site,RemoteHost ##EVENT_LOG_MAX_SIZE = 100000000 |
Event logging, if desired | |
| ##COLLECTOR_MAX_FILE_DESCRIPTORS = 20000 | Allow more file descriptors (only works if HTCondor is started as root) | |
| COLLECTOR0 = $(COLLECTOR) COLLECTOR0_ENVIRONMENT = "_CONDOR_COLLECTOR_LOG=$(LOG)/Collector0Log" COLLECTOR0_ARGS = -f -p 9641 DAEMON_LIST = $(DAEMON_LIST), COLLECTOR0 |
For each secondary HTCondor collector desired, the attributes specified must be set.
For more detailed information on this, refer to the Advanced HTCondor Configuration - Mulitple Collectors document |
|
|
WMS Collector Submit |
MAX_JOBS_RUNNING = 6000 | Allow up to 6K concurrent running jobs. |
| JOB_START_DELAY = 2 JOB_START_COUNT = 50 |
Start max of 50 jobs every 2 seconds | |
| JOB_STOP_DELAY = 1 JOB_STOP_COUNT = 30 |
Stop 30 jobs every seconds This is needed to prevent glexec overload, when used. |
|
| MAX_CONCURRENT_UPLOADS = 100 MAX_CONCURRENT_DOWNLOADS = 100 |
Raise file transfer limits. No upload limits, since JOB_START_DELAY limits that. But do limit downloads, as they are asyncronous |
|
| APPEND_REQ_VANILLA = (Memory>=1) && (Disk>=1) JOB_DEFAULT_REQUESTMEMORY=1 JOB_DEFAULT_REQUESTDISK=1 |
Prevent checking on ImageSize With 7.8.+, the APPEND_REQ_VANILLA is replaced by the JOB_DEFAULT_REQUESTMEMORY/DISK attributes. Having all 3 attributes defined does not present a problem. |
|
| MAXJOBRETIREMENTTIME = $(HOUR) * 24 * 7 | Prevent preemption | |
| SCHEDD_SEND_VACATE_VIA_TCP = True STARTD_SENDS_ALIVES = True |
GCB optimization | |
| ENABLE_USERLOG_FSYNC = False | Reduce disk IO - paranoid fsyncs are usully not needed | |
| SHADOW.GLEXEC_STARTER = True SHADOW.GLEXEC = /bin/false |
Prepare the Shadow for use with glexec-enabled glideins | |
| MAX_SHADOW_LOG = 1000000000 | Limits size of shadow logs | |
| SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE | Submit service only and only if you have enabled the USE_MATCH_AUTH attribute in the Factory and Frontend configuration files. | |
| SHADOW_WORKLIFE = 0 | This is a workaround to a HTCondor problem
if SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE and HTCondor
version <= 7.5.3 See Condor ticket 1481 |
|
|
SHADOW.USE_SHARED_PORT = True SCHEDD.USE_SHARED_PORT = True SHARED_PORT_MAX_WORKERS = 1000 SCHEDD.SHARED_PORT_ARGS = -p 9618 DAEMON_LIST = $(DAEMON_LIST), SHARED_PORT |
SCHEDD Shared Ports If you install the user schedd on a separate host, incoming TCP port 9618 remains to be open (it was 9615 for versions prior to GlideinWMS 3.4.1).For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds using condor_shared_port feature |
|
|
WMS Collector |
SCHEDDGLIDEINS2 = $(SCHEDD) SCHEDDGLIDEINS2_ARGS = -local-name scheddglideins2 SCHEDDGLIDEINS2.SCHEDD_NAME = schedd_glideins2 SCHEDDGLIDEINS2.SCHEDD_LOG = $(LOG)/SchedLog.$(SCHEDDGLIDEINS2.SCHEDD_NAME) SCHEDDGLIDEINS2.LOCAL_DIR_ALT = $(LOCAL_DIR)/$(SCHEDDGLIDEINS2.SCHEDD_NAME) SCHEDDGLIDEINS2.EXECUTE = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/execute SCHEDDGLIDEINS2.LOCK = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/lock SCHEDDGLIDEINS2.PROCD_ADDRESS = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/procd_pipe SCHEDDGLIDEINS2.SPOOL = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/spool SCHEDDGLIDEINS2.JOB_QUEUE_LOG = $(SCHEDDGLIDEINS2.SPOOL)/job_queue.log SCHEDDGLIDEINS2.SCHEDD_ADDRESS_FILE = $(SCHEDDGLIDEINS2.SPOOL)/.schedd_address SCHEDDGLIDEINS2.SCHEDD_DAEMON_AD_FILE = $(SCHEDDGLIDEINS2.SPOOL)/.schedd_classad SCHEDDGLIDEINS2_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridManagerLog.$(SCHEDDGLIDEINS2.SCHEDD_NAME).$(USERNAME)" DAEMON_LIST = $(DAEMON_LIST), SCHEDDGLIDEINS2 DC_DAEMON_LIST = + SCHEDDGLIDEINS2 |
Atttributes for defining multiple schedds. When multiple schedds are used, each additional schedd must specify the unique location for the log file using JOB_QUEUE_LOG. The execute, lock and spool directories for each schedd must be created manually. For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds document |
|
SCHEDDGLIDEINS2_SPOOL_DIR_STRING = "$(SCHEDDGLIDEINS2.SPOOL)" SCHEDDGLIDEINS2.SCHEDD_EXPRS = SPOOL_DIR_STRING |
Allows the Factory to query the schedds directly bypassing the collector and thus improving performance. | |
|
Submit |
SCHEDDJOBS2 = $(SCHEDD) SCHEDDJOBS2_ARGS = -local-name scheddjobs2 SCHEDDJOBS2.SCHEDD_NAME = schedd_jobs2 SCHEDDJOBS2.SCHEDD_LOG = $(LOG)/SchedLog.$(SCHEDDJOBS2.SCHEDD_NAME) SCHEDDJOBS2.LOCAL_DIR_ALT = $(LOCAL_DIR)/$(SCHEDDJOBS2.SCHEDD_NAME) SCHEDDJOBS2.EXECUTE = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/execute SCHEDDJOBS2.LOCK = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/lock SCHEDDJOBS2.PROCD_ADDRESS = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/procd_pipe SCHEDDJOBS2.SPOOL = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/spool SCHEDDJOBS2.JOB_QUEUE_LOG = $(SCHEDDJOBS2.SPOOL)/job_queue.log SCHEDDJOBS2.SCHEDD_ADDRESS_FILE = $(SCHEDDJOBS2.SPOOL)/.schedd_address SCHEDDJOBS2.SCHEDD_DAEMON_AD_FILE = $(SCHEDDJOBS2.SPOOL)/.schedd_classad DAEMON_LIST = $(DAEMON_LIST), SCHEDDJOBS2 DC_DAEMON_LIST = + SCHEDDJOBS2 |
Atttributes for defining multiple schedds. Effective with Condor 7.7.5+, a JOB_QUEUE_LOG attribute was added. When multiple schedds are used, each additional schedd must specify the unique location for that file. The execute, lock and spool directories for each schedd must be created manually. For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds document |
|
SCHEDDJOBS2_SPOOL_DIR_STRING = "$(SCHEDDJOBS2.SPOOL)" SCHEDDJOBS2.SCHEDD_EXPRS = SPOOL_DIR_STRING |
Allows for queries against the schedds directly bypassing the collector
and thus improving performance. Nothing has been implemented at this time to utilize this. Only Factory/WMS Collector communication uses this. |
|
| WMS Collector |
GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE=5000 GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=5000 GRIDMANAGER_MAX_PENDING_REQUESTS=500 GRIDMANAGER_PROXY_REFRESH_TIME=604800 SCHEDD_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridmanagerLog.$(USERNAME)" |
Condor-G attributes The GRIDMANAGER_PROXY_REFRESH_TIME attribute forces Condor-G to re-delegate the proxy as soon as the Frontend provides one, defaulting to 1 week, since we do not expect proxies with longer lifetimes. |
| QUEUE_SUPER_USERS = $(QUEUE_SUPER_USERS), factory_user | Only required when privilege separation is in effect. Allows the Factory user to bypass privilege separation making administration simpler without compromising security. | |
| Submit |
JOB_Site = "$$(GLIDEIN_Site:Unknown)" JOB_GLIDEIN_Entry_Name = "$$(GLIDEIN_Entry_Name:Unknown) JOB_GLIDEIN_Name = "$$(GLIDEIN_Name:Unknown) JOB_GLIDEIN_Factory = "$$(GLIDEIN_Factory:Unknown) JOB_GLIDEIN_Schedd = "$$(GLIDEIN_Schedd:Unknown) JOB_GLIDEIN_ClusterId = "$$(GLIDEIN_ClusterId:Unknown) JOB_GLIDEIN_ProcId = "$$(GLIDEIN_ProcId:Unknown) JOB_GLIDEIN_Site = "$$(GLIDEIN_Site:Unknown) SUBMIT_EXPRS = $(SUBMIT_EXPRS) JOB_Site JOB_GLIDEIN_Entry_Name JOB_GLIDEIN_Name JOB_GLIDEIN_Factory JOB_GLIDEIN_Schedd JOB_GLIDEIN_Schedd JOB_GLIDEIN_ClusterId JOB_GLIDEIN_ProcId JOB_GLIDEIN_Site |
User Job Class Ad Attributes |