GlideinWMS The Glidein-based Workflow Management System

Logical name of the Grid site where the glidein is running. This information is published both in the startd ClassAd and in the user job environment.

Condor expression to use to specify when a user job in the glideins should be held. If any expression is true, the glidein is held. This is usually done to specify "bad" jobs, such as those that claim too much memory.

Condor expression to use to specify when a user job in the glideins should be preempted. If any expression is true, the glidein is preempted. This is usually done to specify custom preemption policies for user jobs.

Used in calculating the Condor RANK

They are summed together, and the user job with the largest rank will run first.

Max amount of time a condor_startd will wait to be matched before giving up and terminating.

Max amount of time a condor_startd will wait after having already completed a job to be matched again. (i.e. the tail of a job).

How long the condor_startd be running before no longer accepting jobs.
The random spread is used to improve the efficiencies, so the actual value used by Condor will be anywhere between GRT-GRTS and GRT.

Max allowed time for the glidein.
This variable is optional. If you specify this variable, then Condor startup scripts will calculate the GLIDEIN_Retire_Time for the glidein as GLIDEIN_MAX_Walltime - GLIDEIN_Job_Max_Time. If GLIDEIN_Retire_Time is also specified, it will be ignored and only the calculated value is used. Note that if GLIDEIN_Job_Max_Time is specified but is greater than GLIDEIN_Max_Walltime or does not allow enough time for graceful shutdown, it will be reduced by the glidein. See Lifetime of a glidein for details.

Once DAEMON_SHUTDOWN is reached and the glidein pilot enters the Retiring state, this amount passes to allow the startd and job to gracefully shutdown before forcefully terminating the glidein. See Lifetime of a glidein for details.

Ability to control whether the pseduo-interactive monitoring slot is started on the worker node. Set to False if you do not want the monitoring slot started.

Special purpose resources added in separate slots or the main slot of the glidein. The separate slots will never be available for the regular jobs. This string is a semicolon separated list of comma separated resource descriptions. Each resource description contains the name (case insensitive) and optionally the number of resource instances (default is 1 (*), the total memory reserved (default is 128MB time the number of resources), a type option to say if the resource is added to the main slot (main) or has a new dedicated slot (partitionable) or many dedicated slots (one per instance, static). The default is partitionable, unless there is only one resource instance, then is static, and the disk reserved (default is auto, where HTCondor splits evenly). When adding resources to the main slots: the memory and disk parameters are ignored; HTCondor splits automatically resources depending on the number of CPUs and having a partitionable main slot or not; if the number of resources is not equal (or an exact multiple) to the number of CPUs, then you must select partitionable slots (slots_layout="partitionable" in the config/submit section of the entry configuration or SLOTS_LAYOUT partitionable in the frontend configuration) otherwise the startd may fail due to impossible configuration (check the HTCondor manual to learn more on how resources are split)

The parameters in a resource description can be listed or specified using their name: name, number, memory, type, disk (see the last example below).

Characters may be appended to a numerical value of memory to indicate percentage (%) or units. K or KB indicates KiB, $2^{10}$ numbers of bytes. M or MB indicates MiB, $2^{20}$ numbers of bytes. G or GB indicates GiB, $2^{30}$ numbers of bytes. T or TB indicates TiB, $2^{40}$ numbers of bytes. Disk space must be a fraction or percentage: 1/4, 25%. auto lets HTCondor do the splitting. Check the HTCondor manual for more about resource splitting.

Jobs submission must use request_RESOURCE=N with N>0 to use these slots or to use the resource in the main slot, e.g. request_ioslot=1

Examples:

• value="ioslot,2,200,static" creates statically partitioned 2 ioslot resources with 100MB memory each.
• value="ioslot" creates 1 ioslot resource with 128MB
• value="ioslot,2,disk=1GB;monitor;GPUs,3,,main" creates (1) a partitionable slot with 2 ioslot resources, 256MB total memory and 1GB total disk (2) 1 monitor slot resource with 128MB memory (3) adds 3 GPUs to the main slot/slots
• value="GPUs,type=main" discovers (*) how many GPUs are available and adds them to the main slot

(*) GPUs is a special resource name. If no number is specified, the glidein will invoke HTCondor's GPU discovery mechanism and get the number from there, which could be 0 if there are no GPUs. Your job will find in the HTCondor ClassAd also all the special attributes about the GPUs. If no number is specified and the auto-discovery fails 0 GPUs are assumed. Since you don't know the number of GPUs that will be added to the main slot, the slot_layout must be partitionable.

If set, Condor will launch all user jobs via glexec, thus running the job under the appropriate local account. This is important both for glideinWMS security and for accounting purposes.
A special value of "OSG" can be used to automatically find the locally installed gLExec on OSG worker nodes.

This variable is renamed to GLEXEC in the condor config.

If set to False, the condor_starter is run sharing the same UID as the user job. This has security implications.

If running Condor 7.1.3 or later, it is recommended to turn this on and have the condor_starter be protected from the user jobs.

Should process group monitoring be enabled?

This is a Condor optimization parameter. Unfortunately, it negatively interferes with the batch systems used by the Grid sites, so it should not be turned on unless you have a very good reason to do so.

If True, forces the glidein to use TCP updates.
The collector must be configured in the same way for this to work.

Also see the Condor documentation for implications and side effects.

If True, enable the startd UDP command socket (Condor default).

Using the UDP command socket is a Condor optimization that makes working over firewalls and NATs very difficult. It is thus recommended you leave it disabled in the glideins.

Please note if you leave it disabled, that you must configure the schedd with
SCHEDD_SEND_VACATE_VIA_TCP = True
and the negotiator with
NEGOTIATOR_INFORM_STARTD = False
to have a functional system.

If set to False, the schedd will be sending keepalives to the startd.

Setting this to True instructs the startd to send keepalives to the schedd. This improves the glidein behavior when running behind a firewall or a NAT.

Please note that the schedd must be configured in the same way for this to work.

Security related settings. Please notice that the glideins always require GSI authentication.

For more details see the configuration page or the Condor manual.

Another security setting.

If set to True, the schedd and the startd will use a low overhead protocol. See the configuration page or the Condor manual.

What is the maximum size the logs should grow.

Setting them too low will made debugging difficult.
Setting them too high may fill up the disk in anomalous situations, both on the work nodes and on the glidein factory.

If set to True, it will enable HTCondor Connection Brokering (CCB), which is needed if the glideins are behind a firewall or a NAT.

If set to True, it will enable the shared port daemon, which will reduce the number of connections between the glidein and the collectors.

Number of CPUs glidein should use. GLIDEIN_CPUS is used to set NUM_CPUS for the HTCondor started by the glidein. Use "slot" (or "auto") to let glidein determine this from the job slot assigned to the glidein, use "node" to let glidein determine this from the hardware of the worker node (e.g. if you are sure that only your job is running on the node), set a value to force a different number. In case of static partitioning, glidein will create GLIDEIN_CPUS number of slots. In case of dynamic partitioning, the slots will be created automatically based on the CPUs required by the job and GLIDEIN_CPUS is the sum of the slots made available. Refer to HTCondor manual for info on NUM_CPUS

Amount of memory glidein should use. If set, GLIDEIN_MaxMemMBs is used to set total MEMORY used for the HTCondor and Startd and jobs started by the glidein. If not GLIDEIN_MaxMemMBs is not set and GLIDEIN_MaxMemMBs_Estimate is TRUE, GLIDEIN_MaxMemMBs is calculated based on the memory available on the worker node. If GLIDEIN_MaxMemMBs is not set and GLIDEIN_MaxMemMBs_Estimate is not TRUE, glidein lets HTCondor decide the amount of memory. Refer to HTCondor manual for info on MEMORY

Used in conjunction with GLIDEIN_MaxMemMBs. See GLIDEIN_MaxMemMBs for the description.

This attribute regulates advertising of validation failures to the Factory collector.

• NEVER - The glidein will just fail, and do nothing.
• ALIVEONLY - The glidein will advertise a Master ClassAd in Drained state and Retiring activity for the lifetime of the glidein script.
• ALWAYS - Similar to ALIVEONLY, but will also send a final ClassAd in Drained state and Killing activity just before dying.

When advertised, the classad is flagged GLIDEIN_Failed=True, the error is recorded in the GLIDEIN_FAILURE_REASON and GLIDEIN_EXIT_CODE attributes, and the failing script is recorded in the GLIDEIN_LAST_SCRIPT attribute.

Logical name of the Glidein Factory machine (like “osg1”).

Identification name of the Glidein Factory instance (like “v1_0”).

Identification name of the entry point (like “ucsd5”).

Type of Grid resource (like “gt2”).

URI of the Grid gatekeeper (like “osg1.ucsd.edu/jobmanager-pbs”)

Optional RSL string (like "(condor_submit=('+ProdSlot' 'TRUE'))")

Optional URL of the site Web proxy.

A special value “OSG” can be used to automatically discover the local Web proxy on OSG worker nodes.

This variable is exported as GLIDEIN_Proxy_URL to the use job environment.

This setting can be either:

• “std” - Default mode, where all interesting debug information is reported back to the Glidein Factory and the glidein will wait 20 minutes on a worker node that failed validation to minimize the black hole effect.

• “nodebug” - Disable most diagnostic messages. This can be useful for very stable setups. The glidein still waits 20 minutes on a worker node that failed validation to minimize the black hole effect.

• Fast – All debugging is enabled and the glidein waits only 2 minutes on a worker node that failed validation. This mode is useful when debugging a misbehaving Grid site.

Identification name of the VO frontend request (like “ucsd5@v1_0@osg1@cms4”).

List of Collector URIs used by the VO Condor pool (like “cc.cms.edu:9620,cc.cms.edu:9621”).

One of the URIs in the list will be selected and used as HEAD_NODE in the condor_config.

Condor expression to use to specify when a user job in the glideins should be held. If any expression is true, the glidein is held. This is usually done to specify "bad" jobs, such as those that claim too much memory.

Condor expression to use to specify when a user job in the glideins should be preempted. If any expression is true, the glidein is preempted. This is usually done to specify custom preemption policies.

Used in calculating the Condor RANK

They are summed together, and the user job with the largest rank will run first.

Max allowed time for the job to end.
This is used by the condor_startd. Once the GLIDEIN_Retire_Time expires, the glidein will allow the job to run for this amount of time before activating DAEMON_SHUTDOWN and killing the job. Note that, if GLIDEIN_Max_Walltime is specified, this variable could be reduced by the glidein in order to fit within Glidein_Max_Walltime. See Lifetime of a glidein for details.

If False, the user job environment will contain only glidein factory provided variables.

If True, the user job environment will also contain the environment variables defined at glidein startup.

See JOB_INHERITS_STARTER_ENVIRONMENT documentation for more details.

By default, the glidein will unset the variable X509_USER_PROXY for security reasons to prevent the user jobs from accessing the pilot proxy. Setting this to true will override this behavior and keep the X509_USER_PROXY in the environment.

• fixed - create N single core slots, and split memory and disk equally mong them
• partitionable - start up with a single partitionable slot and let HTCondor do the spliting as needed

This attribute regulates advertising of validation failures to the user collector.

• NEVER - The glidein will just fail, and do nothing. (default)
• ALIVEONLY - The glidein will advertise a Machine ClassAd in Drained state and Retiring activity for the lifetime of the glidein script.
• ALWAYS - Similar to ALIVEONLY, but will also send a final ClassAd in Drained state and Killing activity just before dying.

When advertised, the classad is flagged GLIDEIN_Failed=True, the error is recorded in the GLIDEIN_FAILURE_REASON and GLIDEIN_EXIT_CODE attributes, and the failing script is recorded in the GLIDEIN_LAST_SCRIPT attribute.

CLAIM_WORKLIFE for non-dynamic slots. Defaults to -1 i.e. HTCondor will treat this as an infinite claim worklife and schedd will hold claim to the slot until jobs are preempted or user runs out of jobs.

CLAIM_WORKLIFE for dynamically partitionable slots. Defaults to 3600. This controls how frequently the dynamically partionable slots will coalesce.

Max lifetime of the VM. When this timer is reached the glideinwms-pilot service will terminate and glidein process, shutdown the glideinwms-pilot service and issue a VM shutdown

Disables VM from automatically shutting down after the glideinwms-pilot service has exited.
NOTE: This variable is for debugging purposes only and should not be set to True in the Production environment. Disabling auto shutdown could result in accumulating financial charges in case of paid clouds.