GlideinWMS The Glidein-based Workflow Management System

Integration with gLExec

As you may have noticed, all of the glideins are submitted with the same service proxy. While this has the advantage of simplifying the architecture and improve both efficiency and VO control, it does have a few problems:

• All glidein scripts and HTCondor daemons, AND user jobs all run under the same Unix UID. So users can interfere with the glidein tasks, possibly hacking the system. 
  Plus, when several glideins start on the same node (on multi processor/core machines), one user job can interfere with another user job.

• The real user is never authenticated against the Grid site authorization infrastructure. This makes it impossible for the sites to enforce their policies, nor can they analyze the usage of their resources; they see only glideins. This makes them very unfriendly toward the glidein based WMS.

To solve this problem, some Grid sites are deploying gLExec on the worker nodes. gLExec is a service that will take the following:

• the user proxy, and
• the desired command

It will contact the local authorization and mapping system, switch to the UID of the user (as opposed to the glidein UID), and execute the provided command as that UID.

By using gLExec, a Glidein Factory can get rid of both of the above problems, and still keep all the advantages.

To enable gLExec support, you need to specify:

For most current gLExec installation this comes down to:

More details about scripts in general can be found in the "custom code" section.

You will also need to properly configure the shadow config files on the submit machine, by adding the following to the condor_config:

GLEXEC_STARTER = True
GLEXEC = /bin/false

Just some history information to understand about glexec: In the old gLExec operation mode, condor_startd invoked condor_starter through gLExec. The result was that ciondor_starter was running under the same UID as the user job, leaving it vulnerable to attack from a malicious user. The new operating mode solves this by having condor_starter run the user jobs via glexec; this adds a little more overhead to handle the user jobs, but makes the system much more secure.

To enable the new operation mode, add the following line to your configuration file:

Note that you still need to set GLEXEC_BIN, too.

Troubleshooting

gLExec installations on at least one site had problems with delegated proxies. If in doubt, try to disable the delegation.

To disable delegation, add the following to the shadow configuration file:

DELEGATE_JOB_GSI_CREDENTIALS=False
SEC_DEFAULT_ENCRYPTION=PREFERRED

Then, set the following tags in the glidein creation file:

Another thing to consider is the startup directory; it must be accessible by both: the starting user and the target user(s). The directory you usually start in the Grid is most often not readable by any other user, so you must select something else. Both HTCondor and OSG should be fine, or you can specify any other fixed, WN-local location.

Private networks and firewalls

HTCondor daemons need two way communication in order to work properly. This clashes with the network policies of most Grid sites, that have worker nodes in private networks or implement a restrictive firewall.

HTCondor provides the CCB mechanism to address this. It was providing also a second mechanism, GCB, but it is no more supported in recent versions, so remove it if you still have it in your configuration.

CCB - Condor Connection Broker

The detailed description of CCB is beyond the scope of this manual and you should refer to the HTCondor documentation available here. , where you will find only the parameters needed to enable it in the glideins.

To use HTCondor with CCB, you need to specify:

and you are done. Just make sure you follow the suggested scalability guidelines described in the HTCondor manual.

Security handles

As mentioned in the startup page, the glidein pool must be properly configured to protect it from hackers and malicious users. The same page also describes what needs to be done on the collector machine.
The glidein itself can also be configured. The default configuration works fine for most users, but you may need to change them.

The values are set using the <attr /> option, and the default values are:

HTCondor also supports a more efficient authentication mechanism between the condor_schedd/condor_shadow and condor_startd/condor_starter. This method uses the match ClaimId as a shared password for authentication between these daemons. Since using a shared secret is much cheaper that using GSI authentication, this should be used every time it is feasible.

This option is enabled by default. <attr /> option:

<attr name=USE_MATCH_AUTH ... value=True.. /> ... enabled
<attr name=USE_MATCH_AUTH ... value=False.. /> ... disabled

When enabled, this HTCondor attribute must be set in the condor_config of the submit machine.
This option is not used by the HTCondor negotiator or collector and therefore not needed if they are installed separately.

SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = True

Using TCP to send updates to the Collector

By default, HTCondor uses UDP packets to communicate between the glideins and the Collector. While more efficient than TCP, UDP packets are often blocked at the firewall, or lost on the WAN.

To disable TCP updates, specify, with the <attr/> option:

UPDATE_COLLECTOR_WITH_TCP=False

In GlideinWMS, we enable the glideins to update the user collector using TCP by default.
Please be aware that this will configure the glideins only; you still need to properly configure the Collector machine. See Condor documentation for more details.

Multiple Collectors

By default, HTCondor uses only one Collector for the glidein (user) pool. However, if the load becomes too high on the collector, you can configure multiple collectors in a chain.

You will need a master and a set of slave collectors. Each slave collector will service a portion of the pool and will forward communication between the startd daemons to the master collector. Machine ClassAds from these startd's will be sent to the master collector. The negotiator and the schedds will talk to the master collector, and the startds will talk to one of the slave ones. This will reduce load on the central manager.

To set up slave collector in the glidein (user) pool, one way is to set the following env variables before starting up the condor_master:

COLH=`condor_config_val COLLECTOR_HOST`
LD=`condor_config_val LOCAL_DIR`
export _CONDOR_COLLECTOR_HOST=$COLH:
export _CONDOR_MASTER_NAME=collector_
export _CONDOR_DAEMON_LIST="MASTER, COLLECTOR"
export _CONDOR_LOCAL_DIR=$LD/$_CONDOR_MASTER_NAME
export _CONDOR_LOCK=$_CONDOR_LOCAL_DIR/lock
# Forward all the traffic to the main collector
export _CONDOR_CONDOR_VIEW_HOST=localhost:9618

Using localhost to communicate to the master collector is more efficient. But will not work if they are on different hosts with HTCondor version 8.2. For those case you'll need: export _CONDOR_CONDOR_VIEW_HOST=$COLH:9618

Once you have the slave collectors set up, you will want to use them.

The VO Frontend will have to point the Factory to a list of collectors.

The configuration internally will add a line in the Factory configuration file that will set up the glideins to handle the multiple collectors. (You should now see a line like: "<file absfname="web_base/collector_setup.sh" executable="True"/>" after reconfiguring).

Setting the glidein start and rank condition

As with any HTCondor pool, you may need to set the startd start and rank conditions.
For a glidein, you can set this with the <attr/> options:

GLIDEIN_Start=expression
GLIDEIN_Rank=expression

For example:

Internal Configuration

• job.descript => read by the daemon do decide how to work
• attributes.cfg => fixed values, these are published in the Factory ClassAd
• params.cfg => for values the Frontend will change, also published in the Factory ClassAd

Multiple Condor Tarballs

One frequent problem is that one particular condor binary will not run on all compute nodes. Entry points require different architectures, or have different versions of glibc.

The solution is to have multiple condor binaries. The way to do this is to specify a tarball tag in the Factory configuration file.

1. Download the HTCondor binary from the University of Wisconsin site. (Alternatively, You can build it from scratch on the architecture. Refer to HTCondor instructions for this.)

   The glideinwms pilot uses a subset of the condor binaries/libraries. The create_condor_tarball script can be used to reduce space needed on your Factory node. Details on this script can be found in the Components - Tools section of the documentation.

2. Add a new condor_tarball tag to the Factory configuration file:
   There are two different ways you can do this:
   • Put the tarball in a directory owned by the wmsfactory and enter the condor_tarball tag as:

     <glidein ... >
     ...
       <condor_tarballs >
         <condor_tarball os="OS" arch="Arch" tar_file="ZIPPED_TARFILE" version="Condor_Version" />

   • Put the tarball in a directory owned by the wmsfactory and unzip/untar it. Then, enter the condor_tarball tag as:

     <glidein ... >
     ...
       <condor_tarballs >
         <condor_tarball os="OS" arch="Arch" base_dir="DIR_OF_UNTARRED_BINARY" version="Condor_Version" />

   To simplify the configuration, os, arch and version support comma values. This can drastically reduce the number of condor_tarball entries needed in the configuration file.
   
   Consider an example below for default OS as rhel5 and default arch as x86_64. If the Factory admin also wants os and arch information explicitly available, the configuration needs following entries to cover possible combinations.

   <condor_tarball os="default" arch="default" base_dir="dir" version="default"/>
   <condor_tarball os="rhel6" arch="x86_64" base_dir="dir" version="default"/>
   <condor_tarball os="rhel6" arch="default" base_dir="dir" version="default"/>
   <condor_tarball os="default" arch="x86_64" base_dir="dir" version="default"/>

   Above example can be easily consolidated into a single condor_tarball entry as below and the Factory reconfiguration process will internally consider all the combinations. This also applies to the version.

   <condor_tarball os="default,rhel6" arch="default,x86_64" base_dir="dir" version="default"/>

3. Verify your entry point attributes. Each entry point will have the following attr set up. Make sure that this matches the above condor_tarball parameters:
   

   <entry>
      <attrs>
        <attr name="CONDOR_ARCH" const="True" parameter="True" glidein_publish="False" job_publish="False" publish="False" type="string" value="Arch"/>
        <attr name="CONDOR_OS" const="True" parameter="True" glidein_publish="False" job_publish="False" publish="False" type="string" value="Condor_Version"/>
        <attr name="GLEXEC_JOB" const="True" parameter="True" glidein_publish="False" job_publish="False" publish="False" type="string" value="True"/>
      </attrs>
   

   The CONDOR_OS and the CONDOR_ARCH should match the os and arch defined in the tarball tag. If set to "auto", the glidein will decide the appropriate tarball to use for that worker node. By default, the CONDOR_VERSION will be defined globally in <glidein><attrs> and should match the version in the condor_tarball tag. You can overwrite this global version and define one locally in the entry if needed.
4. Reconfigure the Factory using the command:

   ./factory_startup reconfig ../CONFIG_DIR/glideinWMS.xml

5. After reconfig, you can see the tar_file created from the condor distribution in the condor_tarball element in the configuration as (if you populated just tar_file in step 2

   <condor_tarball arch="default" os="default" tar_file="FACTORY_DIR/condor-8.7tgz" version="default"/>

   or (if you populated just base_dir in step 2)

   <condor_tarball arch="default" base_dir="/opt/glideins/git-xen21-master-ps.ini/condor-wms" os="default" tar_file="/var/www/html/glideinwms/factory_service/stage/glidein_master/condor_bin_0.d7bbk9.tgz" version="default"/>

Limiting time spent on a Grid resource

The whole concept of gliding into Grid resources is based on the idea that you are getting those resources on a temporary basis. This implies that you need to leave the slot as soon as possible, else your jobs will simply be killed by the annoyed Grid administrators.
On the other hand, submitting new glideins is not cost free, so you want to keep the resource for at least some period of time.

The glideins have two mechanisms to regulate this:

1. After a specified amount of time, the glidein will enter the RETIRING state. This means, it will wait for the current job to finish (or kill it if it does not end within a configurable timeout) and exit immediately afterwards. This obviously implies that no new jobs will start after it entered that state.
   The two timeouts can be set with the <attr /> options:

   GLIDEIN_Retire_Time=nr_of_seconds
   GLIDEIN_Job_Max_Time=nr_of_seconds

   The two default to 2 and 100 hours.

2. If a glidein is not claimed within a configurable timeout, the glidein will exit.
   The timeout can be set with:

   GLIDEIN_Max_Idle=nr_of_seconds GLIDEIN_Max_Tail=nr_of_seconds

   There are two configurable parameters for this timeout behavior. The first, GLIDEIN_Max_Idle, affects how long a glidein will wait for its first job. The second parameter is how long a glidein will wait to get a subsequent job once its finished its job. The defaults for these are 1200 and 400 seconds, respectively.

An example:

Old-style pseudo-interactive monitoring

The pseudo-interactive monitoring uses a dedicated startd in the glideins for monitoring purposes. This allows for monitoring even when the job starter enters the “Retiring” activity.

The side effect is that you do not have anymore the cross-VM statistics and the names of the slots is also different.

To enable the old mode, use:

Adding custom files/scripts to the glideins

While provided code should cover most of the general purpose use cases, some administrators may have additional needs. For these cases, the Glidein framework provides the possibility to download and process additional files. Both , Factory and Frontend configuration allow to specify lists of files either for all their glideins, for a specific entry (Factory), or job group (Frontend). These files can be scripts (executables), regular files or tarballs and depending on the options can be downloaded at different times (see the custom scripts document for more detils).

Note: Files and subsystems will be downloaded before the scripts. User provided scripts will be executed in the specified order, and before the HTCondor daemons are started up.

• <glidein>
   [<entries><entry>]
   <files>
    <file absfname="script name" executable="True" prefix="cron_prefix" comment="comment"/>

  Path to the custom script. The script will be copied in the Web-accessible area, and when a glidein starts, the glidein startup script will pull it and execute it. If any parameters are needed, they can be specified using <attr />, or stored in a file (see below).
  For more detailed information, see the page dedicated to writing custom scripts.

  <glidein>
   [<entries><entry>]
   <files>
     <file absfname="script name" wrapper="True" comment="comment"/>

  Path to the wrapper custom script. The script will be copied in the Web-accessible area, and will be sourced just before a user job starts starts; i.e. it will become part of the user job wrapper.

• <glidein>
   [<entries><entry>]
   <files>
     <file absfname="local file name" relfname="target file name" const="Bool" executable="False" comment="comment"/>

  Path to the config file. The file will be copied in the Web-accessible area, and pulled by the glidein startup script when a glidein starts. It can be then used by any script (see above).
  Note: Please be cautious in using the const flag; if set to False, the content of the file will not be verified by the glidein startup script and could be tampered in transit by a malicious user. So never put sensitive data (like the switch to disable security checks) in a changeable file.

• <glidein>
   [<entries><entry>]
   <files>
    <file absfname="local file name" untar="True" comment="comment">
      <untar_options cond_attr="conf_sw" dir="dir name" absdir_outattr="attr name"/>

  Sometimes it is useful to transfer a whole set of files, or even directories, and that is much easier to accomplish by means of a tarball. A subsystem is the glidein way to describe a compressed tarball that is delivered to the worker nodes, untarred in a separate directory and advertised to the other scripts.

  • absfname: Path to the custom tar-ball. (like "/tmp/mytar_v12.5.tgz")
  • conf_sw: Name of a configuration switch. (like "ENABLE_KRB5")
    The tarball will be unpacked only if that parameter will be set to 1. Use the <attr /> switch to define the default value. A special name TRUE can be used to always untar it.
  • dir: Name of the subdirectory to untar it in. (like "krb5")
  • absdir_outattr: Name of a variable name. (like "KRB5_SUBSYS_DIR")
    The variable will be set to the absolute path of the directory where the tarball was unpacked, if and only if the unpacking actually happened (else it will not be defined.) ENTRY_ will be prepended if if the <file> directive occurs in an entry.

To solve this, you can set up a Monitoring Collector tree that mirrors that of the VO Pool Collector, and tell the glideins to report there, too.

The configuration syntax is very similar to that of the the VO Pool Collector, but using monitoring_collector instead of collector keyword.
For example:

For more details, see the Frontend documentation.