GlideinWMS The Glidein-based Workflow Management System


This page documents a recipe for enabling and managing HTCondor Token Authorization between Frontends and Glidein Entry Points (CE's).


Requirement Description
GWMS Factory Version >= 3.7 The Factory can be either a fresh install from yum/RPM or a yum upgrade, as long as it meets the version requirements.
GWMS Frontend Version >= 3.7 The Frontend, like the Factory, can be made to meet version requirements via either a fresh install or upgrade.
HTCondor Version >= 8.9.1 HTCondor that the Frontend uses must be a token supporting release. The HTCondor tarballs delivered by Factory entry points also need to be a token supporting release. It is not necessary to upgrade the main Factory HTCondor as long as the GlideinWMS version of the factory complies with the >= 3.7 requirement.
Sudo access for 'frontend' user on gwms-frontend machine The frontend user needs to be able to create passwords and tokens derived from these passwords. The following two lines must be present in /etc/sudoers on the gwms-frontend machine:
frontend ALL=(ALL) NOPASSWD:SETENV: /usr/sbin/condor_store_cred
frontend ALL=(ALL) NOPASSWD:SETENV: /usr/bin/condor_token_create

Upgrade and Configure Frontend Condor

  • Upgrade condor on the frontend machine to a token supporting version. Instructions for installing the from the official HTCondor repo are here . It is often easier to install condor from the OSG repos. At the time of this writing (2/11/20) the easist way to install condor 8.9.5 was:
    # yum -y --enablerepo osg-upcoming install condor
  • Make condor passwords and tokens directories if they don't already exist
    # mkdir -p /etc/condor/passwords.d /etc/condor/tokens.d
  • The utility /usr/local/bin/gwms_condor_ping should be present if 3.7 has been installed correctly. If it is not present, create it now as it is useful for verifying TOKEN functionality
    # cat /usr/local/bin/gwms_condor_ping
    # a convenience wrapper for condor_ping
    # returns a table of authorizations, useful to see
    # if token_auth has been correctly set up
    if [ "$1" = "" ]; then
    condor_ping -addr "<$(host $HOST | sed -e 's/.* //'):9618>" -table ALL
  • As token authentication is not enabled, the output of gwms_condor_ping should show FS or GSI authentication being used:
    # gwms_condor_ping
             Instruction Authentication Encryption Integrity Decision Identity
                   ALLOW             FS       none       MD5    ALLOW
                    READ           none       none      none    ALLOW unauthenticated@unmapped
                   WRITE             FS       none       MD5    ALLOW
              NEGOTIATOR             FS       none       MD5    ALLOW
           ADMINISTRATOR             FS       none       MD5    ALLOW
                   OWNER             FS       none       MD5    ALLOW
                  CONFIG             FS       none       MD5    ALLOW
                  DAEMON             FS       none       MD5    ALLOW
        ADVERTISE_STARTD             FS       none       MD5    ALLOW
        ADVERTISE_SCHEDD             FS       none       MD5    ALLOW
        ADVERTISE_MASTER             FS       none       MD5    ALLOW
  • Add the following lines to the frontends condor configuration:
    ALLOW_READ = *
    QUEUE_SUPER_USER = root, condor
  • Finally:
    • create a condor pool password
    • create a condor admin token
    • restart condor
  • # openssl rand -base64 64 | condor_store_cred -u condor_pool@$HOSTNAME \
    -f /etc/condor/passwords.d/POOL add
    # condor_token_create -identity condor@$HOSTNAME -token condor.$HOSTNAME.token
    # /bin/cp /root/.condor/tokens.d/condor.$HOSTNAME.token /etc/condor/tokens.d/admin.token
    # systemctl restart condor
  • The output of gwms_condor_ping for the root user should now show that it is using TOKEN authentication for most services
    [root@fermicloud158 ~]# gwms_condor_ping
             Instruction Authentication Encryption Integrity Decision Identity
                   ALLOW          TOKEN       none       MD5    ALLOW
                    READ           none       none      none    ALLOW unauthenticated@unmapped
                   WRITE          TOKEN       none       MD5    ALLOW
              NEGOTIATOR          TOKEN       none       MD5    ALLOW
           ADMINISTRATOR          TOKEN       none       MD5    ALLOW
                   OWNER          TOKEN       none       MD5    ALLOW
                  CONFIG          TOKEN       none       MD5    ALLOW
                  DAEMON          TOKEN       none       MD5    ALLOW
        ADVERTISE_STARTD          TOKEN       none       MD5    ALLOW
        ADVERTISE_SCHEDD          TOKEN       none       MD5    ALLOW
        ADVERTISE_MASTER          TOKEN       none       MD5    ALLOW
  • Check the condor logs for unusual authentication errors.

Upgrade Factory Tarballs

  • Go to the HTCondor Downloads page and download a 64bit stripped compressed tarball from the "HTCondor Binaries and Source" section. These instructions have been tested with tarballs condor-8.9.1-x86_64_RedHat7-stripped.tar.gz through condor-8.9.5-x86_64_RedHat7-stripped.tar.gz
  • Copy your downloaded tarball to the /var/lib/gwms-factory/condor directory on the Factory. Untar it
                tar xvzf condor-8.9.5-x86_64_RedHat7-stripped.tar.gz 
  • Add a line to the <condor_tarballs> section of the Factory XML configuration files that points to the condor tarball you just downloaded
          <condor_tarball arch="x86_64" base_dir="/var/lib/gwms-factory/condor/condor-8.9.5-x86_64_RedHat7-stripped" os="rhel7" version="8.9.5"/>
  • Modify the <attrs> section of an <entry> configuration in the Factories XML to use the newly configured condor_tarball
    <entry name="some_site_being_modified" ...>
    <attr name="CONDOR_ARCH" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="x86_64"/>
    <attr name="CONDOR_OS" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="rhel7"/>
    <attr name="CONDOR_VERSION" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="8.9.5"/>
  • Reconfigure the factory so the XML changes take effect
    # systemctl stop gwms-factory.service
    # /usr/sbin/gwms-factory reconfig
    # systemctl start gwms-factory.service

Verify Token Functionality

  • Checklist for Frontend prior to full chain token authentication
    • GWMS >= 3.7 has been installed on Frontend, or Frontend has been modified to run git branch v37/23092. In either case, the following scripts should exist:
      • /usr/sbin/frontend_condor_token
      • /usr/local/bin/gwms_condor_ping
    • gwms_condor_ping shows TOKEN authentication for 'root' user
    • 'frontend' user present in /etc/sudoers as documented in Requirements section
    • GWMS >= 3.7 has been installed on Factory, or Factory has been modified to run git branch v37/23092.
    • An Entry Point on the factory has been configured to deliver a condor tarball >= 8.9.1
  • Submit a condor job to the Frontend with condor requirements that make it run on the newly modified Factory Entry Point. Make sure the job has a 'printenv' or equivalent in it to show the environment the job runs under.
  • When the job completes, check the output from the 'printenv' command. If a token was generated by the Frontend, forwarded to the Factory, forwarded again to the CE, and integrated into the glidein enviroment, GLIDEIN_CONDOR_TOKEN will be set and its value will include the GLIDEIN_Site name, for example for GLIDEIN_Site el7_ GLIDEIN_CONDOR_TOKEN=/tmp/glide_SpYzMq/ticket/el7_osg34_token.

Revoking Tokens for Entry Points

  • Each Glidein_Site that a Frontend submits to is sent condor_tokens generated from a unique password for that site. If the password is changed, all existing tokens for that site become invalid, and token authentication for any running glideins at that site will stop working. New glideins requested by the Frontend will use the new password, and valid tokens will be sent to the Factory and on to the CE's. A quick way to change a token generating password from the command line is
                # export GLIDEIN_SITE='HCC_US_Omaha_crane_gpu'
                # openssl rand -base64 64 | sudo /usr/sbin/condor_store_cred -u frontend@$HOSTNAME -f /etc/condor/passwords.d/${GLIDEIN_SITE} add


  • This recipe enables Condor TOKEN authentication but still allows GSI authentication if a TOKEN is revoked. This may not be the desired behavior.
  • More fine grained authorization scope and management of token lifetime is possible and desirable. Consider this implementation 0.1
  • SciToken authentication and its interactions with Condor and GlideinWMS are not addressed in this document.