GlideinWMS The Glidein-based Workflow Management System

HTCondor IDTokens Authorization

 

 

Jump to:
  1. Description
  2. Requirements
  3. VOFrontend, Factory Processes
    1. Out of the Box
    2. Frontend uid Processes
    3. Factory uid Processes
  4. VOFrontend, glideins
    1. Required Steps
    2. Verify Token Functionality
    3. Revoking Tokens for Entries
  5. Limitations
  6. Useful Links

Description

This page documents several for enabling and managing HTCondor IDTokens authorization between VOFrontends and Factories, and also between VOFrontends and glideins on Compute Elements (CE's).

Requirements

Requirement Description
Compatible HTCondor and GlideinWMS Versions Version numbers and installation instructions

VOFrontend to Factory IDTokens Authentication

Out of the Box

GlideinWMS comes pre-configured with a list of authentication methods to try for HTCondor interprocess communication. Starting with version 3.7.1 the IDTOKENS method was prepended to the existing method list, which was previously "FS, GSI" . HTCondor processes owned by the 'condor' uid (i.e. most of them) on a 3.7.1 installation will use IDTOKENS auth for internal communication 'out of the box'. If IDTOKENS auth fails, FS, then GSI authentication are tried next. A GWMS Factory or VOFrontend that is upgraded to 3.7.1 will mostly authenticate internally with IDTOKENS and externally with GSI.

Frontend uid Processes

The VOFrontend runs some HTCondor processes under the vofrontend owner uid, which is typically 'frontend'. To make these processes authenticate with IDTOKENS, the VOFrontend admin must manually generate a token for these processes to use. One way to do this as is follows:

First, verify that the 'frontend' uid is the one running VOFrontend processes: 

[root@fermicloud170 ~]# ps auxww | grep glidein | grep -v grep
frontend    4017  0.0  1.9 420712 36592 ?        S<   14:18   0:13 python /usr/sbin/glideinFrontend /var/lib/gwms-frontend/vofrontend

Query the HTCondor system for VOFrontend processes and what authentication method they are using: 

[root@fermicloud170 ~]# condor_status -any -constraint 'regexp("glide.*",MyType)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glidefrontendmonitor Frontend-master-v1_0 vofrontend_service@fermicloud170.fnal.gov GSI
glideresource el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov GSI
glideresource el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov GSI
glideresource el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov GSI
[root@fermicloud170 ~]#

We see that we need an IDTOKEN with the subject 'vofrontend_service@fermicloud170.fnal.gov' to replace existing GSI authentication. The following commands accomplish this: 

[root@fermicloud170 ~]# mkdir -p ~frontend/.condor/tokens.d
[root@fermicloud170 ~]# condor_token_create -id vofrontend_service@fermicloud170.fnal.gov -key POOL > ~frontend/.condor/tokens.d/frontend.$HOSTNAME.idtoken
[root@fermicloud170 ~]# chown -R frontend:frontend ~frontend/.condor
[root@fermicloud170 ~]# chmod 600 ~frontend/.condor/tokens.d/*

Verify that the processes are now using IDTOKENS authentication: 

[root@fermicloud170 ~]# condor_status -any -constraint 'regexp("glide.*",MyType)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glidefrontendmonitor Frontend-master-v1_0 vofrontend_service@fermicloud170.fnal.gov IDTOKENS
glideresource el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov IDTOKENS
glideresource el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov IDTOKENS
glideresource el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov IDTOKENS
[root@fermicloud170 ~]#

Factory uid Processes

Most HTCondor Factory processes run under the uid 'condor', and as of version 3.7.1 are already using IDTOKENS authentication. Some processes run under the Factory owner uid, typically 'gfactory'. The factory operator can verify this uid with the following query: 

[root@fermicloud161 ~]# ps -auxww | grep glideinwms | grep -v grep
gfactory    4318  0.1  1.5 416720 28884 ?        S   14:15   0:22 /bin/python /usr/lib/python2.7/site-packages/glideinwms/factory/glideFactoryEntryGroup.pyc 4311 60 5 /var/lib/gwms-factory/work-dir el6_osg34:el7_osg34:el7_osg35 0
[root@fermicloud161 ~]#

Look for HTCondor processes authenticating as user 'gfactory': 

[root@fermicloud161 ~]# condor_status -any  -constraint 'regexp("gfactory.*",AuthenticatedIdentity)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glidefactory el6_osg34@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov FS
glidefactoryclient el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov FS
glidefactory el7_osg34@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov FS
glidefactoryclient el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov FS
glidefactory el7_osg35@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov FS
glidefactoryclient el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov FS
glidefactoryglobal gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov FS
[root@fermicloud161 ~]#

Generate an IDTOKEN for gfactory to use that presents as 'gfactory@fermicloud161.fnal.gov': 

[root@fermicloud161 ~]# mkdir -p ~gfactory/.condor/tokens.d
[root@fermicloud161 ~]# condor_token_create -id gfactory@$HOSTNAME > ~gfactory/.condor/tokens.d/gfactory.$HOSTNAME.idtoken
[root@fermicloud161 ~]# chown -R gfactory:gfactory ~gfactory/.condor/tokens.d
[root@fermicloud161 ~]# chmod 600 ~gfactory/.condor/tokens.d/*
[root@fermicloud161 ~]#

Check that the new IDTOKEN is authenticating: 

[root@fermicloud161 ~]# condor_status -any  -constraint 'regexp("gfactory.*",AuthenticatedIdentity)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glidefactory el6_osg34@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactoryclient el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactory el7_osg34@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactoryclient el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactory el7_osg35@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactoryclient el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactoryglobal gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov IDTOKENS
[root@fermicloud161 ~]#

Factories also have HTCondor processes authenticating from other machines (like VOFrontends) that do not use IDTOKENS auth, but can be made to do so. This can be looked for with a query such as: 

[root@fermicloud161 ~]# condor_status -any  -constraint '!regexp("IDTOKENS",AuthenticationMethod)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glideclient 67748_el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov GSI 
glideclient 67748_el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov GSI
glideclient 67748_el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov GSI
glideclientglobal gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov GSI
[root@fermicloud161 ~]#

These are HTCondor processes controlled by the VOFrontend that we made an IDTOKEN for earlier. For these to authenticate with IDTOKENS, the VOFrontend (fermicloud170) needs to present an IDTOKEN trusted by the factory (fermicloud161). This is done in two steps. First, the Factory admin generates a token the Factory trusts: 

root@fermicloud161 ~]# condor_token_create -id vofrontend_service@fermicloud161.fnal.gov -key POOL > ~/frontend.fermicloud161.fnal.gov.idtoken

Next, the VOFrontend admin securely transfers the new IDTOKEN to where the VOFrontend 'frontend' uid can use it: 

[root@fermicloud170 ~]# scp root@fermicloud161.fnal.gov:/root/frontend.fermicloud161.fnal.gov.idtoken  ~frontend/.condor/tokens.d/frontend.fermicloud161.fnal.gov.idtoken
frontend.fermicloud161.fnal.gov.idtoken                                                                           100%  278   335.3KB/s   00:00
[root@fermicloud170 ~]# chown -R frontend:frontend ~frontend/.condor/tokens.d
[root@fermicloud170 ~]# chmod 600 ~frontend/.condor/tokens.d/*

The Factory admin can now verify that IDTOKENS are being used to authenticate by these processes: 

[root@fermicloud161 ~]# condor_status -any  -constraint 'regexp("vofrontend_service.*",AuthenticatedIdentity)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glideclient 67748_el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov IDTOKENS
glideclient 67748_el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov IDTOKENS
glideclient 67748_el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov IDTOKENS
glideclientglobal gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov IDTOKENS
[root@fermicloud161 ~]#

VOFrontend to glidein IDTokens Authentication

When the VOFrontend requests glideins from a Factory Entry, it checks what version of HTCondor the Factory Entry is sending along with the glidein code to the CE. If the version of HTCondor it finds supports IDTOKENS, the VOFrontend generates an IDTOKEN for the glidein to use to authenticate back to the VOFrontend collector. The newly generated IDTOKEN is encrypted and passed to the Factory, which passes it securely to the CE along with the glidein code. When the glidein starts, if IDTOKENS authentication fails for some reason, GSI is used to authenticate back to the VOFrontend.

Required Steps

  • This authentication chain requires some manual steps by admins:
    • The VOFrontend admin must upgrade to a GWMs version that supports IDTOKENS, 3.7.1 or later.
    • The Factory admin must upgrade to a GWMs version that supports IDTOKENS, 3.7.1 or later.
    • The Factory admin must configure the Factory Entry to use an HTCondor version that supports IDTOKENS, 8.9.1 or later.
    • The Factory admin must ensure that the entry is advertising the afore-mentioned HTCondor version in its classad. Example: 
      <glidein>
   <entries>
      <entry name=..... >
         <attrs>
            <attr name="CONDOR_VERSION"  value="8.9.5" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" />

Verifying Token Functionality in the Glidein

Once the required steps have been implemented, there are a few different ways to verify that IDTOKENS authentication is working. These include:

  • Submit a condor job to the Frontend with condor requirements that make it run on the newly modified Factory Entry. Make sure the job has a 'printenv' or equivalent in it to show the environment the glidein runs the job under. When the job completes, check the output from the 'printenv' command. If a token was generated by the VOFrontend, forwarded to the Factory, forwarded again to the CE, and integrated into the glidein enviroment, GLIDEIN_CONDOR_TOKEN will be set and its value will include the GLIDEIN_Site name, for example if the GLIDEIN_Site is named 'el7_osg34', GLIDEIN_CONDOR_TOKEN will contain the string 'el7_osg34.idtoken'.
  • Edit the CERTIFICATE_MAPFILE of the VOFrontend to comment out the GSI entry for the CE that corresponds to the Factory Entry. This disables GSI communication from the VOFrontend to the CE so if glideins are still connecting back to the VOFrontend from this CE, you are guaranteed to be using IDTOKENS communication .
  • Change the condor debug level on both the VOFrontend and glidein to ALL_DEBUG=D_FULLDEBUG,D_SECURITY. The condor logs will become very verbose, but they will report every authentication attempt and what method was used. It is recommended to reset ALL_DEBUG to something less verbose after verification/debugging.

Revoking Tokens for Entry Points

  • Each Glidein_Site that a VOFrontend submits to is sent IDTOKENS generated from a unique password for that site. If the password is changed, all existing IDTOKENS for that site previously generated become invalid, and IDTOKENS authentication for any running glideins at that site will stop working. New glideins requested by the Frontend will use the new password, and valid tokens will be sent to the Factory and on to the CE's. A quick way to change a token generating password from the command line is 
    # export GLIDEIN_SITE='HCC_US_Omaha_crane_gpu'
# openssl rand -base64 64 | sudo /usr/sbin/condor_store_cred -u frontend@$HOSTNAME -f /etc/condor/passwords.d/${GLIDEIN_SITE} add

Limitations

  • More fine grained management of IDTOKENS lifetime and HTCondor capabilities is possible and desirable. It has not been addressed in this recipe.
  • SciToken authentication and its interactions with Condor and GlideinWMS are addressed here.

Useful Links

Banner image by Madhusudan Katti used under Creative Commons license.
Original Home URL: http://glideinwms.fnal.gov. GlideinWMS email support: glideinwms-support at fnal.gov