GlideinWMS The Glidein-based Workflow Management System

HTCondor IDTokens Authorization

 

 

Jump to:
  1. Description
  2. Requirements
  3. VOFrontend, Factory Processes
    1. Out of the Box
    2. Frontend uid Processes
    3. Factory uid Processes
  4. VOFrontend, glideins
    1. Required Steps
    2. Verify IDTOKENS Functionality
    3. Revoking Tokens for Entries
  5. Limitations
  6. Useful Links

Description

This page is out of date but may contain useful information. Please see JWT configuration for current JWT instructions.

This page documents enabling and managing HTCondor IDTokens authorization between VOFrontends and Factories version 3.7.3 or greater, and also between VOFrontends and remote glideins.

Requirements

Requirement Description
Compatible HTCondor and GlideinWMS Versions Version numbers and installation instructions

VOFrontend to Factory IDTokens Authentication

Out of the Box

GlideinWMS comes pre-configured with a list of authentication methods to try for HTCondor interprocess communication. Starting with the GWMS 3.7 series the IDTOKENS method was prepended to the existing method list, which was previously "FS, GSI" . The effect of this is that HTCondor processes owned by the 'condor' uid (i.e. most of them) on a 3.7.x or greater installation will use IDTOKENS auth for internal communication 'out of the box'. If IDTOKENS auth fails, FS, then GSI authentication are tried next. An existing GWMS Factory or VOFrontend that is upgraded to 3.7.x will mostly authenticate internally with IDTOKENS and externally with GSI.

Frontend uid Processes

The VOFrontend runs some HTCondor processes under the vofrontend owner uid, which is typically 'frontend'. To make these processes authenticate with IDTOKENS, the VOFrontend admin must manually generate a token for these processes to use. One way to do this as is follows:

First, verify that the 'frontend' uid is the one running VOFrontend processes: 

[root@fermicloud170 ~]# ps auxww | grep glidein | grep -v grep
frontend    4017  0.0  1.9 420712 36592 ?        S<   14:18   0:13 python /usr/sbin/glideinFrontend /var/lib/gwms-frontend/vofrontend

Query the HTCondor system for VOFrontend processes and what authentication method they are using: 

[root@fermicloud170 ~]# condor_status -any -constraint 'regexp("glide.*",MyType)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glidefrontendmonitor Frontend-master-v1_0 vofrontend_service@fermicloud170.fnal.gov GSI
glideresource el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov GSI
glideresource el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov GSI
glideresource el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov GSI
[root@fermicloud170 ~]#

We see that we need an IDTOKEN with the subject 'vofrontend_service@fermicloud170.fnal.gov' to replace existing GSI authentication. The following commands accomplish this: 

[root@fermicloud170 ~]# mkdir -p ~frontend/.condor/tokens.d
[root@fermicloud170 ~]# condor_token_create -id vofrontend_service@fermicloud170.fnal.gov -key POOL > ~frontend/.condor/tokens.d/frontend.$HOSTNAME.idtoken
[root@fermicloud170 ~]# chown -R frontend:frontend ~frontend/.condor
[root@fermicloud170 ~]# chmod 600 ~frontend/.condor/tokens.d/*

Verify that the processes are now using IDTOKENS authentication: 

[root@fermicloud170 ~]# condor_status -any -constraint 'regexp("glide.*",MyType)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glidefrontendmonitor Frontend-master-v1_0 vofrontend_service@fermicloud170.fnal.gov IDTOKENS
glideresource el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov IDTOKENS
glideresource el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov IDTOKENS
glideresource el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud170.fnal.gov IDTOKENS
[root@fermicloud170 ~]#

Factory uid Processes

Most HTCondor Factory processes run under the uid 'condor', and in 3.7.x are already using IDTOKENS authentication. Some processes run under the Factory owner uid, typically 'gfactory'. The factory operator can verify this uid with the following query: 

[root@fermicloud161 ~]# ps -auxww | grep glideinwms | grep -v grep
gfactory    4318  0.1  1.5 416720 28884 ?        S   14:15   0:22 /bin/python /usr/lib/python2.7/site-packages/glideinwms/factory/glideFactoryEntryGroup.pyc 4311 60 5 /var/lib/gwms-factory/work-dir el6_osg34:el7_osg34:el7_osg35 0
[root@fermicloud161 ~]#

Look for HTCondor processes authenticating as user 'gfactory':

[root@fermicloud161 ~]# condor_status -any  -constraint 'regexp("gfactory.*",AuthenticatedIdentity)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glidefactory el6_osg34@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov FS
glidefactoryclient el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov FS
glidefactory el7_osg34@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov FS
glidefactoryclient el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov FS
glidefactory el7_osg35@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov FS
glidefactoryclient el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov FS
glidefactoryglobal gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov FS
[root@fermicloud161 ~]#

Generate an IDTOKEN for gfactory to use that presents as 'gfactory@fermicloud161.fnal.gov': 

[root@fermicloud161 ~]# mkdir -p ~gfactory/.condor/tokens.d
[root@fermicloud161 ~]# condor_token_create -id gfactory@$HOSTNAME > ~gfactory/.condor/tokens.d/gfactory.$HOSTNAME.idtoken
[root@fermicloud161 ~]# chown -R gfactory:gfactory ~gfactory/.condor/tokens.d
[root@fermicloud161 ~]# chmod 600 ~gfactory/.condor/tokens.d/*
[root@fermicloud161 ~]

Check that the new IDTOKEN is authenticating:

[root@fermicloud161 ~]# condor_status -any  -constraint 'regexp("gfactory.*",AuthenticatedIdentity)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glidefactory el6_osg34@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactoryclient el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactory el7_osg34@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactoryclient el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactory el7_osg35@gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactoryclient el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main gfactory@fermicloud161.fnal.gov IDTOKENS
glidefactoryglobal gfactory_instance@gfactory_service gfactory@fermicloud161.fnal.gov IDTOKENS
[root@fermicloud161 ~]#

Factories also have HTCondor processes authenticating from other machines (like VOFrontends) that do not use IDTOKENS auth, but can be made to do so. This can be looked for with a query such as: 

[root@fermicloud161 ~]# condor_status -any  -constraint '!regexp("IDTOKENS",AuthenticationMethod)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glideclient 67748_el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov GSI
glideclient 67748_el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov GSI
glideclient 67748_el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov GSI
glideclientglobal gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov GSI
[root@fermicloud161 ~]#

These are HTCondor processes controlled by the VOFrontend that we made an IDTOKEN for earlier. For these to authenticate with IDTOKENS, the VOFrontend (fermicloud170) needs to present an IDTOKEN trusted by the factory (fermicloud161). This is done in two steps. First, the Factory admin generates a token the Factory trusts: 

root@fermicloud161 ~]# condor_token_create -id vofrontend_service@fermicloud161.fnal.gov -key POOL > ~/frontend.fermicloud161.fnal.gov.idtoken

Next, the VOFrontend admin securely transfers the new IDTOKEN to where the VOFrontend 'frontend' uid can use it: 

[root@fermicloud170 ~]# scp root@fermicloud161.fnal.gov:/root/frontend.fermicloud161.fnal.gov.idtoken  ~frontend/.condor/tokens.d/frontend.fermicloud161.fnal.gov.idtoken
frontend.fermicloud161.fnal.gov.idtoken                                                                           100%  278   335.3KB/s   00:00
[root@fermicloud170 ~]# chown -R frontend:frontend ~frontend/.condor/tokens.d
[root@fermicloud170 ~]# chmod 600 ~frontend/.condor/tokens.d/*

The Factory admin can now verify that IDTOKENS are being used to authenticate by these processes: 

[root@fermicloud161 ~]# condor_status -any  -constraint 'regexp("vofrontend_service.*",AuthenticatedIdentity)' -af MyType Name AuthenticatedIdentity AuthenticationMethod
glideclient 67748_el6_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov IDTOKENS
glideclient 67748_el7_osg34@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov IDTOKENS
glideclient 67748_el7_osg35@gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov IDTOKENS
glideclientglobal gfactory_instance@gfactory_service@Frontend-master-v1_0.main vofrontend_service@fermicloud161.fnal.gov IDTOKENS
[root@fermicloud161 ~]#

VOFrontend to glidein IDTokens Authentication

Starting with version 3.7.3, when the VOFrontend requests glideins from a Factory Entry, it sends a self-generated IDTOKEN to be included in the glidein payload. By default, the Frontend generates IDTOKENS lasting 24 hours using the key file /var/lib/gwms-frontend/passwords.d/FRONTEND. Since v3.10.2 the duration, idtoken_lifetime, and key file, idtoken_keyname, are configurable. The newly generated IDTOKEN is encrypted and passed to the Factory, then on to the CE. When the glidein starts, it attempts to use the payload IDTOKEN to authenticate back to the VOFrontend collector. If IDTOKENS authentication fails for some reason, GSI authentication to the VOFrontend is attempted.

NOTE that HTCSS is not renewing this IDTOKEN after the Glidein starts (for HTCondor batch systems) or after the Glidein is queued (all others)

Required Steps

  • This authentication chain requires that the factory and frontend are both upgraded to 3.7.3 or greater.

Verifying IDTOKENS Functionality in the Glidein

Once the required steps have been implemented, there are a few different ways to verify that IDTOKENS authentication between the glidein and VOFrontend Collector is working. These include:

  • Submit a condor job to the Frontend with condor requirements that make it run on the newly modified Factory Entry. Make sure the job has a 'printenv' or equivalent in it to show the environment the glidein runs the job under. When the job completes, check the output from the 'printenv' command. If a token was generated by the VOFrontend, forwarded to the Factory, forwarded again to the CE, and integrated into the glidein enviroment, GLIDEIN_CONDOR_TOKEN will be set and its value will include the GLIDEIN_Site name, for example if the GLIDEIN_Site is named 'el7_osg34', GLIDEIN_CONDOR_TOKEN will contain the string 'el7_osg34.idtoken'.
  • Edit the CERTIFICATE_MAPFILE of the VOFrontend to comment out the GSI entry for the CE that corresponds to the Factory Entry. This disables GSI communication from the VOFrontend to the CE so if glideins are still connecting back to the VOFrontend from this CE, you are guaranteed to be using IDTOKENS communication .
  • Change the condor debug level on both the VOFrontend and glidein to ALL_DEBUG=D_FULLDEBUG,D_SECURITY. The condor logs will become very verbose, but they will report every authentication attempt and what method was used. It is recommended to reset ALL_DEBUG to something less verbose after verification/debugging.

Revoking Tokens for Entry Points

  • Each Glidein_Site that a VOFrontend submits to is sent an IDTOKEN generated from a password in /var/lib/gwms-frontend/passwords.d. HTCondor authenticates these IDTOKENS with its copy of the password in /etc/condor/passwords.d. When the 3.7.3 rpm is installed, a default frontend password is stored in /etc/condor/passwords.d/FRONTEND, which is copied to and MUST MATCH the password file ~frontend/passwords.d/FRONTEND for authentication to succeed. To invalidate all tokens genereated from the default password, change it on the condor side 
    # openssl rand -base64 64 | /usr/sbin/condor_store_cred -u frontend@$HOSTNAME -f /etc/condor/passwords.d/FRONTEND  add
    To set the VOFrontend generating and distributing tokens with this password, copy it to where the frontend expects it: 
    # /bin/cp /etc/condor/passwords.d/FRONTEND ~frontend/passwords.d/FRONTEND
# chown frontend:frontend ~frontend/passwords.d/FRONTEND
    To give each glidein site its own individually revokable password: 
    ###################################################
# #configure uniqe per-site password
# #for glidein site 'HCC_US_Omaha_crane_gpu'
###################################################
# export GLIDEIN_SITE='HCC_US_Omaha_crane_gpu'
# openssl rand -base64 64 | /usr/sbin/condor_store_cred -u frontend@$HOSTNAME -f /etc/condor/passwords.d/${GLIDEIN_SITE} add
# /bin/cp /etc/condor/passwords.d/${GLIDEIN_SITE} ~frontend/passwords.d/${GLIDEIN_SITE}
# chown frontend:frontend ~frontend/passwords.d/${GLIDEIN_SITE}
    • The frontend processes will notice the new password and generate a new IDTOKEN to be sent to that glidein site, invalidating old IDTOKENS held by the site. The new valid IDTOKEN will eventually be passed to the site unless steps are taken to prevent it.

Limitations

  • More fine grained management of IDTOKENS lifetime and HTCondor capabilities is possible and desirable. It has not been addressed in this recipe.
  • SciToken authentication and its interactions with Condor and GlideinWMS are addressed here.

VOFrontend, VOCollector and Schedds

The Frontend needs to query both the Users Collector and Schedds and to publish ClassAds to the former. If they are on the same host, the y will use the FS authentication already in the default configuration. Otherwise, you have to configure a valid authentication method. If GSI is no more supported you can create a Frontend token: 

        condor_token_create -key GPGRID_IDTOKENS_SIGNING_KEY -token FR_NAME.token -lifetime 31556952 -authz READ -authz WRITE -authz DAEMON -identity FR_NAME@FR_HOST@DOMAIN

NOTE The required write and deamon authorizations. The key can be any file in the User Collector key directory, even the FRONTEND key. The token has to be placed in the Frontend token directory, ~FRONTEND_USER/.condor/tokens.d

Useful Links

Banner image by Madhusudan Katti used under Creative Commons license.
Original Home URL: http://glideinwms.fnal.gov. GlideinWMS email support: glideinwms-support at fnal.gov