GlideinWMS The Glidein-based Workflow Management System

Description

This page documents a recipe for enabling and managing SciTokens Authorization between Glidein Factory Entries and the Compute Elements they submit to.

Requirements

htcondor-ce.noarch          4.4.1-1.osg35.el7    @osg
htcondor-ce-client.noarch   4.4.1-1.osg35.el7    @osg
htcondor-ce-condor.noarch   4.4.1-1.osg35.el7    @osg

Obtain a SciToken

Method 1: follow instructions at https://opensciencegrid.org/technology/software/requesting-tokens/#requesting-tokens

Method 2: generate your own self-signed Scitokens. You need the python scitokens library and admin access to a web server that the Factory, Frontend, and CE can all access.

The following is an example of Method 2 on website 'jobsub.fnal.gov'

1.  yum install python2-scitokens or pip install scitokens


2. Generate some keys

scitokens-admin-create-key --create-keys \
                           --pem-private > scitoken.private.pem

scitokens-admin-create-key --private-keyfile scitoken.private.pem\
                           --jwks-private > test.scitoken.private.jwks

scitokens-admin-create-key --private-keyfile test.scitoken.private.pem \
                           --jwks-public > test.scitoken.public.jwks

3. copy keys over to jobsub.fnal.gov, create some other files in the document root directory:
ssh fnalu.fnal.gov
cd /web/sites/j/jobsub.fnal.gov/htdocs
mkdir -p  oauth2/certs  .well-known
# create file  .well-known/openid-configuration with these contents:
cat .well-known/openid-configuration

{
 "issuer":"https://jobsub.fnal.gov",
 "jwks_uri":"https://jobsub.fnal.gov/oauth2/certs/test.scitoken.public.jwks"
}

exit #return to original machine that you generated the keys on

scp test.scitoken.public.jwks fnalu:/web/sites/j/jobsub.fnal.gov/htdocs/oauth2/certs



3. Generate a token to perform condor operations on  an htcondor-ce. NOTE: the scitokens-admin-create-token utility installed by
   pip or yum generates tokens that are only valid for 10 minutes.  If this is not long enough for your tests copy the utility
   (it is a python script) and modify to taste.

#for ce fermicloud349.fnal.gov
scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem \
                             --issuer https://jobsub.fnal.gov sub="condor"\
                             scope="condor"\
                             aud="fermicloud349.fnal.gov:9619"  > fermicloud349.scitoken

#for ce itb-ce2.chtc.wisc.edu
scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem\
                             --issuer https://jobsub.fnal.gov sub="condor"\
                             scope="condor"\
                             aud="itb-ce2.chtc.wisc.edu:9619" > itb-ce2.chtc.wisc.edu.scitoken

          

Configure Factory to submit your SciToken

1. Check that SCITOKENS authentication is enabled on the Factory. For example:

   [root@fermicloud359]# condor_config_val SEC_DEFAULT_AUTHENTICATION_METHODS
   SCITOKENS,TOKEN,FS,GSI
                 

2. Place the correctly named, previously genereated SciToken in the correct directory so that the VOFrontend knows to send it on to the Factory, which uses it to authenticate with the CE.

   In this example, the SciToken must be named el7_osg35.scitoken to associate it with Factory entry el7_osg35 The SciToken goes in directory
   /var/lib/gwms-frontend/tokens.d/
   

   As SciTokens expire, a mechanism must be in place to renew it periodically. A cron job running 'scitokens-admin-create-token' would suffice for development purposes, but some other mechanism would be used in a production environment.

Configure CE to accept your SciToken

1. Check that htcondor on your CE accepts SCITOKENS as an authentication method:

   root@fermicloud349 ~]# condor_ce_config_val -dump | grep SCITOKENS
   COLLECTOR.SEC_ADVERTISE_STARTD_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
   COLLECTOR.SEC_READ_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
   COLLECTOR.SEC_WRITE_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
   MASTER.SEC_DEFAULT_AUTHENTICATION_METHODS = SCITOKENS, FS, GSI
   SCHEDD.SEC_READ_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
   SCHEDD.SEC_WRITE_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
   SCITOKENS_SERVER_AUDIENCE = $(COLLECTOR_HOST)
   SEC_CLIENT_AUTHENTICATION_METHODS = SCITOKENS, FS, GSI, PASSWORD
   SEC_DEFAULT_AUTHENTICATION_METHODS = SCITOKENS,FS, GSI
                 

2. Check that an entry in the CE's CONDOR_MAPFILE assigns your SciToken to a valid user. In this example SciTokens issued by 'https://jobsub.fnal.gov' are mapped to user 'osg'

   [root@fermicloud349 ~]# condor_ce_config_val -dump | grep mapfile
   CERTIFICATE_MAPFILE = /etc/condor-ce/condor_mapfile
   [root@fermicloud349 ~]# cat /etc/condor-ce/condor_mapfile
   SCITOKENS https://jobsub.fnal.gov osg
   GSI (.*) GSS_ASSIST_GRIDMAP
   GSI "(/CN=[-.A-Za-z0-9/= ]+)" \1@unmapped.htcondor.org
   CLAIMTOBE .* anonymous@claimtobe
   FS "^(root|condor)$" \1@daemon.htcondor.org
   FS "(.*)" \1
   [root@fermicloud349 ~]#
                 

Verify Token Functionality

Here we see that the freshly installed SciToken can be decoded by command line tools and that condor_ping to the CE's collector authenticates some condor operations as user 'osg' as configured earlier in fermicloud349's condor_mapfile:

[root@fermicloud359 ~]# _condor_SCITOKENS_FILE=/var/lib/gwms-frontend/tokens.d/el7_osg35.scitoken condor_ping -address "<$(host fermicloud349.fnal.gov | awk '{print $4}'):9619>" -table ALL
         Instruction Authentication Encryption Integrity Decision Identity
               ALLOW      SCITOKENS       none       MD5    ALLOW osg@users.htcondor.org
                READ           none       none      none    ALLOW unauthenticated@unmapped
               WRITE           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
          NEGOTIATOR           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
       ADMINISTRATOR           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
               OWNER           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
              CONFIG           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
              DAEMON           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
    ADVERTISE_STARTD      SCITOKENS       none       MD5    ALLOW osg@users.htcondor.org
    ADVERTISE_SCHEDD           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
    ADVERTISE_MASTER           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
[root@fermicloud359 ~]#

Verify that submitting a job to the Factories entry point uses SciTokens authentication by adding D_SECURITY to the ALL_DEBUG condor setting on the appropriate condor daemons. The condor logs will show what authentication was used, and it should be SCITOKENS between the factory collector and the CE schedd.

A second verification method is to disable GSI communication between the Factory and the CE by commenting out the line in the Factories CONDOR_MAPFILE that does the GSI mapping for the CE, and submit a job. If the factory can still submit jobs to the CE, SCITOKENS authentication must be working.

Useful Links

• SciTokens Claims and Scopes Language A good description of what claims you may want to put in your SciTokens and why.
• WLCG Common JWT Profiles WLCG Tokens are 'nearly' interoperable with SciTokens and are more mature.