Description
This page documents a recipe for enabling and managing SciTokens Authorization between Glidein Factory Entries and the Compute Elements they submit to.Requirements
| Requirement | Description |
| Compatible HTCondor and GlideinWMS Versions For both Factory and VOFrontend | Version numbers and installation instructions |
| An HTCondor Compute Element (htcondor-ce) Compatible with and correctly configured for SciTokens |
Minimal version requirements at the time of this writing
htcondor-ce.noarch 4.4.1-1.osg35.el7 @osg htcondor-ce-client.noarch 4.4.1-1.osg35.el7 @osg htcondor-ce-condor.noarch 4.4.1-1.osg35.el7 @osg |
| A SciToken that the htcondor-ce will accept | Example method for generating a working token is given below. This is not how SciTokens would be generated and used in a production environment. |
Obtain a SciToken
Method 1: follow instructions at https://opensciencegrid.org/technology/software/requesting-tokens/#requesting-tokens
Method 2: generate your own self-signed Scitokens. You need the python scitokens library and admin access to a web server that the Factory, Frontend, and CE can all access.
The following is an example of Method 2 on website 'jobsub.fnal.gov'
1. yum install python2-scitokens or pip install scitokens
2. Generate some keys
scitokens-admin-create-key --create-keys \
--pem-private > scitoken.private.pem
scitokens-admin-create-key --private-keyfile scitoken.private.pem\
--jwks-private > test.scitoken.private.jwks
scitokens-admin-create-key --private-keyfile test.scitoken.private.pem \
--jwks-public > test.scitoken.public.jwks
3. copy keys over to jobsub.fnal.gov, create some other files in the document root directory:
ssh fnalu.fnal.gov
cd /web/sites/j/jobsub.fnal.gov/htdocs
mkdir -p oauth2/certs .well-known
# create file .well-known/openid-configuration with these contents:
cat .well-known/openid-configuration
{
"issuer":"https://jobsub.fnal.gov",
"jwks_uri":"https://jobsub.fnal.gov/oauth2/certs/test.scitoken.public.jwks"
}
exit #return to original machine that you generated the keys on
scp test.scitoken.public.jwks fnalu:/web/sites/j/jobsub.fnal.gov/htdocs/oauth2/certs
3. Generate a token to perform condor operations on an htcondor-ce. NOTE: the scitokens-admin-create-token utility installed by
pip or yum generates tokens that are only valid for 10 minutes. If this is not long enough for your tests copy the utility
(it is a python script) and modify to taste.
#for ce fermicloud349.fnal.gov
scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem \
--issuer https://jobsub.fnal.gov sub="condor"\
scope="condor"\
aud="fermicloud349.fnal.gov:9619" > fermicloud349.scitoken
#for ce itb-ce2.chtc.wisc.edu
scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem\
--issuer https://jobsub.fnal.gov sub="condor"\
scope="condor"\
aud="itb-ce2.chtc.wisc.edu:9619" > itb-ce2.chtc.wisc.edu.scitoken
Configure Factory to submit your SciToken
- Check that SCITOKENS authentication is enabled on the Factory. For example:
[root@fermicloud359]# condor_config_val SEC_DEFAULT_AUTHENTICATION_METHODS SCITOKENS,TOKEN,FS,GSI - Place the correctly named, previously genereated SciToken in the correct directory so that the VOFrontend
knows to send it on to the Factory, which uses it to authenticate with the CE.
In this example, the SciToken must be named el7_osg35.scitoken to associate it with Factory entry el7_osg35 The SciToken goes in directory
/var/lib/gwms-frontend/tokens.d/
As SciTokens expire, a mechanism must be in place to renew it periodically. A cron job running 'scitokens-admin-create-token' would suffice for development purposes, but some other mechanism would be used in a production environment.
Configure CE to accept your SciToken
- Check that htcondor on your CE accepts SCITOKENS as an authentication method:
root@fermicloud349 ~]# condor_ce_config_val -dump | grep SCITOKENS COLLECTOR.SEC_ADVERTISE_STARTD_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI COLLECTOR.SEC_READ_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI COLLECTOR.SEC_WRITE_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI MASTER.SEC_DEFAULT_AUTHENTICATION_METHODS = SCITOKENS, FS, GSI SCHEDD.SEC_READ_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI SCHEDD.SEC_WRITE_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI SCITOKENS_SERVER_AUDIENCE = $(COLLECTOR_HOST) SEC_CLIENT_AUTHENTICATION_METHODS = SCITOKENS, FS, GSI, PASSWORD SEC_DEFAULT_AUTHENTICATION_METHODS = SCITOKENS,FS, GSI - Check that an entry in the CE's CONDOR_MAPFILE assigns your SciToken to a valid user. In this example SciTokens issued by 'https://jobsub.fnal.gov' are mapped to user 'osg'
[root@fermicloud349 ~]# condor_ce_config_val -dump | grep mapfile CERTIFICATE_MAPFILE = /etc/condor-ce/condor_mapfile [root@fermicloud349 ~]# cat /etc/condor-ce/condor_mapfile SCITOKENS https://jobsub.fnal.gov osg GSI (.*) GSS_ASSIST_GRIDMAP GSI "(/CN=[-.A-Za-z0-9/= ]+)" \1@unmapped.htcondor.org CLAIMTOBE .* anonymous@claimtobe FS "^(root|condor)$" \1@daemon.htcondor.org FS "(.*)" \1 [root@fermicloud349 ~]#
Verify Token Functionality
Here we see that the freshly installed SciToken can be decoded by command line tools and that condor_ping to the CE's collector authenticates some condor operations as user 'osg' as configured earlier in fermicloud349's condor_mapfile:
[root@fermicloud359 ~]# _condor_SCITOKENS_FILE=/var/lib/gwms-frontend/tokens.d/el7_osg35.scitoken condor_ping -address "<$(host fermicloud349.fnal.gov | awk '{print $4}'):9619>" -table ALL
Instruction Authentication Encryption Integrity Decision Identity
ALLOW SCITOKENS none MD5 ALLOW osg@users.htcondor.org
READ none none none ALLOW unauthenticated@unmapped
WRITE FAIL FAIL FAIL FAIL FAIL (use -verbose for more info)
NEGOTIATOR FAIL FAIL FAIL FAIL FAIL (use -verbose for more info)
ADMINISTRATOR FAIL FAIL FAIL FAIL FAIL (use -verbose for more info)
OWNER FAIL FAIL FAIL FAIL FAIL (use -verbose for more info)
CONFIG FAIL FAIL FAIL FAIL FAIL (use -verbose for more info)
DAEMON FAIL FAIL FAIL FAIL FAIL (use -verbose for more info)
ADVERTISE_STARTD SCITOKENS none MD5 ALLOW osg@users.htcondor.org
ADVERTISE_SCHEDD FAIL FAIL FAIL FAIL FAIL (use -verbose for more info)
ADVERTISE_MASTER FAIL FAIL FAIL FAIL FAIL (use -verbose for more info)
[root@fermicloud359 ~]#
Verify that submitting a job to the Factories entry point uses SciTokens authentication by adding D_SECURITY to the ALL_DEBUG condor setting on the appropriate condor daemons. The condor logs will show what authentication was used, and it should be SCITOKENS between the factory collector and the CE schedd.
A second verification method is to disable GSI communication between the Factory and the CE by commenting out the line in the Factories CONDOR_MAPFILE that does the GSI mapping for the CE, and submit a job. If the factory can still submit jobs to the CE, SCITOKENS authentication must be working.
Useful Links
- SciTokens Claims and Scopes Language A good description of what claims you may want to put in your SciTokens and why.
- WLCG Common JWT Profiles WLCG Tokens are 'nearly' interoperable with SciTokens and are more mature.