GlideinWMS The Glidein-based Workflow Management System


This page documents a recipe for enabling and managing SciTokens Authorization between Glidein Factory Entries and the Compute Elements they submit to.


Requirement Description
Compatible HTCondor and GlideinWMS Versions For both Factory and VOFrontend Version numbers and installation instructions
An HTCondor Compute Element (htcondor-ce) Compatible with and correctly configured for SciTokens Minimal version requirements at the time of this writing
htcondor-ce.noarch          4.4.1-1.osg35.el7    @osg
htcondor-ce-client.noarch   4.4.1-1.osg35.el7    @osg
htcondor-ce-condor.noarch   4.4.1-1.osg35.el7    @osg
A SciToken that the htcondor-ce will accept Example method for generating a working token is given below. This is not how SciTokens would be generated and used in a production environment.

Obtain a SciToken

Method 1: follow instructions at

Method 2: generate your own self-signed Scitokens. You need the python scitokens library and admin access to a web server that the Factory, Frontend, and CE can all access.

The following is an example of Method 2 on website ''

1.  yum install python2-scitokens or pip install scitokens

2. Generate some keys

scitokens-admin-create-key --create-keys \
                           --pem-private > scitoken.private.pem

scitokens-admin-create-key --private-keyfile scitoken.private.pem\
                           --jwks-private > test.scitoken.private.jwks

scitokens-admin-create-key --private-keyfile test.scitoken.private.pem \
                           --jwks-public > test.scitoken.public.jwks

3. copy keys over to, create some other files in the document root directory:
cd /web/sites/j/
mkdir -p  oauth2/certs  .well-known
# create file  .well-known/openid-configuration with these contents:
cat .well-known/openid-configuration


exit #return to original machine that you generated the keys on

scp test.scitoken.public.jwks fnalu:/web/sites/j/

3. Generate a token to perform condor operations on  an htcondor-ce. NOTE: the scitokens-admin-create-token utility installed by
   pip or yum generates tokens that are only valid for 10 minutes.  If this is not long enough for your tests copy the utility
   (it is a python script) and modify to taste.

#for ce
scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem \
                             --issuer sub="condor"\
                             aud=""  > fermicloud349.scitoken

#for ce
scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem\
                             --issuer sub="condor"\
                             aud="" >


Configure Factory to submit your SciToken

  1. Check that SCITOKENS authentication is enabled on the Factory. For example:
    [root@fermicloud359]# condor_config_val SEC_DEFAULT_AUTHENTICATION_METHODS
  2. Place the correctly named, previously genereated SciToken in the correct directory so that the VOFrontend knows to send it on to the Factory, which uses it to authenticate with the CE.

    In this example, the SciToken must be named el7_osg35.scitoken to associate it with Factory entry el7_osg35 The SciToken goes in directory

    As SciTokens expire, a mechanism must be in place to renew it periodically. A cron job running 'scitokens-admin-create-token' would suffice for development purposes, but some other mechanism would be used in a production environment.

Configure CE to accept your SciToken

  1. Check that htcondor on your CE accepts SCITOKENS as an authentication method:
    root@fermicloud349 ~]# condor_ce_config_val -dump | grep SCITOKENS
  2. Check that an entry in the CE's CONDOR_MAPFILE assigns your SciToken to a valid user. In this example SciTokens issued by '' are mapped to user 'osg'
    [root@fermicloud349 ~]# condor_ce_config_val -dump | grep mapfile
    CERTIFICATE_MAPFILE = /etc/condor-ce/condor_mapfile
    [root@fermicloud349 ~]# cat /etc/condor-ce/condor_mapfile
    GSI "(/CN=[-.A-Za-z0-9/= ]+)" \
    CLAIMTOBE .* anonymous@claimtobe
    FS "^(root|condor)$" \
    FS "(.*)" \1
    [root@fermicloud349 ~]#

Verify Token Functionality

Here we see that the freshly installed SciToken can be decoded by command line tools and that condor_ping to the CE's collector authenticates some condor operations as user 'osg' as configured earlier in fermicloud349's condor_mapfile:

[root@fermicloud359 ~]# _condor_SCITOKENS_FILE=/var/lib/gwms-frontend/tokens.d/el7_osg35.scitoken condor_ping -address "<$(host | awk '{print $4}'):9619>" -table ALL
         Instruction Authentication Encryption Integrity Decision Identity
               ALLOW      SCITOKENS       none       MD5    ALLOW
                READ           none       none      none    ALLOW unauthenticated@unmapped
               WRITE           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
          NEGOTIATOR           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
       ADMINISTRATOR           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
               OWNER           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
              CONFIG           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
              DAEMON           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
    ADVERTISE_STARTD      SCITOKENS       none       MD5    ALLOW
    ADVERTISE_SCHEDD           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
    ADVERTISE_MASTER           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
[root@fermicloud359 ~]#

Verify that submitting a job to the Factories entry point uses SciTokens authentication by adding D_SECURITY to the ALL_DEBUG condor setting on the appropriate condor daemons. The condor logs will show what authentication was used, and it should be SCITOKENS between the factory collector and the CE schedd.

A second verification method is to disable GSI communication between the Factory and the CE by commenting out the line in the Factories CONDOR_MAPFILE that does the GSI mapping for the CE, and submit a job. If the factory can still submit jobs to the CE, SCITOKENS authentication must be working.

Useful Links