Jump to:
Description
This page documents a recipe for enabling and managing HTCondor Token Authorization between Frontends and Glidein Entry Points (CE's).Requirements
| Requirement | Description |
| GWMS Factory Version >= 3.7 | The Factory can be either a fresh install from yum/RPM or a yum upgrade, as long as it meets the version requirements. |
| GWMS Frontend Version >= 3.7 | The Frontend, like the Factory, can be made to meet version requirements via either a fresh install or upgrade. |
| HTCondor Version >= 8.9.1 | HTCondor that the Frontend uses must be a token supporting release. The HTCondor tarballs delivered by Factory entry points also need to be a token supporting release. It is not necessary to upgrade the main Factory HTCondor as long as the GlideinWMS version of the factory complies with the >= 3.7 requirement. |
| Sudo access for 'frontend' user on gwms-frontend machine |
The frontend user needs to be able to create passwords and tokens derived from these passwords. The following two lines must be present in
/etc/sudoers on the gwms-frontend machine:
frontend ALL=(ALL) NOPASSWD:SETENV: /usr/sbin/condor_store_cred
frontend ALL=(ALL) NOPASSWD:SETENV: /usr/bin/condor_token_create
|
Upgrade and Configure Frontend Condor
- Upgrade condor on the frontend machine to a token supporting version. Instructions for installing the from the official HTCondor repo are here . It is often easier to install condor from the OSG repos. At the time of this writing (2/11/20) the easist way to install condor 8.9.5 was:
# yum -y --enablerepo osg-upcoming install condor
- Make condor passwords and tokens directories if they don't already exist
# mkdir -p /etc/condor/passwords.d /etc/condor/tokens.d
- The utility /usr/local/bin/gwms_condor_ping should be present if 3.7 has been installed correctly. If it is not present, create it now as it is useful for verifying TOKEN functionality
# cat /usr/local/bin/gwms_condor_ping #!/bin/sh # a convenience wrapper for condor_ping # returns a table of authorizations, useful to see # if token_auth has been correctly set up # if [ "$1" = "" ]; then HOST=$HOSTNAME else HOST=$1 fi condor_ping -addr "<$(host $HOST | sed -e 's/.* //'):9618>" -table ALL # - As token authentication is not enabled, the output of gwms_condor_ping should show FS or GSI authentication being used:
# gwms_condor_ping Instruction Authentication Encryption Integrity Decision Identity ALLOW FS none MD5 ALLOW condor@fermicloud158.fnal.gov READ none none none ALLOW unauthenticated@unmapped WRITE FS none MD5 ALLOW condor@fermicloud158.fnal.gov NEGOTIATOR FS none MD5 ALLOW condor@fermicloud158.fnal.gov ADMINISTRATOR FS none MD5 ALLOW condor@fermicloud158.fnal.gov OWNER FS none MD5 ALLOW condor@fermicloud158.fnal.gov CONFIG FS none MD5 ALLOW condor@fermicloud158.fnal.gov DAEMON FS none MD5 ALLOW condor@fermicloud158.fnal.gov ADVERTISE_STARTD FS none MD5 ALLOW condor@fermicloud158.fnal.gov ADVERTISE_SCHEDD FS none MD5 ALLOW condor@fermicloud158.fnal.gov ADVERTISE_MASTER FS none MD5 ALLOW condor@fermicloud158.fnal.gov - Add the following lines to the frontends condor configuration:
ALLOW_READ = * ALLOW_DAEMON = $(ALLOW_WRITE) SEC_DEFAULT_AUTHENTICATION_METHODS = TOKEN,FS,GSI ALL_DEBUG = D_SECURITY, D_FULLDEBUG SEC_NEGOTIATOR_AUTHENTICATION_METHODS = $(SEC_DEFAULT_AUTHENTICATION_METHODS) SEC_DAEMON_AUTHENTICATION_METHODS = $(SEC_DEFAULT_AUTHENTICATION_METHODS) QUEUE_SUPER_USER = root, condor RUNTIME_CONFIG_ADMIN = condor@$(FULL_HOSTNAME), root@$(FULL_HOSTNAME) ALLOW_CONFIG = $(RUNTIME_CONFIG_ADMIN)
- Finally:
- create a condor pool password
- create a condor admin token
- restart condor
# openssl rand -base64 64 | condor_store_cred -u condor_pool@$HOSTNAME \
-f /etc/condor/passwords.d/POOL add
# condor_token_create -identity condor@$HOSTNAME -token condor.$HOSTNAME.token
# /bin/cp /root/.condor/tokens.d/condor.$HOSTNAME.token /etc/condor/tokens.d/admin.token
# systemctl restart condor
[root@fermicloud158 ~]# gwms_condor_ping
Instruction Authentication Encryption Integrity Decision Identity
ALLOW TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
READ none none none ALLOW unauthenticated@unmapped
WRITE TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
NEGOTIATOR TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
ADMINISTRATOR TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
OWNER TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
CONFIG TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
DAEMON TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
ADVERTISE_STARTD TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
ADVERTISE_SCHEDD TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
ADVERTISE_MASTER TOKEN none MD5 ALLOW condor@fermicloud158.fnal.gov
Upgrade Factory Tarballs
- Go to the HTCondor Downloads page and download a 64bit stripped compressed tarball from the "HTCondor Binaries and Source" section. These instructions have been tested with tarballs condor-8.9.1-x86_64_RedHat7-stripped.tar.gz through condor-8.9.5-x86_64_RedHat7-stripped.tar.gz
- Copy your downloaded tarball to the /var/lib/gwms-factory/condor directory on the Factory. Untar it
tar xvzf condor-8.9.5-x86_64_RedHat7-stripped.tar.gz - Add a line to the <condor_tarballs> section of the Factory XML configuration files that points to the condor tarball you just downloaded
<condor_tarballs> <condor_tarball arch="x86_64" base_dir="/var/lib/gwms-factory/condor/condor-8.9.5-x86_64_RedHat7-stripped" os="rhel7" version="8.9.5"/> </condor_tarballs> - Modify the <attrs> section of an <entry> configuration in the Factories XML to use the newly configured condor_tarball
<entry name="some_site_being_modified" ...> ... <attrs> <attr name="CONDOR_ARCH" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="x86_64"/> <attr name="CONDOR_OS" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="rhel7"/> <attr name="CONDOR_VERSION" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="8.9.5"/> ... </attrs> - Reconfigure the factory so the XML changes take effect
# systemctl stop gwms-factory.service # /usr/sbin/gwms-factory reconfig # systemctl start gwms-factory.service
Verify Token Functionality
- Checklist for Frontend prior to full chain token authentication
- GWMS >= 3.7 has been installed on Frontend, or Frontend has been modified to run git branch v37/23092. In either case, the following scripts should exist:
- /usr/sbin/frontend_condor_token
- /usr/local/bin/gwms_condor_ping
- gwms_condor_ping shows TOKEN authentication for 'root' user
- 'frontend' user present in /etc/sudoers as documented in Requirements section
- GWMS >= 3.7 has been installed on Factory, or Factory has been modified to run git branch v37/23092.
- An Entry Point on the factory has been configured to deliver a condor tarball >= 8.9.1
- GWMS >= 3.7 has been installed on Frontend, or Frontend has been modified to run git branch v37/23092. In either case, the following scripts should exist:
- Submit a condor job to the Frontend with condor requirements that make it run on the newly modified Factory Entry Point. Make sure the job has a 'printenv' or equivalent in it to show the environment the job runs under.
- When the job completes, check the output from the 'printenv' command. If a token was generated by the Frontend, forwarded to the Factory, forwarded again to the CE, and integrated into the glidein enviroment, GLIDEIN_CONDOR_TOKEN will be set and its value will include the GLIDEIN_Site name, for example for GLIDEIN_Site el7_ GLIDEIN_CONDOR_TOKEN=/tmp/glide_SpYzMq/ticket/el7_osg34_token.
Revoking Tokens for Entry Points
- Each Glidein_Site that a Frontend submits to is sent condor_tokens generated from a unique password for that site. If the password is changed, all existing tokens for that site become invalid, and token authentication for any running glideins at that site will stop working. New glideins requested by the Frontend will use the new password, and valid tokens will be sent to the Factory and on to the CE's. A quick way to change a token generating password from the command line is
# export GLIDEIN_SITE='HCC_US_Omaha_crane_gpu' # openssl rand -base64 64 | sudo /usr/sbin/condor_store_cred -u frontend@$HOSTNAME -f /etc/condor/passwords.d/${GLIDEIN_SITE} add
Limitations
- This recipe enables Condor TOKEN authentication but still allows GSI authentication if a TOKEN is revoked. This may not be the desired behavior.
- More fine grained authorization scope and management of token lifetime is possible and desirable. Consider this implementation 0.1
- SciToken authentication and its interactions with Condor and GlideinWMS are not addressed in this document.