GlideinWMS The Glidein-based Workflow Management System

Components

Tools



gwms-logcat.sh

Provides information about glideins and their HTCondor log files: easily locate stdout and stderr, print a startd log file from the Glidein execution. See below for how to use it.

gwms-logcat.sh -h
./factory/tools/gwms-logcat.sh [options] LOG_TYPE LOGFILE
./factory/tools/gwms-logcat.sh [options] LOG_TYPE ENTRY [JOB_ID]
./factory/tools/gwms-logcat.sh -f URL [options] LOG_TYPE
./factory/tools/gwms-logcat.sh -r [options] LOG_TYPE JOB_ID
./factory/tools/gwms-logcat.sh -l
  LOG_TYPE HTCondor log to extract from the job logfile:
           all (all logs), master, startd, starter, startdhistory, xml
  LOGFILE  Job log file (stderr from a glidein)
  ENTRY    Entry name
  JOB_ID   HTCondor job (glidein) id. By default picks the last job with a valid log file
  -v       verbose
  -h       print this message
  -l       list all entries (arguments are ignored)
  -a       list only entries that were active (has at least one job) - used with '-l', ignored otherwise
  -u USER  to use a different user (job owner) from the default Frontend one
  -r       Remote running jobs. pilot_launcher.log is fetched from the VM
  -c FNAME Factory configuration file (default: /etc/gwms-factory/glideinWMS.xml)
  -f URL   Forward the information (to a fodler: file:///path/ via copy or a URL http:// via post)

gwms-logcat.sh can be used interactively to examin the HTCondor log files or the XML file produced by the Glideins and stored at the Factory.

Files from a Glidein running on the cloud and spinning a VM are not transferred back to the Factory but you can explore them if the VM is still running using the remote option (-r) which fetches pilot_launcher.log from the VM.

The option -f is forwarding the information in batch mode: it is processing all the desired log files to the destination directory or URL. It is also saving a checkpoint file to avoid reprocessing multiple times the same files. Ideally can be run periodically, e.g. as cron job.

gwms-logcat.sh -f /var/www/html/glidein-logs/ startd

glidecondor_addDN

The glidecondor_addDN script (found in the ./install directory of the GlideinWMS tree) can help to manage HTCondor mapfiles. The management of DNs and authorization to specific users can be confusing and this utility can help.

It has two usage modes:

  1. Add a single DN:
    glidecondor_addDN [options] [-daemon comment] DN_or_certfile user
  2. Add DNs from a list file:
    glidecondor_addDN [options] -import listfile*

Where [options] is any combination of:

-m fname By default, the script will find the certificate_mapfile by invoking
condor_config_val CERTIFICATE_MAPFILE
Use this option to specify a different file name (mostly useful for prototyping).
-d fname By default, the script will put the daemon DNs into 90_gwms_dns.config in the HTCondor's LOCAL_CONFIG_DIR (obtained via condor_config_val).
Use this option to specify a different file name (mostly useful for prototyping).
-allow-alterate If present, the tool will look for alternate config files (e.g. condor_config.local) to put daemon DNs into.
-disable-checks Without it, valid config files must already be present.
-recreate By default, the tool will add to existing files.
If this option is present, it will destroy any existing configuration files, and create new ones instead.
-allow-others By default when -recreate is used, the new config files will override any existing DN attributes (i.e. GSI_DAEMON_NAME).
If you want to preserve the settings from other config files, use this option.
-q Quiet operation.

In both operation modes, the tool will add properly encoded (DN,username) entries in the certificate mapfile. Also, if the DN is tagged as belonging to a trusted daemon, also in the DN config file.

In both modes, the DN can be the actual DN string, or a certificate from which to extract the string from. The tool will try to guess which one you are using, but you can also force it by prepending it with either dn: or file:.

The listfile contains one DN per line; lines starting with a hash (#) or that are completely empty, are ignored.
Each line must have exactly three columns, separated by any number of spaces:

user type DN_or_certfile

The DN_or_cerfile can contain spaces, and MUST NOT be quoted.

The type columns must be either daemon or client.

Example of the first type:

glidecondor_addDN -daemon "My own host certificate DN" file:/etc/grid-security/hostcert.pem condor

Example of the second type:

cat pool_dns.list
# This file contains the pool DNs
# Format
# user type   DN
condor daemon file:/etc/grid-security/hostcert.pem
ce2    daemon /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=osg-glidein-ce-2.example.edu
ce1    daemon dn:/DC=org/DC=doegrids/OU=Services/CN=osg-glidein-ce.example.edu
pilot1 daemon /DC=org/DC=doegrids/OU=Services/CN=osgpilot01/osg-glidein-fe.example.gov
fe1    client /DC=org/DC=doegrids/OU=Services/CN=osg-glidein-fe.example.gov
glidecondor_addDN -recreate -disable-checks -import pool_dns.list

glidecondor_createSecSched

The glidecondor_createSecSched script (found in the ./install directory of the GlideinWMS tree) can help manage the HTCondor's secondary schedds. It can be used when HTCondor's configuration is stored in LOCAL_CONFIG_DIR.

Usage:

glidecondor_createSecSched [-commonlog] [-nocreatedirs] scheddlist

This script will re-create the 11_gwms_secondary_schedds.config file at each invocation. The node admin can thus easily change the number of secondary schedds as needed.

By default, each secondary schedd will have its own log file. If a shared log is desired, use -commonlog.
Please note that the tool does not try to guess what the existing config(if any) is using.

By default, the tool will create any needed directories. If you are sure the directories are already in place, e.g. beacuse you are just reducing the number of secondary schedds, you can use -nocreatedirs to speed it up.

Example:

glidecondor_createSecSched schedd_glideins1,schedd_glideins2

glidecondor_createSecCol

The glidecondor_createSecCol script (found in the ./install directory of the GlideinWMS tree) can help manage the HTCondor's secondary collectors. It can be used when HTCondor's configuration is stored in LOCAL_CONFIG_DIR.

Usage:

glidecondor_createSecCol [-useportasname] [-commonlog] portrange

This script will re-create the 11_gwms_secondary_collectors.config file file at each invocation. The node admin can thus easily change the number of secondary collectors as needed.

Since each HTCondor Daemon needs a unique name, each collector will be called COLLECTORxxx. By default, the numbering starts with 0 for the first collector in the port range, and is incremented by one for each additional one.
If -useportasname is used, xxx is instead defined as the port number used by the collector.
Please note that the tool does not try to guess what the existing config (if any) is using.

By default, each secondary collector will have its own log file. If a shared log is desired, use -commonlog.

Example:

glidecondor_createSecCol 9620-9639

create_condor_tarball

The create_condor_tarball script (found in the ./factory/creation directory of the GlideinWMS tree) can help reduce the disk space needs on the Factory node.

It can be used to create a condor tarball for use by the GlideinWMS pilot process for various entry points. The GlideinWMS pilot does not need all the files/processes contained in a normal HTCondor tarfile. This utility extracts only the needed processes/libraries thereby reducing the space needed on your Factory node.

Usage: create_condor_tarball output_filename input_tarfile_or_dir

   output_filename - file name of the output tarball created

   input_tarfile_or_dir - this is the source tarfile or if already expanded,
                          the top level directory.
Once the output condor tarball is created, additional configuration is required in the Factory configuration file. Refer to the Multiple HTCondor Tarballs section of the Factory Configuration document for details.

Example:

create_condor_tarball /tmp/condor-8.7.6 condor-8.7.6-x86_64_RedHat6-stripped.tar.gz

Other monitoring tools

The tools analyze_entries,analyze_queues, and analyze_frontends exist in the factory/tools directory. They can be used to monitor recent status of the Factory, entries, and connected frontends. See Factory monitoring for more details.