gwms-logcat.sh
Provides information about glideins and their HTCondor log files: easily locate stdout and stderr, print a startd log file form the Glidein execution. See below for how to use it.
gwms-logcat.sh -h
./factory/tools/gwms-logcat.sh [options] LOG_TYPE LOGFILE
./factory/tools/gwms-logcat.sh [options] LOG_TYPE ENTRY [JOB_ID]
./factory/tools/gwms-logcat.sh -f URL [options] LOG_TYPE
./factory/tools/gwms-logcat.sh -r [options] LOG_TYPE JOB_ID
./factory/tools/gwms-logcat.sh -l
LOG_TYPE HTCondor log to extract from the job logfile:
all (all logs), master, startd, starter, startdhistory, xml
LOGFILE Job log file (stderr from a glidein)
ENTRY Entry name
JOB_ID HTCondor job (glidein) id. By default picks the last job with a valid log file
-v verbose
-h print this message
-l list all entries (arguments are ignored)
-a list only entries that were active (has at least one job) - used with '-l', ignored otherwise
-u USER to use a different user (job owner) from the default frontend one
-r Remote running jobs. pilot_launcher.log is fetched form the VM
-c FNAME Factory configuration file (default: /etc/gwms-factory/glideinWMS.xml)
-f URL Forward the information (to a fodler: file:///path/ via copy or a URL http:// via post)
gwms-logcat.sh can be used interactively to examin the HTCondor log files or the XML file produced by the Glideins and stored at the Factory.
Files from a Glidein running on the cloud and spinning a VM are not transferred back to the factory but you can explore them if the VM is still running using the remote option (-r) which fetches pilot_launcher.log from the VM.
The option -f is forwarding the information in batch mode: it is processing all the desired log files to the destination directory or URL. It is also saving a checkpoint file to avoid reprocessing multiple times the same files. Ideally can be run periodically, e.g. as cron job.
gwms-logcat.sh -f /var/www/html/glidein-logs/ startd
glidecondor_addDN
The glidecondor_addDN script (found in the ./install directory of the glideinWMS tree) can help manage Condor mapfiles. The management of DNs and authorization to specific users can be confusing and this utility can help.
It has two usage modes:
- Add a single DN:
glidecondor_addDN [options] [-daemon comment] DN_or_certfile user
- Add DNs from a list file:
glidecondor_addDN [options] -import listfile*
Where [options] is any combination of:
| -m fname | By default, the script will find the certificate_mapfile by invoking
condor_config_val CERTIFICATE_MAPFILEUse this option to specify a different file name (mostly useful for prototyping). |
| -d fname | By default, the script will put the daemon DNs into 90_gwms_dns.config in the Condor's LOCAL_CONFIG_DIR (obtained via condor_config_val). Use this option to specify a different file name (mostly useful for prototyping). |
| -allow-alterate | If present, the tool will look for alternate config files (e.g. condor_config.local) to put daemon DNs into. |
| -disable-checks | Without it, valid config files must already be present. |
| -recreate | By default, the tool will add to existing files. If this option is present, it will destroy any existing configuration files, and create new ones instead. |
| -allow-others | By default when -recreate is used, the new config files will override any existing DN attributes (i.e. GSI_DAEMON_NAME). If you want to preserve the settings from other config files, use this option. |
| -q | Quiet operation. |
In both operation modes, the tool will add properly encoded (DN,username) entries in the certificate mapfile, and, iif the DN is tagged as belonging to a trusted daemon, also in the DN config file.
In both modes, the DN can be the actual DN string, or a certificate from which to extract the string from. The tool will try to guess which one you are using, but you can also force it by prepending it with either dn: or file:.
The listfile contains one DN per line; lines starting with a hash (#) or that are completely empty, are ignored.
Each line must have exactly three columns, separated by any number of spaces:
user type DN_or_certfile
The DN_or_cerfile can contain spaces, and MUST NOT be quoted.
The type columns must be either daemon or client.
Example of the first type:
glidecondor_addDN -daemon "My own host certificate DN" file:/etc/grid-security/hostcert.pem condor
Example of the second type:
cat pool_dns.list# This file contains the pool DNs # Format # user type DN condor daemon file:/etc/grid-security/hostcert.pem ce2 daemon /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=osg-glidein-ce-2.example.edu ce1 daemon dn:/DC=org/DC=doegrids/OU=Services/CN=osg-glidein-ce.example.edu pilot1 daemon /DC=org/DC=doegrids/OU=Services/CN=osgpilot01/osg-glidein-fe.example.gov fe1 client /DC=org/DC=doegrids/OU=Services/CN=osg-glidein-fe.example.govglidecondor_addDN -recreate -disable-checks -import pool_dns.list
glidecondor_createSecSched
The glidecondor_createSecSched script (found in the ./install directory of the glideinWMS tree) can help manage the HTCondor's secondary schedds. It can be used when HTCondor's configuration is stored in LOCAL_CONFIG_DIR.
Usage:
glidecondor_createSecSched [-commonlog] [-nocreatedirs] scheddlist
This script will re-create the 11_gwms_secondary_schedds.config file at each invocation. The node admin can thus easily change the number of secondary schedds as needed.
By default, each secondary schedd will have its own log file.
If a shared log is desired, use -commonlog.
Please note that the tool does not try to guess what the existing config, if any, is using.
By default, the tool will create any needed directories. If you are sure the directories are already in place, e.g. beacuse you are just reducing the number of secondary schedds, you can use -nocreatedirs to speed it up.
Example:
glidecondor_createSecSched schedd_glideins1,schedd_glideins2
glidecondor_createSecCol
The glidecondor_createSecCol script (found in the ./install directory of the glideinWMS tree) can help manage the HTCondor's secondary collectors. It can be used when HTCondor's configuration is stored in LOCAL_CONFIG_DIR.
Usage:
glidecondor_createSecCol [-useportasname] [-commonlog] portrange
This script will re-create the 11_gwms_secondary_collectors.config file file at each invocation. The node admin can thus easily change the number of secondary collectors as needed.
Since each Condor Daemon needs a unique name, each collector will be called COLLECTORxxx.
By default, the numbering starts with 0 for the first collector in the port range, and is incremented by one for each additional one.
If -useportasname is used, xxx is instread defined as the port number used by the collector.
Please note that the tool does not try to guess what the existing config, if any, is using.
By default, each secondary collector will have its own log file.
If a shared log is desired, use -commonlog.
Please note that the tool does not try to guess what the existing config, if any, is using.
Example:
glidecondor_createSecCol 9620-9639
create_condor_tarball
The create_condor_tarball script (found in the ./factory/creation directory of the glideinWMS tree) can help reduce the disk space needs on the factory node.
It can be used to create a condor tarball for use by the glideinwms pilot process for various entry points. The glideinwms pilot does not need all the files/processes contained in a normal HTCondor tarfile. This utility extracts only the needed processes/libraries thereby reducing the space needed on your factory node.
Usage: create_condor_tarball output_filename input_tarfile_or_dir
output_filename - file name of the output tarball created
input_tarfile_or_dir - this is the source tarfile or if already expanded,
the top level directory.
Once the output condor tarball is created, additional configuration is required in
the factory configuration file. Refer to the
Multiple Condor Tarballs section of the Factory Configuration document for details.
Example:
create_condor_tarball /tmp/condor-7.7.5 condor-7.7.5-x86_64_rhap_5-stripped.tar.gz
Other monitoring tools
The tools analyze_entries,analyze_queues, and analyze_frontends exist in the factory/tools directory. They can be used to monitor recent status of the factory, entries, and connected frontends. See Factory monitoring for more details.