glideinwms.factory.tools package

Subpackages

Submodules

glideinwms.factory.tools.OSG_autoconf module

glideinwms.factory.tools.cat_MasterLog module

Print out the MasterLog for a glidein output file

Usage: cat_MasterLog.py logname

glideinwms.factory.tools.cat_MasterLog.main()[source]

glideinwms.factory.tools.cat_StartdHistoryLog module

Print out the StartdHistoryLog for a glidein output file

Usage: cat_StartdHistoryLog.py logname

glideinwms.factory.tools.cat_StartdHistoryLog.main()[source]

glideinwms.factory.tools.cat_StartdLog module

Print out the StartdLog for a glidein output file

Usage: cat_StartdLog.py logname

glideinwms.factory.tools.cat_StartdLog.main()[source]

glideinwms.factory.tools.cat_StarterLog module

Print out the StarterLog for a glidein output file

Usage: cat_StarterLog.py logname

glideinwms.factory.tools.cat_StarterLog.main()[source]

glideinwms.factory.tools.cat_XMLResult module

Print out the XML Result for a glidein output file

Usage: cat_XMLResult.py [-raw] [-forcemulti] logname+

If -raw is present, do not wrap multiple XMLs into a ResultSet If -forcemulti is present, make it a ResultSet even if only one file present

glideinwms.factory.tools.cat_XMLResult.main(args)[source]

glideinwms.factory.tools.cat_logs module

Print out the logs for a certain date

Usage: cat_logs.py <factory> YY/MM/DD [hh:mm:ss]

glideinwms.factory.tools.cat_logs.main()[source]
glideinwms.factory.tools.cat_logs.parse_args()[source]

glideinwms.factory.tools.cat_named_log module

Print out the named HTCondor Log for a glidein output file

Usage: cat_named_log.py [-monitor] HTC_log GWMS_logname

glideinwms.factory.tools.cat_named_log.main()[source]

glideinwms.factory.tools.find_StartdLogs module

Print out the StartdLogs for a certain date

Usage: find_StartdLogs.py <factory> YY/MM/DD [hh:mm:ss]

glideinwms.factory.tools.find_StartdLogs.main()[source]
glideinwms.factory.tools.find_StartdLogs.parse_args()[source]

glideinwms.factory.tools.find_logs module

Find the logs for a certain date

Usage: find_logs.py <factory> YY/MM/DD [hh:mm:ss]

glideinwms.factory.tools.find_logs.main()[source]
glideinwms.factory.tools.find_logs.parse_args()[source]

glideinwms.factory.tools.get_tarballs module

This script download HTCondor tarballs from the official condor website.

# Exit codes: # 0: All good # 1: Configuration file does not exist

A configuration file is specified through the GET_TARBALLS_CONFIG environment variable. Alternatively, the script looks for a file named get_tarballs.yaml in the same directory of get_tarball.py

This is a sample configuration file:

DESTINATION_DIR: “/var/lib/gwms-factory/condor/” TARBALL_BASE_URL: “https://research.cs.wisc.edu/htcondor/tarball/” DEFAULT_TARBALL_VERSION: [ “9.0.16” ] # Can be set to “latest” CONDOR_TARBALL_LIST:

  • MAJOR_VERSION: “9.0” WHITELIST: [ “9.0.7”, “9.0.16”, “latest” ]

  • MAJOR_VERSION: “10.0” WHITELIST: [ “latest” ]

  • MAJOR_VERSION: “10.x” DOWNLOAD_LATEST: True # Same as adding “latest” to a WHITELIST. Default False

  • MAJOR_VERSION: “23.0” WHITELIST: [ “23.0.0” ]

  • MAJOR_VERSION: “23.x” WHITELIST: [ “23.0.0” ]

FILENAME_LIST: [ “condor-{version}-x86_64_CentOS7-stripped.tar.gz”, “condor-{version}-x86_64_CentOS8-stripped.tar.gz”, “condor-{version}-x86_64_AlmaLinux8-stripped.tar.gz”, “condor-{version}-x86_64_Ubuntu18-stripped.tar.gz”, “condor-{version}-x86_64_Ubuntu20-stripped.tar.gz”, “condor-{version}-aarch64_Stream8-stripped.tar.gz”, “condor-{version}-ppc64le_CentOS8-stripped.tar.gz”, “condor-{version}-ppc64le_AlmaLinux8-stripped.tar.gz”, “condor-{version}-aarch64_AlmaLinux8-stripped.tar.gz” ] OS_MAP: { “CentOS7”:”default,rhel7,linux-rhel7”, “CentOS8”:”rhel8,linux-rhel8”, “AlmaLinux8”:”rhel8,linux-rhel8”, “Ubuntu18”:”ubuntu18,linux-ubuntu18”, “Ubuntu20”:”ubuntu20,linux-ubuntu20”} ARCH_MAP: { “x86_64”:”default”, “ppc64le”:”ppc64le”, “aarch64”:”aarch64” } XML_OUT: “/etc/gwms-factory/config.d/01-condor-tarballs.xml”

# Not specifying BLACKLIST or WHITELIST download everything # Blacklist is ignored if whitelist is specified as well # WHITELIST: download only those releases # BLACKLIST: do not download the releases but download all the rest # CHECK_LATEST: print a warning if latest version of the major series is not on the factory xml

class glideinwms.factory.tools.get_tarballs.Config[source]

Bases: UserDict

Used to store information about the configuration file. yaml file gets converted to a dictionary by using yaml.load.

_abc_impl = <_abc._abc_data object>
validate()[source]

Validate the configuration file, sort the whitelists and blacklists, etc

class glideinwms.factory.tools.get_tarballs.TarballManager(release_url, filenames, destination, verbose=False)[source]

Bases: HTMLParser

This class manages the HTCondor tarballs for a major release (e.g.: 23.0).

In the constructor, it builds a list of releases by parsing the major release web page looking for minor releases (e.g.: 23.0.0) that have been released (they have a release directory). The list of releases are solely used for validation purposes, and to know the latest release.

It then offers a method to download a tarball and save it locally, and a method to generate the xml snippet to add to the glideinWMS.xml tarball configuration section.

download_tarballs(version)[source]
Download a specific set of condor tarballs from the release_url link

All the OS and architecture tarballs for the specified condor version are downloaded. The set of OS and architecture files are specified in the constructure using filenames

The method also checks the tarball checksum (by downloading the sha256sum.txt file) If a tarball already exist and its checksum is correct then it is skipped. If a specific os/architecture tarball is not avalable it is skipped, and a message is printed on stdout if verbose has been set to True in the constructor.

Parameters:

version – The condor version to download among this major release. E.g.: “23.0.1”

generate_xml(os_map, arch_map, whitelist, blacklist, default_tarball_version)[source]

Generate the XML snipped to be used in the <tarball> section of the glideinWMS.xml configuration.

Parameters:

  • os_map – A map that indicates how to translate the OS in the tarball filename to the os attribute in the xml. See OS_MAP in the configuration template.

  • arch_map – A map that indicates how to translate the ARCH in the tarball filename to the os attribute in the xml. See ARCH_MAP in the configuration template.

  • whitelist – The whitelist that tells the method which versions of condor have been downloaded. Can be “latest”.

  • blacklist – The blacklist in case it was used.

  • default_tarball_version – The default condor tarball version, “default” will be addded to the version attribute in the xml

handle_data(data)[source]

Internal method. Override the base class handle_data

verify_checksum(tname, checksums)[source]

Internal function to verify the checksum of a file

glideinwms.factory.tools.get_tarballs.check_xml(release)[source]

Deprecate?

glideinwms.factory.tools.get_tarballs.main()[source]

The main. You happy pylint?

glideinwms.factory.tools.get_tarballs.parse_opts()[source]

Parse the command line using ArgumentParser. Only option is –verbose

glideinwms.factory.tools.get_tarballs.save_xml(dest_xml, xml)[source]

Wrapper function to save the tarball xml snippet to disk. Also adds the necessary xml tags.

glideinwms.factory.tools.gfdiff module

This script allows to compare two different entries

glideinwms.factory.tools.gfdiff.check_dict_diff(*args, **kw)[source]

The wrapper function

glideinwms.factory.tools.gfdiff.check_list_diff(list_a, list_b)[source]

Scan the two list for differences

glideinwms.factory.tools.gfdiff.count_tabs(function_to_decorate)[source]

Decorator function that keeps track of how many intentation level are required. In other words, the decorator counts how many times the decorated function is called

glideinwms.factory.tools.gfdiff.get_entry_text(entry, conf)[source]

Get an entry text from the xml configuration file

glideinwms.factory.tools.gfdiff.handle_mergely(entry_a, conf_a, entry_b, conf_b, mergely_only)[source]

Function that prints the link to the mergely website

glideinwms.factory.tools.gfdiff.main()[source]

The main

glideinwms.factory.tools.gfdiff.parse_opts()[source]

Parse the command line options for this command

glideinwms.factory.tools.manual_glidein_submit module

Module contents