GlideinWMS The Glidein-based Workflow Management System

WMS Factory

Tarball Downloader

Jump to:
  1. Description
  2. Exit Codes
  3. Configuration
  4. Sample Execution
  5. Sample Generated XML ile
Related Links:

Description

The get_tarball script is a Python script that downloads HTCondor tarballs from the official HTCondor website.
The tarballs are downloaded by looking at the URL set using TARBALL_BASE_URL in the configuration file. The files are downloaded and stored in the DESTINATION_DIR directory. Checkusm are verified after a file is downloaded. If a file already exist, or if its checksum is wrong or missing, the file is skipped.
A list of major condor version has to be specified using CONDOR_TARBALL_LIST. Each major version has to be a directory here (assuming this is the link set in the configuration). For each major version, all the tarballs in the WHITELIST are downloaded. Alternatively, all tarballs available are downloaded excluding the one in the BLACKLIST.
For each minor version (e.g.: "9.0.7"), the script attempts to download all the files in the FILENAME_LIST. The attempted URL is TARBALL_BASE_URL/MAJOR_VERSION/WHITELIST[i]/release/FILENAME_LIST[j], for example: https://research.cs.wisc.edu/htcondor/tarball/23.0/23.0.0/release/condor-23.0.0-x86_64_CentOS7-stripped.tar.gz
The next three parameters, OS_MAP, ARCH_MAP, and XML_OUT determines if and how an xml file with the tarballs has to be generated. If XML_OUT is not set then nothing is done and the other two parameters can be missing. Otherwise an xml file will be generated and stored there. OS_MAP and ARCH_MAP indicates how the operating system and the architecture in the tarball filename has to be translated in the xml corresponding fields.
get_tarball is a standalone script and does not require any glideinWMS dependency or installation.

Exit Codes

  • 0: All operations completed successfully.
  • 1: Configuration file does not exist.

Configuration

To specify the configuration for the downloader, you can use the following methods:

  • Set the GET_TARBALLS_CONFIG environment variable.
  • Place a configuration file named get_tarballs.yaml in the same directory as get_tarball.py.

Sample Configuration File:

DESTINATION_DIR: "/var/lib/gwms-factory/condor/"
TARBALL_BASE_URL: "https://research.cs.wisc.edu/htcondor/tarball/"
DEFAULT_TARBALL_VERSION: ["9.0.16"] # Can be set to "latest"
CONDOR_TARBALL_LIST:
   - MAJOR_VERSION: "9.0"
     WHITELIST: ["9.0.7", "9.0.16", "latest"]
   - MAJOR_VERSION: "10.0"
     WHITELIST: ["latest"]
     CHECK_LATEST: True
   - MAJOR_VERSION: "10.x"
     DOWNLOAD_LATEST: True # Same as adding "latest" to a WHITELIST. Default False
   - MAJOR_VERSION: "23.0"
     WHITELIST: ["23.0.0"]
FILENAME_LIST: ["condor-{version}-x86_64_CentOS7-stripped.tar.gz", "condor-{version}-x86_64_CentOS8-stripped.tar.gz",
    "condor-{version}-x86_64_AlmaLinux8-stripped.tar.gz", "condor-{version}-x86_64_Ubuntu18-stripped.tar.gz", "condor-{version}-x86_64_Ubuntu20-stripped.tar.gz",
    "condor-{version}-aarch64_Stream8-stripped.tar.gz", "condor-{version}-ppc64le_CentOS8-stripped.tar.gz", "condor-{version}-ppc64le_AlmaLinux8-stripped.tar.gz",
    "condor-{version}-aarch64_AlmaLinux8-stripped.tar.gz"]
OS_MAP: { "CentOS7": "default, rhel7, linux-rhel7", "CentOS8": "rhel8, linux-rhel8", "AlmaLinux8": "rhel8, linux-rhel8", "Ubuntu18": "ubuntu18, linux-ubuntu18",
    "Ubuntu20": "ubuntu20, linux-ubuntu20" }
ARCH_MAP: { "x86_64": "default", "ppc64le": "ppc64le", "aarch64": "aarch64" }
XML_OUT: "/etc/gwms-factory/config.d/01-condor-tarballs.xml"

Sample Execution

This is the standard execution output. More output can be obtained using --verbose. 
$ get_tarballs
Handling major version 9.0
	File condor-9.0.7-x86_64_CentOS7-stripped.tar.gz successfully downloaded
	File condor-9.0.7-x86_64_CentOS8-stripped.tar.gz successfully downloaded
	File condor-9.0.7-x86_64_Ubuntu18-stripped.tar.gz successfully downloaded
	File condor-9.0.7-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
	File condor-9.0.16-x86_64_CentOS7-stripped.tar.gz successfully downloaded
	File condor-9.0.16-x86_64_CentOS8-stripped.tar.gz successfully downloaded
	File condor-9.0.16-x86_64_Ubuntu18-stripped.tar.gz successfully downloaded
	File condor-9.0.16-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
	File condor-9.0.18-x86_64_CentOS7-stripped.tar.gz successfully downloaded
	File condor-9.0.18-x86_64_CentOS8-stripped.tar.gz successfully downloaded but
	     checksum not available at https://research.cs.wisc.edu/htcondor/tarball/9.0/9.0.18/release/ (check 'sha256sum.txt')
	File condor-9.0.18-x86_64_Ubuntu18-stripped.tar.gz successfully downloaded
	File condor-9.0.18-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
Handling major version 10.0
	File condor-10.0.9-x86_64_CentOS7-stripped.tar.gz successfully downloaded
	File condor-10.0.9-x86_64_AlmaLinux8-stripped.tar.gz successfully downloaded
	File condor-10.0.9-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
	File condor-10.0.9-ppc64le_AlmaLinux8-stripped.tar.gz successfully downloaded
	File condor-10.0.9-aarch64_AlmaLinux8-stripped.tar.gz successfully downloaded
Handling major version 10.x
	File condor-10.9.0-x86_64_CentOS7-stripped.tar.gz successfully downloaded
	File condor-10.9.0-x86_64_AlmaLinux8-stripped.tar.gz successfully downloaded
	File condor-10.9.0-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
	File condor-10.9.0-ppc64le_AlmaLinux8-stripped.tar.gz successfully downloaded
	File condor-10.9.0-aarch64_AlmaLinux8-stripped.tar.gz successfully downloaded
Handling major version 23.0
	File condor-23.0.0-x86_64_CentOS7-stripped.tar.gz successfully downloaded
	File condor-23.0.0-x86_64_AlmaLinux8-stripped.tar.gz successfully downloaded
	File condor-23.0.0-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
	File condor-23.0.0-ppc64le_AlmaLinux8-stripped.tar.gz successfully downloaded
	File condor-23.0.0-aarch64_AlmaLinux8-stripped.tar.gz successfully downloaded

Sample Generated XML File

An xml configuration file can be generated and placed in the /etc/gwms-factory/config.d/ directory to be picked up and used for the final glideinWMS.xml file. Here is a sample generated xml: 
<glidein>
   <condor_tarballs>
      <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-9.0.7-x86_64_CentOS7-stripped.tar.gz" version="9.0.7"/>
      <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-9.0.7-x86_64_CentOS8-stripped.tar.gz" version="9.0.7"/>
      <condor_tarball arch="default" os="ubuntu18, linux-ubuntu18" tar_file="/tmp/condor-9.0.7-x86_64_Ubuntu18-stripped.tar.gz" version="9.0.7"/>
      <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-9.0.7-x86_64_Ubuntu20-stripped.tar.gz" version="9.0.7"/>
      <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-9.0.16-x86_64_CentOS7-stripped.tar.gz" version="9.0.16,default"/>
      <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-9.0.16-x86_64_CentOS8-stripped.tar.gz" version="9.0.16,default"/>
      <condor_tarball arch="default" os="ubuntu18, linux-ubuntu18" tar_file="/tmp/condor-9.0.16-x86_64_Ubuntu18-stripped.tar.gz" version="9.0.16,default"/>
      <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-9.0.16-x86_64_Ubuntu20-stripped.tar.gz" version="9.0.16,default"/>
      <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-9.0.18-x86_64_CentOS7-stripped.tar.gz" version="9.0.18,9.0.x"/>
      <condor_tarball arch="default" os="ubuntu18, linux-ubuntu18" tar_file="/tmp/condor-9.0.18-x86_64_Ubuntu18-stripped.tar.gz" version="9.0.18,9.0.x"/>
      <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-9.0.18-x86_64_Ubuntu20-stripped.tar.gz" version="9.0.18,9.0.x"/>
      <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-10.0.9-x86_64_CentOS7-stripped.tar.gz" version="10.0.9,10.0.x"/>
      <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.0.9-x86_64_AlmaLinux8-stripped.tar.gz" version="10.0.9,10.0.x"/>
      <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-10.0.9-x86_64_Ubuntu20-stripped.tar.gz" version="10.0.9,10.0.x"/>
      <condor_tarball arch="ppc64le" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.0.9-ppc64le_AlmaLinux8-stripped.tar.gz" version="10.0.9,10.0.x"/>
      <condor_tarball arch="aarch64" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.0.9-aarch64_AlmaLinux8-stripped.tar.gz" version="10.0.9,10.0.x"/>
      <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-10.9.0-x86_64_CentOS7-stripped.tar.gz" version="10.9.0,10.x"/>
      <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.9.0-x86_64_AlmaLinux8-stripped.tar.gz" version="10.9.0,10.x"/>
      <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-10.9.0-x86_64_Ubuntu20-stripped.tar.gz" version="10.9.0,10.x"/>
      <condor_tarball arch="ppc64le" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.9.0-ppc64le_AlmaLinux8-stripped.tar.gz" version="10.9.0,10.x"/>
      <condor_tarball arch="aarch64" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.9.0-aarch64_AlmaLinux8-stripped.tar.gz" version="10.9.0,10.x"/>
      <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-23.0.0-x86_64_CentOS7-stripped.tar.gz" version="23.0.0,23.0.x"/>
      <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-23.0.0-x86_64_AlmaLinux8-stripped.tar.gz" version="23.0.0,23.0.x"/>
      <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-23.0.0-x86_64_Ubuntu20-stripped.tar.gz" version="23.0.0,23.0.x"/>
      <condor_tarball arch="ppc64le" os="rhel8, linux-rhel8" tar_file="/tmp/condor-23.0.0-ppc64le_AlmaLinux8-stripped.tar.gz" version="23.0.0,23.0.x"/>
      <condor_tarball arch="aarch64" os="rhel8, linux-rhel8" tar_file="/tmp/condor-23.0.0-aarch64_AlmaLinux8-stripped.tar.gz" version="23.0.0,23.0.x"/>
   </condor_tarballs>
</glidein>
Optionally, you can create a pre configuration hook to fully automate the tarballs download and setup.
Banner image by Madhusudan Katti used under Creative Commons license.
Original Home URL: https://glideinwms.fnal.gov. GlideinWMS email support: glideinwms-support at fnal.gov