glideinwms.creation.lib package

Submodules

glideinwms.creation.lib.cWConsts module

glideinwms.creation.lib.cWConsts.get_timestr(when=1716130687.624341)

Get a 6 char alphanumeric string based on the timestamp.

The string increases monotonically for each 62 years period starting from 2000. A later time corresponds to a bigger string in lexicographic order (within the period).

Parameters:

when (float) – time to convert in string (seconds from epoch, only the integer part is used)

Returns:

6 chars string depending on the time

Return type:

str

glideinwms.creation.lib.cWConsts.insert_timestr(instr)

insert timestr just before the last ‘.’ (dot)

Parameters:

instr (str) – dot separated string, e.g. file name

Returns:

input string with TIMESTR, dot separated, before the last dot

Return type:

str

glideinwms.creation.lib.cWDictFile module

Classes needed to handle dictionary files And other support functions

class glideinwms.creation.lib.cWDictFile.ChmodDirSupport(dir, chmod, dir_name)

Bases: SimpleDirSupport

create_dir(fail_if_exists=True)

Create the directory

Parameters:

fail_if_exists (bool) – fail with RuntimeError if the directory exists already

Returns:

True if the dir was created, false else

Return type:

bool

Raises:

RuntimeError – if failed the operation failed and the dircetory is not already there or if it is there and fail_if_exists is True

class glideinwms.creation.lib.cWDictFile.DescriptionDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFileTwoKeys

Dictionary file used for descriptions.

These are file lists sent to the Glidein and parsed to download the listed files

format_val(key, want_comments)

Return a string with the formatted (space+tab separated) key, value

Parameters:

• key – key of the key, value pair to format

• want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

formatted key, value

Return type:

str

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

class glideinwms.creation.lib.cWDictFile.DictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: object

Dictionaries serialized in files, one line per item

Comments start w/ ‘#’ at the beginning of the line Empty lines are nor tolerated by the parser in the glidein (bash)

Files have to be compatible across GlideinWMS versions running in different Python versions In this Python 3 version: using binary files and ‘latin-1’ encoding to preserve bytes (0x80…0xff) through round-tripping from byte streams to Unicode and back

add(key, val, allow_overwrite=False)

Add a key, value pair to the dictionary (self) if not already there

Changes the content of the dictionary and set self.changed to True if the value was added.

Parameters:

• key (str) – dictionary key

• val – Any value stored in the dictionary, usually a tuple (e.g. attributes and value)

• allow_overwrite (bool) – Allow to overwrite an exising value if True (default: False)

Raises:

DictFileError – when trying to modify a read only DictFile or overriding an existing item when prohibited or using an incompatible value

erase()

file_footer(want_comments)

Return the file footer as comment (None for this class)

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

None

file_header(want_comments)

Return the file header line (file name) as comment, None is want_comments is False

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

The file header, a comment containing the file name

Return type:

str

format_val(key, want_comments)

Return a string with the formatted (space+tab separated) key, value

Parameters:

• key – key of the key, value pair to format

• want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

formatted key, value

Return type:

str

get(key, default=None)

get_dir()

get_filepath()

get_fname()

has_key(key)

is_compatible(old_val, new_val)

is_equal(other, compare_dir=False, compare_fname=False, compare_keys=None)

Parametrised comparison of DictFile objects

Parameters:

• other (DictFile) – second object in the comparison, must be of the same class

• compare_dir (bool) – if True, compare also the directories of the files

• compare_fname (bool) – if True, compare also the file names

• compare_keys (bool) – if True, compare the order of the keys. If None, use order_matters

Returns:

True if self and other have the same representation

Return type:

bool

load(dir=None, fname=None, change_self=True, erase_first=True, set_not_changed=True)

if dir and/or fname are not specified, use the defaults specified in __init__, if they are, and change_self is True, change the self.

Parameters:

• dir (str,Path,None) – directory to store the dictionary file

• fname (str, Path, None) – file name to use for the dictionary

• change_self (bool) – if True, update the dir and file attributes (must provide dir and file)

• erase_first (bool) – if True, delete old content first

• set_not_changed (bool) – if True, set self.changed to False

Returns:

load_from_fd(fd, erase_first=True, set_not_changed=True)

Load a dictionary in memory from a binary file encoded w/ BINARY_ENCODING

Values (on lines) are loaded into the dictionary using parse_val, which expects str

Parameters:

• fd – file descriptor to load the dictionary from

• erase_first (bool) – if True, delete old content of the dictionary first

• set_not_changed (bool) – if True, set self.changed to False

Raises:

DictFileError – problem parsong a line

load_from_str(data, erase_first=True, set_not_changed=True)

Load data from a string into the object (self)

Parameters:

• data (str) – string to load from

• erase_first (bool) – if True, delete old content of the dictionary first

• set_not_changed (bool) – if True, set self.changed to False

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

remove(key, fail_if_missing=False)

save(dir=None, fname=None, sort_keys=None, set_readonly=True, reset_changed=True, save_only_if_changed=True, want_comments=True)

Save the dictionary into a binary file encoded w/ BINARY_ENCODING (dump function)

save_into_fd() is the actual function writing to file DictFile.save_into_fd() receives str from .format_val() and is encoding them using BINARY_ENCODING File permission is 644, to avoid accidental execution of configuration files

If dir and/or fname are not specified, use the defaults specified in __init__

Parameters:

• dir (str) – folder containing the dictionary, override the object value

• fname (str) – file name (the file path is the concatenation of dir and fname), override the object value

• sort_keys (bool) – True if keys should be sorted, override the object value

• set_readonly (bool) – if True, set read only after saving it (this is a flag in the file dict, unrelated to the persistent file permission)

• reset_changed (bool) – if False, do not reset self.changed

• save_only_if_changed (bool) – if False, save also if it was not changed

• want_comments (bool) – if True, comments are saved as well

save_into_bytes(sort_keys=None, set_readonly=True, reset_changed=True, want_comments=True)

Save the dictionary into a bytes array

Parameters:

• sort_keys (bool)

• set_readonly (bool) – if True (default) set also the dictionary as read-only

• reset_changed (bool) – if True (default) set also the dictionary as not changed (all changes have been saved)

• want_comments (bool) – False if you want to skip the comments

Returns:

the content of the dictionary

Return type:

bytes

save_into_fd(fd, sort_keys=None, set_readonly=True, reset_changed=True, want_comments=True)

Save into a BINARY_ENCODING (latin-1) encoded binary file It could be an actual file or a byte array or a string

Parameters:

• fd – binary file

• sort_keys (bool) – whether the keys should be sorted before writing the dictionary (default: None, use the object setting)

• set_readonly (bool) – False to set the DictFile read-write

• reset_changed (bool) – False not to record the save (self.changed remains True)

• want_comments (bool) – False to disable comments

save_into_str(sort_keys=None, set_readonly=True, reset_changed=True, want_comments=True)

Save the dictionary into a string. E.g. to compare it with another dictionary

Same as save_into_bytes, but returns a string

Parameters:

• sort_keys (bool)

• set_readonly (bool) – if True (default) set also the dictionary as read-only

• reset_changed (bool) – if True (default) set also the dictionary as not changed (all changes have been saved)

• want_comments (bool) – False if you want to skip the comments

Returns:

the content of the dictionary

Return type:

str

set_readonly(readonly=True)

exception glideinwms.creation.lib.cWDictFile.DictFileError(msg)

Bases: RuntimeError

class glideinwms.creation.lib.cWDictFile.DictFileTwoKeys(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

DictFile double keyed: both key and val are keys

add(key, val, allow_overwrite=False)

Parameters:

• key

• val

• allow_overwrite

Raises:

DictFileError –

erase()

get_val2(key)

Retrieve value associated with reverse key

Parameters:

key – reverse key (value)

Returns:

value associated with reverse key (key)

has_key2(key)

Check reverse dictionary keys

Parameters:

key – reverse key (value)

Returns:

reverse key in value list

is_compatible2(old_val2, new_val2)

is_equal(other, compare_dir=False, compare_fname=False, compare_keys=None)

Compare two DictFileDoubleKey objects (and optionally their file)

Parameters:

• other – other dictionary, object of the same class

• compare_dir – if True compare also the file directory (Default: False)

• compare_fname – if True compare also the file name (Default: False)

• compare_keys – if True compare also the keys lists. If None, use order_matters (Default: False)

Returns:

remove(key, fail_if_missing=False)

Parameters:

• key

• fail_if_missing

Raises:

DictFileError – KeyError, the key does not exist

class glideinwms.creation.lib.cWDictFile.DirSupport

Bases: object

Abstract class for a directory creation

create_dir(fail_if_exists=True)

Create the directory

Parameters:

fail_if_exists (bool) – fail with RuntimeError if the directory exists already

Returns:

True if the dir was created, false else

Return type:

bool

Raises:

RuntimeError – if failed the operation failed and the dircetory is not already there or if it is there and fail_if_exists is True

delete_dir()

class glideinwms.creation.lib.cWDictFile.DirsSupport

Bases: object

add_dir_obj(dir_obj)

create_dirs(fail_if_exists=True)

delete_dirs()

class glideinwms.creation.lib.cWDictFile.ExeFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: SimpleFile

Dictionary file (:class:DictFile) with the content of an executable file

This class holds the content of the whole file in the single bytes value with key ‘content’. Any other key is invalid. When saving the content to the file, it will set the permissions to make it executable

save(dir=None, fname=None, sort_keys=None, set_readonly=True, reset_changed=True, save_only_if_changed=True, want_comments=True)

Parameters:

• dir

• fname

• sort_keys

• set_readonly

• reset_changed

• save_only_if_changed

• want_comments

Returns:

class glideinwms.creation.lib.cWDictFile.FileCommonDicts

Bases: object

has_key(key)

keys()

set_readonly(readonly=True)

class glideinwms.creation.lib.cWDictFile.FileDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: SimpleFileDictFile

Dictionary file for files (file list). Used for list of transferred files.

It is using a dictionary (key, value) from DictFile, serialized to file. The key is the file ID The value (line) on file has DATA_LENGTH (7) components: the key and the first DATA_LENGTH-1 attributes below. The value in memory has DATA_LENGTH components (real_fname,cache/exec,period,prefix,cond_download,config_out, data), the key is used as key for the dictionary and the data (file content) is added reading the file. Here the attributes stored as tuple in the dictionary value: 1. real_fname, i.e file name 2. cache/exec/… keyword identifying the file type: regular, nocache, exec (:s modifier to run in singularity), untar, wrapper 3. period period in seconds at which an executable is re-invoked (only for periodic executables, 0 otherwise) 4. prefix startd_cron variables prefix (default is GLIDEIN_PS_) 5. cond_download has a special value of TRUE 6. config_out has a special value of FALSE 7. data - String containing the data extracted from the file (real_fname) (not in the serialized dictionary) For placeholders, the real_name is empty (and the tuple starts w/ an empty string). Placeholders cannot be serialized (saved into file). Empty strings would cause error when parsed back.

DATA_LENGTH = 7

PLACEHOLDER_VALUE = ('', '', 0, '', '', '', '')

add(key, val, allow_overwrite=False, allow_overwrite_placeholder=True)

Add a file to the list

Invoke add_from_str if the content is provided (6th component of val), add_from_file otherwise

Parameters:

• key (str) – file ID

• val (tuple) – lists of 6 or 7 components (see class definition)

• allow_overwrite (bool) – if True the existing files can be replaced (default: False)

• allow_overwrite_placeholder (bool) – if True, placeholder files can be replaced even if allow_overwrite is False (default: True)

Raises:

DictFileError –

add_from_bytes(key, val, data, allow_overwrite=False, allow_overwrite_placeholder=True)

Add a file to the list, the content is provided separately (not in the val tuple)

Parameters:

• key (str) – file ID

• val (tuple) – lists of 6 or 7 components (see class definition)

• data (bytes) – bytes string w/ data to add

• allow_overwrite (bool) – if True the existing files can be replaced (default: False)

• allow_overwrite_placeholder (bool) – if True, placeholder files can be replaced even if allow_overwrite is False (default: True)

Raises:

DictFileError –

add_from_str(key, val, data, allow_overwrite=False, allow_overwrite_placeholder=True)

Add a file to the list, the content is provided separately (not in the val tuple)

Parameters:

• key (str) – file ID

• val (tuple) – lists of 6 or 7 components (see class definition)

• data (str) – string w/ data to add

• allow_overwrite (bool) – if True the existing files can be replaced (default: False)

• allow_overwrite_placeholder (bool) – if True, placeholder files can be replaced even if allow_overwrite is False (default: True)

Raises:

DictFileError –

add_placeholder(key, allow_overwrite=True)

file_header(want_comments)

Return the file header line (file name) as comment, None is want_comments is False

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

The file header, a comment containing the file name

Return type:

str

format_val(key, want_comments)

Print lines: only the file name (key) the first item of the value tuple if not None

Parameters:

• key – file name (dictionary key)

• want_comments – NOT USED, required by inheritance

Returns:

Formatted string with key (file name) and values (tuple of options for that file)

Only the file name if there are no values

Return type:

str

get_file_fname(key)

get_immutable_files()

is_placeholder(key)

static make_val_tuple(file_name, file_type, period=0, prefix='GLIDEIN_PS_', cond_download='TRUE', config_out='FALSE')

Make a tuple with the DATA_LENGTH-1 attributes in the correct order using the defaults

Parameters:

• file_name – name of the file (aka real_fname)

• file_type – type of the file (regular, nocache, exec, untar, wrapper). ‘exec allows modifiers like ‘:s’

• period – period for periodic executables (ignored otherwise, default: 0)

• prefix – prefix for periodic executables (ignored otherwise, default: GLIDEIN_PS_)

• cond_download – conditional download (default: ‘TRUE’)

• config_out – config out (default: ‘FALSE’)

Returns:

tuple with the DATA_LENGTH-1 attributes

See class definition for more information about the attributes

parse_val(line)

Parse a line of serialized FileDictFile files and add it to the dictionary

Each line is a tab separated tuple w/ the key and the attributes describing the entry (see class description )

Parameters:

line – string with the line content

Returns:

tuple with DATA_LENGTH-1 values

Return type:

tuple

reuse(other, compare_dir=False, compare_fname=False, compare_files_fname=False)

Reuse the entry value (and file) if an item in the “other” dictionary shares the same attributes and content

Parameters:

• other (FileDictFile) – other dictionary

• compare_dir (bool) – reuse only if the serialized dictionary is in the same directory (Default: False)

• compare_fname (bool) – reuse only if the serialized dictionary has the same name (Default: False)

• compare_files_fname (bool) – reuse only if the item file name is the same (Default: False)

static val_to_file_name(val)

class glideinwms.creation.lib.cWDictFile.FileDicts(work_dir, stage_dir, sub_list=[], workdir_name='work', simple_work_dir=False, log_dir=None)

Bases: object

This Class contains both the main and the sub dicts for a file dictionary

create_dirs(fail_if_exists=True)

delete_dirs()

erase(destroy_old_subs=True)

Erase the fileDict If destroy_old_subs is True, then the sub list and dict are reset; if it is False, then erase() is called recursively on the sub_list items but the sub names is preserved.

Parameters:

destroy_old_subs (bool) – if false, the sub names will be preserved

get_sub_name_from_sub_stage_dir(stage_dir)

This must be redefined by the child and return the sub (e.g. Entry) name

Parameters:

stage_dir (str, Path) – sub item stage directory

Returns:

sub item name (e.g. Entry name)

Return type:

str

is_equal(other, compare_work_dir: bool = False, compare_stage_dir: bool = False, compare_fnames: bool = False)

Compare 2 fileDict main and sub dictionaries. Return False if the content is different and optionally also if file name or staging or work directory differ.

Parameters:

• other (FileDicts) – other FileDicts object to compare to self. Must be of the same class as self

• compare_work_dir (bool) – if True, fail if the 2 fileDict are not in the same directory (same work directory)

• compare_stage_dir (bool) – if True, fail if the 2 fileDict don’t use the same staging directory

• compare_fnames (bool) – if True, fail if the file name is different

Returns:

True if the 2 fileDict are the same (following the specified options)

Return type:

bool

load(destroy_old_subs=True)

Load all the dictionaries (from files). If destroy_old_subs is False, existing (in the FileDicts dictionary) items are preserved if not over-written by a loaded one with the same name. If it is True, old items are dropped and only the new ones will be in the dictionary.

Parameters:

destroy_old_subs (bool) – if false, overwrite the subs you load, but leave the others as they are

new_MainDicts()

This should be redefined by the child and return a child of FileMainDicts

Returns:

return a main dictionary of the same type (child of FileMainDicts)

Return type:

FileMainDicts

new_SubDicts(sub_name)

This should be redefined by the child and return a child of FileSubDicts

Parameters:

sub_name (str) – sub dictionary name

Returns:

return a sub dictionary of the same type (child of FileSubDicts)

Return type:

FileSubDicts

reuse(other)

Populate the dictionary (re)using as much as possible of other. I.e. all the items with the same key.

Parameters:

other (FileDicts) – File dictionary to reuse. Must be of the same class as self

Raises:

RuntimeError – if the file dict is incompatible, i.e. the work_dir or stage_dir are different

save(set_readonly=True)

Save (to file) all the fileDict and sub dict. Invoking SUB.save(), MAIN.save(), and SUB.save_final()

Parameters:

set_readonly (bool) – set the file as read only if True

set_readonly(readonly=True)

class glideinwms.creation.lib.cWDictFile.FileMainDicts(work_dir, stage_dir, workdir_name, simple_work_dir=False, log_dir=None)

Bases: FileCommonDicts, DirsSupport

This Class contains the main dicts (dicts is a dict of DictFiles)

erase()

get_daemon_log_dir(base_dir)

get_main_dicts()

Interface

Returns:

keys are standard depending on the dictionary

Return type:

dict of DictFile

get_summary_signature()

is_equal(other, compare_work_dir=False, compare_stage_dir=False, compare_fnames=False)

load()

populate(params=None)

reuse(other)

save(set_readonly=True)

class glideinwms.creation.lib.cWDictFile.FileSubDicts(base_work_dir, base_stage_dir, sub_name, summary_signature, workdir_name, simple_work_dir=False, base_log_dir=None)

Bases: FileCommonDicts, DirsSupport

This Class contains the sub dicts

erase()

get_sub_dicts()

get_sub_log_dir(base_dir)

get_sub_stage_dir(base_dir)

get_sub_work_dir(base_dir)

is_equal(other, compare_sub_name=False, compare_fnames=False)

Parameters:

• other – other must be of the same class (child of FileSubDicts)

• compare_sub_name (bool)

• compare_fnames (bool)

Returns:

load()

reuse(other)

reuse_nocheck(other)

save(set_readonly=True)

save_final(set_readonly=True)

class glideinwms.creation.lib.cWDictFile.GridMapDict(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFileTwoKeys

Dictionary file (:class:DictFile) with the GrigMap file information

The dictionary is keyed both by DN and user

file_header(want_comments)

Return the file header line (file name) as comment, None is want_comments is False

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

The file header, a comment containing the file name

Return type:

str

format_val(key, want_comments)

Return a string with the formatted (space+tab separated) key, value

Parameters:

• key – key of the key, value pair to format

• want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

formatted key, value

Return type:

str

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

class glideinwms.creation.lib.cWDictFile.LogDirSupport(log_dir, dir_name='log')

Bases: SimpleDirSupport

class glideinwms.creation.lib.cWDictFile.LogSymlinkSupport(log_dir, work_dir, symlink_subdir='log', dir_name='log')

Bases: SymlinkSupport

class glideinwms.creation.lib.cWDictFile.MonitorDirSupport(monitor_dir, dir_name='monitor')

Bases: DirSupport, DirsSupport

create_dir(fail_if_exists=True)

Create the directory

Parameters:

fail_if_exists (bool) – fail with RuntimeError if the directory exists already

Returns:

True if the dir was created, false else

Return type:

bool

Raises:

RuntimeError – if failed the operation failed and the dircetory is not already there or if it is there and fail_if_exists is True

delete_dir()

class glideinwms.creation.lib.cWDictFile.MonitorFileDicts(work_dir, stage_dir, sub_list=[], workdir_name='work', simple_work_dir=False)

Bases: object

erase(destroy_old_subs=True)

get_sub_name_from_sub_stage_dir(sign_key)

load(destroy_old_subs=True)

new_MainDicts()

new_SubDicts(sub_name)

save(set_readonly=True)

set_readonly(readonly=True)

class glideinwms.creation.lib.cWDictFile.MonitorWLinkDirSupport(monitor_dir, work_dir, work_subdir='monitor', monitordir_name='monitor')

Bases: MonitorDirSupport

class glideinwms.creation.lib.cWDictFile.MultiSimpleDirSupport(list_of_dirs, dir_name)

Bases: DirSupport, DirsSupport

create_dir(fail_if_exists=True)

Create the directory

Parameters:

fail_if_exists (bool) – fail with RuntimeError if the directory exists already

Returns:

True if the dir was created, false else

Return type:

bool

Raises:

RuntimeError – if failed the operation failed and the dircetory is not already there or if it is there and fail_if_exists is True

delete_dir()

class glideinwms.creation.lib.cWDictFile.ReprDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: ReprDictFileInterface, DictFile

Dictionary file (:class:DictFile) saving the Python representation of the items

Uses the :class:ReprDictFileInterface interface and inherits from the actual DictFile

class glideinwms.creation.lib.cWDictFile.ReprDictFileInterface

Bases: object

Interface for a dictionary file (:class:DictFile) saving the Python representation

Will convert values into python format before writing them out Given that it does not call any parent methods, implement an interface first

format_val(key, want_comments)

interface_fake_init()

parse_val(line)

class glideinwms.creation.lib.cWDictFile.SHA1DictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

Dictionary file (:class:DictFile) with SHA1 signatures of files

This is used to send to the Glidein files checksums. Saved as “SHA1 FNAME” lines

add_from_file(filepath, allow_overwrite=False, key=None)

Add the SHA1 digest of the file to the dictionary (keyed by the file name)

Parameters:

• filepath (str) – path of the file to calculate the digest

• allow_overwrite (bool) – allow overwrite if True

• key (str) – if key==None, use basefname (file name extracted form filepath)

Returns:

format_val(key, want_comments)

Format values into a line

Parameters:

• key (str) – key

• want_comments (bool) – not used

Returns:

line for the file

Return type:

str

parse_val(line)

Parse a line into values for the dictionary

Comments are ignored

Parameters:

line (str) – file line

Returns:

class glideinwms.creation.lib.cWDictFile.SimpleDirSupport(dir, dir_name)

Bases: DirSupport

create_dir(fail_if_exists=True)

Create the directory

Parameters:

fail_if_exists (bool) – fail with RuntimeError if the directory exists already

Returns:

True if the dir was created, false else

Return type:

bool

Raises:

RuntimeError – if failed the operation failed and the dircetory is not already there or if it is there and fail_if_exists is True

delete_dir()

class glideinwms.creation.lib.cWDictFile.SimpleFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

Dictionary file (:class:DictFile) with the content of a file

This class holds the content of the whole file in the single bytes value with key ‘content’. Any other key is invalid.

add(key, val, allow_overwrite=False)

Add a key, value pair to the dictionary (self) if not already there

Changes the content of the dictionary and set self.changed to True if the value was added.

Parameters:

• key (str) – dictionary key

• val – Any value stored in the dictionary, usually a tuple (e.g. attributes and value)

• allow_overwrite (bool) – Allow to overwrite an exising value if True (default: False)

Raises:

DictFileError – when trying to modify a read only DictFile or overriding an existing item when prohibited or using an incompatible value

file_header(want_comments)

Return the file header line (file name) as comment, None is want_comments is False

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

The file header, a comment containing the file name

Return type:

str

format_val(key, want_comments)

Format the content of the file. Only the ‘content’ key is accepted

Parameters:

• key

• want_comments (bool) – not used

Returns:

content of the file

Return type:

str

Raises:

RuntimeError – if the key is not ‘content’

load_from_fd(fd, erase_first=True, set_not_changed=True)

Load the content from a binary file (binary mode used also for text files)

Parameters:

• fd – binary file to read the data from

• erase_first (bool) – if True, default, delete the old content first

• set_not_changed (bool) – if True, set self.changed to False

Returns:

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

class glideinwms.creation.lib.cWDictFile.SimpleFileDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

Dictionary (:class:DictFile) of files that holds also the content of the file as the last element in the values.

Value is a tuple. The dictionary is serialized using a file (dictionary file), one item per line. Each item is a file, identified by a file name, with an optional attribute (value) and the file content. File names are key. All files are in the same directory of the dictionary (self.dir). The values are a tuple and the last item is the file content. The file content in the value can be a binary blob (bytes) so it should read accordingly w/o attempting a decode. Only the file name and the first element of the value are saved in the dictionary file (serialized dictionary). SimpleFileDictFile is used for file lists.

Dictionary saved as “FNAME VALUE”, where FNAME is the key and VALUE the file attributes, and a series of separate files w/ the content

add(key, val, allow_overwrite=False)

Add an entry to the dictionary, e.g. a file to the list

Parameters:

• key (str) – file name (dictionary key)

• val – parameters (not the file content), usually file attributes

• allow_overwrite (bool) – if True, allows to override existing content in the dictionary (Default: False)

add_from_bytes(key, val, data, allow_overwrite=False)

Add an entry to the dictionary, parameters and content are both available

Same as add_from_str(), but the value is here is binary, BINARY_ENCODING encoded if it was text

Parameters:

• key (str) – file name (key)

• val – parameters (not the file content), usually file attributes (tuple, list or scalar)

• data (bytes) – bytes string with the file content added to the dictionary (this is binary or BINARY_ENCODING encoded text)

• allow_overwrite (bool) – if True, allows to override existing content in the dictionary

add_from_fd(key, val, fd, allow_overwrite=False)

Add an entry to the dictionary using a file object - has a read() method that provides the content

Parameters:

• key (str) – file name (key)

• val – parameters (not the file content), usually file attributes (tuple, list or scalar)

• fd – file object - has a read() method, opened in binary mode not to try a decode of the content

• allow_overwrite (bool) – if True, allows to override existing content in the dictionary

add_from_file(key, val, filepath, allow_overwrite=False)

Add data from a file. Add an entry to the dictionary using a file path

The file could be either a text or a binary file. Opened as binary file.

Parameters:

• key (str) – file name (key)

• val – parameters (not the file content), usually file attributes (tuple, list or scalar)

• filepath (str) – full path of the file

• allow_overwrite (bool) – if True, allows to override existing content in the dictionary

Raises:

DictFileError – if the file could not be opened (IOError from the system)

add_from_str(key, val, data, allow_overwrite=False)

Add an entry to the dictionary, parameters and content are both available

Same as add_from_bytes(), but the value is here is text, will be BINARY_ENCODING encoded before calling add_from_bytes()

Parameters:

• key (str) – file name (key)

• val – parameters (not the file content), usually file attributes (tuple, list or scalar)

• data (str) – string with the file content added to the dictionary (this is decoded, not bytes) It will be encoded using BINARY_ENCODING

• allow_overwrite (bool) – if True, allows to override existing content in the dictionary

format_val(key, want_comments)

Print lines: only the file name (key) the first item of the value tuple if not None

Parameters:

• key – file name (dictionary key)

• want_comments – NOT USED, required by inheritance

Returns:

Formatted string with key (file name) and values (tuple of options for that file)

Only the file name if there are no values

Return type:

str

get_file_fname(key)

get_immutable_files()

parse_val(line)

Parse line and add value and content to the dictionary

Skip comments (start with #) or empty lines and do nothing, otherwise add to the dictionary: First item is the file name (key), the rest are the parameter, data is read form the file (file name) Used to parse the line of a file list and add the files to a DictFile object

Parameters:

line (str) – line to be parsed

save_files(allow_overwrite=False)

Write the content of the files referred in the dictionary

For each item self.vals[key][-1] is the content of the file It should be bytes, if it is not it will be encoded using BINARY_ENCODING This methos is not saving the dictionary itself (key, values)

Parameters:

allow_overwrite (bool) – if True allow to over write existing files

Raises:

DictFileError – if an error occurred in writing into the file or the file data (last element) is str instead of bytes

class glideinwms.creation.lib.cWDictFile.SimpleWorkDirSupport(dir, dir_name)

Bases: SimpleDirSupport

class glideinwms.creation.lib.cWDictFile.StageDirSupport(stage_dir, dir_name='stage')

Bases: SimpleDirSupport

class glideinwms.creation.lib.cWDictFile.StrDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

Dictionary file (:class:DictFile) holding only strings

All values are converted to strings when added to the dictionary. And they are saved to the file and loaded from the file as string.

add(key, val, allow_overwrite=False)

Add a key, value pair to the dictionary (self) if not already there

Changes the content of the dictionary and set self.changed to True if the value was added.

Parameters:

• key (str) – dictionary key

• val – Any value stored in the dictionary, usually a tuple (e.g. attributes and value)

• allow_overwrite (bool) – Allow to overwrite an exising value if True (default: False)

Raises:

DictFileError – when trying to modify a read only DictFile or overriding an existing item when prohibited or using an incompatible value

class glideinwms.creation.lib.cWDictFile.StrWWorkTypeDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: StrDictFile

Dictionary file (:class:DictFile) saving only strings in the file

This extends the :class:StrDictFile dictionary file. The values are converted to strings when added, but typed_vals, retrievable with :func:get_typed_val, contain the typed values. Values are saved to the file and loaded from the file as string. It is not guaranteed to have typed values on (re-)load.

add(key, val, allow_overwrite=False)

Add a key, value pair to the dictionary (self) if not already there

Changes the content of the dictionary and set self.changed to True if the value was added.

Parameters:

• key (str) – dictionary key

• val – Any value stored in the dictionary, usually a tuple (e.g. attributes and value)

• allow_overwrite (bool) – Allow to overwrite an exising value if True (default: False)

Raises:

DictFileError – when trying to modify a read only DictFile or overriding an existing item when prohibited or using an incompatible value

erase()

get_typed_val(key)

remove(key, fail_if_missing=False)

class glideinwms.creation.lib.cWDictFile.SummarySHA1DictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

Dictionary file (:class:DictFile) with a Summary w/ SHA1 signatures

Values are (sha1, fname2) Saved as “SHA1 FNAME2 FNAME” lines

add(key, val, allow_overwrite=False)

Add a SHA1 signature to the dictionary

Parameters:

• key (str) – key, file name

• val (tuple) – (sha1, fname2) tuples

• allow_overwrite

Returns:

add_from_file(filepath, fname2=None, allow_overwrite=False, key=None)

Add a file and its SHA1 signature to a summary dictionary

Parameters:

• filepath (str) – full path of the file to add to the dictionary

• fname2 (str) – if fname2==None, use basefname

• allow_overwrite (bool) – allow overwrite if True

• key (str) – if key==None, use basefname (file name extracted form filepath)

Returns:

format_val(key, want_comments)

Format the dictionary elements into a line

Parameters:

• key (str) – key, file name

• want_comments (bool) – not used

Returns:

parse_val(line)

Parse a line and add the values to the dictionary

Parameters:

line (str) – line to parse

Returns:

class glideinwms.creation.lib.cWDictFile.SymlinkSupport(target_dir, symlink, dir_name)

Bases: DirSupport

Symlink to a directory

create_dir(fail_if_exists=True)

Create the directory

Parameters:

fail_if_exists (bool) – fail with RuntimeError if the directory exists already

Returns:

True if the dir was created, false else

Return type:

bool

Raises:

RuntimeError – if failed the operation failed and the dircetory is not already there or if it is there and fail_if_exists is True

delete_dir()

class glideinwms.creation.lib.cWDictFile.VarsDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

Dictionary file (:class:DictFile) to store variables’ information.

This is used store or to transfer to the Glidein variables lists. The key is the variable name. Values are (Type,Default,CondorName,Required,Export,UserName). The CondorName is the name to use in the HTCSS configuration

(special keyworks: + the same as the variable name)

The UserName is the name to use in the job environment

(special keyworks: + the same as the variable name, @ the same as the HTCSS name, - do not export)

add(key, val, allow_overwrite=0)

Add a key, value pair to the dictionary (self) if not already there

Changes the content of the dictionary and set self.changed to True if the value was added.

Parameters:

• key (str) – dictionary key

• val – Any value stored in the dictionary, usually a tuple (e.g. attributes and value)

• allow_overwrite (bool) – Allow to overwrite an exising value if True (default: False)

Raises:

DictFileError – when trying to modify a read only DictFile or overriding an existing item when prohibited or using an incompatible value

add_extended(key, type, val_default, condor_name, required, export_condor, user_name, allow_overwrite=0)

file_header(want_comments)

Return the file header line (file name) as comment, None is want_comments is False

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

The file header, a comment containing the file name

Return type:

str

format_val(key, want_comments)

Return a string with the formatted (space+tab separated) key, value

Parameters:

• key – key of the key, value pair to format

• want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

formatted key, value

Return type:

str

is_compatible(old_val, new_val)

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

class glideinwms.creation.lib.cWDictFile.WorkDirSupport(work_dir, workdir_name)

Bases: MultiSimpleDirSupport

glideinwms.creation.lib.cWDictFile.validate_node(nodestr, allow_range=False, check_dns=True)

Validate HTCondor endpoint (node) string

This can be a node, node:port, node:port-range or a shared port sinful string node[:port]?[var=val&]sock=collectorN1[-N2][&var=val] or a schedd schedd_name@node:port[?sock=collector&var=val] ‘sock’ cannot appear more than once ranges can be either in ports or in ‘sock’, not in both at the same time

Parameters:

• nodestr (str) – endpoint (node) string

• allow_range (bool) – True if a port range is allowed (e.g. for secondary collectors or CCBs)

• check_dns (bool) – False if the DNS check should raise only a RuntimeWarning (e.g. for schedds)

Raises:

• RuntimeWarning if the DNS check fails and check_dns is False –

• RuntimeError if the validation fails in any other way –

glideinwms.creation.lib.cWExpand module

Contains the algorithms needed for attribute expansion

glideinwms.creation.lib.cWExpand.expand_DD(qstr, attr_dict)

glideinwms.creation.lib.cWExpand.expand_DLR(qstr, attr_dict)

glideinwms.creation.lib.cWParamDict module

glideinwms.creation.lib.cWParamDict.add_file_unparsed(user_file, dicts, is_factory)

Add a user file residing in the stage area file as described by Params.file_defaults :param user_file: file from the config files “files” sections :param dicts: parameters dictionaries :param is_factory: True if invoked for the factory (cgWParamDict.py), false for the frontend (cvWParamDict.py) :return: None (dictionaries are modified)

glideinwms.creation.lib.cWParamDict.has_file_wrapper(dicts)

glideinwms.creation.lib.cWParamDict.has_file_wrapper_params(file_params)

glideinwms.creation.lib.cWParams module

This module contains the generic params classes

class glideinwms.creation.lib.cWParams.CommentedOrderedDict(indict=None)

Bases: OrderedDict

Ordered dictionary with comment support

class glideinwms.creation.lib.cWParams.CommonParams(usage_prefix, src_dir, argv)

Bases: Params

extract_attr_val(attr_obj)

get_subparams_class()

init_support_defaults()

class glideinwms.creation.lib.cWParams.CommonSubParams(data)

Bases: SubParams

_abc_impl = <_abc._abc_data object>

extract_attr_val(attr_obj)

class glideinwms.creation.lib.cWParams.Params(usage_prefix, src_dir, argv)

Bases: object

abstract class

Children must define:

get_top_element(self) init_defaults(self) derive(self) get_xml_format(self)

derive()

get_description(indent='', width=80)

get_subparams_class()

get_xml()

load_file(fname)

Load from a file one element per line

-opt val

Parameters:

fname

Returns:

save_into_file(fname, set_ro=False)

Save into a file The file should be usable for reload

Parameters:

• fname

• set_ro

Returns:

save_into_file_wbackup(fname, set_ro=False)

Save into a file (making a backup) The file should be usable for reload

Parameters:

• fname

• set_ro

Returns:

class glideinwms.creation.lib.cWParams.SubParams(data)

Bases: Mapping

Read-only dictionary containing Configuration info

__get_el(name)

Element getter, used by both __getitem__ and __getattr__

Parameters:

name (str) – key or attribute name

Returns:

value

Raises:

KeyError – when the key/attribute name is not in self.data

_abc_impl = <_abc._abc_data object>

use_defaults(defaults)

Put default values where there is nothing

Parameters:

defaults

Returns:

validate(base, path_text)

Validate input against base template (i.e. the defaults)

Parameters:

• base

• path_text

Returns:

glideinwms.creation.lib.cWParams.col_wrap(text, width, indent)

Wrap a text string to a fixed length

Parameters:

• text (str) – string to wrap

• width (int) – length

• indent (str) – indentation string

Returns:

glideinwms.creation.lib.cWParams.defdict2string(defaults, indent, width=80)

Convert defaults to a string

Parameters:

• defaults

• indent

• width

Returns:

glideinwms.creation.lib.cWParams.extract_attr_val(attr_obj)

Return attribute value in the proper python format

INTERNAL, don’t use directly Use the class definition instead

Parameters:

attr_obj

Returns:

glideinwms.creation.lib.cWParams.is_valid_name(name)

Check if a string can be used as a valid name

Whitelist based:

only allow ascii characters, numbers and a few punctuations no spaces, no special characters or other punctuation

Parameters:

name (str) – name to validate

Returns:

True if the name is not empty and has only valid characters, False otherwise

Return type:

bool

glideinwms.creation.lib.cWParams.shorten_text(text, width)

Shorten text, make sure you properly account tabs

Tabs are every 8 spaces (counted as number of chars to the next tab stop)

Parameters:

• text (str) – text to shorten

• width (int) – length

Returns (tuple):

shorten text (str): shortened text next char (str): remainder

glideinwms.creation.lib.cgWConsts module

glideinwms.creation.lib.cgWConsts.get_entry_log_dir(log_dir, entry_name)

glideinwms.creation.lib.cgWConsts.get_entry_monitor_dir(monitor_dir, entry_name)

glideinwms.creation.lib.cgWConsts.get_entry_name_from_entry_monitor_dir(entry_monitor_dir)

glideinwms.creation.lib.cgWConsts.get_entry_name_from_entry_stage_dir(entry_stage_dir)

glideinwms.creation.lib.cgWConsts.get_entry_name_from_entry_submit_dir(entry_submit_dir)

glideinwms.creation.lib.cgWConsts.get_entry_stage_dir(stage_dir, entry_name)

glideinwms.creation.lib.cgWConsts.get_entry_submit_dir(submit_dir, entry_name)

glideinwms.creation.lib.cgWConsts.get_entry_userlog_dir(log_dir, entry_name)

glideinwms.creation.lib.cgWConsts.get_entry_userproxies_dir(proxies_dir, entry_name)

glideinwms.creation.lib.cgWCreate module

Functions needed to create files used by the glidein entry points

class glideinwms.creation.lib.cgWCreate.GlideinSubmitDictFile(dir, fname, sort_keys=False, order_matters=False, jobs_in_cluster=None, fname_idx=None)

Bases: CondorJDLDictFile

finalize(sign, entry_sign, descript, entry_descript)

Since the arguments will be built by the submit script now, just pass. Keeping the function here because the code is so obtuse, if I remove the function, I may create unintended effects.

populate(exe_fname, entry_name, conf, entry)

Since there are only two parameters that ever were passed that didn’t already exist in the params dict or the sub_params dict, the function signature has been greatly simplified into just those two parameters and the two dicts.

This has the added benefit of being “future-proof” for as long as we maintain this particular configuration method. Any new attribute that may be in params or sub_params can be accessed here without having to add yet another parameter to the function.

Parameters:

• exe_fname (str, Path) – startup file name (glidein_startup.sh)

• entry_name (str) – Name of the Entry

• conf (factoryXmlConfig.Config) – XML configuration

• entry (factoryXmlConfig.EntryElement) – Entry element XML configuration

populate_condorc_grid()

populate_ec2_grid()

populate_gce_grid()

populate_glidein_classad(proxy_url)

populate_standard_grid(rsl, auth_method, gridtype, entry_enabled, entry_name, enc_input_files=None)

Create a standard condor jdl file to submit to OSG grid

Parameters:

• rsl (str)

• auth_method (str) – grid_proxy, voms_proxy, key_pair, cert_pair, username_password, optionally with +vm_type, +vm_id, and/or +project_id

• grid_type (str) – ‘condor’, ‘cream’, ‘nordugrid_rsl’ currently supported

• entry_enabled (expr) – if evaluates to False, does not write jdl

• entry_name (str) – factory entry name

• enc_input_files (list) – files to be appended to Transfer_input_files and encrypt_input_files in the jdl

Returns:

None

Raises:

RunTimeError when jdl file should not be created –

populate_submit_attrs(submit_attrs, gridtype, attr_prefix='')

glideinwms.creation.lib.cgWCreate.copy_exe(filename, work_dir, org_dir, overwrite=False)

Copy a file from one dir to another and changes the permissions to 0555. Can overwrite an existing file.

Parameters:

• filename – base name of the file

• work_dir – destination directory

• org_dir – source directory

• overwrite – if True, delete the destination file before making a copy of the source. If false shutil.copy2 (the command used underneath) will overwrite the destination

Returns:

None

Raises:

RuntimeError – if the copy failed raising IOError

glideinwms.creation.lib.cgWCreate.copy_file(infile, outfile)

Copy a file from infile to outfile preserving permission mode and all file metadata

It follows symlinks and overwrites the destination if existing

Parameters:

• infile – source file path

• outfile – destination file path or directory

Returns:

None

Raises:

RuntimeError – if the copy failed raising IOError

glideinwms.creation.lib.cgWCreate.create_condor_tar_fd(condor_base_dir)

Extract only components from a full condor distribution needed to run a glidein on a CE. This code is only run when a factory reconfig or upgrade is triggered.

Parameters:

condor_base_dir (str) – an untarred tarball of an HTCondor Distribution

Returns:

representation of tarfile.

Return type:

StringIO

glideinwms.creation.lib.cgWCreate.create_initd_startup(startup_fname, factory_dir, glideinWMS_dir, cfg_name, rpm_install='')

Creates the factory startup script from the template.

glideinwms.creation.lib.cgWCreate.get_condor_dlls(condor_dir, files=[], libdirs=['lib', 'lib/condor'])

Given list of condor files return all the libraries referenced by the files

@type condor_dir: string @param condor_dir: Location containing condor binaries @type files: list @param files: List of files relative to condor_dir @type libdirs: list @param libdirs: List of dirs relative to condor_dir that contain libs

@return: List containing linked libraries required by all the files.

Paths a relative to the condor_dir

@rtype: list

glideinwms.creation.lib.cgWCreate.get_factory_log_recipients(entry)

Read the log recipients specified by the Factory in its configuration

Parameters:

entry – dict-like object representing the entry configuration

Returns:

list contaning the URLs of the log servers, empty if none present

Return type:

list

glideinwms.creation.lib.cgWCreate.get_link_chain(link)

Given a filepath, checks if it is a link and processes all the links until the actual file is found

@type link: string @param link: Full path to the file/link

@return: List containing links in the chain @rtype: list

glideinwms.creation.lib.cgWCreate.get_template(template_name, glideinWMS_dir)

glideinwms.creation.lib.cgWCreate.ldd(file)

Given a file return all the libraries referenced by the file

@type file: string @param file: Full path to the file

@return: List containing linked libraries required by the file @rtype: list

glideinwms.creation.lib.cgWDictFile module

Dictionary files used in the Factory

Main dictionaries are in the global configuration. Entry dictionaries are in the entry configuration. Common dictionaries are in both. The versioned dictionary are stored in timestamp versioned files

Common dictionaries: attrs (cWDictFile.ReprDictFile, cgWConsts.ATTRS_FILE): attributes in the “entry” XML tag and “attrs” set as constants description (cWDictFile.DescriptionDictFile, cWConsts.DESCRIPTION_FILE, versioned): factory/entry description consts (cWDictFile.StrDictFile, cWConsts.CONSTS_FILE, versioned): constant attrs - TODO: not written? params (cWDictFile.ReprDictFile, cgWConsts.PARAMS_FILE): attrs that are not constant (const=”False”, default) submit (cWDictFile.ReprDictFile, cgWConsts.SUBMIT_ATTRS_FILE): submit configuration attributes vars (cWDictFile.VarsDictFile, cWConsts.VARS_FILE, versioned): condor variables untar_cfg (cWDictFile.StrDictFile, cWConsts.UNTAR_CFG_FILE, versioned): instructions for files to un-compress file_list (cWDictFile.FileDictFile, cWConsts.FILE_LISTFILE, versioned): list of files to transfers signature (cWDictFile.SHA1DictFile, cWConsts.SIGNATURE_FILE, versioned): files signatures for tampering verification

Main dictionaries (common and): summary_signature (cWDictFile.SummarySHA1DictFile, cWConsts.SUMMARY_SIGNATURE_FILE): glidein (cWDictFile.StrDictFile, cgWConsts.GLIDEIN_FILE): build_cvmfsexec (cWDictFile.ReprDictFile, cgWConsts.CVMFSEXEC_BUILD_FILE): cvmfsexec releases frontend_descript (cWDictFile.ReprDictFile, cgWConsts.FRONTEND_DESCRIPT_FILE): frontend description gridmap (cWDictFile.GridMapDict, cWConsts.GRIDMAP_FILE, versioned): at_file_list (cWDictFile.FileDictFile, cgWConsts.AT_FILE_LISTFILE, versioned): list of files w/ intermediate priority (close to entry) after_file_list (cWDictFile.FileDictFile, cgWConsts.AFTER_FILE_LISTFILE, versioned): list of files loaded later (after entry)

Entry dictionaries (common and): job_descript (cWDictFile.StrDictFile, cgWConsts.JOB_DESCRIPT_FILE) infosys (InfoSysDictFile, cgWConsts.INFOSYS_FILE) mongroup (MonitorGroupDictFile, cgWConsts.MONITOR_CONFIG_FILE)

class glideinwms.creation.lib.cgWDictFile.CondorJDLDictFile(dir, fname, sort_keys=False, order_matters=False, jobs_in_cluster=None, fname_idx=None)

Bases: DictFile

Creating the condor submit file

NOTE: the ‘environment’ attribute should be in the new syntax format, to allow characters like ‘;’ in the values

value all double quoted, var=var_val space separated, var_val can be single quoted

add_environment(val)

append(key, val)

file_footer(want_comments)

Return the file footer as comment (None for this class)

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

None

format_val(key, want_comments)

Return a string with the formatted (space+tab separated) key, value

Parameters:

• key – key of the key, value pair to format

• want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

formatted key, value

Return type:

str

is_equal(other, compare_dir=False, compare_fname=False, compare_keys=None)

Parametrised comparison of DictFile objects

Parameters:

• other (DictFile) – second object in the comparison, must be of the same class

• compare_dir (bool) – if True, compare also the directories of the files

• compare_fname (bool) – if True, compare also the file names

• compare_keys (bool) – if True, compare the order of the keys. If None, use order_matters

Returns:

True if self and other have the same representation

Return type:

bool

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

class glideinwms.creation.lib.cgWDictFile.InfoSysDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

add(key, val, allow_overwrite=0)

Add a key, value pair to the dictionary (self) if not already there

Changes the content of the dictionary and set self.changed to True if the value was added.

Parameters:

• key (str) – dictionary key

• val – Any value stored in the dictionary, usually a tuple (e.g. attributes and value)

• allow_overwrite (bool) – Allow to overwrite an exising value if True (default: False)

Raises:

DictFileError – when trying to modify a read only DictFile or overriding an existing item when prohibited or using an incompatible value

add_extended(infosys_type, server_name, ref_str, allow_overwrite=0)

file_header(want_comments)

Return the file header line (file name) as comment, None is want_comments is False

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

The file header, a comment containing the file name

Return type:

str

format_val(key, want_comments)

Return a string with the formatted (space+tab separated) key, value

Parameters:

• key – key of the key, value pair to format

• want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

formatted key, value

Return type:

str

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

class glideinwms.creation.lib.cgWDictFile.MonitorGroupDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

add(key, val, allow_overwrite=0)

Add a key, value pair to the dictionary (self) if not already there

Changes the content of the dictionary and set self.changed to True if the value was added.

Parameters:

• key (str) – dictionary key

• val – Any value stored in the dictionary, usually a tuple (e.g. attributes and value)

• allow_overwrite (bool) – Allow to overwrite an exising value if True (default: False)

Raises:

DictFileError – when trying to modify a read only DictFile or overriding an existing item when prohibited or using an incompatible value

add_extended(group_name, allow_overwrite=0)

file_footer(want_comments)

Return the file footer as comment (None for this class)

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

None

file_header(want_comments)

Return the file header line (file name) as comment, None is want_comments is False

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

The file header, a comment containing the file name

Return type:

str

format_val(key, want_comments)

Return a string with the formatted (space+tab separated) key, value

Parameters:

• key – key of the key, value pair to format

• want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

formatted key, value

Return type:

str

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

class glideinwms.creation.lib.cgWDictFile.baseClientDirSupport(user, dir, dir_name='client')

Bases: MultiSimpleDirSupport

class glideinwms.creation.lib.cgWDictFile.chmodClientDirSupport(user, dir, chmod, dir_name)

Bases: clientDirSupport

create_dir(fail_if_exists=True)

Create the directory

Parameters:

fail_if_exists (bool) – fail with RuntimeError if the directory exists already

Returns:

True if the dir was created, false else

Return type:

bool

Raises:

RuntimeError – if failed the operation failed and the dircetory is not already there or if it is there and fail_if_exists is True

class glideinwms.creation.lib.cgWDictFile.clientDirSupport(user, dir, dir_name)

Bases: SimpleDirSupport

create_dir(fail_if_exists=True)

Create the directory

Parameters:

fail_if_exists (bool) – fail with RuntimeError if the directory exists already

Returns:

True if the dir was created, false else

Return type:

bool

Raises:

RuntimeError – if failed the operation failed and the dircetory is not already there or if it is there and fail_if_exists is True

delete_dir()

class glideinwms.creation.lib.cgWDictFile.clientLogDirSupport(user, log_dir, dir_name='clientlog')

Bases: clientDirSupport

class glideinwms.creation.lib.cgWDictFile.clientProxiesDirSupport(user, proxies_dir, proxiesdir_name='clientproxies')

Bases: chmodClientDirSupport

class glideinwms.creation.lib.cgWDictFile.clientSymlinksSupport(user_dirs, work_dir, symlink_base_subdir, dir_name)

Bases: MultiSimpleDirSupport

glideinwms.creation.lib.cgWDictFile.get_common_dicts(submit_dir, stage_dir)

Return a dictionary with new instances of all the common (on both global and entry) dictionaries

Parameters:

• submit_dir

• stage_dir

Returns:

dictionary with new instances of all the common (in both global and entry) dictionaries

Return type:

dict

glideinwms.creation.lib.cgWDictFile.get_entry_dicts(entry_submit_dir, entry_stage_dir, entry_name)

Return a dictionary with new instances of all the entry only dictionaries

Parameters:

• entry_submit_dir

• entry_stage_dir

• entry_name (str) – name of the entry

Returns:

dictionary with new instances of all the entry only dictionaries

Return type:

dict

glideinwms.creation.lib.cgWDictFile.get_main_dicts(submit_dir, stage_dir)

Return a dictionary with new instances of all the main (global) only dictionaries

Parameters:

• submit_dir

• stage_dir

Returns:

dictionary with new instances of all the main (global) only dictionaries

Return type:

dict

class glideinwms.creation.lib.cgWDictFile.glideinDicts(work_dir, stage_dir, log_dir, client_log_dirs, client_proxies_dirs, entry_list=[], workdir_name='submit')

Bases: FileDicts

get_sub_name_from_sub_stage_dir(sign_key)

This must be redefined by the child and return the sub (e.g. Entry) name

Parameters:

stage_dir (str, Path) – sub item stage directory

Returns:

sub item name (e.g. Entry name)

Return type:

str

new_MainDicts()

This should be redefined by the child and return a child of FileMainDicts

Returns:

return a main dictionary of the same type (child of FileMainDicts)

Return type:

FileMainDicts

new_SubDicts(sub_name)

This should be redefined by the child and return a child of FileSubDicts

Parameters:

sub_name (str) – sub dictionary name

Returns:

return a sub dictionary of the same type (child of FileSubDicts)

Return type:

FileSubDicts

class glideinwms.creation.lib.cgWDictFile.glideinEntryDicts(base_work_dir, base_stage_dir, sub_name, summary_signature, workdir_name, base_log_dir, base_client_log_dirs, base_client_proxies_dirs)

Bases: FileSubDicts

This Class contains the entry and entry set dicts

get_sub_dicts()

get_sub_log_dir(base_dir)

get_sub_stage_dir(base_dir)

get_sub_work_dir(base_dir)

load()

reuse(other)

reuse_nocheck(other)

save(set_readonly=True)

save_final(set_readonly=True)

class glideinwms.creation.lib.cgWDictFile.glideinMainDicts(work_dir, stage_dir, workdir_name, log_dir, client_log_dirs, client_proxies_dirs)

Bases: FileMainDicts

get_daemon_log_dir(base_dir)

get_main_dicts()

Interface

Returns:

keys are standard depending on the dictionary

Return type:

dict of DictFile

load()

reuse(other)

save(set_readonly=True)

glideinwms.creation.lib.cgWDictFile.load_common_dicts(dicts, description_el)

Load (from file) the common (to global and entry) dictionaries

Parameters:

• dicts (dict) – dictionary with the common dictionaries updated in place this dictionary may contain also other (global or entry) dictionaries that are not updated

• description_el

glideinwms.creation.lib.cgWDictFile.load_entry_dicts(entry_dicts, entry_name, summary_signature)

Load (from file) the entry dictionaries Loads also the common ones, calling load_common_dicts

Parameters:

• entry_dicts (dict) – dictionary with entry dictionaries updated in place

• entry_name (str) – entry name

• summary_signature

glideinwms.creation.lib.cgWDictFile.load_main_dicts(main_dicts)

Load (from file) the main dictionaries Loads also the common ones, calling load_common_dicts

Parameters:

main_dicts (dict) – dictionary with main (global) dictionaries (updated in place)

glideinwms.creation.lib.cgWDictFile.refresh_description(dicts)

glideinwms.creation.lib.cgWDictFile.refresh_file_list(dicts, is_main, files_set_readonly=True, files_reset_changed=True)

Update in place the file lists dictionaries

Parameters:

• dicts

• is_main (bool) – True if this is the file list for main (and not entries or entry lists)

• files_set_readonly (bool) – do not set read only if false

• files_reset_changed (bool) – do not reset the changed flag if False

glideinwms.creation.lib.cgWDictFile.refresh_signature(dicts)

glideinwms.creation.lib.cgWDictFile.reuse_common_dicts(dicts, other_dicts, is_main, all_reused)

glideinwms.creation.lib.cgWDictFile.reuse_entry_dicts(entry_dicts, other_entry_dicts, entry_name)

glideinwms.creation.lib.cgWDictFile.reuse_file_dict(dicts, other_dicts, key)

glideinwms.creation.lib.cgWDictFile.reuse_main_dicts(main_dicts, other_main_dicts)

glideinwms.creation.lib.cgWDictFile.reuse_simple_dict(dicts, other_dicts, key, compare_keys=None)

glideinwms.creation.lib.cgWDictFile.save_common_dicts(dicts, is_main, set_readonly=True)

Save the common dictionaries (in both the main and entry dictionaries lists) If they are there, they will be updated in place This function calls the save method of each individual dictionary in the common list

Parameters:

• dicts (dict) – dictionary of all dictionaries

• is_main (bool) – True if called while saving the main dictionaries (and not entries or entry lists)

• set_readonly (bool) – True (default) to also set the dictionary read only

glideinwms.creation.lib.cgWDictFile.save_entry_dicts(entry_dicts, entry_name, summary_signature, set_readonly=True)

Save the entry dictionaries (corresponding to the entry configuration) If they are there, they will be updated in place This function calls the save method of each individual dictionary in the entry list, including the common ones by calling save_common_dicts

Parameters:

• entry_dicts (dict) – dictionary of all dictionaries

• entry_name (str) – name of the Entry

• summary_signature – summary signature dict file, used to insert the entry sifnature dict file

• set_readonly (bool) – True (default) to also set the dictionary read only

glideinwms.creation.lib.cgWDictFile.save_main_dicts(main_dicts, set_readonly=True)

Save the main dictionaries (corresponding to the global configuration) If they are there, they will be updated in place This function calls the save method of each individual dictionary in the main list, including the common ones by calling save_common_dicts

Parameters:

• main_dicts (dict) – dictionary of all dictionaries

• set_readonly (bool) – True (default) to also set the dictionary read only

glideinwms.creation.lib.cgWParamDict module

Glidein creation module Classes and functions needed to handle dictionary files created out of the parameter object

exception glideinwms.creation.lib.cgWParamDict.UnconfiguredScheddError(schedd)

Bases: Exception

glideinwms.creation.lib.cgWParamDict.add_attr_unparsed(attr, dicts, description)

glideinwms.creation.lib.cgWParamDict.add_attr_unparsed_real(attr, dicts)

glideinwms.creation.lib.cgWParamDict.add_file_unparsed(user_file, dicts, is_factory)

glideinwms.creation.lib.cgWParamDict.add_submit_attr_unparsed(attr, dicts, description, resource_type='main')

Adds a submit_attr to the main/entry dictionary. The value is added raw, not parsed

Parameters:

• attr – submit_attr element from the XML file

• dicts – Entry or main dictionary. This is updated adding a value in dicts[“submit_attrs”]

• description (str) – name of the entry (“main” for the main/global configuration)

• resource_type (str) – Resource type of the entry where the attribute is defined. Use “main” for the main config

Raises:

RuntimeError – if the submit_attr configuration is incorrect

glideinwms.creation.lib.cgWParamDict.calc_monitoring_collectors_string(collectors)

glideinwms.creation.lib.cgWParamDict.calc_primary_monitoring_collectors(collectors)

glideinwms.creation.lib.cgWParamDict.copy_file(infile, outfile)

glideinwms.creation.lib.cgWParamDict.get_valid_condor_tarballs(condor_tarballs)

class glideinwms.creation.lib.cgWParamDict.glideinDicts(conf, sub_list=None)

Bases: glideinDicts

This Class contains all the main, the entry, and the entry set dicts

new_MainDicts()

This should be redefined by the child and return a child of FileMainDicts

Returns:

return a main dictionary of the same type (child of FileMainDicts)

Return type:

FileMainDicts

new_SubDicts(sub_name)

This should be redefined by the child and return a child of FileSubDicts

Parameters:

sub_name (str) – sub dictionary name

Returns:

return a sub dictionary of the same type (child of FileSubDicts)

Return type:

FileSubDicts

populate(other=None)

Will update params (or self.params) using the values from other. When other is provided, set the schedd to use for an Entry preserving the ones set in other and populate the common element in the entry using the main dictionary content.

Parameters:

other (glideinDicts|None) – other dictionary of the same Class

reuse(other)

Parameters:

other

Returns:

sortit(unsorted_dict)

A temporary method for sorting a dictionary based on the value of the dictionary item. In python 2.4+, a ‘key’ arguement can be used in the ‘sort’ and ‘sorted’ functions. This is not available in python 2.3.4/SL4 platforms. Returns a sorted list of the dictionary items based on their value.

class glideinwms.creation.lib.cgWParamDict.glideinEntryDicts(conf, sub_name, summary_signature, workdir_name)

Bases: glideinEntryDicts

erase()

load()

Load also the condor_jdl. Other dictionaries are loaded using the parent method cgWDictFile.glideinEntryDicts.load()

populate(entry, schedd, main_dicts)

Populate the entry dictionaries

Parameters:

• entry (_type_) – entry section from the Factory XML configuration

• schedd (str) – schedd used for the job submission

• main_dicts (_type_) – main dictionaries

reuse(other)

save_final(set_readonly=True)

Save the condor_jdl dictionaries last, since if contains the signatures of the other dictionaries

Parameters:

set_readonly (bool) – True (default) if it should save and also set read-only

class glideinwms.creation.lib.cgWParamDict.glideinMainDicts(conf, workdir_name)

Bases: glideinMainDicts

This Class contains the main dicts

find_parent_dir(search_path, name)

Given a search path, determine if the given file exists somewhere in the path. Returns: if found. returns the parent directory

if not found, raises an Exception

populate(other=None)

reuse(other)

save(set_readonly=True)

save_monitor()

save_monitor_config(work_dir, glidein_dict)

save_pub_key()

glideinwms.creation.lib.cgWParamDict.iter_to_dict(dictObject)

Traverses a iterable (DictMixin) recursively to convert to proper dict any nested classes

glideinwms.creation.lib.cgWParamDict.itertools_product(*args, **kwds)

itertools.product() from Python 2.6

glideinwms.creation.lib.cgWParamDict.old_get_valid_condor_tarballs(params)

glideinwms.creation.lib.cgWParamDict.populate_cvmfsexec_build_config(cvmfsexec_dict, conf)

glideinwms.creation.lib.cgWParamDict.populate_factory_descript(work_dir, glidein_dict, active_sub_list, disabled_sub_list, conf)

Modifies the glidein_dict to contain the factory configuration values and the active_sub_list and disabled_sub_list to contain the active and disabled entries

Parameters:

• work_dir

• glidein_dict

• active_sub_list

• disabled_sub_list

• conf – Factory global configuration

glideinwms.creation.lib.cgWParamDict.populate_frontend_descript(frontend_dict, conf)

glideinwms.creation.lib.cgWParamDict.populate_gridmap(conf, gridmap_dict)

glideinwms.creation.lib.cgWParamDict.populate_job_descript(work_dir, job_descript_dict, num_factories, sub_name, entry, schedd, attrs_dict, enable_expansion)

Modifies the job_descript_dict to contain the factory configuration values.

Parameters:

• work_dir (str) – location of entry files

• job_descript_dict (dict) – contains the values of the job.descript file

• num_factories

• sub_name (str) – entry name

• entry

• schedd

• attrs_dict (dict) – dictionary of attributes

• enable_expansion (bool) – whether or not expand the attribute values with a $ in them

Returns:

glideinwms.creation.lib.cgWParamDict.validate_attribute(attr_name, attr_val)

Check the attribute value is valid. Otherwise throw RuntimeError

glideinwms.creation.lib.cgWParamDict.validate_condor_tarball_attrs(conf)

glideinwms.creation.lib.cgWParamDict.validate_submit_attr(attr_name, attr_val, always_publish, resource_type)

Validate if a submit_attr is configured correctly. Raise RuntimeError if incorrect

Parameters:

• attr_name (str) – attribute name

• attr_val (str) – attribute value

• always_publish (bol) – True if the attribute should be puublished on all resources (not only the ones supporting condor attributes)

• resource_type (str) – type of the resource used

Raises:

RuntimeError – if the submit_attr configuration is incorrect

glideinwms.creation.lib.cgWParams module

class glideinwms.creation.lib.cgWParams.GlideinParams(usage_prefix, src_dir, argv)

Bases: CommonParams

Contains all the factory configuration values as params. Used in create_glideins and recreate_glideins.

buildDir(factoryVersioning, basedir)

derive()

get_top_element()

get_xml_format()

init_defaults()

Populates the defaults for all the factory configuration values.

glideinwms.creation.lib.cgWParams.find_condor_base_dir()

glideinwms.creation.lib.check_config_frontend module

glideinwms.creation.lib.check_python3_expr module

Validate expressions in frontend.xml for compatibility with Python 3

glideinwms.creation.lib.check_python3_expr._log(text, silent=False)

glideinwms.creation.lib.check_python3_expr.check_2to3(code, patch=False, refactoring_tool=<lib2to3.refactor.RefactoringTool object>)

Evaluates an expression using 2to3 and returns refactoring suggestions.

Parameters:

• code (str) – Code to evaluate.

• patch (bool) – If True, returns a patch with the suggested changes.

• refactoring_tool (RefactoringTool) – Used to by 2to3 to evaluate the expression.

Returns:

2to3 suggested code. None if the expression conforms with Python 3.

Return type:

str

glideinwms.creation.lib.check_python3_expr.check_syntax(code)

Validates the Python 3 syntax of a code.

Parameters:

code (str) – Code to validate.

Returns:

None if code is valid. Error message if the code is invalid.

Return type:

str

glideinwms.creation.lib.check_python3_expr.check_types(expression, factory_attrs, job_attrs)

Validates the types of match_attrs in a match_expr.

Parameters:

• expression (str) – Code to validate.

• factory_attrs

• job_attrs

Returns:

None if code is valid. Error message if the code is invalid.

Return type:

str

glideinwms.creation.lib.check_python3_expr.element_name(element)

Finds the name attribute of element. Returns None if nothing is found.

Parameters:

element (Element) – Element to search.

Returns:

Element name.

Return type:

str

glideinwms.creation.lib.check_python3_expr.findall_path(root, tag, elements=[])

Finds all elements in root of tag type preserving their paths.

Parameters:

• root (Element) – Root element to be searched.

• tag (str) – Tag to search.

• elements (list, optional) – List of found elements. To be used with recursive calls. Defaults to [].

Returns:

List of found elements.

Return type:

list

glideinwms.creation.lib.check_python3_expr.main(config_file, enforce_2to3=False, silent=False, refactoring_tool=<lib2to3.refactor.RefactoringTool object>)

Parse the Frontend configuration in config_file and validate Python code.

Parameters:

• config_file (str) – Path to the frontend configuration file.

• enforce_2to3 (bool, optional) – Treats 2to3 suggestions as errors. Defaults to False.

• silent (bool, optional)

• refactoring_tool (RefactoringTool) – Used to by 2to3 to evaluate the expression

Returns:

True if the file is valid and False otherwise. list: List of results for every element evaluated

Return type:

bool

glideinwms.creation.lib.check_python3_expr.match_attrs_to_tuples(match_attrs)

Converts a match_attrs element to a list of tuples.

Parameters:

match_attrs (Element) – match_attrs element.

Returns:

List of tuples.

Return type:

list

glideinwms.creation.lib.cvWConsts module

glideinwms.creation.lib.cvWConsts.get_group_log_dir(log_dir, group_name)

glideinwms.creation.lib.cvWConsts.get_group_monitor_dir(monitor_dir, group_name)

glideinwms.creation.lib.cvWConsts.get_group_name_from_group_monitor_dir(group_monitor_dir)

glideinwms.creation.lib.cvWConsts.get_group_name_from_group_stage_dir(group_stage_dir)

glideinwms.creation.lib.cvWConsts.get_group_name_from_group_work_dir(group_work_dir)

glideinwms.creation.lib.cvWConsts.get_group_stage_dir(stage_dir, group_name)

glideinwms.creation.lib.cvWConsts.get_group_work_dir(work_dir, group_name)

glideinwms.creation.lib.cvWCreate module

glideinwms.creation.lib.cvWCreate.create_client_condor_config(config_fname, mapfile_fname, collector_nodes, classad_proxy)

glideinwms.creation.lib.cvWCreate.create_client_mapfile(mapfile_fname, my_DN, factory_DNs, schedd_DNs, collector_DNs, pilot_DNs=[])

Write a HTCondor map file and add all the provided DNs and map them to the corresponding condor user

Used to create a frontend-specific mapfile used by the tools

Parameters:

• mapfile_fname (str) – path to the map file

• my_DN (list) – list of DNs corresponding to the Frontend (mapped to me)

• factory_DNs (list) – list of DNs corresponding to the Factory (mapped to factory)

• schedd_DNs (list) – list of DNs corresponding to the User schedds (mapped to schedd)

• collector_DNs (list) – list of DNs corresponding to the User collector/s (mapped to collector)

• pilot_DNs (list) – list of DNs corresponding to the pilots (mapped to pilot)

glideinwms.creation.lib.cvWCreate.create_initd_startup(startup_fname, frontend_dir, glideinWMS_dir, cfg_name, rpm_install='')

Creates the frontend startup file and changes the permissions. Can overwrite an existing file.

glideinwms.creation.lib.cvWCreate.filter_unwanted_config_attrs(attrs, mlcd)

Places ‘#’ in front of unwanted condor configuration settings prior to printing it all to the frontend.condor_config file

Args:

attrs: list of strings, output from condor_config_val -dump mlcd: multi line config dict

given multi line input in a condor config file like so: CONDOR_SETTING_1 @=xx

REQUIREMENTS regexp(“^docker://[^/]+$”, SingularityImage) COPY SingularityImage orig_SingularityImage EVALSET SingularityImage replace(“^docker://(.+)”, SingularityImage, “docker://docker.io/library/1”)

@xx

condor_config_val -dump munges it into something indigestable in a config file:

CONDOR_SETTING_1 = REQUIREMENTS regexp(“^docker://[^/]+$”, SingularityImage) COPY SingularityImage orig_SingularityImage EVALSET SingularityImage replace(“^docker://(.+)”, SingularityImage, “docker://docker.io/library/1”)

an mlcd entry will be

mlcd[‘CONDOR_SETTING_1’] = “@=xx

REQUIREMENTS regexp(“^docker://[^/]+$”, SingularityImage)……. @xx”

the (incorrect) settings from above going through condor_config_val -dump will be filtered out of attrs, and replaced with correct formatting from contents of mlcd

Returns:

attrs: list of strings reformatted as valid condor config for frontend

glideinwms.creation.lib.cvWCreate.find_multilines(config_text)

Parses condor config file looking for multiline entries :param config_text: string, contents of a condor configuration file

Returns:

dictionary. keys are first line of multi line config

values are the rest of the multi line config keeping original formatting

see parse_configs_for_multis() below for example muli dict

Return type:

multi

glideinwms.creation.lib.cvWCreate.get_template(template_name, glideinWMS_dir)

glideinwms.creation.lib.cvWCreate.parse_configs_for_multis(conf_list)

parse list of condor config files searching for multi line configurations Args:

conf_list: string, output of condor_config_val -config

Returns:

multi: dictionary. keys are first line of multi line config

values are the rest of the multi line config keeping original formatting

example: this paragraph in a condor_configuration :

JOB_ROUTER_CREATE_IDTOKEN_atlas @=end

sub = “Atlasfetime = 900” lifetime = 900 scope = “ADVERTISE_STARTD, ADVERTISE_MASTER, READ” dir = “$(LOCAL_DIR)/jrtokens” filename = “ce_atlas.idtoken” owner = “atlas”

@end

would generate a multi entry like this:

multi[“JOB_ROUTER_CREATE_IDTOKEN_atlas”] =

‘@=end

sub = “Atlas” lifetime = 900

….. @end

‘

these entries will be rendered into the frontend.condor_config with proper spacing and line returns unlike how they would be rendered by condor_config_val –dump

KNOWN PROBLEM: if condor config has two multi-line configs with same name and different lines generated config file may be incorrect. The condor config is probably incorrect as well :)

glideinwms.creation.lib.cvWDictFile module

class glideinwms.creation.lib.cvWDictFile.ParamsDictFile(dir, fname, sort_keys=False, order_matters=False, fname_idx=None)

Bases: DictFile

add(key, val, allow_overwrite=0)

Add a key, value pair to the dictionary (self) if not already there

Changes the content of the dictionary and set self.changed to True if the value was added.

Parameters:

• key (str) – dictionary key

• val – Any value stored in the dictionary, usually a tuple (e.g. attributes and value)

• allow_overwrite (bool) – Allow to overwrite an exising value if True (default: False)

Raises:

DictFileError – when trying to modify a read only DictFile or overriding an existing item when prohibited or using an incompatible value

add_extended(key, is_expression, val, allow_overwrite=0)

file_header(want_comments)

Return the file header line (file name) as comment, None is want_comments is False

Parameters:

want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

The file header, a comment containing the file name

Return type:

str

format_val(key, want_comments)

Return a string with the formatted (space+tab separated) key, value

Parameters:

• key – key of the key, value pair to format

• want_comments (bool) – If True print also comment lines (lines w/o values)

Returns:

formatted key, value

Return type:

str

get_true_val(key)

get_val_type(key)

parse_val(line)

Parse a line and add it to the dictionary

Changes the dictionary and self.changed via self.add()

Parameters:

line (str) – splitting w/ the default separator should yeld the key and the value

Raises:

DictFileError, from self.add() –

class glideinwms.creation.lib.cvWDictFile.frontendDicts(work_dir, stage_dir, group_list=[], workdir_name='submit', simple_work_dir=False, log_dir=None)

Bases: FileDicts

get_sub_name_from_sub_stage_dir(sign_key)

This must be redefined by the child and return the sub (e.g. Entry) name

Parameters:

stage_dir (str, Path) – sub item stage directory

Returns:

sub item name (e.g. Entry name)

Return type:

str

new_MainDicts()

This should be redefined by the child and return a child of FileMainDicts

Returns:

return a main dictionary of the same type (child of FileMainDicts)

Return type:

FileMainDicts

new_SubDicts(sub_name)

This should be redefined by the child and return a child of FileSubDicts

Parameters:

sub_name (str) – sub dictionary name

Returns:

return a sub dictionary of the same type (child of FileSubDicts)

Return type:

FileSubDicts

class glideinwms.creation.lib.cvWDictFile.frontendGroupDicts(base_work_dir, base_stage_dir, sub_name, summary_signature, workdir_name, simple_work_dir=False, base_log_dir=None)

Bases: FileSubDicts

get_sub_dicts()

get_sub_log_dir(base_dir)

get_sub_stage_dir(base_dir)

get_sub_work_dir(base_dir)

load()

reuse(other)

reuse_nocheck(other)

save(set_readonly=True)

save_final(set_readonly=True)

class glideinwms.creation.lib.cvWDictFile.frontendMainDicts(work_dir, stage_dir, workdir_name, simple_work_dir=False, assume_groups=True, log_dir=None)

Bases: FileMainDicts

get_daemon_log_dir(base_dir)

get_main_dicts()

Interface

Returns:

keys are standard depending on the dictionary

Return type:

dict of DictFile

load()

reuse(other)

save(set_readonly=True)

glideinwms.creation.lib.cvWDictFile.get_common_dicts(work_dir, stage_dir, simple_work_dir)

glideinwms.creation.lib.cvWDictFile.get_group_dicts(group_work_dir, group_stage_dir, group_name, simple_work_dir)

glideinwms.creation.lib.cvWDictFile.get_main_dicts(work_dir, stage_dir, simple_work_dir, assume_groups)

glideinwms.creation.lib.cvWDictFile.load_common_dicts(dicts, description_el)

glideinwms.creation.lib.cvWDictFile.load_group_dicts(group_dicts, group_name, summary_signature)

glideinwms.creation.lib.cvWDictFile.load_main_dicts(main_dicts)

glideinwms.creation.lib.cvWDictFile.refresh_description(dicts)

glideinwms.creation.lib.cvWDictFile.refresh_file_list(dicts, is_main, files_set_readonly=True, files_reset_changed=True)

glideinwms.creation.lib.cvWDictFile.refresh_signature(dicts)

glideinwms.creation.lib.cvWDictFile.reuse_common_dicts(dicts, other_dicts, is_main, all_reused)

glideinwms.creation.lib.cvWDictFile.reuse_file_dict(dicts, other_dicts, key)

glideinwms.creation.lib.cvWDictFile.reuse_group_dicts(group_dicts, other_group_dicts, group_name)

glideinwms.creation.lib.cvWDictFile.reuse_main_dicts(main_dicts, other_main_dicts)

glideinwms.creation.lib.cvWDictFile.reuse_simple_dict(dicts, other_dicts, key, compare_keys=None)

glideinwms.creation.lib.cvWDictFile.save_common_dicts(dicts, is_main, set_readonly=True)

glideinwms.creation.lib.cvWDictFile.save_group_dicts(group_dicts, group_name, summary_signature, set_readonly=True)

glideinwms.creation.lib.cvWDictFile.save_main_dicts(main_dicts, set_readonly=True)

glideinwms.creation.lib.cvWParamDict module

Frontend creation module Classes and functions needed to handle dictionary files created out of the parameter object

glideinwms.creation.lib.cvWParamDict.add_attr_unparsed(attr_name, params, dicts, description)

glideinwms.creation.lib.cvWParamDict.add_attr_unparsed_real(attr_name, params, dicts)

glideinwms.creation.lib.cvWParamDict.apply_group_singularity_policy(descript_dict, sub_params, params)

glideinwms.creation.lib.cvWParamDict.apply_multicore_policy(descript_dict)

glideinwms.creation.lib.cvWParamDict.calc_glidein_ccbs(collectors)

Return a string usable for GLIDEIN_CCB

Parameters:

collectors (list) – list of CCB collectors elements (dict)

Returns:

string usable for the GLIDEIN_CCB attribute

Return type:

str

glideinwms.creation.lib.cvWParamDict.calc_glidein_collectors(collectors)

Return a string usable for GLIDEIN_Collector

Parameters:

collectors (list) – list of collectors elements (dict)

Returns:

string usable for the GLIDEIN_Collector attribute

Return type:

str

glideinwms.creation.lib.cvWParamDict.derive_and_validate_match(group_name, match_expr_pair, factory_attr_list_pair, job_attr_list_pair, attr_dict_pair, policy_files_pair)

Validate match strings, by first concatenating and then evaluating them Since the eval will likely use the external dictionaries,

create a mock version of them, just making sure the types are correct

The complete list of attributes is created by merging main and group dictionaries

Parameters:

• group_name (str) – name of the group (frontend for the global attributes only)

• match_expr_pair (tuple) – Pair of (main,group) match strings to validate

• factory_attr_list_pair (tuple) – Pair of (main,group) descriptions of the queried factory attributes

• job_attr_list_pair (tuple) – Pair of (main,group) descriptions of the queried user job attributes

• attr_dict_pair (tuple) – Pair of (main,group) descriptions of the frontend attributes

• policy_modules_pair (tuple) – Pair of (main,group) descriptions of the frontend attributes

Returns:

class glideinwms.creation.lib.cvWParamDict.frontendDicts(params, sub_list=None)

Bases: frontendDicts

local_populate(params)

new_MainDicts()

This should be redefined by the child and return a child of FileMainDicts

Returns:

return a main dictionary of the same type (child of FileMainDicts)

Return type:

FileMainDicts

new_SubDicts(sub_name)

This should be redefined by the child and return a child of FileSubDicts

Parameters:

sub_name (str) – sub dictionary name

Returns:

return a sub dictionary of the same type (child of FileSubDicts)

Return type:

FileSubDicts

populate(params=None)

reuse(other)

Populate the dictionary (re)using as much as possible of other. I.e. all the items with the same key.

Parameters:

other (FileDicts) – File dictionary to reuse. Must be of the same class as self

Raises:

RuntimeError – if the file dict is incompatible, i.e. the work_dir or stage_dir are different

class glideinwms.creation.lib.cvWParamDict.frontendGroupDicts(params, sub_name, summary_signature, workdir_name)

Bases: frontendGroupDicts

populate(promote_dicts, main_dicts, params=None)

reuse(other)

Reuse as much of the other as possible other must be of the same class

@type other: frontendGroupDicts @param other: Object to reuse

save(set_readonly=True)

save_client_security()

class glideinwms.creation.lib.cvWParamDict.frontendMainDicts(params, workdir_name)

Bases: frontendMainDicts

find_parent_dir(search_path, name)

Given a search path, determine if the given file exists somewhere in the path. Returns: if found. returns the parent directory

if not found, raises an Exception

populate(params=None)

Populate the main dictionary. Return a dictionary of attributes that must go into the group section

Parameters:

params

Returns:

dictionary of attributes that must go into the group section

Return type:

dict

reuse(other)

Reuse as much of the other as possible other must be of the same class

@type other: frontendMainDicts @param other: Object to reuse

save(set_readonly=True)

save_client_security()

save_monitor()

glideinwms.creation.lib.cvWParamDict.get_pool_list(credential)

glideinwms.creation.lib.cvWParamDict.match_attrs_to_array(match_attrs)

glideinwms.creation.lib.cvWParamDict.populate_common_attrs(dicts)

glideinwms.creation.lib.cvWParamDict.populate_common_descript(descript_dict, params)

Populate info common for both frontend (global) and group in the descript dict. descript_dict will be modified in this function

Parameters:

• descript_dict (cWDictFile.StrDictFile) – description dictionary, modified in this function (side effect)

• params – params or sub_params from the config file

Raises:

RuntimeError when no schedd is known to DNS (or via invoked validation functions) –

glideinwms.creation.lib.cvWParamDict.populate_frontend_descript(work_dir, frontend_dict, active_sub_list, params)

glideinwms.creation.lib.cvWParamDict.populate_gridmap(params, gridmap_dict)

glideinwms.creation.lib.cvWParamDict.populate_group_descript(work_dir, group_descript_dict, sub_name, sub_params)

glideinwms.creation.lib.cvWParamDict.populate_group_security(client_security, params, sub_params, group_name)

Populate the DNs in client_security (factory_DNs, schedd_DNs, pilot_DNs)

There is no return. Only via side effects

Parameters:

• client_security (dict) – Frontend security info

• params – parameters form the configuration

• sub_params

• group_name (str) – group name

glideinwms.creation.lib.cvWParamDict.populate_main_security(client_security, params)

glideinwms.creation.lib.cvWParamDict.translate_match_attrs(loc_str, match_attrs_name, match_attrs)

Translate the passed factory/job match_attrs to a format useful for match validation step

Parameters:

• loc_str

• match_attrs_name

• match_attrs

Returns:

glideinwms.creation.lib.cvWParamDict.validate_attribute(attr_name, attr_val)

Check the attribute value is valid. Otherwise throw RuntimeError

glideinwms.creation.lib.cvWParamDict.validate_credential_type(cred_type)

glideinwms.creation.lib.cvWParamDict.validate_match(loc_str, match_str, factory_attrs, job_attrs, attr_dict, policy_modules)

Validate match_expr, factory_match_attrs, job_match_attrs, <attrs> and their equivalents in policy_modules, by actually evaluating the match_expr string. Since it will likely use the external dictionaries,

create a mock version of them, just making sure the types are correct

Parameters:

• loc_str (str) – Section to be validated. i.e. ‘frontend’ or ‘group x’

• match_str (str) – match_expr to be applied to this section

• factory_attrs (dict) – factory_match_attrs for this section

• job_attrs (dict) – job_match_attrs for this section

• attr_dict (dict) – attrs for this section

• policy_modules (list) – policy modules

glideinwms.creation.lib.cvWParamDict.validate_singularity(descript_dict, sub_params, params, name)

If Singularity is enabled in a group, there should be at least one user wrapper for that group

@param descript_dict: dictionaries with user files @param sub_params: attributes in the group section of the XML file @param params: attributes in the general section of the XML file @param name: group name @return:

glideinwms.creation.lib.cvWParams module

This module contains the create_frontend params class

class glideinwms.creation.lib.cvWParams.VOFrontendParams(usage_prefix, src_dir, argv)

Bases: CommonParams

buildDir(frontendVersioning, basedir)

derive()

extract_attr_val(attr_obj)

get_subparams_class()

get_top_element()

get_xml_format()

Return xml formatting for the config

init_defaults()

update_match_attrs()

validate_names()

Validate frontend, group name and attr name

class glideinwms.creation.lib.cvWParams.VOFrontendSubParams(data)

Bases: CommonSubParams

_abc_impl = <_abc._abc_data object>

extract_attr_val(attr_obj)

glideinwms.creation.lib.cvWParams.extract_attr_val(attr_obj)

glideinwms.creation.lib.factoryXmlConfig module

class glideinwms.creation.lib.factoryXmlConfig.CondTarElement(tag, *args, **kwargs)

Bases: DictElement

_abc_impl = <_abc._abc_data object>

validate()

class glideinwms.creation.lib.factoryXmlConfig.Config(tag, *args, **kwargs)

Bases: DictElement

_abc_impl = <_abc._abc_data object>

check_recoverable_exitcodes()

get_client_log_dirs()

get_client_proxy_dirs()

get_entries()

get_log_dir()

get_monitor_dir()

get_stage_dir()

get_submit_dir()

get_web_url()

set_client_log_dirs()

set_client_proxy_dirs()

set_log_dir()

set_monitor_dir()

set_num_factories()

set_stage_dir()

set_submit_dir()

set_web_url()

validate()

class glideinwms.creation.lib.factoryXmlConfig.CvmfsexecDistroElement(tag, *args, **kwargs)

Bases: DictElement

_abc_impl = <_abc._abc_data object>

getPlatforms()

getSources()

setPlatforms()

validate()

class glideinwms.creation.lib.factoryXmlConfig.EntryElement(tag, *args, **kwargs)

Bases: DictElement

_abc_impl = <_abc._abc_data object>

getName()

validate()

validate_sub_elements()

class glideinwms.creation.lib.factoryXmlConfig.EntrySetElement(tag, *args, **kwargs)

Bases: EntryElement

_abc_impl = <_abc._abc_data object>

getName()

The name for entry sets is actaully called alias

get_subentries()

select(entry)

validate()

class glideinwms.creation.lib.factoryXmlConfig.FactAttrElement(tag, *args, **kwargs)

Bases: AttrElement

_abc_impl = <_abc._abc_data object>

check_overwrite_soundness()

If the attribute is defined in the global attributes section then check that the “const” key is True/False for both

validate()

class glideinwms.creation.lib.factoryXmlConfig.FactFileElement(tag, *args, **kwargs)

Bases: FileElement

_abc_impl = <_abc._abc_data object>

validate()

class glideinwms.creation.lib.factoryXmlConfig.FrontendElement(tag, *args, **kwargs)

Bases: DictElement

_abc_impl = <_abc._abc_data object>

validate()

glideinwms.creation.lib.factoryXmlConfig._parse(file, default=False)

The root element (glidein) is registered as type Config

Parameters:

• file – file path

• default – if True return a default configuration (no file)

Returns:

the document root

Return type:

Config

glideinwms.creation.lib.factoryXmlConfig.parse(file)

parse a config file

The root element (glidein) is registered as type Config

Parameters:

file – file path

Returns:

the document root

Return type:

Config

glideinwms.creation.lib.matchPolicy module

This module contains the Match Policy related class

class glideinwms.creation.lib.matchPolicy.MatchPolicy(file, search_path=[])

Bases: object

loadMatchAttrs()

If given match_attr i.e. factory_match_attr or job_match_attr exits load it from the pyObject

policyFileToPyModuleName()

exception glideinwms.creation.lib.matchPolicy.MatchPolicyContentError(file, attr, expected_type, actual_type)

Bases: Exception

exception glideinwms.creation.lib.matchPolicy.MatchPolicyLoadError(file='', search_path=[])

Bases: Exception

glideinwms.creation.lib.xmlConfig module

XML configuration handler Currently used only in the Factory configuration, factoryXmlConfig, parsing glideinXML.xml and imported elements

class glideinwms.creation.lib.xmlConfig.AttrElement(tag, *args, **kwargs)

Bases: DictElement

_abc_impl = <_abc._abc_data object>

get_val()

validate()

class glideinwms.creation.lib.xmlConfig.DictElement(tag, *args, **kwargs)

Bases: Element, MutableMapping

_abc_impl = <_abc._abc_data object>

add_child(child)

check_boolean(flag)

check_missing(attr)

clear_lists()

err_str(str)

get_child(tag)

get_child_list(tag)

has_child(tag)

merge(other)

merge_default_attrs(default)

merge_defaults(default)

class glideinwms.creation.lib.xmlConfig.Element(tag, file='default', line_no=None, parent=None)

Bases: object

add_child(child)

clear_lists()

get_config_node()

Get the node containing the whole configuration

merge(other)

merge_defaults(default)

validate()

class glideinwms.creation.lib.xmlConfig.FileElement(tag, *args, **kwargs)

Bases: DictElement

_abc_impl = <_abc._abc_data object>

validate()

class glideinwms.creation.lib.xmlConfig.Handler(file=None)

Bases: ContentHandler

endElement(name)

Signals the end of an element in non-namespace mode.

The name parameter contains the name of the element type, just as with the startElement event.

startElement(name, attrs)

Signals the start of an element in non-namespace mode.

The name parameter contains the raw XML 1.0 name of the element type as a string and the attrs parameter holds an instance of the Attributes class containing the attributes of the element.

class glideinwms.creation.lib.xmlConfig.ListElement(tag, *args, **kwargs)

Bases: Element

add_child(child)

check_sort_key()

clear_lists()

get_children()

merge(other)

merge_defaults(default)

glideinwms.creation.lib.xmlConfig.register_list_elements(tag_list)

glideinwms.creation.lib.xmlConfig.register_root(tag)

glideinwms.creation.lib.xmlConfig.register_tag_classes(map_dict)

glideinwms.creation.lib.xslt module

glideinwms.creation.lib.xslt.xslt_xml(old_xmlfile, xslt_plugin_dir)

Take an XML file, transform it via any XSLT in the xslt_plugin_dir, and return the output.

Parameters:

• old_xmlfile (str) – Path of the XML file

• xslt_plugin_dir

Returns:

Module contents