glideinwms.lib package

Submodules

glideinwms.lib.classadSupport module

class glideinwms.lib.classadSupport.Classad(adType, advertiseCmd, invalidateCmd)

Bases: object

Base class describing a classad.

adParams

try:

self.adParams

except:

self.adParams = {}

update(params_dict, prefix='')

Update or Add ClassAd attributes

Parameters:

• params_dict – new attributes

• prefix – prefix used for the attribute names (Default: “”)

writeToFile(filename, append=True)

Write a ClassAd to file, adding a blank line if in append mode to separate the ClassAd

There can be no empty line at the beginning of the file: https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5147

Parameters:

• filename – file to write to

• append – write mode if False, append if True (Default)

class glideinwms.lib.classadSupport.ClassadAdvertiser(pool=None, multi_support=False, tcp_support=False)

Bases: object

Base Class to handle the advertisement of classads to condor pools. It contains a dictionary of classads keyed by the classad name and functions to do advertisement and invalidation of classads

addClassad(name, ad_obj)

Adds the classad to the classad dictionary

@type name: string @param name: Name of the classad @type ad_obj: ClassAd @param ad_obj: Actual classad object

advertiseAllClassads()

Advertise all the known classads to the pool

advertiseClassad(ad)

Advertise the classad to the pool

@type ad: string @param ad: Name of the classad

advertiseClassads(ads=None)

Advertise multiple classads to the pool

@type ads: list @param ads: classad names to advertise

classadToFile(ad)

Write classad to the file and return the filename

@type ad: string @param ad: Name of the classad

@rtype: string @return: Name of the file

classadsToFile(ads)

Write multiple classads to a file and return the filename. Use only when multi advertise is supported by condor.

@type ads: list @param ads: Classad names

@rtype: string @return: Filename containing all the classads to advertise

doAdvertise(fname)

Do the actual advertisement of classad(s) in the file

@type fname: string @param fname: File name containing classad(s)

getAllClassads()

Return all the known classads

@rtype: string @return: All the known classads delimited by empty line

getUniqClassadFilename()

Return a uniq file name for advertising/invalidating classads

@rtype: string @return: Filename

invalidateAllClassads()

Invalidate all the known classads

invalidateClassad(ad)

Invalidate the classad from the pool

@type type: string @param type: Name of the classad

invalidateConstrainedClassads(constraint)

Invalidate classads from the pool matching the given constraints

@type type: string @param type: Condor constraints for filtering the classads

glideinwms.lib.classadSupport.exe_condor_advertise(fname, command, pool, is_multi=False, use_tcp=False)

Wrapper to execute condorAdvertise from the condorManager

glideinwms.lib.classadSupport.generate_classad_filename(prefix='gwms_classad')

Return a uniq file name for advertising/invalidating classads

@type prefix: string @param prefix: Prefix to be used for the filename

@rtype: string @return: Filename

glideinwms.lib.cleanupSupport module

class glideinwms.lib.cleanupSupport.Cleanup

Bases: object

add_cleaner(cleaner)

cleanup()

start_background_cleanup()

wait_for_cleanup()

class glideinwms.lib.cleanupSupport.CredCleanup

Bases: Cleanup

Cleans up old credential files.

cleanup(in_use_proxies)

class glideinwms.lib.cleanupSupport.DirCleanup(dirname, fname_expression, maxlife, should_log=True, should_log_warnings=True)

Bases: object

cleanup()

delete_file(fpath)

get_files_wstats()

class glideinwms.lib.cleanupSupport.DirCleanupCredentials(dirname, fname_expression, maxlife)

Bases: DirCleanup

Used to cleanup old credential files saved to disk by the factory for glidein submission (based on ctime).

cleanup(in_use_creds)

class glideinwms.lib.cleanupSupport.DirCleanupWSpace(dirname, fname_expression, maxlife, minlife, maxspace, should_log=True, should_log_warnings=True)

Bases: DirCleanup

cleanup()

glideinwms.lib.condorExe module

exception glideinwms.lib.condorExe.CondorExeError(err_str)

Bases: RuntimeError

Base class for condorExe module errors

exception glideinwms.lib.condorExe.ExeError(err_str)

Bases: CondorExeError

exception glideinwms.lib.condorExe.UnconfigError(err_str)

Bases: CondorExeError

glideinwms.lib.condorExe.exe_cmd(condor_exe, args, stdin_data=None, env={})

Execute an arbitrary condor command and return its output as a list of lines Fails if stderr is not empty

Parameters:

• condor_exe (str) – condor_exe uses a relative path to $CONDOR_BIN

• args (str) – arguments for the command

• stdin_data (str) – Data that will be fed to the command via stdin

• env (dict) – Environment to be set before execution

Returns:

Lines of stdout from the command

Raises:

• UnconfigError –

• ExeError –

glideinwms.lib.condorExe.exe_cmd_sbin(condor_exe, args, stdin_data=None, env={})

glideinwms.lib.condorExe.generate_bash_script(cmd, environment)

Print to a string a shell script setting the environment in ‘environment’ and running ‘cmd’ If ‘cmd’ last argument is a file it will be printed as well in the string

Parameters:

• cmd (str) – command string

• environment (dict) – environment as a dictionary

Returns:

multi-line string with environment, command and eventually the input file

Return type:

str

glideinwms.lib.condorExe.iexe_cmd(cmd, stdin_data=None, child_env=None)

Fork a process and execute cmd - rewritten to use select to avoid filling up stderr and stdout queues.

Parameters:

• cmd (str) – Sting containing the entire command including all arguments

• stdin_data (str) – Data that will be fed to the command via stdin

• child_env (dict) – Environment to be set before execution

Returns:

Lines of stdout from the command

Return type:

list of str

Raises:

ExeError –

glideinwms.lib.condorExe.init()

Set both Set condor_bin_path and condor_sbin_path

glideinwms.lib.condorExe.init1()

Set condor_bin_path

glideinwms.lib.condorExe.init2()

Set condor_sbin_path

glideinwms.lib.condorExe.set_path(new_condor_bin_path, new_condor_sbin_path=None)

Set path to condor binaries, if needed

Works changing the global variables condor_bin_path and condor_sbin_path

Parameters:

• new_condor_bin_path (str) – directory where the HTCondor binaries are located

• new_condor_sbin_path (str) – directory where the HTCondor system binaries are located

glideinwms.lib.condorLogParser module

class glideinwms.lib.condorLogParser.cacheDirClass(logClass, dirname, log_prefix, log_suffix='.log', cache_ext='.cifpk', inactive_files=None, inactive_timeout=86400, cache_dir=None, wrapperClass=None, username=None)

Bases: object

This is the base class for all the directory log Parser classes. It parses some/all log files in a directory. It should generally not be called directly. Rather, call one of the inherited classes.

cdInit(logClass, dirname, log_prefix, log_suffix='.log', cache_ext='.cifpk', inactive_files=None, inactive_timeout=86400, cache_dir=None, wrapperClass=None, username=None)

@param logClass: this is an actual class, not an object @param inactive_files: if None, will be reloaded from cache @param inactive_timeout: how much time must elapse before a file can be

declared inactive

@param cache_dir: If None, use dirname for the cache directory.

diff(other)

Diff self data with other info

This is a virtual function that just calls the class diff() function.

getFileList(active_only)

Lists the directory and returns files that match the prefix/suffix extensions and are active (modified within inactivity_timeout)

@return: a list of log files

has_changed()

Checks all the files in the list to see if any have changed.

@return: True/False

load(active_only=True)

For each file in the filelist, call the appropriate load() function for that file. Merge all the data from all the files into temporary array mydata then set it to self.data. It will save the list of inactive_files it finds in a cache for quick access.

This function should set self.data.

class glideinwms.lib.condorLogParser.cachedLogClass

Bases: object

This is the base class for most Log Parsers in lib/condorLogParser and factory/glideFactoryLogParser. (I{virtual, do not use})

The Constructor for inherited classes needs to define logname and cachename (possibly by using clInit) as well as the methods

loadFromLog, merge and isActive.

init method to be used by real constructors

clInit(logname, cache_dir, cache_ext)

has_changed()

Compare to cache, and tell if the log file has changed since last checked

load()

Load data from most recent file. Update the cache if needed. If the file has not changed, use the cache instead (typically named something like filename.ftstpk) in a pickle format. If file is newer, uses inherited class’s loadFromLog method. Then, save in pickle cache.

loadCache()

loadFromLog()

saveCache()

glideinwms.lib.condorLogParser.countAndInterpretRawStatuses(jobs_raw)

Given a dictionary of job statuses (like the one got from parseSubmitLogFastRaw) will return a dictionary of status counts

for example: {‘Completed’: 30170, ‘Removed’: 148, ‘Running’: 5013}

@param jobs_raw: Dictionary of job statuses @return: Dictionary of status counts

glideinwms.lib.condorLogParser.countStatuses(jobs)

Given a dictionary of job statuses (like the one got from parseSubmitLogFastRaw) will return a dictionary of sstatus counts

for example: {‘009’: 25170, ‘012’: 418, ‘005’: 1503}

glideinwms.lib.condorLogParser.diffTimes(start_time, end_time, year)

Get two condor time strings and compute the difference The start_time must be before the end_time

glideinwms.lib.condorLogParser.diffTimeswWrap(start_time, end_time, year, wrap_time)

Get two condor time strings and compute the difference The start_time must be before the end_time

class glideinwms.lib.condorLogParser.dirCompleted(dirname, log_prefix, log_suffix='.log', cache_ext='.cifpk', inactive_files=None, inactive_timeout=86400, cache_dir=None)

Bases: cacheDirClass

This class will keep track of:

• counts of statuses (Wait, Idle, Running, Held, Completed, Removed)

• list of completed jobs

This data is available in self.data dictionary for example self.data={‘completed_jobs’:[‘123.002’,’555.001’], ‘counts’:{‘Idle’: 1145, ‘Completed’: 2}}

class glideinwms.lib.condorLogParser.dirCounts(dirname, log_prefix, log_suffix='.log', cache_ext='.cifpk', inactive_files=None, inactive_timeout=86400, cache_dir=None)

Bases: cacheDirClass

This class will keep track of counts of statuses (Wait, Idle, Running, Held, Completed, Removed) These data is available in self.data dictionary for example self.data={‘Idle’: 1145, ‘Completed’: 2}

class glideinwms.lib.condorLogParser.dirSummary(dirname, log_prefix, log_suffix='.log', cache_ext='.cifpk', inactive_files=None, inactive_timeout=86400, cache_dir=None)

Bases: cacheDirClass

This class will keep track of: jobs in various of statuses (Wait, Idle, Running, Held, Completed, Removed) This data is available in self.data dictionary For example, self.data={‘Idle’:[‘123.003’,’123.004’],’Running’:[‘123.001’,’123.002’]}

class glideinwms.lib.condorLogParser.dirSummaryTimings(dirname, log_prefix, log_suffix='.log', cache_ext='.cifpk', inactive_files=None, inactive_timeout=86400, cache_dir=None)

Bases: cacheDirClass

This class will keep track of: jobs in various of statuses (Wait, Idle, Running, Held, Completed, Removed) This data is available in self.data dictionary For example self.data={‘Idle’:[(‘123.003’,’09/28 01:38:53’, ‘09/28 01:42:23’, ‘09/28 08:06:33’),(‘123.004’,’09/28 02:38:53’, ‘09/28 02:42:23’, ‘09/28 09:06:33’)], ‘Running’:[(‘123.001’,’09/28 01:32:53’, ‘09/28 01:43:23’, ‘09/28 08:07:33’),(‘123.002’,’09/28 02:38:53’, ‘09/28 03:42:23’, ‘09/28 06:06:33’)]}

glideinwms.lib.condorLogParser.get_new_status(old_status, new_status)

Given a job with an old and new status, will return the appropriate status to register to the job.

@param old_status: Globus job status @param new_status: Globus job status @return: Appropriate status for the job

glideinwms.lib.condorLogParser.interpretStatus(status, default_status='Idle')

Transform a integer globus status to either Wait, Idle, Running, Held, Completed or Removed

glideinwms.lib.condorLogParser.listAndInterpretRawStatuses(jobs_raw, invert_function)

Given a dictionary of job statuses (whatever the invert_function recognises) will return a dictionary of jobs in each status (syntax depends on the invert_function)

for example with linvert_funtion==istStatuses: {‘Completed’: [“2.003”,”5.001”], ‘Removed’: [“41.001”], ‘Running’: [“408.003”]}

@param jobs_raw: A dictionary of job statuses @param invert_function: function to turn a job status into “Completed”,”Removed”,”Running”, etc @return: Dictionary of jobs in each category.

glideinwms.lib.condorLogParser.listStatuses(jobs)

Given a dictionary of job statuses (like the one got from parseSubmitLogFastRaw) will return a dictionary of jobs in each status

For example: {‘009’: [“1.003”,”2.001”], ‘012’: [“418.001”], ‘005’: [“1503.001”,”1555.002”]}

@param jobs: Dictionary of job statuses @return: Dictionary of jobs in each status category

glideinwms.lib.condorLogParser.listStatusesTimings(jobs)

Given a dictionary of job statuses + timings (like the one got from parseSubmitLogFastRawTimings) will return a dictionary of jobs +timings in each status

For example: {‘009’: [(“1.003”,’09/28 01:38:53’, ‘’, ‘09/28 01:38:53’),(“2.001”,’09/28 03:38:53’, ‘’, ‘09/28 04:38:53’)], ‘005’: [(“1503.001”, ‘09/28 01:48:52’, ‘09/28 16:11:23’, ‘09/28 20:31:53’),(“1555.002”, ‘09/28 02:48:52’, ‘09/28 18:11:23’, ‘09/28 23:31:53’)]}

@param jobs: Dictionary of job statuses and timings @return: Dictionary of jobs+timings in each status category

glideinwms.lib.condorLogParser.loadCache(fname)

Loads a pickle file from a filename and returns the resulting data.

@param fname: Filename to load @return: data retrieved from file

class glideinwms.lib.condorLogParser.logCompleted(logname, cache_dir)

Bases: cachedLogClass

This class will keep track of:

• counts of statuses (Wait, Idle, Running, Held, Completed, Removed)

• list of completed jobs

This data is available in self.data dictionary

For example self.data= {‘completed_jobs’:[‘123.002’,’555.001’], ‘counts’:{‘Idle’: 1145, ‘Completed’: 2}}

diff(other)

Diff self.data with other info. For use in comparing previous iteration with current iteration

Uses symmetric difference of sets.

isActive()

loadFromLog()

Load information from condor_activity logs Then parse globus statuses. Finally, parse and add counts.

merge(other)

Merge self data with other info @return: merged data, may modify other

class glideinwms.lib.condorLogParser.logCounts(logname, cache_dir)

Bases: cachedLogClass

This class will keep track of counts of statuses (Wait, Idle, Running, Held, Completed, Removed) This data is available in self.data dictionary For example self.data={‘Idle’: 1145, ‘Completed’: 2}

diff(other)

Diff self data with other info @return: diff of counts

isActive()

loadFromLog()

merge(other)

Merge self data with other info @return: merged data, may modify other

class glideinwms.lib.condorLogParser.logSummary(logname, cache_dir)

Bases: cachedLogClass

This class will keep track of: jobs in various of statuses (Wait, Idle, Running, Held, Completed, Removed) This data is available in self.data dictionary for example self.data={‘Idle’:[‘123.003’,’123.004’],’Running’:[‘123.001’,’123.002’]}

diff(other)

diff self data with other info. Used to compare previous iteration with current iteration

Performs symmetric difference on the two sets and creates a dictionary for each status.

@return: data[status][‘Entered’|’Exited’] - list of jobs

isActive()

loadFromLog()

Parse the condor activity log and interpret the globus status code. Stores in self.data

merge(other)

Merge self data with other info @return: merged data, may modify other

class glideinwms.lib.condorLogParser.logSummaryTimings(logname, cache_dir)

Bases: cachedLogClass

This class will keep track of: jobs in various of statuses (Wait, Idle, Running, Held, Completed, Removed) This data is available in self.data dictionary for example self.data={‘Idle’:[‘123.003’,’123.004’],’Running’:[‘123.001’,’123.002’]}

diff(other)

diff self data with other info @return: data[status][‘Entered’|’Exited’] - list of jobs

isActive()

loadFromLog()

merge(other)

merge self data with other info @return: merged data, may modify other

glideinwms.lib.condorLogParser.parseSubmitLogFast(fname)

Reads a Condor submit log, return a dictionary of jobIds each having the last status.

For example {(1583,4)’: 0, (3616,8): 9}

@param fname: filename to parse @return: Dictionary of jobIDs and last status

glideinwms.lib.condorLogParser.parseSubmitLogFastRaw(fname)

Read a condor submit log.

@return: a dictionary of jobStrings each having the last statusString For example {‘1583.004’: ‘000’, ‘3616.008’: ‘009’}

glideinwms.lib.condorLogParser.parseSubmitLogFastRawCallback(fname, callback)

Read a condor submit log for each new event, call a callback

@param fname: Condor submit file to parse @param callname: def callback(new_status_str,timestamp_str,job_str)

glideinwms.lib.condorLogParser.parseSubmitLogFastRawTimings(fname)

Read a condor submit log. Returns a dictionary of jobStrings each having (the last statusString,firstTime,runningTime,lastTime) plus the first and last date in the file

for example {‘9568.001’:(‘000’, ‘09/28 01:38:53’, ‘’, ‘09/28 01:38:53’),’9868.003’:(‘005’, ‘09/28 01:48:52’, ‘09/28 16:11:23’, ‘09/28 20:31:53’)},’09/28 01:38:53’,’09/28 20:31:53’

@return: a dictionary of jobStrings

glideinwms.lib.condorLogParser.parseSubmitLogFastTimings(fname, year=None)

Reads a Condor submit log, return a dictionary of jobIds each having (the last status, seconds in queue, if status==5, seconds running)

For example {(1583,4)’: (0,345,None), (3616,8): (5,7777,4532)}

@param fname: filename to parse @param year: if no year, then use the current one @return: Dictionary of jobIDs

glideinwms.lib.condorLogParser.rawJobId2Nr(str)

Convert the log representation into (ClusterId,ProcessId)

Return (-1,-1) in case of error

glideinwms.lib.condorLogParser.rawTime2cTime(instr, year)

Convert the log representation into ctime

@return: ctime or -1 in case of error

glideinwms.lib.condorLogParser.rawTime2cTimeLastYear(instr)

Convert the log representation into ctime, works only for the past year

@return: ctime or -1 in case of error

glideinwms.lib.condorLogParser.saveCache(fname, data)

Creates a temporary file to store data in, then moves the file into the correct place. Uses pickle to store data.

@param fname: Filename to write to. @param data: data to store in pickle format

glideinwms.lib.condorManager module

glideinwms.lib.condorManager.cached_exe_cmd(cmd, arg_str, schedd_name, pool_name, schedd_lookup_cache)

glideinwms.lib.condorManager.condorAdvertise(classad_fname, command, use_tcp=False, is_multi=False, pool_name=None)

glideinwms.lib.condorManager.condorHold(constraint, schedd_name=None, pool_name=None, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

glideinwms.lib.condorManager.condorHoldOne(cluster_or_uname, schedd_name=None, pool_name=None, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

glideinwms.lib.condorManager.condorRelease(constraint, schedd_name=None, pool_name=None, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

glideinwms.lib.condorManager.condorReleaseOne(cluster_or_uname, schedd_name=None, pool_name=None, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

glideinwms.lib.condorManager.condorRemove(constraint, schedd_name=None, pool_name=None, do_forcex=False, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

glideinwms.lib.condorManager.condorRemoveOne(cluster_or_uname, schedd_name=None, pool_name=None, do_forcex=False, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

glideinwms.lib.condorManager.condorReschedule(schedd_name=None, pool_name=None, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

glideinwms.lib.condorManager.condorSubmitOne(submit_file, schedd_name=None, pool_name=None, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

glideinwms.lib.condorManager.ismulti2str(is_multi)

glideinwms.lib.condorManager.pool2str(pool_name)

glideinwms.lib.condorManager.schedd2str(schedd_name)

glideinwms.lib.condorManager.usetcp2str(use_tcp)

glideinwms.lib.condorMonitor module

class glideinwms.lib.condorMonitor.AbstractQuery

Bases: object

Pure virtual class to have a minimum set of methods defined

fetch(constraint=None, format_list=None)

Fetch the classad attributes specified in the format_list matching the constraint

@rtype: dict @return: Dict containing classad attributes and values

fetchStored(constraint_func=None)

@param constraint_func: A boolean function, with only one argument

(data el). If None, return all the data.

@type constraint_func: Boolean function

@rtype: dict @return: Same as fetch(), but limited to constraint_func(el)==True

load(constraint=None, format_list=None)

Fetch the data and store it in self.stored_data

class glideinwms.lib.condorMonitor.BaseSubQuery(query, subquery_func)

Bases: StoredQuery

Virtual class that implements fetch for StoredQuery()

fetch(constraint=None)

Fetch the classad attributes specified in the format_list matching the constraint

@rtype: dict @return: Dict containing classad attributes and values

load(constraint=None)

Fetch the data and store it in self.stored_data

class glideinwms.lib.condorMonitor.CondorQ(schedd_name=None, pool_name=None, security_obj=None, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

Bases: CondorQuery

Class to implement condor_q. Uses htcondor-python bindings if possible.

fetch(constraint=None, format_list=None)

Return the results obtained using HTCondor commands or python bindings

Parameters:

• constraint (str) – query constraint

• format_list (list) – Classad attr & type. [(attr1, ‘i’), (‘attr2’, ‘s’)]

Returns (dict): Dict containing the query results

fetch_using_bindings(constraint=None, format_list=None)

Fetch the condor_q results using htcondor-python bindings

Parameters:

• constraint (str) – Constraints to be applied to the query

• format_list (list) – Classad attr & type. [(attr1, ‘i’), (‘attr2’, ‘s’)]

Returns (dict): Dict containing the results

class glideinwms.lib.condorMonitor.CondorQEdit(pool_name=None, schedd_name=None)

Bases: object

Fully implemented class that executes condorq_edit commands. Only provides a method to do bulk updates of jobs using transaction-based API and the condor python bindings. Cannot be used without python bindings

executeAll(joblist=None, attributes=None, values=None)

Given equal sized lists of job ids, attributes and values, executes in one large transaction a single qedit for each job.

class glideinwms.lib.condorMonitor.CondorQLite(schedd_name=None, pool_name=None, security_obj=None, schedd_lookup_cache=<glideinwms.lib.condorMonitor.LocalScheddCache object>)

Bases: CondorQuery

fetch(constraint=None, format_list=None)

Return the results obtained using HTCondor commands or python bindings

Parameters:

• constraint (str) – query constraint

• format_list (list) – Classad attr & type. [(attr1, ‘i’), (‘attr2’, ‘s’)]

Returns (dict): Dict containing the query results

class glideinwms.lib.condorMonitor.CondorQuery(exe_name, resource_str, group_attribute, pool_name=None, security_obj=None, env={})

Bases: StoredQuery

Fully implemented class that executes condor commands

fetch(constraint=None, format_list=None)

Return the results obtained using HTCondor commands or python bindings

Parameters:

• constraint (str) – query constraint

• format_list (list) – Classad attr & type. [(attr1, ‘i’), (‘attr2’, ‘s’)]

Returns (dict): Dict containing the query results

fetch_using_bindings(constraint=None, format_list=None)

Fetch the results using htcondor-python bindings

Parameters:

• constraint (str) – Constraints to be applied to the query

• format_list (list) – Classad attr & type. [(attr1, ‘i’), (‘attr2’, ‘s’)]

Returns (dict): Dict containing the results

Raises:

NotImplementedError – the operation is not implemented using bindings

fetch_using_exe(constraint=None, format_list=None)

Return the results obtained from executing the HTCondor query command

Parameters:

• constraint (str) – Constraints to be applied to the query

• format_list (list) – Classad attr & type. [(attr1, ‘i’), (‘attr2’, ‘s’)]

Returns (dict): Dict containing the results

get_requested_encryption()

Get the current encryption settings

Returns: None->None; REQUIRED->True; OPTIONAL->False

get_requested_integrity()

Get the current integrity settings

Returns: None->None; REQUIRED->True; OPTIONAL->False

load(constraint=None, format_list=None)

Fetch the results and cache it in self.stored_data

require_encryption(requested_encryption)

Set client encryption settings to use for condor commands

Parameters:

requested_encryption (str) – HTCondor encryption level

require_integrity(requested_integrity)

Set client integrity settings to use for condor commands

Parameters:

requested_integrity (str) – HTCondor integrity level

class glideinwms.lib.condorMonitor.CondorStatus(subsystem_name=None, pool_name=None, security_obj=None)

Bases: CondorQuery

Class to implement condor_status. Uses htcondor-python bindings if possible.

fetch(constraint=None, format_list=None)

Return the results obtained using HTCondor commands or python bindings

Parameters:

• constraint (str) – query constraint

• format_list (list) – Classad attr & type. [(attr1, ‘i’), (‘attr2’, ‘s’)]

Returns (dict): Dict containing the query results

fetch_using_bindings(constraint=None, format_list=None)

Fetch the results using htcondor-python bindings

class glideinwms.lib.condorMonitor.Group(query, group_key_func, group_data_func)

Bases: BaseSubQuery

Sub Query class with grouping functionality Each element has a value that is the summary of the values in a group

class glideinwms.lib.condorMonitor.LocalScheddCache

Bases: NoneScheddCache

Schedd caching class The schedd can be found either through -name attr or through the local disk lookup. Remember which one to use.

disable()

enable()

getScheddId(schedd_name, pool_name)

Given the schedd name and pool name, get a tuple containing (ScheddId, dict)

@param schedd_name: Name of the schedd @type schedd_name: string

@param pool_name: Name of the pool @type pool_name: string

@return: tuple (schedd string, env)

iGetEnv(schedd_name, pool_name)

class glideinwms.lib.condorMonitor.NestedGroup(query, group_key_func, group_element_func=None)

Bases: BaseSubQuery

Sub Query class with grouping functionality to create nested results Each element is a dictionary with elements reduced from the original elements in the group

class glideinwms.lib.condorMonitor.NoneDiskCache

Bases: object

Dummy class used if a regular DiskCache is not specified

get(objid)

save(objid, obj)

class glideinwms.lib.condorMonitor.NoneScheddCache

Bases: object

Dummy caching class, when you don’t want caching. Used as base class below, too

getScheddId(schedd_name, pool_name)

Given the schedd name and pool name, return a tuple (ScheddId, dict)

@param schedd_name: Name of the schedd @type schedd_name: string

@param pool_name: Name of the pool @type pool_name: string

@return: tuple (schedd string, LOCAL_DIR)

iGetCmdScheddStr(schedd_name)

exception glideinwms.lib.condorMonitor.PBError(err_str)

Bases: RuntimeError

Thrown when there are exceptions using htcondor python bindings

exception glideinwms.lib.condorMonitor.QueryError(err_str)

Bases: RuntimeError

Thrown when there are exceptions using htcondor python bindings or commands

class glideinwms.lib.condorMonitor.StoredQuery

Bases: AbstractQuery

Virtual class that implements fetchStored

fetchStored(constraint_func=None)

@param constraint_func: A boolean function, with only one argument

(data el). If None, return all the data.

@type constraint_func: Boolean function

@rtype: dict @return: Same as fetch(), but limited to constraint_func(el)==True

stored_data = {}

class glideinwms.lib.condorMonitor.SubQuery(query, constraint_func=None)

Bases: BaseSubQuery

Fully usable subquery class, typically used on CondorQ() or CondorStatus()

class glideinwms.lib.condorMonitor.Summarize(query, hash_func=<function Summarize.<lambda>>)

Bases: object

Summarizing classes

count(constraint=None, hash_func=None, flat_hash=False)

countStored(constraint_func=None, hash_func=None, flat_hash=False)

getHash(hash_func)

list(constraint=None, hash_func=None)

listStored(constraint_func=None, hash_func=None)

class glideinwms.lib.condorMonitor.SummarizeMulti(queries, hash_func=<function SummarizeMulti.<lambda>>)

Bases: object

count(constraint=None, hash_func=None)

countStored(constraint_func=None, hash_func=None)

glideinwms.lib.condorMonitor.addDict(base_dict, new_dict)

Recursively add two dictionaries Do it in place, using the first one

Parameters:

• base_dict – first dictionary

• new_dict – dictionary to be added

glideinwms.lib.condorMonitor.applyConstraint(data, constraint_func)

Return a subset of data that satisfies constraint_function If constraint_func is None, return back entire data

Parameters:

• data –

• constraint_func –

Returns:

glideinwms.lib.condorMonitor.bindings_friendly_attrs(format_list)

Convert the format_list into attrs that can be used with python bindings Python bindings should take care of the typing

glideinwms.lib.condorMonitor.bindings_friendly_constraint(constraint)

Convert the constraint to format that can be used with python bindings

glideinwms.lib.condorMonitor.complete_format_list(in_format_list, req_format_els)

glideinwms.lib.condorMonitor.condorq_attrs(q_constraint, attribute_list)

Retrieves a list of a single item from the all the factory queues.

glideinwms.lib.condorMonitor.doGroup(indata, group_key_func, group_data_func)

Group the indata based on the keys that satisfy group_key_func (applied to the value) Return a dict of groups summarized by group_data_func Each group returned by group_data_func must be a dictionary, possibly similar to the original value of the indata elements

Parameters:

• indata –

• group_key_func –

• group_data_func –

Returns:

dict of groups summarized by group_data_func

glideinwms.lib.condorMonitor.doNestedGroup(indata, group_key_func, group_element_func=None)

Group the indata based on the keys that satisfy group_key_func (applied to the value) Return a dict of dictionaries created by group_element_func Each each value of the dictionaries returned by group_element_func must be a dictionary, possibly similar to the original value of the indata elements

If group_element_func is None (not provided), then the dictionaries in the groups are a copy of the original dictionaries in indata

Parameters:

• indata – data to group

• group_key_func – group_by function

• group_element_func – how to handle the data in each group (by default is a copy of the original one)

Returns:

dictionary of dictionaries with grouped indata

Return type:

dict

glideinwms.lib.condorMonitor.fetch2count(data, hash_func)

Nested count of the hash values returned from all the elements in data

Parameters:

• data – data from a fetch()

• hash_func –

  Hashing function One argument: classad dictionary Returns: hash value

  if None, will not be counted if a list, all elements will be used

Returns:

dictionary of hash values

Elements are counts (or more dictionaries if hash returns lists)

Return type:

dict

glideinwms.lib.condorMonitor.fetch2count_flat(data, hash_func)

Count the hash values returned from all the elements in data

Parameters:

• data – data from a fetch()

• hash_func –

  Hashing function One argument: classad dictionary Returns: flat hash value (for hashing functions returning also lists, use fetch2count

  if None, will not be counted

Returns:

a dictionary with a count of the hash values returned

glideinwms.lib.condorMonitor.fetch2list(data, hash_func)

glideinwms.lib.condorMonitor.htcondor_full_reload()

glideinwms.lib.condorMonitor.list2dict(list_data, attr_name)

Convert a list to a dictionary where the keys are tuples with the values of the attributes listed in attr_name

Original description: Convert a list to a dictionary and group the results based on attributes specified by attr_name

This function has a couple of quirks, but is OK because the ways it is used (MM) The way it is used, attr_name is the job cluster, process, which are both present in all jobs from condor_q and unique, or the Name that is always present and unique in condor_status, so the quirks should not cause problems 1. Type checking (of attr_name) probably should use isistance() 2. dict_name is a tuple including elements of attr_name translated to value in list_el

if them or the lowercase is a key in list_el BUT from the value ( dict_data[dict_name] ) only exact match is excluded, not the lowercase version

3. keys (dict_name) may have different cardinality if one or some of the elements is not matching list_el keys

4. if 2 or more list_el have the same dict_name (same valies in attr_list attributes), the newest ones overwrite the older ones without any warning AND the original description mentions … “and group the results” … there is no grouping

5. ‘Undefined’ attributes are not added to the dict_el (dict elements may have different keys)

6. using ‘%s’%a_value != ‘Undefined’ and str(list_el[a]) != ‘Undefined’ for the same. Use twice the better one

Parameters:

• list_data – list of dictionaries to convert

• attr_name – string (1 attribute) or list or tuple (one or more attributes) with the attributes to use as key

Returns:

dictionary of dictionaries

Return type:

dict

glideinwms.lib.condorMonitor.resource_str_to_py_adtype(resource_str)

Given the resource string return equivalent classad type

glideinwms.lib.condorMonitor.set_path(new_condor_bin_path)

glideinwms.lib.condorMonitor.xml2list(xml_data)

glideinwms.lib.condorMonitor.xml2list_char_data(data)

glideinwms.lib.condorMonitor.xml2list_end_element(name)

glideinwms.lib.condorMonitor.xml2list_start_element(name, attrs)

glideinwms.lib.condorSecurity module

class glideinwms.lib.condorSecurity.EnvProtoState(filter=None)

Bases: SecEnvState

class glideinwms.lib.condorSecurity.EnvState(filter)

Bases: object

load()

restore()

class glideinwms.lib.condorSecurity.GSIRequest(x509_proxy=None, allow_fs=True, allow_idtokens=True, proto_requests=None)

Bases: ProtoRequest

enforce_requests()

restore_state()

save_state()

class glideinwms.lib.condorSecurity.ProtoRequest(requests=None)

Bases: SecEnvRequest

get(context, feature)

set(context, feature, value)

class glideinwms.lib.condorSecurity.SecEnvRequest(requests=None)

Bases: object

enforce_requests()

get(context, feature)

has_saved_state()

restore_state()

save_state()

set(context, feature, value)

class glideinwms.lib.condorSecurity.SecEnvState(sec_filter)

Bases: EnvState

glideinwms.lib.condorSecurity.convert_sec_filter(sec_filter)

glideinwms.lib.config_util module

glideinwms.lib.defaults module

Collections of constants that are used throughout the GlideinWMS project

glideinwms.lib.defaults.force_bytes(instr, encoding='utf8')

Forces the output to be bytes, encoding the input if it is a unicode string (str)

AnyStr is str or bytes types

Parameters:

• instr (AnyStr) – string to be converted

• encoding (str) – a valid encoding, utf8, ascii, latin-1

Returns:

instr as bytes string

Return type:

bytes

Raises:

ValueError – if it detects an improper str conversion (b’’ around the string)

glideinwms.lib.disk_cache module

This module helps managing objects that needs to be cached. Objects are cached both in memory and on disk so that the cache can be leveraged by multiple processes.

Each object you want to save needs to have a string id that identifies it. The id is used to locate the object in the memory (a key in a dictionary), and on the disk (the filename).

class glideinwms.lib.disk_cache.DiskCache(cache_dir, cache_duration=3600)

Bases: object

The class that manages the cache. Objects expires after a cache_duration time (defaults to one hour). Objects are pickled into a file. The directory to save those files has to be specified. Methods to save and load an object by its id are provided.

get(objid)

Returns the cached object given its object id objid. Returns None if the object is not in the cache, or if it has expired.

First we check if the object is in the memory dictionary, otherwise we look for its corresponding cache file, and we loads it from there.

Parameters:

objid (str) – the string representing the object id you want to get

Returns:

The cached object, or None if the object does not exist or the cache is expired

get_fname(objid)

Simple auxiliary function that returns the cache filename given a cache object id

save(objid, obj)

Save an object into the cache.

Objects are saved both in memory and into the corresponding cache file (one file for each object id). Objects are saved paired with the timestamp representing the time when it has been saved.

Parameters:

• objid (str) – The id of the object you are saving

• obj – the python object that you want to save

glideinwms.lib.disk_cache.get_lock(name)

Create a “name”.lock file and, using fcnt, lock it (or wait for the lock to be released) before proceeding

N.B. The “name”.lock file is not removed after the lock is released, it is kept to be reused: we only care about the lock status.

Params:

name (str): the name of the file you want to lock. A lockfile name.lock will be created

glideinwms.lib.exprParser module

glideinwms.lib.exprParser.exp_compare(node1, node2)

Compare 2 AST trees to verify if they are the same

Parameters:

• node1 (ast.AST) – AST tree

• node2 (ast.AST) – AST tree

Returns:

True if node1 and node2 are the same expression

Return type:

bool

glideinwms.lib.exprParser.exp_compile(obj)

Convert an ast object into a code object

Parameters:

obj (ast.AST) – AST object to compile

Returns:

code object

glideinwms.lib.exprParser.exp_parse(expression)

Convert an expression string into an ast object

Parameters:

expression (str) – expression string

Returns:

ast tree from the expression, starting from ast.Expression node

Return type:

ast.AST

glideinwms.lib.exprParser.exp_unparse(obj, raise_on_unknown=False)

Convert an ast object back into a string

Parameters:

• obj (ast.AST) – ast object to convert back to string

• raise_on_unknown (bool) –

Returns:

string with the expression

Return type:

str

glideinwms.lib.fork module

exception glideinwms.lib.fork.FetchError(msg)

Bases: ForkError

exception glideinwms.lib.fork.ForkError(msg)

Bases: RuntimeError

Base class for this module’s errors

class glideinwms.lib.fork.ForkManager

Bases: object

add_fork(key, function, *args)

bounded_fork_and_collect(max_forks, log_progress=True, sleep_time=0.01)

fork_and_collect()

fork_and_wait()

exception glideinwms.lib.fork.ForkResultError(nr_errors, good_results, failed=[])

Bases: ForkError

glideinwms.lib.fork.fetch_fork_result(r, pid)

Used with fork clients Can raise: OSError if Bad file descriptor or file already closed or if waitpid syscall returns -1 FetchError if a os.read error was encountered Possible errors from os.read (catched here): - EOFError if the forked process failed an nothing was written to the pipe, if cPickle finds an empty string - IOError failure for an I/O-related reason, e.g., “pipe file not found” or “disk full”. - OSError other system-related error

@type r: pipe @param r: Input pipe

@type pid: int @param pid: pid of the child

@rtype: Object @return: Unpickled object

glideinwms.lib.fork.fetch_fork_result_list(pipe_ids)

Read the output pipe of the children, used after forking to perform work and after forking to entry.writeStats()

@type pipe_ids: dict @param pipe_ids: Dictinary of pipe and pid

@rtype: dict @return: Dictionary of fork_results

glideinwms.lib.fork.fetch_ready_fork_result_list(pipe_ids)

Read the output pipe of the children, used after forking. If there is data on the pipes to consume, read the data and close the pipe. and after forking to entry.writeStats()

@type pipe_ids: dict @param pipe_ids: Dictinary of pipe and pid

@rtype: dict @return: Dictionary of work_done

glideinwms.lib.fork.fork_in_bg(function_torun, *args)

Fork and call a function with args

This function returns right away, returning the pid and a pipe to the stdout of the function process where the output of the function will be pickled

Example

def add(i, j): return i+j d = fork_in_bg(add, i, j)

Parameters:

• function_torun (function) – function to call after forking the process

• *args – arguments list to pass to the function

Returns:

dict with {‘r’: fd, ‘pid’: pid} where fd is the stdout from a pipe.

Return type:

dict

glideinwms.lib.fork.wait_for_pids(pid_list)

Wait for all pids to finish. Throw away any stdout or err

glideinwms.lib.glideinWMSVersion module

class glideinwms.lib.glideinWMSVersion.GlideinWMSDistro(chksumFile='checksum')

Bases: object

class __impl(chksumFile='checksum')

Bases: object

Implementation of the singleton interface

createVersionString()

version()

__instance = None

glideinwms.lib.glideinWMSVersion.usage()

glideinwms.lib.glideinWMSVersion.version(chksumFile=None)

glideinwms.lib.hashCrypto module

hashCrypto - This module defines classes to perform hash based cryptography

It uses M2Crypto: https://github.com/mcepl/M2Crypto a wrapper around OpenSSL: https://www.openssl.org/docs/man1.1.1/man3/

NOTE get_hash() and extract_hash() both return Unicode utf-8 (defaults.BINARY_ENCODING_CRYPTO) strings and

get_hash() accepts byte-like objects or utf-8 encoded Unicode strings. Same for all the get_XXX or extract_XXX that use those functions. Other class methods and functions use bytes for input and output

class glideinwms.lib.hashCrypto.Hash(hash_algo)

Bases: object

Generic hash class

Available hash algorithms:

‘sha1’ ‘sha224’ ‘sha256’, ‘ripemd160’ ‘md5’

compute(data)

Compute hash inline

len(data) must be less than len(key)

Parameters:

data (bytes) – data to calculate the hash of

Returns:

digest value as bytes string (OpenSSL final and digest together)

Return type:

bytes

compute_base64(data)

like compute, but base64 encoded

compute_hex(data)

like compute, but hex encoded

extract(fname, block_size=1048576)

Extract hash from a file

len(data) must be less than len(key)

Parameters:

• fname (str) – input file path (binary file)

• block_size –

Returns:

digest value as bytes string (OpenSSL final and digest together)

Return type:

bytes

extract_base64(fname, block_size=1048576)

like extract, but base64 encoded

extract_hex(fname, block_size=1048576)

like extract, but hex encoded

redefine(hash_algo)

class glideinwms.lib.hashCrypto.HashMD5

Bases: Hash

class glideinwms.lib.hashCrypto.HashSHA1

Bases: Hash

class glideinwms.lib.hashCrypto.HashSHA256

Bases: Hash

glideinwms.lib.hashCrypto.extract_hash(hash_algo, fname, block_size=1048576)

Compute hash from file

Parameters:

• hash_algo (str) – hash algorithm to use

• fname (str) – file path (file will be open in binary mode)

• block_size (int) – block size

Returns:

utf-8 encoded hash

Return type:

str

glideinwms.lib.hashCrypto.extract_md5(fname, block_size=1048576)

glideinwms.lib.hashCrypto.extract_sha1(fname, block_size=1048576)

glideinwms.lib.hashCrypto.extract_sha256(fname, block_size=1048576)

glideinwms.lib.hashCrypto.get_hash(hash_algo, data)

Compute hash inline

Parameters:

• hash_algo (str) – hash algorithm to use

• data (AnyStr) – data of which to calculate the hash

Returns:

utf-8 encoded hash

Return type:

str

glideinwms.lib.hashCrypto.get_md5(data)

glideinwms.lib.hashCrypto.get_sha1(data)

glideinwms.lib.hashCrypto.get_sha256(data)

glideinwms.lib.logSupport module

class glideinwms.lib.logSupport.GlideinHandler(filename, maxDays=1, minDays=0, maxMBytes=10, backupCount=5, compression=None)

Bases: BaseRotatingHandler

Custom logging handler class for glideinWMS. It combines the decision tree for log rotation from the TimedRotatingFileHandler with the decision tree from the RotatingFileHandler. This allows us to specify a lifetime AND file size to determine when to rotate the file.

This class assumes that the lifetime specified is in days. (24 hour periods)

@type filename: string @ivar filename: Full path to the log file. Includes file name. @type interval: int @ivar interval: Number of days to keep log file before rotating @type maxBytes: int @param maxMBytes: Maximum size of the logfile in MB before file rotation (used with min days) @type minDays: int @param minDays: Minimum number of days (used with max bytes) @type backupCount: int @ivar backupCount: How many backups to keep

_open_new_log()

This function is here to bridge the gap between the old (python 2.4) way of opening new log files and the new (python 2.7) way.

check_and_perform_rollover()

doRollover()

do a rollover; in this case, a date/time stamp is appended to the filename when the rollover happens. If there is a backup count, then we have to get a list of matching filenames, sort them and remove the one with the oldest suffix.

getFilesToDelete()

Determine the files to delete when rolling over.

More specific than the earlier method, which just used glob.glob().

shouldRollover(record, empty_record=False)

Determine if rollover should occur.

Basically, we are combining the checks for size and time interval

@type record: string @param record: The message that will be logged.

@attention: Due to the architecture decision to fork “workers” we run into an issue where the child that was forked could cause a log rotation. However the parent will never know and the parent’s file descriptor will still be pointing at the old log file (now renamed by the child). This will in turn cause the parent to immediately request a log rotate, which results in what appears to be truncated logs. To handle this we add a flag to disable log rotation. By default this is set to False, but anywhere we want to fork a child (or in any object that will be forked) we set the flag to True. Then in the parent, we initiate a log function that will log and rotate if necessary.

class glideinwms.lib.logSupport.MsgFilter(msg_type_list)

Bases: Filter

Filter used in handling records for the info logs.

filter(rec)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

msg_type_list = [20]

glideinwms.lib.logSupport.add_processlog_handler(logger_name, log_dir, msg_types, extension, maxDays, minDays, maxMBytes, backupCount=5, compression=None)

Adds a handler to the GlideinLogger logger referenced by logger_name.

glideinwms.lib.logSupport.alternate_log(msg)

When an exceptions happen within the logging system (e.g. when the disk is full while rotating a log file) an alternate logging is necessary, e.g. writing to stderr

glideinwms.lib.logSupport.format_dict(unformated_dict, log_format='   %-25s : %s\n')

Convenience function used to format a dictionary for the logs to make it human readable.

@type unformated_dict: dict @param unformated_dict: The dictionary to be formatted for logging @type log_format: string @param log_format: format string for logging

glideinwms.lib.logSupport.roll_all_logs()

glideinwms.lib.pidSupport module

exception glideinwms.lib.pidSupport.AlreadyRunning

Bases: RuntimeError

class glideinwms.lib.pidSupport.PidSupport(pid_fname)

Bases: object

format_pid_file_content()

load_registered()

parse_pid_file_content(lines)

register(pid=None, started_time=None)

relinquish()

reset_to_default()

class glideinwms.lib.pidSupport.PidWParentSupport(pid_fname)

Bases: PidSupport

format_pid_file_content()

parse_pid_file_content(lines)

register(parent_pid, pid=None, started_time=None)

reset_to_default()

glideinwms.lib.pidSupport.check_pid(pid)

glideinwms.lib.pidSupport.register_sighandler()

glideinwms.lib.pidSupport.termsignal(signr, frame)

glideinwms.lib.pidSupport.unregister_sighandler()

glideinwms.lib.pubCrypto module

pubCrypto - This module defines classes to perform public key cryptography

It uses M2Crypto: https://github.com/mcepl/M2Crypto a wrapper around OpenSSL: https://www.openssl.org/docs/man1.1.1/man3/

NOTE For convenience and consistency w/ previous versions of this module, Encryption/Signing functions

(b64, hex and .encrypt() ) accept bytes-like objects (bytes, bytearray) and also Unicode strings utf-8 encoded (defaults.BINARY_ENCODING_CRYPTO). B64 and hex Decryption functions, consistent w/ Python’s binascii.a2b_* functions, accept bytes and Unicode strings containing only ASCII characters, .decrypt() only accepts bytes-like objects (such as bytes, bytearray and other objects that support the buffer protocol). All these functions return bytes.

Keys can be loaded from AnyStr (str, bytes, bytearray). Keys are returned as bytes string. Key files are binary.

exception glideinwms.lib.pubCrypto.PubCryptoError(msg)

Bases: Exception

Exception masking M2Crypto exceptions, to ease error handling in modules importing pubCrypto

class glideinwms.lib.pubCrypto.PubRSAKey(key_str=None, key_fname=None, encryption_padding=4, sign_algo='sha256')

Bases: object

Public part of the RSA key

_load_from_bio(bio)

Load the key into the object

Protected, overridden by child classes. Used by load

Parameters:

bio (M2Crypto.BIO.BIO) – BIO to retrieve the key from (file or memory buffer)

_save_to_bio(bio)

Save the key from the object

Protected, overridden by child classes. Used by save and get

Parameters:

bio (M2Crypto.BIO.BIO) – BIO object to save the key to (file or memory buffer)

Returns:

status returned by M2Crypto.m2.rsa_write_pub_key

Return type:

int

Raises:

KeyError – if the key is not defined

encrypt(data)

Encrypt the data

Parameters:

data (AnyStr) – string to encrypt. bytes-like or str. If unicode, it is encoded using utf-8 before being encrypted. len(data) must be less than len(key)

Returns:

encrypted data

Return type:

bytes

encrypt_base64(data)

like encrypt, but base64 encoded

encrypt_hex(data)

like encrypt, but hex encoded

get()

Retrieve the key

Returns:

key

Return type:

bytes

load(key_str=None, key_fname=None)

Load key from a string or a file

Only one of the two can be defined (not None) Load the key into self.rsa_key

Parameters:

• key_str (str/bytes) – string w/ base64 encoded key Must be bytes-like object or ASCII string, like base64 inputs

• key_fname (str) – file name

Raises:

ValueError – if both key_str and key_fname are defined

save(key_fname)

Save the key to a file

The file is binary and is written using M2Crypto.BIO

Parameters:

key_fname (str) – file name

Returns:

verify(data, signature)

Verify that the signature gets you the data

Parameters:

• data (AnyStr) – string to verify. bytes-like or str. If unicode, it is encoded using utf-8 before being encrypted. :

• signature (bytes) – signature to use in the verification

Returns:

True if the signature gets you the data

Return type:

bool

Raises:

KeyError – if the key is not defined

verify_base64(data, signature)

like verify, but the signature is base64 encoded

verify_hex(data, signature)

like verify, but the signature is hex encoded

class glideinwms.lib.pubCrypto.RSAKey(key_str=None, key_fname=None, private_cipher='aes_256_cbc', private_callback=<function _default_callback>, encryption_padding=4, sign_algo='sha256')

Bases: PubRSAKey

Public and private part of the RSA key

PubRSAKey()

Return the public part only. Downgrade to PubRSAKey

Returns:

an object w/ only the public part of the key

Return type:

PubRSAKey

_load_from_bio(bio)

Load the key into the object

Internal, overrides the parent _load_from_bio. Used by load

Parameters:

bio (M2Crypto.BIO.BIO) –

_save_to_bio(bio)

Save the key from the object

Protected, overridden by child classes. Used by save and get

Parameters:

bio (M2Crypto.BIO.BIO) – BIO to save the key into (file or memory buffer)

Returns:

Raises:

KeyError – if the key is not defined

decrypt(data)

Decrypt data inline

Parameters:

data (bytes) – data to decrypt

Returns:

decrypted string

Return type:

bytes

Raises:

KeyError – if the key is not defined

decrypt_base64(data)

like decrypt, but base64 encoded

decrypt_hex(data)

like decrypt, but hex encoded

new(key_length=None, exponent=65537)

Refresh/Generate a new key and store it in the object

Parameters:

• key_length (int/None) – if no key_length provided, use the length of the existing one

• exponent (int) – exponent

sign(data)

Sign data inline. Same as private_encrypt

Parameters:

data (AnyStr) – string to encrypt. If unicode, it is encoded using utf-8 before being encrypted. len(data) must be less than len(key)

Returns:

encrypted data

Return type:

bytes

sign_base64(data)

like sign, but base64 encoded

sign_hex(data)

like sign, but hex encoded

glideinwms.lib.pubCrypto._default_callback(*args)

Return a dummy passphrase

Good for service key processing where human not present. Used as a callback in the :mod:M2Crypto module:

A Python callable object that is invoked to acquire a passphrase with which to unlock the key. The default is :func:M2Crypto.util.passphrase_callback

def passphrase_callback(v: bool, prompt1: str = 'Enter passphrase:', prompt2: str = 'Verify passphrase:'
                       ): -> Optional[str]

Parameters:

*args –

Returns:

str or None

Return type:

Optional[str]

glideinwms.lib.pubCrypto.passphrase_callback(v: bool, prompt1: str = 'Enter passphrase:', prompt2: str = 'Verify passphrase:')

glideinwms.lib.rrdSupport module

class glideinwms.lib.rrdSupport.BaseRRDSupport(rrd_obj)

Bases: object

create_rrd(rrdfname, rrd_step, rrd_archives, rrd_ds)

Create a new RRD archive

Parameters:

• archive (rrdfname - File path name of the RRD) –

• seconds (rrd_step - base interval in) –

• tuples (rrd_archives - list of) – CF - consolidation function (usually AVERAGE) xff - xfiles factor (fraction that can be unknown) steps - how many of these primary data points are used to build a consolidated data point rows - how many generations of data values are kept

• fileds (each containing the following) – CF - consolidation function (usually AVERAGE) xff - xfiles factor (fraction that can be unknown) steps - how many of these primary data points are used to build a consolidated data point rows - how many generations of data values are kept

• fields (rrd_ds - a tuple containing the following) – ds-name - attribute name DST - Data Source Type (usually GAUGE) heartbeat - the maximum number of seconds that may pass between two updates before it becomes unknown min - min value max - max value

For more details see

http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

create_rrd_multi(rrdfname, rrd_step, rrd_archives, rrd_ds_arr)

Create a new RRD archive

Parameters:

• archive (rrdfname - File path name of the RRD) –

• seconds (rrd_step - base interval in) –

• tuples (rrd_ds_arr - list of) – CF - consolidation function (usually AVERAGE) xff - xfiles factor (fraction that can be unknown) steps - how many of these primary data points are used to build a consolidated data point rows - how many generations of data values are kept

• fileds (each containing the following) – CF - consolidation function (usually AVERAGE) xff - xfiles factor (fraction that can be unknown) steps - how many of these primary data points are used to build a consolidated data point rows - how many generations of data values are kept

• tuples – ds-name - attribute name DST - Data Source Type (usually GAUGE) heartbeat - the maximum number of seconds that may pass between two updates before it becomes unknown min - min value max - max value

• fields (each containing the following) – ds-name - attribute name DST - Data Source Type (usually GAUGE) heartbeat - the maximum number of seconds that may pass between two updates before it becomes unknown min - min value max - max value

For more details see

http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

fetch_rrd(filename, CF, resolution=None, start=None, end=None, daemon=None)

Fetch will analyze the RRD and try to retrieve the data in the resolution requested.

Parameters:

• from (filename -the name of the RRD you want to fetch data) –

• data (CF -the consolidation function that is applied to the) – you want to fetch (AVERAGE, MIN, MAX, LAST)

• have (resolution -the interval you want your values to) – (default 300 sec)

• series (end -end of the time) –

• series –

• specified (daemon -Address of the rrdcached daemon. If) – command is sent to the server before reading the RRD files. This allows rrdtool to return fresh data even if the daemon is configured to cache values for a long time.

• flush (a) – command is sent to the server before reading the RRD files. This allows rrdtool to return fresh data even if the daemon is configured to cache values for a long time.

For more details see

http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

get_disk_lock(fname)

get_graph_lock(fname)

isDummy()

rrd2graph(fname, rrd_step, ds_name, ds_type, start, end, width, height, title, rrd_files, cdef_arr=None, trend=None, img_format='PNG')

Create a graph file out of a set of RRD files

Parameters:

• file (fname - File path name of the graph) –

• files (rrd_files - list of RRD) –

• files –

• files –

• start –

• format (end - Time points in utime) –

• width –

• graph (title - Title to put in the) –

• graph –

• files – rrd_id - logical name of the RRD file (will be the graph label) rrd_fname - name of the RRD file graph_type - Graph type (LINE, STACK, AREA) grpah_color - Graph color in rrdtool format

• of (each being a tuple) – rrd_id - logical name of the RRD file (will be the graph label) rrd_fname - name of the RRD file graph_type - Graph type (LINE, STACK, AREA) grpah_color - Graph color in rrdtool format

• values (cdef_arr - list of derived RRD) –

  if present, only the cdefs will be plotted

  each elsement is a tuple of (in order)

  rrd_id - logical name of the RRD file (will be the graph label) cdef_formula - Derived formula in rrdtool format graph_type - Graph type (LINE, STACK, AREA) grpah_color - Graph color in rrdtool format

• seconds (trend - Trend value in) –

For more details see

http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

rrd2graph_multi(fname, rrd_step, start, end, width, height, title, rrd_files, cdef_arr=None, trend=None, img_format='PNG')

Create a graph file out of a set of RRD files

Parameters:

• file (img_format - format of the graph) –

• files (rrd_files - list of RRD) –

• start –

• format (end - Time points in utime) –

• width –

• graph (title - Title to put in the) –

• graph –

• files – rrd_id - logical name of the RRD file (will be the graph label) rrd_fname - name of the RRD file ds_name - Which attribute should I use in the RRD files ds_type - Which type should I use in the RRD files graph_type - Graph type (LINE, STACK, AREA) graph_color - Graph color in rrdtool format

• of (each being a tuple) – rrd_id - logical name of the RRD file (will be the graph label) rrd_fname - name of the RRD file ds_name - Which attribute should I use in the RRD files ds_type - Which type should I use in the RRD files graph_type - Graph type (LINE, STACK, AREA) graph_color - Graph color in rrdtool format

• values (cdef_arr - list of derived RRD) –

  if present, only the cdefs will be plotted

  each elsement is a tuple of (in order)

  rrd_id - logical name of the RRD file (will be the graph label) cdef_formula - Derived formula in rrdtool format graph_type - Graph type (LINE, STACK, AREA) grpah_color - Graph color in rrdtool format

• seconds (trend - Trend value in) –

• file –

For more details see

http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

rrd2graph_multi_now(fname, rrd_step, period, width, height, title, rrd_files, cdef_arr=None, trend=None, img_format='PNG')

Create a graph file out of a set of RRD files

Parameters:

• file (img_format - format of the graph) –

• files (rrd_files - list of RRD) –

• start=now-period (period -) –

• end=now –

• width –

• graph (title - Title to put in the) –

• graph –

• files – rrd_id - logical name of the RRD file (will be the graph label) rrd_fname - name of the RRD file ds_name - Which attribute should I use in the RRD files ds_type - Which type should I use in the RRD files graph_type - Graph type (LINE, STACK, AREA) graph_color - Graph color in rrdtool format

• of (each being a tuple) – rrd_id - logical name of the RRD file (will be the graph label) rrd_fname - name of the RRD file ds_name - Which attribute should I use in the RRD files ds_type - Which type should I use in the RRD files graph_type - Graph type (LINE, STACK, AREA) graph_color - Graph color in rrdtool format

• values (cdef_arr - list of derived RRD) –

  if present, only the cdefs will be plotted

  each elsement is a tuple of (in order)

  rrd_id - logical name of the RRD file (will be the graph label) cdef_formula - Derived formula in rrdtool format graph_type - Graph type (LINE, STACK, AREA) grpah_color - Graph color in rrdtool format

• seconds (trend - Trend value in) –

• file –

For more details see

http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

rrd2graph_now(fname, rrd_step, ds_name, ds_type, period, width, height, title, rrd_files, cdef_arr=None, trend=None, img_format='PNG')

Create a graph file out of a set of RRD files

Parameters:

• file (fname - File path name of the graph) –

• files (rrd_files - list of RRD) –

• files –

• files –

• start=now-period (period -) –

• end=now –

• width –

• graph (title - Title to put in the) –

• graph –

• files – rrd_id - logical name of the RRD file (will be the graph label) rrd_fname - name of the RRD file graph_type - Graph type (LINE, STACK, AREA) grpah_color - Graph color in rrdtool format

• of (each being a tuple) – rrd_id - logical name of the RRD file (will be the graph label) rrd_fname - name of the RRD file graph_type - Graph type (LINE, STACK, AREA) grpah_color - Graph color in rrdtool format

• values (cdef_arr - list of derived RRD) –

  if present, only the cdefs will be plotted

  each elsement is a tuple of (in order)

  rrd_id - logical name of the RRD file (will be the graph label) cdef_formula - Derived formula in rrdtool format graph_type - Graph type (LINE, STACK, AREA) grpah_color - Graph color in rrdtool format

• seconds (trend - Trend value in) –

For more details see

http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

update_rrd(rrdfname, time, val)

Create an RRD archive with a new value

Parameters:

• archive (rrdfname - File path name of the RRD) –

• taken (time - When was the value) –

• value (val - What vas the) –

update_rrd_multi(rrdfname, time, val_dict)

Create an RRD archive with a set of values (possibly all of the supported)

Parameters:

• archive (rrdfname - File path name of the RRD) –

• taken (time - When was the value) –

• value (val_dict - What was the) –

verify_rrd(filename, expected_dict)

Verifies that an rrd matches a dictionary of datastores. This will return a tuple of arrays ([missing],[extra]) attributes

@param filename: filename of the rrd to verify @param expected_dict: dictionary of expected values @return: A two-tuple of arrays ([missing attrs],[extra attrs])

class glideinwms.lib.rrdSupport.DummyDiskLock

Bases: object

close()

class glideinwms.lib.rrdSupport.ExeRRDSupport

Bases: BaseRRDSupport

class glideinwms.lib.rrdSupport.ModuleRRDSupport

Bases: BaseRRDSupport

glideinwms.lib.rrdSupport.addDataStore(filenamein, filenameout, attrlist)

Add a list of data stores to a rrd export file This will essentially add attributes to the end of a rrd row

@param filenamein: filename path of a rrd exported with rrdtool dump @param filenameout: filename path of output xml with datastores added @param attrlist: array of datastores to add

glideinwms.lib.rrdSupport.dummy_disk_lock()

class glideinwms.lib.rrdSupport.rrdSupport

Bases: BaseRRDSupport

class glideinwms.lib.rrdSupport.rrdtool_exe

Bases: object

This class is a wrapper around the rrdtool client (binary) and is used in place of the rrdtool python module, if that one is not available

create(*args)

dump(*args)

Run rrd_tool dump

Input is usually just the file name. Output is a list of lines, as returned from rrdtool.

Parameters:

*args – rrdtool dump arguments, joined in single string for the command line

Returns:

multi-line string, output of rrd dump

Return type:

str

fetch(*args)

graph(*args)

info(*args)

restore(*args)

update(*args)

glideinwms.lib.rrdSupport.string_quote_join(arglist)

glideinwms.lib.servicePerformance module

class glideinwms.lib.servicePerformance.PerfMetric(name)

Bases: object

Class to store performance metrics for different events in a service

deregister_event(event_name)

event_end(event_name, t=None)

event_lifetime(event_name, check_active_event=True)

event_start(event_name, t=None)

register_event_time(event_name, t_tag, t=None)

glideinwms.lib.servicePerformance.endPerfMetricEvent(name, event_name, t=None)

glideinwms.lib.servicePerformance.getPerfMetric(name)

Given the name of the service, return the PerfMetric object

glideinwms.lib.servicePerformance.getPerfMetricEventLifetime(name, event_name)

glideinwms.lib.servicePerformance.startPerfMetricEvent(name, event_name, t=None)

glideinwms.lib.subprocessSupport module

exception glideinwms.lib.subprocessSupport.CalledProcessError2(returncode, cmd, output=None)

Bases: Exception

This exception is raised when a process run by check_call() or check_output() returns a non-zero exit status. The exit status will be stored in the returncode attribute; check_output() will also store the output in the output attribute.

glideinwms.lib.subprocessSupport.iexe_cmd(cmd, useShell=False, stdin_data=None, child_env=None, text=True)

Fork a process and execute cmd - rewritten to use select to avoid filling up stderr and stdout queues.

The useShell value of True should be used sparingly. It allows for executing commands that need access to shell features such as pipes, filename wildcards. Refer to the python manual for more information on this. When used, the ‘cmd’ string is not tokenized.

One possible improvement would be to add a function to accept an array instead of a command string.

Parameters:

• cmd (str) – String containing the entire command including all arguments

• useShell (bool) – if True run the command in a shell (passed to Popen as shell)

• stdin_data (str/bytes) – Data that will be fed to the command via stdin. It will be bytes if text is False, str otherwise

• child_env (dict) – Environment to be set before execution

• text (bool) – if False, then stdin_data and the return value are bytes instead of str (default: True)

Returns:

output of the command. It will be bytes if text is False,

str otherwise

Return type:

str/bytes

glideinwms.lib.symCrypto module

symCrypto - This module defines classes to perform symmetric key cryptography (shared or hidden key)

It uses M2Crypto: https://github.com/mcepl/M2Crypto a wrapper around OpenSSL: https://www.openssl.org/docs/man1.1.1/man3/

NOTE For convenience and consistency w/ previous versions of this module, Encryption/Signing functions

(b64, hex and .encrypt() ) accept bytes-like objects (bytes, bytearray) and also Unicode strings utf-8 encoded (defaults.BINARY_ENCODING_CRYPTO). B64 and hex Decryption functions, consistent w/ Python’s binascii.a2b_* functions, accept bytes and Unicode strings containing only ASCII characters, .decrypt() only accepts bytes-like objects (such as bytes, bytearray and other objects that support the buffer protocol). All these functions return bytes.

Key definitions accept AnyStr (str, bytes, bytearray), key_str are iv_str bytes, key_iv_code is a str, so is the key

class glideinwms.lib.symCrypto.AutoSymKey(key_iv_code=None)

Bases: MutableSymKey

Symmetric Keys from code strings. Get cypher name from key_iv_code

auto_load(key_iv_code=None)

Load a new key_iv_key and extract the cypher

Parameters:

key_iv_code (AnyStr) – cypher byte string. str is encoded using BINARY_ENCODING_CRYPTO

Raises:

ValueError – if the format of the code is incorrect

class glideinwms.lib.symCrypto.MutableSymKey(cypher_name=None, key_len=None, iv_len=None, key_str=None, iv_str=None, key_iv_code=None)

Bases: SymKey

SymKey class, allows to change the crypto after instantiation

get_wcrypto()

Get the stored key and the crypto name

Returns:

cypher name bytes: key string bytes: iv string

Return type:

str

is_valid()

Return true if the key is valid.

Redefine, as null crypto name could be used in this class

Returns:

True if both the key string and cypher name are not None

Return type:

bool

redefine(cypher_name=None, key_len=None, iv_len=None, key_str=None, iv_str=None, key_iv_code=None)

Load a new crypto type and a new key

Parameters:

• cypher_name –

• key_len –

• iv_len –

• key_str –

• iv_str –

• key_iv_code –

Returns:

class glideinwms.lib.symCrypto.ParametrizedSymKey(cypher_name, key_str=None, iv_str=None, key_iv_code=None)

Bases: SymKey

Helper class to build different types of Symmetric Keys from a parameter dictionary (cypher_dict).

class glideinwms.lib.symCrypto.Sym3DESKey(key_str=None, iv_str=None, key_iv_code=None)

Bases: ParametrizedSymKey

class glideinwms.lib.symCrypto.SymAES128Key(key_str=None, iv_str=None, key_iv_code=None)

Bases: ParametrizedSymKey

class glideinwms.lib.symCrypto.SymAES256Key(key_str=None, iv_str=None, key_iv_code=None)

Bases: ParametrizedSymKey

class glideinwms.lib.symCrypto.SymBlowfishKey(key_str=None, iv_str=None, key_iv_code=None)

Bases: ParametrizedSymKey

class glideinwms.lib.symCrypto.SymDESKey(key_str=None, iv_str=None, key_iv_code=None)

Bases: ParametrizedSymKey

class glideinwms.lib.symCrypto.SymKey(cypher_name, key_len, iv_len, key_str=None, iv_str=None, key_iv_code=None)

Bases: object

Symmetric keys cryptography

You probably don’t want to use this, use the child classes instead

self.key_str and self.iv_str are bytes (strings) with HEX encoded data

Available ciphers, too many to list them all, try man enc, a few of them are:

‘aes_128_cbc’ ‘aes_128_ofb ‘aes_256_cbc’ ‘aes_256_cfb’ ‘bf_cbc’ ‘des3’

decrypt(data)

Decrypt data inline

Parameters:

data (bytes) – data to decrypt

Returns:

decrypted data

Return type:

bytes

Raises:

KeyError – if there is no valid crypto key

decrypt_base64(data)

like decrypt, but the input is base64 encoded

Parameters:

data (AnyStrASCII) – Base64 input data. bytes or ASCII encoded Unicode str

Returns:

decrypted data

Return type:

bytes

decrypt_hex(data)

like decrypt, but the input is hex encoded

Parameters:

data (AnyStrASCII) – HEX input data. bytes or ASCII encoded Unicode str

Returns:

decrypted data

Return type:

bytes

encrypt(data)

Encrypt data inline

Parameters:

data (AnyStr) – data to encrypt

Returns:

encrypted data

Return type:

bytes

Raises:

KeyError – if there is no valid crypto key

encrypt_base64(data)

like encrypt, but the result is base64 encoded

encrypt_hex(data)

like encrypt, but the result is hex encoded

get()

Get the key and initialization vector

Returns:

(key, iv) tuple wehere both key and iv are bytes

Return type:

tuple

get_code()

Return the key code: cypher, key, iv, as a comma separated string

Returns:

key description in the string

Return type:

str

is_valid()

Return true if the key is valid

Returns:

True if the key string is not None

Return type:

bool

load(key_str=None, iv_str=None, key_iv_code=None)

Load a new key from text (str/bytes)

Parameters:

• key_str (str/bytes) – string w/ base64 encoded key Must be bytes-like object or ASCII string, like base64 inputs

• iv_str (str/bytes) – initialization vector

• key_iv_code (str/bytes) – comma separated text with cypher, key, iv

Returns:

new(random_iv=True)

Generate a new key

Set self.key_str and self.iv_str

Parameters:

random_iv (bool) – if False, set iv to 0

glideinwms.lib.tarSupport module

exception glideinwms.lib.tarSupport.FileDoesNotExist(full_path)

Bases: Exception

File does not exist exception

@note: Include the file name in the full_path @ivar full_path: The full path to the missing file. Includes the file name

class glideinwms.lib.tarSupport.GlideinTar

Bases: object

This class provides a container for creating tarballs. The class provides methods to add files and string data (ends up as a file in the tarball). The tarball can be written to a file on disk or written to memory.

add_file(filename, arc_dirname)

Add a filepath to the files list

@type filename: string @param filename: The file path to the file that will eventually be written to the tarball. @type arc_dirname: string @param arc_dirname: This is the directory that the file will show up under in the tarball

add_string(name, string_data)

Add a string to the string dictionary.

@type name: string @param name: A string specifying the “filename” within the tarball that the string_data will be written to. @type string_data: string @param string_data: The contents that will be written to a “file” within the tarball.

create_tar(tf)

Takes the provided tar file object and adds all the specified data to it. The strings dictionary is parsed such that the key name is the file name and the value is the file data in the tar file.

@type tf: Tar File @param tf: The Tar File Object that will be written to

create_tar_blob(compression='gz')

Creates a tarball and writes it out to memory

@Note: we don’t have to worry about ReadError, since we don’t allow

appending. We only write to a tarball on create.

@param fd: The file that the tarball will be written to @param compression: The type of compression that should be used

@raise glideinwms_tarfile.CompressionError: This exception can be raised is an

invalid compression type has been passed in

create_tar_file(archive_full_path, compression='gz')

Creates a tarball and writes it out to the file specified in fd

@Note: we don’t have to worry about ReadError, since we don’t allow

appending. We only write to a tarball on create.

@param fd: The file that the tarball will be written to @param compression: The type of compression that should be used

@raise glideinwms_tarfile.CompressionError: This exception can be raised is an

invalid compression type has been passed in

is_tarfile(full_path)

Checks to see if the tar file specified is valid and can be read. Returns True if the file is a valid tar file and it can be read. Returns False if not valid or it cannot be read.

@param full_path: The full path to the tar file. Includes the file name

@return: True/False

glideinwms.lib.timeConversion module

glideinwms.lib.timeConversion.extractHuman(time_str)

glideinwms.lib.timeConversion.extractISO8601_Local(time_str)

glideinwms.lib.timeConversion.extractISO8601_UTC(time_str)

glideinwms.lib.timeConversion.extractRFC2822_Local(time_str)

glideinwms.lib.timeConversion.extractRFC2822_UTC(time_str)

glideinwms.lib.timeConversion.extractSeconds(time_str)

glideinwms.lib.timeConversion.getHuman(now=None)

glideinwms.lib.timeConversion.getISO8601_Local(now=None)

glideinwms.lib.timeConversion.getISO8601_UTC(now=None)

glideinwms.lib.timeConversion.getRFC2822_Local(now=None)

glideinwms.lib.timeConversion.getRFC2822_UTC(now=None)

glideinwms.lib.timeConversion.getSeconds(now=None)

glideinwms.lib.timeConversion.getTZval(t)

glideinwms.lib.timeConversion.get_time_in_format(now=None, time_format=None)

glideinwms.lib.token_util module

glideinwms.lib.unparser module

Usage (for testing, will do roundtrips on the files): unparse.py <path to source files> unparse.py –testdir <path to source files directories>

class glideinwms.lib.unparser.Unparser(tree, file=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

Bases: object

Methods in this class recursively traverse an AST and output source code for the abstract syntax; original formatting is disregarded.

_AnnAssign(t)

_Assert(t)

_Assign(t)

_AsyncFor(t)

_AsyncFunctionDef(t)

_AsyncWith(t)

_Attribute(t)

_AugAssign(t)

_Await(t)

_BinOp(t)

_BoolOp(t)

_Break(t)

_Bytes(t)

_Call(t)

_ClassDef(t)

_Compare(t)

_Constant(t)

_Continue(t)

_Delete(t)

_Dict(t)

_DictComp(t)

_Ellipsis(t)

_ExceptHandler(t)

_Exec(t)

_Expr(tree)

_Expression(tree)

_ExtSlice(t)

_For(t)

_FormattedValue(t)

_FunctionDef(t)

_GeneratorExp(t)

_Global(t)

_If(t)

_IfExp(t)

_Import(t)

_ImportFrom(t)

_Index(t)

_Interactive(tree)

_JoinedStr(t)

_Lambda(t)

_List(t)

_ListComp(t)

_Module(tree)

_Name(t)

_NameConstant(t)

_NamedExpr(tree)

_Nonlocal(t)

_Num(t)

_Pass(t)

_Print(t)

_Raise(t)

_Repr(t)

_Return(t)

_Set(t)

_SetComp(t)

_Slice(t)

_Starred(t)

_Str(tree)

_Subscript(t)

_Try(t)

_Tuple(t)

_UnaryOp(t)

_While(t)

_With(t)

_Yield(t)

_YieldFrom(t)

__For_helper(fill, t)

__FunctionDef_helper(t, fill_suffix)

_alias(t)

_arg(t)

_arguments(t)

_comprehension(t)

_fstring_Constant(t, write)

_fstring_FormattedValue(t, write)

_fstring_JoinedStr(t, write)

_fstring_Str(t, write)

_generic_With(t, async_=False)

_keyword(t)

_withitem(t)

_write_constant(value)

binop = {'Add': '+', 'BitAnd': '&', 'BitOr': '|', 'BitXor': '^', 'Div': '/', 'FloorDiv': '//', 'LShift': '<<', 'MatMult': '@', 'Mod': '%', 'Mult': '*', 'Pow': '**', 'RShift': '>>', 'Sub': '-'}

boolops = {<class '_ast.And'>: 'and', <class '_ast.Or'>: 'or'}

cmpops = {'Eq': '==', 'Gt': '>', 'GtE': '>=', 'In': 'in', 'Is': 'is', 'IsNot': 'is not', 'Lt': '<', 'LtE': '<=', 'NotEq': '!=', 'NotIn': 'not in'}

dispatch(tree)

Dispatcher function, dispatching tree type T to method _T.

enter()

Print ‘:’, and increase the indentation.

fill(text='')

Indent a piece of text, according to the current indentation level

leave()

Decrease the indentation level.

unop = {'Invert': '~', 'Not': 'not', 'UAdd': '+', 'USub': '-'}

write(text)

Append a piece of text to the current line.

glideinwms.lib.unparser.interleave(inter, f, seq)

Call f on each item in seq, calling inter() in between.

glideinwms.lib.unparser.main(args)

glideinwms.lib.unparser.roundtrip(filename, output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

glideinwms.lib.unparser.testdir(a)

glideinwms.lib.util module

exception glideinwms.lib.util.ExpiredFileException

Bases: Exception

The file is too old to be used

glideinwms.lib.util.chmod(*args, **kwargs)

Wrapper for os.chmod that supresses PermissionError exceptions

Parameters:

• *args – Positional arguments to pass to os.chmod

• **kwargs – Keyword arguments to pass to os.chmod

Returns:

None

glideinwms.lib.util.conditional_raise(mask_exceptions)

Auxiliary function to handle conditional raising

Parameters:

mask_exceptions – callback function and arguments to use if an exception happens (Default: None) The callback function can access the exception via sys.exc_info() If a function is not provided, the exception is re-risen if provided it is called using mask_exceptions[0](*mask_exceptions[1:])

Returns:

None

glideinwms.lib.util.dict_normalize(in_dict, keys=None, prefix='', suffix='', default=None)

Change the keys of a dictionary

Parameters:

• in_dict – input dictionary

• keys – key list, if None it is using in_dict.keys() (Default: None)

• prefix – prefix for the keys (Default “”)

• suffix – suffix for the keys (Default “”)

• default – default value passed to get (Default: None)

Returns:

normalized dictionary

glideinwms.lib.util.dict_to_flat(in_dict, prefix='', suffix='', sep='')

Flatten a multi-level dictionary to one level The resulting keys are the string concatenation of the original ones A separator can be added between keys

NOTE: Value could be clobbered if there are duplicates in the strings resulting from concatenating keys, e.g. {‘a’:{‘b’:1}, ‘ab’:2} A separator will not solve the problem if it is a valid character for the keys

Parameters:

• in_dict – input dictionary

• prefix – prefix for the keys (Default “”)

• suffix – suffix for the keys (Default “”)

• sep – separator between keys (Default: “”)

Returns:

flattened dictionary

glideinwms.lib.util.dict_to_flat_slow(in_dict, prefix='', suffix='')

Flatten a multi-level dictionary to one level The resulting keys are the string concatenation of the original ones

Parameters:

• in_dict – input dictionary

• prefix – prefix for the keys (Default “”)

• suffix – suffix for the keys (Default “”)

Returns:

flattened dictionary

glideinwms.lib.util.file_get_tmp(fname=None, tmp_type=None)

Get the name of a temporary file Depending on the option chosen this may be unsafe: .tmp suffix is OK only if no one else will use this file .$PID.tmp is OK if no multithreading is used and there are no safety concerns (name easy to guess, attacks prone) tempfile from tempfile.mkstemp() that guarantees uniqueness and safety

@param fname: original file name @param tmp_type: type of temporary file name (Default: None):

• None (or anything False) - ‘.tmp’ suffix added to the file name (unsafe, may be conflicts if )

• PID - ‘.$PID.tmp’ suffix added to the file name

• REAL (or anything else) - real tempfile (unique and safe, using tempfile.mkstemp())

@return: tamporary file name

glideinwms.lib.util.file_pickle_dump(fname, content, tmp_type='PID', mask_exceptions=None, protocol=4)

Serialize and save content

To avoid inconsistent content @param fname: file storing the serialized content @param content: content to serialize @param tmp_type: tmp file type as defined in file_get_tmp (Default: PID, .$PID.tmp suffix) @param mask_exceptions: callback function and arguments to use if an exception happens (Default: None)

The callback function can access the exception via sys.exc_info() If a function is not provided, the exception is re-risen if provided it is called using mask_exceptions[0](*mask_exceptions[1:])

@param protocol: Pickle protocol to be used (Default: pickle.HIGHEST_PROTOCOL, 5 as of py3.8) @return: True if the saving was successful, False or an exception otherwise

glideinwms.lib.util.file_pickle_load(fname, mask_exceptions=None, default=None, expiration=- 1, remove_expired=False, last_time={})

Load a serialized dictionary

This implementation does not use file locking, it relies on the atomicity of file movement/replacement and deletion @param fname: name of the file with the serialized data @param mask_exceptions: callback function and arguments to use if an exception happens (Default: None)

The callback function can access the exception via sys.exc_info() If a function is not provided, the exception is re-risen if provided it is called using mask_exceptions[0](*mask_exceptions[1:])

@param default: value returned if the unpickling fails (Default: None) @param expiration: input file expiration in seconds (Default: -1)

-1 file never expires 0 file always expires after reading

@param remove_expired: remove expired file (Default: False)

NOTE: if you remove the obsolete file from the reader you may run into a race condition with undesired effects: 1. the reader detects the obsolete file, 2. the writer writes a new version, 3. the reader deletes the new version This can happen only in cycles where there is an obsolete data file to start with, so the number of data files lost because of this is smaller than the occurrences of obsoleted files. When the expiration time is much bigger than the loop time of the writer this is generally acceptable.

@param last_time: last time a file has been used, persistent to keep history (Default: {}, first time called) @return: python objects (e.g. data dictionary)

glideinwms.lib.util.file_tmp2final(fname, tmp_fname=None, bck_fname=None, do_backup=True, mask_exceptions=None)

Complete an atomic write by moving a file new version to its destination.

If do_backup is True it removes the previous backup and copies the file to bak_fname. Moves tmp_fname to fname.

Parameters:

• fname – name of the file

• tmp_fname – name of the temporary file with the new version of the content (Default: <fname>.tmp)

• bck_fname – name of a backup of the old version (Default: <fname>~)

• do_backup – do a backup of the old version only if True (Default: True)

• mask_exceptions – callback function and arguments to use if an exception happens (Default: None) The callback function can access the exception via sys.exc_info() If a function is not provided, the exception is re-risen if provided it is called using mask_exceptions[0](*mask_exceptions[1:])

Returns:

False if the move caused an exception. True otherwise

glideinwms.lib.util.flattenDict(d, join=<built-in function add>, lift=<function <lambda>>)

Flexible flattening of a dictionary

Parameters:

• d – dictionary to flatten

• join – join function for the keys (allows to concatenate strings)

• lift – lift function for the keys (allows to create tuples)

Returns:

list with flattened dictionary

>>> testData = {
    'a':1,
    'b':2,
    'c':{
        'aa':11,
        'bb':22,
        'cc':{
            'aaa':111
        }
    }
}
>>> from pprint import pprint as pp

>>> pp(dict( flattenDict(testData, lift=lambda x:(x,)) ))
{('a',): 1,
 ('b',): 2,
 ('c', 'aa'): 11,
 ('c', 'bb'): 22,
 ('c', 'cc', 'aaa'): 111}

>>> pp(dict( flattenDict(testData, join=lambda a,b:a+'_'+b) ))
{'a': 1, 'b': 2, 'c_aa': 11, 'c_bb': 22, 'c_cc_aaa': 111}

>>> pp(dict( (v,k) for k,v in flattenDict(testData, lift=hash, join=lambda a,b:hash((a,b))) ))
{1: 12416037344,
 2: 12544037731,
 11: 5470935132935744593,
 22: 4885734186131977315,
 111: 3461911260025554326}

glideinwms.lib.util.handle_hooks(basedir, script_dir)

The function itaretes over the script_dir directory and executes any script found there

glideinwms.lib.util.hash_nc(data, len=None)

Returns a non-cryptographic MD5 hash encoded in base32

Parameters:

• data (AnyStr) – Data to hash

• len (int, optional) – Hash length. Defaults to None.

Returns:

Hash

Return type:

str

glideinwms.lib.util.print_funct(*args, **kwargs)

Print function that can be used as mask exception (the print statement cannot be passed as parameter)

Parameters:

• args – list of what to print, converted into string and separated by ‘sep’

• kwargs – keywords, valid keywords: ‘sep’ separator, default is space

Returns:

None

glideinwms.lib.util.safe_boolcomp(value, expected)

Safely do a boolean comparison.

This works even if the value you wantto compare is a string.

Parameters:

• value – what you want to safely compare

• expected (bool) – What you want to compare value with

Returns:

True if str(value).lower() is True

Return type:

bool

glideinwms.lib.util.str2bool(val)

Convert u”True” or u”False” to boolean or raise ValueError

glideinwms.lib.x509Support module

glideinwms.lib.x509Support.extract_DN(fname)

Extract a Distinguished Name from an X.509 proxy.

@type fname: string @param fname: Filename containing the X.509 proxy

glideinwms.lib.xmlFormat module

glideinwms.lib.xmlFormat.class2file(fd, inst, inst_name, params={}, subclass_params={}, dicts_params=None, lists_params=None, tree_params=None, text_params=None, indent_tab='   ', leading_tab='', debug_str='')

glideinwms.lib.xmlFormat.class2head(inst, inst_name, params, dicts_params, lists_params, tree_params, text_params, leading_tab, debug_str)

glideinwms.lib.xmlFormat.class2string(inst, inst_name, params={}, subclass_params={}, dicts_params=None, lists_params=None, tree_params=None, text_params=None, indent_tab='   ', leading_tab='', debug_str='', override_dictionary_type=None)

glideinwms.lib.xmlFormat.complete_class_params(class_params)

glideinwms.lib.xmlFormat.complete_dict_params(dict_params)

glideinwms.lib.xmlFormat.complete_list_params(list_params)

glideinwms.lib.xmlFormat.dict2file(fd, dict_data, dict_name, el_name, dict_attr_name='name', el_attr_name=None, params={}, subtypes_params={}, indent_tab='   ', leading_tab='', debug_str='')

glideinwms.lib.xmlFormat.dict2string(dict_data, dict_name, el_name, dict_attr_name='name', el_attr_name=None, params={}, subtypes_params={}, indent_tab='   ', leading_tab='', debug_str='')

glideinwms.lib.xmlFormat.list2file(fd, list_data, list_name, el_name, el_attr_name=None, params={}, subtypes_params={}, indent_tab='   ', leading_tab='', debug_str='')

glideinwms.lib.xmlFormat.list2string(list_data, list_name, el_name, el_attr_name=None, params={}, subtypes_params={}, indent_tab='   ', leading_tab='', debug_str='')

glideinwms.lib.xmlFormat.time2xml(the_time, outer_tag, indent_tab='   ', leading_tab='')

glideinwms.lib.xmlFormat.tree2file(fd, tree, tree_name, child_element, indent_tab='   ', leading_tab='', debug_str='')

glideinwms.lib.xmlFormat.tree2string(tree, tree_name, child_element, indent_tab='   ', leading_tab='', debug_str='')

glideinwms.lib.xmlFormat.xml_quoteattr(el)

glideinwms.lib.xmlParse module

exception glideinwms.lib.xmlParse.CorruptXML

Bases: Exception

class glideinwms.lib.xmlParse.OrderedDict

Bases: dict

class glideinwms.lib.xmlParse.OrderedDict2(dict=None)

Bases: UserDict

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 41

_abc_registry = <_weakrefset.WeakSet object>

clear() → None.  Remove all items from D.

copy()

items() → a set-like object providing a view on D's items

keys() → a set-like object providing a view on D's keys

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values() → an object providing a view on D's values

glideinwms.lib.xmlParse.domel2dict(doc, use_ord_dict=False, always_singular_list=[])

Recursive function transforming XML elements in a dictionary or list. If the node is unique (or it has attributes and the kids have no ‘name’ attribute), then a dictionary with all the attributes is returned If the element is singular of the parent (english word is analyzed):

if it has a ‘name’ attribute or the parent has attributes, a dictionary is added to the parent (name is the key) if if has no name and the parent is empty or a list, then is added to the parent (list)

Parameters:

• doc – document or ELEMENT_NODE

• use_ord_dict – use ordinate dictionary if True

• always_singular_list – these are considered unique singular even if the word is singular form of a plural

Returns:

dictionary or list with the content

glideinwms.lib.xmlParse.getXMLAttributes(element, use_ord_dict)

glideinwms.lib.xmlParse.getXMLElements(element)

glideinwms.lib.xmlParse.is_singular_of(mysin, myplu, always_singular_list=[])

glideinwms.lib.xmlParse.xmlfile2dict(fname, use_ord_dict=False, always_singular_list=[])

glideinwms.lib.xmlParse.xmlstring2dict(instr, use_ord_dict=False, always_singular_list=[])

Module contents