glideinwms.frontend package

Subpackages

• glideinwms.frontend.tools package
  • Subpackages
    • glideinwms.frontend.tools.lib package
      • Submodules
      • glideinwms.frontend.tools.lib.frontenvparse module
      • Module contents
  • Module contents

Submodules

glideinwms.frontend.checkFrontend module

glideinwms.frontend.glideinFrontend module

glideinwms.frontend.glideinFrontendConfig module

class glideinwms.frontend.glideinFrontendConfig.AttrsDescript(base_dir, group_name)

Bases: JoinConfigFile

class glideinwms.frontend.glideinFrontendConfig.BaseSignatureDescript(config_dir, signature_fname, signature_type, validate=None)

Bases: ConfigFile

split_func(line, convert_function)

class glideinwms.frontend.glideinFrontendConfig.ConfigFile(config_dir, config_file, convert_function=<built-in function repr>, validate=None)

Bases: object

derive()

load(fname, convert_function, validate=None)

open(fname)

split_func(line, convert_function)

validate_func(data, validate, fname)

class glideinwms.frontend.glideinFrontendConfig.ElementDescript(base_dir, group_name)

Bases: GroupConfigFile

class glideinwms.frontend.glideinFrontendConfig.ElementMergedDescript(base_dir, group_name)

Bases: object

merge()

split_list(val)

class glideinwms.frontend.glideinFrontendConfig.ExtStageFiles(base_URL, descript_fname, validate_algo, signature_hash)

Bases: StageFiles

get_condor_vars()

get_constants()

load_preentry_file_list()

class glideinwms.frontend.glideinFrontendConfig.FrontendConfig

Bases: object

class glideinwms.frontend.glideinFrontendConfig.FrontendDescript(config_dir)

Bases: ConfigFile

class glideinwms.frontend.glideinFrontendConfig.GroupConfigFile(base_dir, group_name, config_file, convert_function=<built-in function repr>, validate=None)

Bases: ConfigFile

class glideinwms.frontend.glideinFrontendConfig.GroupSignatureDescript(base_dir, group_name)

Bases: object

class glideinwms.frontend.glideinFrontendConfig.HistoryFile(base_dir, group_name, load_on_init=True, default_factory=None)

Bases: object

empty()

get(keyid, defaultval=None)

has_key(keyid)

load(raise_on_error=False)

save(raise_on_error=False)

class glideinwms.frontend.glideinFrontendConfig.JoinConfigFile(base_dir, group_name, config_file, convert_function=<built-in function repr>, main_validate=None, group_validate=None)

Bases: ConfigFile

class glideinwms.frontend.glideinFrontendConfig.MergeStageFiles(base_URL, validate_algo, main_descript_fname, main_signature_hash, group_name, group_descript_fname, group_signature_hash)

Bases: object

get_condor_vars()

get_constants()

class glideinwms.frontend.glideinFrontendConfig.ParamsDescript(base_dir, group_name)

Bases: JoinConfigFile

class glideinwms.frontend.glideinFrontendConfig.SignatureDescript(config_dir)

Bases: ConfigFile

split_func(line, convert_function)

class glideinwms.frontend.glideinFrontendConfig.StageFiles(base_URL, descript_fname, validate_algo, signature_hash)

Bases: object

get_file_list(list_type)

get_stage_file(fname, repr)

glideinwms.frontend.glideinFrontendConfig.get_group_dir(base_dir, group_name)

glideinwms.frontend.glideinFrontendDowntimeLib module

class glideinwms.frontend.glideinFrontendDowntimeLib.DowntimeFile(fname)

Bases: object

addPeriod(start_time, end_time, create_if_empty=True)

checkDowntime(check_time=None)

endDowntime(end_time=None)

printDowntime(check_time=None)

read(raise_on_error=False)

startDowntime(start_time=None, end_time=None, create_if_empty=True)

glideinwms.frontend.glideinFrontendDowntimeLib.addPeriod(fname, start_time, end_time, create_if_empty=True)

glideinwms.frontend.glideinFrontendDowntimeLib.checkDowntime(fname, check_time=None)

glideinwms.frontend.glideinFrontendDowntimeLib.endDowntime(fname, end_time=None)

glideinwms.frontend.glideinFrontendDowntimeLib.printDowntime(fname, check_time=None)

glideinwms.frontend.glideinFrontendDowntimeLib.read(fname, raise_on_error=False)

glideinwms.frontend.glideinFrontendElement module

glideinwms.frontend.glideinFrontendInterface module

glideinwms.frontend.glideinFrontendLib module

glideinwms.frontend.glideinFrontendLib.appendRealRunning(condorq_dict, status_dict)

Adds provenance information from condor_status to the condor_q dictionary The name of static or pslots is the value of RemoteHost NOTE: HTC 8.5 may change RemoteHost to be the DynamicSlot name

Parameters:

• condorq_dict – adding ‘RunningOn’ to each job

• status_dict – running jobs from condor_status

Returns:

glideinwms.frontend.glideinFrontendLib.countCondorQ(condorq_dict)

glideinwms.frontend.glideinFrontendLib.countCondorStatus(status_dict)

glideinwms.frontend.glideinFrontendLib.countCoresCondorStatus(status_dict, state='TotalCores')

glideinwms.frontend.glideinFrontendLib.countGlideinsCondorStatus(status_dict)

Return the number of glideins in the dictionary

Parameters:

status_dict – the output of getCondorStatus

Returns:

number of glideins in the dictionary (integer)

A Glidein is an execution of the glidein_startup.sh script

• may be different from job submitted by the factory (for multinode jobs - future)

• is different from a slot (or schedd or vm)

It defines GLIDEIN_MASTER_NAME which is the part after ‘@’ in the slot name Sets from different collectors are assumed disjunct

glideinwms.frontend.glideinFrontendLib.countIdleCoresCondorStatus(status_dict)

Counts the cores in the status dictionary The status is redundant in part but necessary to handle correctly partitionable slots which are 1 glidein but may have some running cores and some idle cores @param status_dict: a dictionary with the Machines to count @type status_dict: str

glideinwms.frontend.glideinFrontendLib.countMatch(match_obj, condorq_dict, glidein_dict, attr_dict, ignore_down_entries, condorq_match_list=None, match_policies=[], group_name=None)

Get the number of jobs that match each glidein

Parameters:

• match_obj – output of re.compile(match string,’<string>’,’eval’)

• condorq_dict (dictionary: sched_name->CondorQ object) – output of getidleCondorQ

• glidein_dict (dictionary: glidein_name->dictionary of params and attrs) – output of interface.findGlideins

• attr_dict – dictionary of constant attributes

• condorq_match_list – list of job attributes from the XML file

Returns:

tuple of 4 elements, where first 3 are a dictionary of glidein name where elements are number of jobs matching First tuple : Straight match Second tuple : The entry proportion based on unique subsets Third tuple : Elements that can only run on this site Forth tuple : The entry proportion glideins to be requested based

on unique subsets after considering multicore jobs, GLIDEIN_CPUS/GLIDEIN_ESTIMATED_CPUS (cores in glideins) GLIDEIN_NODES (number of nodes in multinode submissions)

A special ‘glidein name’ of (None, None, None) is used for jobs that don’t match any ‘real glidein name’ in all 4 tuples above

glideinwms.frontend.glideinFrontendLib.countRealRunning(match_obj, condorq_dict, glidein_dict, attr_dict, condorq_match_list=None, match_policies=[])

Counts all the running jobs on an entry

Parameters:

• match_obj – selection for the jobs

• condorq_dict – result of condor_q, keyed by schedd name

• glidein_dict – glideins, keyed by entry (glidename)

• attr_dict – entry attributes, NOT USED

• condorq_match_list – match attributes used for clustering

• match_policies –

Returns: Tuple with the job counts (used for stats) and glidein counts (used for glidein_max_run)

Both are dictionaries keyed by glidename (entry)

glideinwms.frontend.glideinFrontendLib.countRunningCondorStatus(status_dict)

glideinwms.frontend.glideinFrontendLib.countRunningCoresCondorStatus(status_dict)

Counts the cores in the status dictionary The status is redundant in part but necessary to handle correctly partitionable slots which are 1 glidein but may have some running cores and some idle cores @param status_dict: a dictionary with the Machines to count @type status_dict: str

glideinwms.frontend.glideinFrontendLib.countTotalCoresCondorStatus(status_dict)

Counts the cores in the status dictionary The status is redundant in part but necessary to handle correctly partitionable slots which are 1 glidein but may have some running cores and some idle cores @param status_dict: a dictionary with the Machines to count @type status_dict: str

glideinwms.frontend.glideinFrontendLib.evalParamExpr(expr_obj, frontend, glidein)

glideinwms.frontend.glideinFrontendLib.getClientCondorStatus(status_dict, frontend_name, group_name, request_name)

glideinwms.frontend.glideinFrontendLib.getClientCondorStatusCredIdOnly(status_dict, cred_id)

glideinwms.frontend.glideinFrontendLib.getClientCondorStatusPerCredId(status_dict, frontend_name, group_name, request_name, cred_id)

glideinwms.frontend.glideinFrontendLib.getCondorQ(schedd_names, constraint=None, format_list=None, want_format_completion=True, job_status_filter=(1, 2))

Return a dictionary of schedds containing interesting jobs Each element is a condorQ

If not all the jobs of the schedd has to be considered, specify the appropriate constraint

Parameters:

• schedd_names –

• constraint (str) – constraint string or None

• format_list –

• want_format_completion (bool) –

• job_status_filter –

Returns:

glideinwms.frontend.glideinFrontendLib.getCondorQConstrained(schedd_names, type_constraint, constraint=None, format_list=None)

glideinwms.frontend.glideinFrontendLib.getCondorQUsers(condorq_dict)

glideinwms.frontend.glideinFrontendLib.getCondorStatus(collector_names, constraint=None, format_list=None, want_format_completion=True, want_glideins_only=True)

Return a dictionary of collectors containing interesting classads Each element is a condorStatus @param collector_names: @param constraint: @param format_list: @param want_format_completion: @param want_glideins_only: @return:

glideinwms.frontend.glideinFrontendLib.getCondorStatusConstrained(collector_names, type_constraint, constraint=None, format_list=None, subsystem_name=None)

glideinwms.frontend.glideinFrontendLib.getCondorStatusNonDynamic(status_dict)

Return a dictionary of collectors containing static+partitionable slots and exclude any dynamic slots

Each element is a condorStatus Use the output of getCondorStatus

glideinwms.frontend.glideinFrontendLib.getCondorStatusSchedds(collector_names, constraint=None, format_list=None, want_format_completion=True)

glideinwms.frontend.glideinFrontendLib.getFactoryEntryList(status_dict)

glideinwms.frontend.glideinFrontendLib.getFailedCondorStatus(status_dict)

glideinwms.frontend.glideinFrontendLib.getGlideinCpusNum(glidein, estimate_cpus=True)

Given the glidein data structure, get the GLIDEIN_CPUS and GLIDEIN_ESTIMATED_CPUS configured. If estimate_cpus is false translate keywords to numerical equivalent (auto/slot -> -1, node -> 0), otherwise estimate CPUs If GLIDEIN_CPUS is not configured ASSUME it to be 1, if it is set to auto/slot/-1 or node/0, use GLIDEIN_ESTIMATED_CPUS if provided, otherwise ASSUME it to be 1 In the future there should be better guesses

glideinwms.frontend.glideinFrontendLib.getGlideinNodesNum(glidein, estimate_nodes=True)

Given the glidein data structure, get the GLIDEIN_NODES configured. If estimate_nodes is false translate keywords to numerical equivalent (and raise ValueError if no valid keyword), otherwise estimate nodes. If GLIDEIN_NODES is not configured, ASSUME it to be 1 Currently no keyword is allowed. estimate_nodes is there for future expansions.

glideinwms.frontend.glideinFrontendLib.getHACheckInterval(frontend_data)

Given the frontendDescript return if this frontend is to be run in ‘master’ or ‘slave’ mode

glideinwms.frontend.glideinFrontendLib.getHAMode(frontend_data)

Given the frontendDescript return if this frontend is to be run in ‘master’ or ‘slave’ mode

glideinwms.frontend.glideinFrontendLib.getHASettings(frontend_data)

glideinwms.frontend.glideinFrontendLib.getIdleCondorQ(condorq_dict)

glideinwms.frontend.glideinFrontendLib.getIdleCondorStatus(status_dict)

glideinwms.frontend.glideinFrontendLib.getIdleCoresCondorStatus(status_dict)

glideinwms.frontend.glideinFrontendLib.getIdleVomsCondorQ(condorq_dict)

glideinwms.frontend.glideinFrontendLib.getOldCondorQ(condorq_dict, min_age)

glideinwms.frontend.glideinFrontendLib.getRunningCondorQ(condorq_dict)

glideinwms.frontend.glideinFrontendLib.getRunningCondorStatus(status_dict)

Return a dictionary of collectors containing running(claimed) slots Each element is a condorStatus

Parameters:

status_dict – output of getCondorStatus

Returns:

dictionary of collectors containing running(claimed) slots

glideinwms.frontend.glideinFrontendLib.getRunningCoresCondorStatus(status_dict)

glideinwms.frontend.glideinFrontendLib.getRunningJobsCondorStatus(status_dict)

Return a dictionary of collectors containing running(claimed) slots This includes Fixed slots and Dynamic slots (no partitionable slots) Each one is matched with a single job (gives number of running jobs) Each element is a condorStatus

Parameters:

status_dict – output of getCondorStatus

Returns:

dictionary of collectors containing running(claimed) slots

glideinwms.frontend.glideinFrontendLib.getRunningPSlotCondorStatus(status_dict)

Return a dictionary of collectors containing running(claimed) partitionable slots Each element is a condorStatus

Parameters:

status_dict – output of getCondorStatus

Returns:

collectors containing running(claimed) partitionable slots

glideinwms.frontend.glideinFrontendLib.hashJob(condorq_el, condorq_match_list=None)

glideinwms.frontend.glideinFrontendLib.uniqueSets(in_sets)

glideinwms.frontend.glideinFrontendMonitorAggregator module

class glideinwms.frontend.glideinFrontendMonitorAggregator.MonitorAggregatorConfig

Bases: object

config_frontend(monitor_dir, groups)

glideinwms.frontend.glideinFrontendMonitorAggregator.aggregateStatus()

glideinwms.frontend.glideinFrontendMonitorAggregator.verifyHelper(filename, dict, fix_rrd=False)

Helper function for verifyRRD. Checks one file, prints out errors. if fix_rrd, will attempt to dump out rrd to xml, add the missing attributes, then restore. Original file is obliterated.

@param filename: filename of rrd to check @param dict: expected dictionary @param fix_rrd: if true, will attempt to add missing attrs

glideinwms.frontend.glideinFrontendMonitorAggregator.verifyRRD(fix_rrd=False)

Go through all known monitoring rrds and verify that they match existing schema (could be different if an upgrade happened) If fix_rrd is true, then also attempt to add any missing attributes.

glideinwms.frontend.glideinFrontendMonitorAggregator.write_one_rrd(name, updated, data, fact=0)

glideinwms.frontend.glideinFrontendMonitoring module

class glideinwms.frontend.glideinFrontendMonitoring.MonitoringConfig

Bases: object

establish_dir(relative_dname)

write_file(relative_fname, output_str)

write_rrd_multi(relative_fname, ds_type, time, val_dict, min_val=None, max_val=None)

Create a RRD file, using rrdtool.

class glideinwms.frontend.glideinFrontendMonitoring.factoryStats

Bases: object

get_data()

get_total()

get_xml_data(indent_tab='   ', leading_tab='')

get_xml_total(indent_tab='   ', leading_tab='')

get_xml_updated(indent_tab='   ', leading_tab='')

logClientMonitor(client_name, client_monitor, client_internals)

client_monitor is a dictinary of monitoring info client_internals is a dictinary of internals

At the moment, it looks only for

‘Idle’ ‘Running’ ‘GlideinsIdle’ ‘GlideinsRunning’ ‘GlideinsTotal’ ‘LastHeardFrom’

logJobs(client_name, qc_status)

logRequest(client_name, requests, params)

requests is a dictinary of requests params is a dictinary of parameters

At the moment, it looks only for

‘IdleGlideins’ ‘MaxRunningGlideins’

write_file()

class glideinwms.frontend.glideinFrontendMonitoring.groupStats

Bases: object

get_factories_data()

get_factory_dict(factory)

get_states_data()

get_total()

get_xml_factories_data(indent_tab='   ', leading_tab='')

get_xml_states_data(indent_tab='   ', leading_tab='')

get_xml_total(indent_tab='   ', leading_tab='')

get_xml_updated(indent_tab='   ', leading_tab='')

logFactAttrs(factory, attrs, blacklist)

logFactDown(factory, isDown)

logFactReq(factory, reqIdle, reqMaxRun, params)

logGlideins(slots_data)

logJobs(jobs_data)

logMatchedGlideins(factory, total, idle, running, failed, totalcores, idlecores, runningcores)

logMatchedJobs(factory, idle, effIdle, oldIdle, running, realRunning)

write_file()

write_one_rrd(name, data, fact=0)

glideinwms.frontend.glideinFrontendMonitoring.sanitize(name)

glideinwms.frontend.glideinFrontendMonitoring.tmp2final(fname)

This exact method is also in glideFactoryMonitoring.py

glideinwms.frontend.glideinFrontendMonitoring.write_frontend_descript_xml(frontendDescript, monitor_dir)

Writes out the frontend descript.xml file in the monitor web area.

@type frontendDescript: FrontendDescript @param frontendDescript: contains the data in the frontend.descript file in the frontend instance dir @type monitor_dir: string @param monitor_dir: filepath the the monitor dir in the frontend instance dir

glideinwms.frontend.glideinFrontendPidLib module

class glideinwms.frontend.glideinFrontendPidLib.ElementPidSupport(startup_dir, group_name)

Bases: PidWParentSupport

class glideinwms.frontend.glideinFrontendPidLib.FrontendPidSupport(startup_dir)

Bases: PidSupport

format_pid_file_content()

parse_pid_file_content(lines)

register(action_type, pid=None, started_time=None)

See parent for full description We add action_type here

Parameters:

• action_type –

• pid – if None, will default to os.getpid()

• started_time – if None, use time.time()

reset_to_default()

glideinwms.frontend.glideinFrontendPidLib.get_element_pid(startup_dir, group_name)

Raise an exception if not running

Parameters:

• startup_dir –

• group_name –

Returns:

Raises:

RuntimeError – if the Group element process is not running or has no parent

glideinwms.frontend.glideinFrontendPidLib.get_frontend_action_type(startup_dir)

Get the action type (). Raise an exception if not running

Parameters:

startup_dir –

Returns:

Raises:

RuntimeError – if the Frontend is not running

glideinwms.frontend.glideinFrontendPidLib.get_frontend_pid(startup_dir)

Return the Frontend pid. Raise an exception if not running

Parameters:

startup_dir –

Returns:

Raises:

RuntimeError – if the Frontend is not running or is unable to find the pid

glideinwms.frontend.glideinFrontendPlugins module

glideinwms.frontend.gwms_renew_proxies module

Automatical renewal of proxies necessary for a glideinWMS frontend

exception glideinwms.frontend.gwms_renew_proxies.ConfigError

Bases: BaseException

Catch-all class for errors in proxies.ini or system VO configuration

class glideinwms.frontend.gwms_renew_proxies.Proxy(cert, key, output, lifetime, uid=0, gid=0, rfc=True, pathlength='20', bits='2048')

Bases: object

Class for holding information related to the proxy

_voms_proxy_info(*opts)

Run voms-proxy-info. Returns stdout, stderr, and return code of voms-proxy-info

actimeleft()

Safely return the remaining lifetime of the proxy’s VOMS AC, in seconds (returns 0 if unexpected stdout)

cleanup()

Cleanup temporary proxy files

timeleft()

Safely return the remaining lifetime of the proxy, in seconds (returns 0 if unexpected stdout)

write()

Move output proxy from temp location to its final destination

class glideinwms.frontend.gwms_renew_proxies.VO(vo, fqan)

Bases: object

Class for holding information related to VOMS attributes

glideinwms.frontend.gwms_renew_proxies._run_command(command)

Runs the specified command, specified as a list. Returns stdout, stderr and return code

glideinwms.frontend.gwms_renew_proxies._safe_int(string_var)

Convert a string to an integer. If the string cannot be cast, return 0.

glideinwms.frontend.gwms_renew_proxies.main()

Main entrypoint

glideinwms.frontend.gwms_renew_proxies.parse_vomses(vomses_contents)

Parse the contents of a vomses file with the the following format per line:

“<VO ALIAS> ” “<VOMS ADMIN HOSTNAME>” “<VOMS ADMIN PORT>” “<VOMS CERT DN>” “<VO NAME>”

And return two mappings:

1. Case insensitive VO name to their canonical versions

2. VO certificate DN to URI, i.e. HOSTNAME:PORT

glideinwms.frontend.gwms_renew_proxies.voms_proxy_fake(proxy, vo_info)

Create a valid proxy without contacting a VOMS Admin server. VOMS attributes are created from user config. Returns stdout, stderr, and return code of voms-proxy-fake

glideinwms.frontend.gwms_renew_proxies.voms_proxy_init(proxy, voms_attr=None)

Create a proxy using voms-proxy-init, using the proxy information and optionally VOMS attribute. Returns stdout, stderr, and return code of voms-proxy-init

glideinwms.frontend.manageFrontendDowntimes module

glideinwms.frontend.manageFrontendDowntimes.add(opt_dict)

glideinwms.frontend.manageFrontendDowntimes.delay2time(delayStr)

glideinwms.frontend.manageFrontendDowntimes.down(opt_dict)

glideinwms.frontend.manageFrontendDowntimes.get_args(argv)

glideinwms.frontend.manageFrontendDowntimes.get_downtime_fd(work_dir)

glideinwms.frontend.manageFrontendDowntimes.main(argv)

glideinwms.frontend.manageFrontendDowntimes.printtimes(opt_dict)

glideinwms.frontend.manageFrontendDowntimes.str2time(timeStr)

glideinwms.frontend.manageFrontendDowntimes.strtxt2time(timeStr)

glideinwms.frontend.manageFrontendDowntimes.up(opt_dict)

glideinwms.frontend.manageFrontendDowntimes.usage()

glideinwms.frontend.stopFrontend module

glideinwms.frontend.stopFrontend.get_element_pids(work_dir, frontend_pid)

glideinwms.frontend.stopFrontend.main(work_dir, force=False)

Module contents