glideinwms.factory package

Subpackages

• glideinwms.factory.tools package
  • Subpackages
    • glideinwms.factory.tools.lib package
      • Submodules
      • glideinwms.factory.tools.lib.analyze module
      • glideinwms.factory.tools.lib.gWftArgsHelper module
      • glideinwms.factory.tools.lib.gWftLogParser module
      • Module contents
  • Submodules
  • glideinwms.factory.tools.OSG_autoconf module
  • glideinwms.factory.tools.cat_MasterLog module
  • glideinwms.factory.tools.cat_StartdHistoryLog module
  • glideinwms.factory.tools.cat_StartdLog module
  • glideinwms.factory.tools.cat_StarterLog module
  • glideinwms.factory.tools.cat_XMLResult module
  • glideinwms.factory.tools.cat_logs module
  • glideinwms.factory.tools.cat_named_log module
  • glideinwms.factory.tools.find_StartdLogs module
  • glideinwms.factory.tools.find_logs module
  • glideinwms.factory.tools.gfdiff module
  • glideinwms.factory.tools.manual_glidein_submit module
  • Module contents

Submodules

glideinwms.factory.checkFactory module

glideinwms.factory.glideFactory module

glideinwms.factory.glideFactoryConfig module

class glideinwms.factory.glideFactoryConfig.ConfigFile(config_file, convert_function=<built-in function repr>)

Bases: object

has_key(key_name)

load(fname, convert_function)

class glideinwms.factory.glideFactoryConfig.EntryConfigFile(entry_name, config_file, convert_function=<built-in function repr>)

Bases: ConfigFile

class glideinwms.factory.glideFactoryConfig.FactoryConfig

Bases: object

class glideinwms.factory.glideFactoryConfig.FrontendDescript

Bases: ConfigFile

Contains the security identity and username mappings for the Frontends that are authorized to use this factory.

Contains dictionary of dictionaries: obj.data[frontend][‘ident’]=identity obj.data[frontend][‘usermap’][sec_class]=username

get_all_frontend_sec_classes()

Get a list of all frontend:sec_class

get_all_usernames()

Gets all the usernames assigned to all the frontends.

@return: list of usernames

get_frontend_name(identity)

Get the frontend:sec_class mapping for the given identity

get_identity(frontend)

Gets the identity for the given frontend. If the Frontend is unknown, returns None.

@type frontend: string @param frontend: frontend name

@return identity

get_username(frontend, sec_class)

Gets the security name mapping for the given frontend and security class. If not found or not authorized, returns None.

@type frontend: string @param frontend: frontend name @type sec_class: string @param sec_class: security class name

@return: security name

class glideinwms.factory.glideFactoryConfig.GlideinDescript

Bases: ConfigFile

backup_and_load_old_key()

Backup existing key and load the key object

backup_rsa_key()

Backup existing rsa key.

load_old_rsa_key()

Load the old key object.

load_pub_key(recreate=False)

Load the key object. Create the key if required

@type recreate: bool @param recreate: Create a new key overwriting the old one. Defaults to False

remove_old_key()

class glideinwms.factory.glideFactoryConfig.GlideinKey(pub_key_type, key_fname=None, recreate=False)

Bases: object

extract_sym_key(enc_sym_key)

Extracts the symkey from encrypted fronted attribute

Parameters:

enc_sym_key (AnyStrASCII) – encrypted symmetric key

Returns:

symmetric key (SymKey child object)

Return type:

SymKey

Raises:

RuntimeError – if the key type is not RSA

get_pub_key_id()

get_pub_key_type()

get_pub_key_value()

load(key_fname=None, recreate=False)

Create the key if required and initialize it

Parameters:

• key_fname (str) – Filename of the key

• recreate (bool) – Create a new key if True else load existing key. Defaults to False.

Raises:

RuntimeError – if asking for a key type different from RSA

class glideinwms.factory.glideFactoryConfig.JobAttributes(entry_name)

Bases: JoinConfigFile

class glideinwms.factory.glideFactoryConfig.JobDescript(entry_name)

Bases: EntryConfigFile

class glideinwms.factory.glideFactoryConfig.JobParams(entry_name)

Bases: JoinConfigFile

class glideinwms.factory.glideFactoryConfig.JoinConfigFile(entry_name, config_file, convert_function=<built-in function repr>)

Bases: ConfigFile

class glideinwms.factory.glideFactoryConfig.SignatureFile

Bases: ConfigFile

load(fname, convert_function)

Load the signatures.sha1 file into the class as a dictionary. The convert_function is completely ignored here. The line format is different from all the other class in that there are three values with the key being the last value. The internal dictionary has the following structure:

where:

line[0] is the sign for the line line[1] is the descript file for the line line[2] is the key for the line

for each line:

line[2]_sign = line[0] line[2]_descript = line[1]

glideinwms.factory.glideFactoryCredentials module

exception glideinwms.factory.glideFactoryCredentials.CredentialError

Bases: Exception

defining new exception so that we can catch only the credential errors here and let the “real” errors propagate up

class glideinwms.factory.glideFactoryCredentials.SubmitCredentials(username, security_class)

Bases: object

Data class containing all information needed to submit a glidein.

add_factory_credential(cred_type, absfname)

Adds a factory provided security credential.

add_identity_credential(cred_type, cred_str)

Adds an identity credential.

add_security_credential(cred_type, filename)

Adds a security credential.

glideinwms.factory.glideFactoryCredentials.check_security_credentials(auth_method, params, client_int_name, entry_name, scitoken_passthru=False)

Verify that only credentials for the given auth method are in the params

Parameters:

• auth_method – (string): authentication method of an entry, defined in the config

• params – (dictionary): decrypted params passed in a frontend (client) request

• client_int_name (string) – internal client name

• entry_name – (string): name of the entry

• scitoken_passthru – (bool): if True, scitoken present in credential. Override checks for ‘auth_method’ and proceded with glidein request

Raises:

CredentialError – if the credentials in params don’t match what is defined for the auth method

glideinwms.factory.glideFactoryCredentials.compress_credential(credential_data)

glideinwms.factory.glideFactoryCredentials.get_globals_classads(factory_collector='default')

glideinwms.factory.glideFactoryCredentials.get_key_obj(pub_key_obj, classad)

Gets the symmetric key object from the request classad

@type pub_key_obj: object @param pub_key_obj: The factory public key object. This contains all the encryption and decryption methods @type classad: dictionary @param classad: a dictionary representation of the classad

glideinwms.factory.glideFactoryCredentials.process_global(classad, glidein_descript, frontend_descript)

glideinwms.factory.glideFactoryCredentials.safe_update(fname, credential_data)

glideinwms.factory.glideFactoryCredentials.update_credential_file(username, client_id, credential_data, request_clientname)

Updates the credential file

Parameters:

• username – credentials’ username

• client_id – id used for tracking the submit credentials

• credential_data – the credentials to be advertised

• request_clientname – client name passed by frontend

:return:the credential file updated

glideinwms.factory.glideFactoryCredentials.validate_frontend(classad, frontend_descript, pub_key_obj)

Validates that the frontend advertising the classad is allowed and that it claims to have the same identity that Condor thinks it has.

@type classad: dictionary @param classad: a dictionary representation of the classad @type frontend_descript: class object @param frontend_descript: class object containing all the frontend information @type pub_key_obj: object @param pub_key_obj: The factory public key object. This contains all the encryption and decryption methods

@return: sym_key_obj - the object containing the symmetric key used for decryption @return: frontend_sec_name - the frontend security name, used for determining the username to use.

glideinwms.factory.glideFactoryDowntimeLib module

class glideinwms.factory.glideFactoryDowntimeLib.DowntimeFile(fname)

Bases: object

addPeriod(start_time, end_time, entry='All', frontend='All', security_class='All', comment='', create_if_empty=True)

checkDowntime(entry='Any', frontend='Any', security_class='Any', check_time=None)

endDowntime(end_time=None, entry='All', frontend='All', security_class='All', comment='')

printDowntime(entry='Any', check_time=None)

purgeOldPeriods(cut_time=None, raise_on_error=False)

read(raise_on_error=False)

startDowntime(start_time=None, end_time=None, entry='All', frontend='All', security_class='All', comment='', create_if_empty=True)

glideinwms.factory.glideFactoryDowntimeLib.addPeriod(fname, start_time, end_time, entry='All', frontend='All', security_class='All', comment='', create_if_empty=True)

glideinwms.factory.glideFactoryDowntimeLib.checkDowntime(fname, entry='Any', frontend='Any', security_class='Any', check_time=None)

glideinwms.factory.glideFactoryDowntimeLib.endDowntime(fname, end_time=None, entry='All', frontend='All', security_class='All', comment='')

glideinwms.factory.glideFactoryDowntimeLib.printDowntime(fname, entry='Any', check_time=None)

glideinwms.factory.glideFactoryDowntimeLib.purgeOldPeriods(fname, cut_time=None, raise_on_error=False)

glideinwms.factory.glideFactoryDowntimeLib.read(fname, raise_on_error=False)

glideinwms.factory.glideFactoryEntry module

glideinwms.factory.glideFactoryEntryGroup module

glideinwms.factory.glideFactoryInterface module

class glideinwms.factory.glideFactoryInterface.EntryClassad(factory_name, glidein_name, entry_name, trust_domain, auth_method, supported_signtypes, pub_key_obj=None, glidein_attrs={}, glidein_params={}, glidein_monitors={}, glidein_stats={}, glidein_web_attrs={}, glidein_config_limits={})

Bases: Classad

This class describes the glidefactory classad. Factory advertises the glidefactory classad to the user pool as an UPDATE_AD_GENERIC type classad

class glideinwms.factory.glideFactoryInterface.FactoryConfig

Bases: object

class glideinwms.factory.glideFactoryInterface.FactoryGlobalClassad(factory_name, glidein_name, supported_signtypes, pub_key_obj)

Bases: Classad

This class describes the glidefactoryglobal classad. Factory advertises the glidefactoryglobal classad to the user pool as an UPDATE_AD_GENERIC type classad

glidefactory and glidefactoryglobal classads must be of the same type because they may be invalidated together (with a single command)

class glideinwms.factory.glideFactoryInterface.MultiAdvertizeGlideinClientMonitoring(factory_name, glidein_name, entry_name, glidein_attrs, factory_collector='default')

Bases: object

add(client_name, client_int_name, client_int_req, client_params={}, client_monitors={}, limits_triggered={})

do_advertize()

do_advertize_iterate()

do_advertize_multi()

writeToMultiClassadFile(filename=None, append=True)

exception glideinwms.factory.glideFactoryInterface.MultiExeError(arr)

Bases: ExeError

glideinwms.factory.glideFactoryInterface._remove_if_there(fname)

Remove the file and ignore errors (e.g. file not there)

glideinwms.factory.glideFactoryInterface.advertizeGlideinClientMonitoring(factory_name, glidein_name, entry_name, client_name, client_int_name, client_int_req, glidein_attrs={}, client_params={}, client_monitors={}, factory_collector='default')

glideinwms.factory.glideFactoryInterface.advertizeGlideinClientMonitoringFromFile(fname, remove_file=True, is_multi=False, factory_collector='default')

glideinwms.factory.glideFactoryInterface.advertizeGlideinFromFile(fname, remove_file=True, is_multi=False, factory_collector='default')

glideinwms.factory.glideFactoryInterface.advertizeGlobal(factory_name, glidein_name, supported_signtypes, pub_key_obj, stats_dict={}, factory_collector='default')

Creates the glidefactoryglobal classad and advertises.

@type factory_name: string @param factory_name: the name of the factory @type glidein_name: string @param glidein_name: name of the glidein @type supported_signtypes: string @param supported_signtypes: suppported sign types, i.e. sha1 @type pub_key_obj: GlideinKey @param pub_key_obj: for the frontend to use in encryption @type stats_dict: dict @param stats_dict: completed jobs statistics @type factory_collector: string or None @param factory_collector: the collector to query, special value ‘default’ will get it from the global config

@todo add factory downtime?

glideinwms.factory.glideFactoryInterface.createGlideinClientMonitoringFile(fname, factory_name, glidein_name, entry_name, client_name, client_int_name, client_int_req, glidein_attrs={}, client_params={}, client_monitors={}, limits_triggered={}, do_append=False)

glideinwms.factory.glideFactoryInterface.deadvertizeAllGlideinClientMonitoring(factory_name, glidein_name, entry_name, factory_collector='default')

Deadvertize monitoring classads for the given entry.

glideinwms.factory.glideFactoryInterface.deadvertizeFactory(factory_name, glidein_name, factory_collector='default')

Deadvertize all entry and global classads for this factory.

glideinwms.factory.glideFactoryInterface.deadvertizeFactoryClientMonitoring(factory_name, glidein_name, factory_collector='default')

Deadvertize all monitoring classads for this factory.

glideinwms.factory.glideFactoryInterface.deadvertizeGlidein(factory_name, glidein_name, entry_name, factory_collector='default')

Removes the glidefactory classad advertising the entry from the WMS Collector.

glideinwms.factory.glideFactoryInterface.deadvertizeGlobal(factory_name, glidein_name, factory_collector='default')

Removes the glidefactoryglobal classad advertising the factory globals from the WMS Collector.

glideinwms.factory.glideFactoryInterface.exe_condor_advertise(fname, command, is_multi=False, factory_collector=None)

glideinwms.factory.glideFactoryInterface.findGroupWork(factory_name, glidein_name, entry_names, supported_signtypes, pub_key_obj=None, additional_constraints=None, factory_collector='default')

Find request classAds that have my (factory, glidein name, entries) and create the dictionary of dictionary of work request information. Example: work[entry_name][frontend] = {‘params’:’value’, ‘requests’:’value}

@type factory_name: string @param factory_name: name of the factory

@type glidein_name: string @param glidein_name: name of the glidein instance

@type entry_names: list @param entry_names: list of factory entry names

@type supported_signtypes: list @param supported_signtypes: only support one kind of signtype, ‘sha1’, default is None

@type pub_key_obj: string @param pub_key_obj: only support ‘RSA’, defaults to None

@type additional_constraints: string @param additional_constraints: any additional constraints to include for querying the WMS collector, default is None

@type factory_collector: string or None @param factory_collector: the collector to query, special value ‘default’ will get it from the global config

@rtype: dict @return: Dictionary of work to perform. Return format is work[entry_name][frontend] = {‘params’:’value’, ‘requests’:’value}

glideinwms.factory.glideFactoryInterface.findWork(factory_name, glidein_name, entry_name, supported_signtypes, pub_key_obj=None, additional_constraints=None, factory_collector='default')

Find request classAds that have my (factory, glidein name, entry name) and create the dictionary of work request information.

@type factory_name: string @param factory_name: name of the factory @type glidein_name: string @param glidein_name: name of the glidein instance @type entry_name: string @param entry_name: name of the factory entry @type supported_signtypes: list @param supported_signtypes: only support one kind of signtype, ‘sha1’, default is None @type pub_key_obj: string @param pub_key_obj: only support ‘RSA’ @type additional_constraints: string @param additional_constraints: any additional constraints to include for querying the WMS collector, default is None

@type factory_collector: string or None @param factory_collector: the collector to query, special value ‘default’ will get it from the global config

@return: dictionary, each key is the name of a frontend. Each value has a ‘requests’ and a ‘params’ key. Both refer to classAd dictionaries.

glideinwms.factory.glideFactoryInterface.workGroupByEntries(work)

Given the dictionary of work items, group the work based on the entry Example: grouped_work[entry][w]

glideinwms.factory.glideFactoryLib module

class glideinwms.factory.glideFactoryLib.ClientWeb(client_web_url, client_signtype, client_descript, client_sign, client_group, client_group_web_url, client_group_descript, client_group_sign, factoryConfig=None)

Bases: object

get_glidein_args()

class glideinwms.factory.glideFactoryLib.FactoryConfig

Bases: object

config_dirs(submit_dir, log_base_dir, client_log_base_dir, client_proxies_base_dir)

config_remove_freq(sleepBetweenRemoves, maxRemovesXCycle)

config_submit_freq(sleepBetweenSubmits, maxSubmitsXCycle)

config_whoamI(factory_name, glidein_name)

get_client_log_dir(entry_name, username)

get_client_proxies_dir(username)

class glideinwms.factory.glideFactoryLib.GlideinTotals(entry_name, frontendDescript, jobDescript, entry_condorQ, log=None)

Bases: object

Keeps track of all glidein totals.

add_idle_glideins(nr_glideins, frontend_name)

Updates the totals with the additional glideins.

can_add_idle_glideins(nr_glideins, frontend_name, log=None, factoryConfig=<glideinwms.factory.glideFactoryLib.FactoryConfig object>)

Determines how many more glideins can be added. Does not compare against request max_glideins. Does not update totals.

get_max_held(frontend_name)

Returns max held for the given frontend:sec_class.

has_entry_exceeded_max_glideins()

has_entry_exceeded_max_held()

has_entry_exceeded_max_idle()

has_sec_class_exceeded_max_held(frontend_name)

Compares the current held for a security class to the security class limit.

glideinwms.factory.glideFactoryLib.clean_glidein_queue(remove_excess_tp, glidein_totals, condorQ, req_min_idle, req_max_glideins, frontend_name, log=None, factoryConfig=None)

Cleans up the glideins queue (removes any excesses) per the frontend request.

We are not adjusting the glidein totals with what has been removed from the queue. It may take a cycle (or more) for these totals to occur so it would be difficult to reflect the true state of the system.

TODO: req_min_idle=0 when remove_excess_tp.frontend_req_min_idle is not means that a limit was reached in the Factory

or some component (Factory/Entry) is in downtime. Check if the removal behavior should change

Parameters:

• remove_excess_tp (tuple) – remove_excess_str (NO, WAIT, IDLE, ALL), remove_excess_margin, frontend_req_min_idle remove_excess_str and remove_excess_margin are the removal request form the Frontend The frontend_req_min_idle item of the tuple indicates the original frontend pressure. We use this instead of req_min_idle for the IDLE pilot removal because the factory could set req_min_idle to 0 if an entry is in downtime, or the factory limits are reached. We do not want to remove idle pilots in these cases!

• glidein_totals (dict) – Number of Glideins in different states for each Frontend

• condorQ (dict) – Results of condor_q, classified

• req_min_idle – min_idle requested by the Frontend (NOT USED, used frontend_req_min_idle in remove_excess_tp instead to avoid Factory limits effects)

• req_max_glideins – max_glideins requested by the Frontend

• frontend_name (str) – Name of the Frontend, to use as key

• log (logging.Logger) – logger

• factoryConfig (FactoryConfig) – configuration object

Returns:

1 if some glideins were removed, 0 otherwise

Return type:

int

TODO:V could return the number of glideins removed

glideinwms.factory.glideFactoryLib.days2sec(days)

glideinwms.factory.glideFactoryLib.diffList(base_list, subtract_list)

glideinwms.factory.glideFactoryLib.env_list2dict(env, sep='=')

glideinwms.factory.glideFactoryLib.escapeParam(param_str)

glideinwms.factory.glideFactoryLib.executeSubmit(log, factoryConfig, username, schedd, exe_env, submitFile)

glideinwms.factory.glideFactoryLib.extractHeldSimple(q, factoryConfig=None)

All Held Glideins: JobStatus == 5

q: dictionary of Glideins from condor_q factoryConfig (FactoryConfig): Factory configuartion (NOT USED, for interface)

Returns:

dictionary of Held Glideins from condor_q

Return type:

dict

glideinwms.factory.glideFactoryLib.extractIdleQueued(q, factoryConfig=None)

All Idle Glideins already submitted: with hash_status 1xxx except 1001

hash_status 1xxx implies JobStatus 1

Parameters:

• q – dictionary of Glideins from condor_q

• factoryConfig (FactoryConfig) – Factory configuartion (NOT USED, for interface)

Returns:

dictionary of Idle and Submitted Glideins from condor_q

Return type:

dict

glideinwms.factory.glideFactoryLib.extractIdleSimple(q, factoryConfig=None)

All Idle Glideins: JobStatus == 1

q: dictionary of Glideins from condor_q factoryConfig (FactoryConfig): Factory configuartion (NOT USED, for interface)

Returns:

dictionary of Idle Glideins from condor_q

Return type:

dict

glideinwms.factory.glideFactoryLib.extractIdleUnsubmitted(q, factoryConfig=None)

All Idle Glideins not yet submitted (Unsubmitted, aka Waiting): with hash_status 1001

hash_status 1xxx implies JobStatus 1

q: dictionary of Glideins from condor_q factoryConfig (FactoryConfig): Factory configuartion (NOT USED, for interface)

Returns:

dictionary of Idle not Submitted Glideins from condor_q

Return type:

dict

glideinwms.factory.glideFactoryLib.extractJobId(submit_out)

Extracts the number of jobs and cluster id from a condor output.

Parameters:

submit_out (list) – Condor output. Expects a list of str.

Raises:

condorExe.ExeError – When it failts to apply a regular expression to a line of the output.

Returns:

Number of jobs and cluster id.

Return type:

tuple

glideinwms.factory.glideFactoryLib.extractNonRunSimple(q, factoryConfig=None)

All NOT Running Glideins: JobStatus != 2

q: dictionary of Glideins from condor_q factoryConfig (FactoryConfig): Factory configuartion (NOT USED, for interface)

Returns:

dictionary of Not Running Glideins from condor_q

Return type:

dict

glideinwms.factory.glideFactoryLib.extractRecoverableHeldSimple(q, factoryConfig=None)

glideinwms.factory.glideFactoryLib.extractRecoverableHeldSimpleWithinLimits(q, factoryConfig=None)

glideinwms.factory.glideFactoryLib.extractRunSimple(q, factoryConfig=None)

All Running Glideins: JobStatus == 2

q: dictionary of Glideins from condor_q factoryConfig (FactoryConfig): Factory configuartion (NOT USED, for interface)

Returns:

dictionary of Running Glideins from condor_q

Return type:

dict

glideinwms.factory.glideFactoryLib.extractRunStale(q, factoryConfig=None)

glideinwms.factory.glideFactoryLib.extractStaleSimple(q, factoryConfig=None)

glideinwms.factory.glideFactoryLib.extractUnrecoverableHeldForceX(q, factoryConfig=None)

glideinwms.factory.glideFactoryLib.extractUnrecoverableHeldSimple(q, factoryConfig=None)

glideinwms.factory.glideFactoryLib.getCondorQCredentialList(factoryConfig=None)

Returns a list of all currently used proxies based on the glideins in the queue.

glideinwms.factory.glideFactoryLib.getCondorQData(entry_name, client_name, schedd_name, factoryConfig=None)

Get Condor data, given the glidein name To be passed to the main functions if client_name=None, return all clients

glideinwms.factory.glideFactoryLib.getCondorStatusData(entry_name, client_name, pool_name=None, factory_startd_attribute=None, glidein_startd_attribute=None, entry_startd_attribute=None, client_startd_attribute=None, factoryConfig=None)

glideinwms.factory.glideFactoryLib.getQClientNames(condorq, factoryConfig=None)

Return a dictionary grouping the condorQ by client_names (factoryConfig.client_schedd_attribute)

Parameters:

condorq – condor_q query object from condorMonitor.CondorQ

Returns:

list of client names (factoryConfig.client_schedd_attribute)

glideinwms.factory.glideFactoryLib.getQCredentials(condorq, client_name, creds, client_sa, cred_secclass_sa, cred_id_sa)

Get the current queue status for a given client and credential (condorQ sub-query). v3 equivalent for getQProxySecClass

Parameters:

• condorq – condor_q query to select within

• client_name (param) – client name (e.g. the Frontend requesting the jobs)

• creds – credential object (to extract sec class and id)

• client_sa – schedd attribute to compare the client name

• cred_secclass_sa – schedd attribute to compare the security class

• cred_id_sa – schedd attribute to compare the credential ID

Returns:

sub-query with only the desired jobs

glideinwms.factory.glideFactoryLib.getQProxSecClass(condorq, client_name, proxy_security_class, client_schedd_attribute=None, credential_secclass_schedd_attribute=None, factoryConfig=None)

Get the current queue status for client and security class.

glideinwms.factory.glideFactoryLib.getQStatus(condorq)

Return a dictionary with detailed_jobStatus/numJobs, where detailed_jobStatus is returned by hash_status Idle jobs may be: 1001, 1002, 1010 depending on the GridJobStatus Running maybe: 2 or 4010 if in stage-out 1100 is used for unknown GridJobStatus

glideinwms.factory.glideFactoryLib.getQStatusSF(condorq)

Return a dictionary where keys are GlideinEntrySubmitFile(s) and values is a jobStatus/numJobs dict NOTE: this has not the same level of detail as getQStatus, e.g. Idle jobs are not split depending on GridJobStatus

glideinwms.factory.glideFactoryLib.getQStatusStale(condorq)

Return a dictionary with jobStatus, stale_info/numJobs, where stale_info is 1 if the status information is old

glideinwms.factory.glideFactoryLib.get_status_glideidx(el)

glideinwms.factory.glideFactoryLib.get_submit_environment(entry_name, client_name, submit_credentials, client_web, params, idle_lifetime, log=None, factoryConfig=None)

glideinwms.factory.glideFactoryLib.group_unclaimed(el_list)

glideinwms.factory.glideFactoryLib.hash_status(el)

glideinwms.factory.glideFactoryLib.hash_statusStale(el)

glideinwms.factory.glideFactoryLib.hrs2sec(hrs)

glideinwms.factory.glideFactoryLib.in_submit_environment(entry_name, exe_env)

glideinwms.factory.glideFactoryLib.isGlideinHeldNTimes(jobInfo, factoryConfig=None, n=20)

This function looks at the glidein job’s information and returns if the CondorG job is held for more than N(defaults to 20) iterations

This is useful to remove Unrecoverable glidein (CondorG job) with forcex option.

Parameters:

jobInfo (dict) – Dictionary containing glidein job’s classad information

Returns:

True if job is held more than N(defaults to 20) iterations, False if otherwise.

Return type:

bool

glideinwms.factory.glideFactoryLib.isGlideinUnrecoverable(jobInfo, factoryConfig=None, glideinDescript=None)

This function looks at the glidein job’s information and returns if the CondorG job is unrecoverable. Condor hold codes are available at: https://htcondor.readthedocs.io/en/v8_9_4/classad-attributes/job-classad-attributes.html

This is useful to change to status of glidein (CondorG job) from hold to idle.

In 3.6.2 the behavior of the function changed. Instead of having a list of unrecoverable codes in the function (that got outdated once gt was deprecated), we consider each code unrecoverable and give the operators the possibility of specify a list of recoverable codes in the config.

Parameters:

jobInfo (dict) – Dictionary containing glidein job’s classad information

Returns:

True if job is unrecoverable, False if recoverable

Return type:

bool

glideinwms.factory.glideFactoryLib.isGlideinWithinHeldLimits(jobInfo, factoryConfig=None)

This function looks at the glidein job’s information and returns if the CondorG job can be released.

This is useful to limit how often held jobs are released.

Parameters:

jobInfo (dict) – Dictionary containing glidein job’s classad information

Returns:

True if job is within limits, False if it is not

Return type:

bool

glideinwms.factory.glideFactoryLib.is_str_safe(s)

glideinwms.factory.glideFactoryLib.keepIdleGlideins(client_condorq, client_int_name, req_min_idle, req_max_glideins, idle_lifetime, remove_excess, submit_credentials, glidein_totals, frontend_name, client_web, params, log=None, factoryConfig=None)

Looks at the status of the queue and determines how many glideins to submit. Returns the number of newly submitted glideins.

If the system is unable to submit glideins because has reached one of the limits (request, entry, frontend:security_class), and the frontend asks for removal (RemoveExcess) in the request, it will try to remove excess glideins.

Parameters:

• client_condorq (CondorQ) – Condor queue filtered by security class

• client_int_name (str) – internal representation of the client name

• req_min_idle (int) – min number of idle glideins needed from the frontend request

• req_max_glideins (int) – max number of running glideins allowed in the frontend request

• idle_lifetimei (int) – how much to wait before removing glideins that are idle

• remove_excess (tuple) – remove_excess_str (NO, WAIT, IDLE, ALL), remove_excess_margin

• submit_credentials (SubmitCredentials) – all the information needed to submit the glideins

• glidein_totals (GlideinTotals) – entry and frontend glidein counts

• frontend_name (str) – frontend name, used to map frontend totals in glidein_totals (“frontend:sec_class”)

• client_web (glideFactoryLib.ClientWeb) – client web values

• params (dict) – params from the entry configuration or frontend to be passed to the glideins

• log (logger) – factory logger

• factoryConfig – factory configuration

Raises:

condorExe.ExeError – in case of issues executing condor commands

glideinwms.factory.glideFactoryLib.logStats(condorq, client_int_name, client_security_name, proxy_security_class, log=None, factoryConfig=None)

Sum to the current schedd statistics of this entry (from condor_q on the Factory) to the values already stored in factoryConfig.client_stats, factoryConfig.qc_stats

Parameters:

• condorq – condorQ object, containing a list of all jobs of the schedd (.data) for the entry invoking this

• client_int_name – client name (from the requestor/Frontend)

• client_security_name – security name used by the client

• proxy_security_class – credential security class used by the client

• log – to log errors/info/…

• factoryConfig – common data block for the entry to get schedd statistics (client_stats, qc_stats)

glideinwms.factory.glideFactoryLib.logStatsAll(condorq, log=None, factoryConfig=None)

Sum to the current schedd statistics of this entry (from condor_q on the Factory) to the values already stored in factoryConfig.client_stats, factoryConfig.qc_stats Do that for all the clients that have jobs on this entry

Parameters:

• condorq – condorQ object, containing a list of all jobs of the schedd (.data) for the entry invoking this

• log – to log errors/info/…

• factoryConfig – common data block for the entry to get schedd statistics (client_stats, qc_stats)

glideinwms.factory.glideFactoryLib.logWorkRequest(client_int_name, client_security_name, proxy_security_class, req_idle, req_max_run, remove_excess, work_el, fraction=1.0, log=None, factoryConfig=None)

Logs work requests

Parameters:

• client_int_name – client internal name

• client_security_name – client security name

• proxy_security_class – security ID

• req_idle – requested idle glideins

• req_max_run – max running glideins

• remove_excess – tuple, remove_excess_str, remove_excess_margin

• work_el – Work requests, temporary workaround; the requests should always be processed at the caller level

• fraction – fraction for this entry

• log –

• factoryConfig –

glideinwms.factory.glideFactoryLib.releaseGlideins(schedd_name, jid_list, log=None, factoryConfig=None)

glideinwms.factory.glideFactoryLib.removeGlideins(schedd_name, jid_list, force=False, log=None, factoryConfig=None)

glideinwms.factory.glideFactoryLib.sanitizeGlideins(condorq, log=None, factoryConfig=None)

glideinwms.factory.glideFactoryLib.schedd_name2str(schedd_name)

glideinwms.factory.glideFactoryLib.secClass2Name(client_security_name, proxy_security_class)

glideinwms.factory.glideFactoryLib.set_condor_integrity_checks()

glideinwms.factory.glideFactoryLib.submitGlideins(entry_name, client_name, nr_glideins, idle_lifetime, frontend_name, submit_credentials, client_web, params, status_sf, log=None, factoryConfig=None)

Submit the glidein

Parameters:

• entry_name (str) –

• client_name (str) –

• nr_glideins (int) –

• idle_lifetime (int) –

• frontend_name (str) –

• submit_credentials (dict) –

• client_web (client_web) – None means client did not pass one, backwards compatibility

• params –

• status_sf (dict) – keys are GlideinEntrySubmitFile(s) and values is a jobStatus/numJobs dict

• log –

• factoryConfig –

glideinwms.factory.glideFactoryLib.sum_idle_count(qc_status)

Add the summary of idle jobs to the statistics passed as input

Parameters:

qc_status – Query count summary with job_status/number_of_jobs

Returns:

Adds qc_status[1] to qc_status

glideinwms.factory.glideFactoryLib.update_x509_proxy_file(entry_name, username, client_id, proxy_data, factoryConfig=None)

glideinwms.factory.glideFactoryLib.which(program)

Implementation of which command in python.

Returns:

Path to the binary

Return type:

str

glideinwms.factory.glideFactoryLogParser module

class glideinwms.factory.glideFactoryLogParser.dirSummarySimple(obj)

Bases: object

dirSummary Simple

for now it is just a constructor wrapper Further on it will need to implement glidein exit code checks

diff(other)

merge(other)

mkTempLogObj()

class glideinwms.factory.glideFactoryLogParser.dirSummaryTimingsOut(dirname, cache_dir, client_name, user_name, inactive_files=None, inactive_timeout=86400)

Bases: cacheDirClass

This class uses a lambda function to initialize an instance of cacheDirClass. The function chooses all condor_activity files in a directory that correspond to a particular client.

get_simple()

class glideinwms.factory.glideFactoryLogParser.dirSummaryTimingsOutFull(dirname, cache_dir, inactive_files=None, inactive_timeout=86400)

Bases: cacheDirClass

This class uses a lambda function to initialize an instance of cacheDirClass. The function chooses all condor_activity files in a directory regardless of client name.

get_simple()

glideinwms.factory.glideFactoryLogParser.extractLogData(fname)

Given a filename of a job file “path/job.NUMBER.out” extract the statistics of the job duration, etc.

@param fname: Filename to extract @return: a dictionary with keys:

• glidein_duration - integer, how long did the glidein run

• validation_duration - integer, how long before starting condor

• condor_started - Boolean, did condor even start (if false, no further entries)

• condor_duration - integer, how long did Condor run

• stats - dictionary of stats (as in KNOWN_SLOT_STATS), each having

• jobsnr - integer, number of jobs started

• secs - integer, total number of secods used

For example {‘glidein_duration’:20305,’validation_duration’:6,’condor_started’ : 1, ‘condor_duration’: 20298, ‘stats’: {‘badSignal’: {‘secs’: 0, ‘jobsnr’: 0}, ‘goodZ’: {‘secs’ : 19481, ‘jobsnr’: 1}, ‘Total’: {‘secs’: 19481, ‘jobsnr’: 1}, ‘goodNZ’: {‘secs’: 0, ‘jobsnr’: 0}, ‘badOther’: {‘secs’: 0, ‘jobsnr’: 0}}}

class glideinwms.factory.glideFactoryLogParser.logSummaryTimingsOut(logname, cache_dir, username)

Bases: logSummaryTimings

Class logSummaryTimingsOut logs timing and status of a job. It declares a job complete only after the output file has been received The format is slightly different than the one of logSummaryTimings; we add the dirname in the job id When a output file is found, it adds a 4th parameter to the completed jobs See extractLogData below for more details

diff(other)

Diff self.data with other for use in comparing current iteration data with previous iteration.

Uses diff_raw to perform symmetric difference of self.data and other and puts it into data[status][‘Entered’|’Exited’] Completed jobs are augmented with data from the log

@return: data[status][‘Entered’|’Exited’] - list of jobs

diff_raw(other)

Diff self.data with other info, add glidein log data to Entered/Exited. Used to compare current data with previous iteration.

Uses symmetric difference of sets to compare the two dictionaries.

@type other: dictionary of statuses -> jobs @return: data[status][‘Entered’|’Exited’] - list of jobs

loadFromLog()

This class inherits from cachedLogClass. So, load() will first check the cached files. If changed, it will call this function. This uses the condorLogParser to load the log, then does some post-processing to check the job.NUMBER.out files to see if the job has finished and to extract some data.

class glideinwms.factory.glideFactoryLogParser.logSummaryTimingsOutWrapper

Bases: object

getObj(logname=None, cache_dir=None, username='all')

glideinwms.factory.glideFactoryMonitorAggregator module

class glideinwms.factory.glideFactoryMonitorAggregator.MonitorAggregatorConfig

Bases: object

config_factory(monitor_dir, entries, log)

glideinwms.factory.glideFactoryMonitorAggregator.aggregateJobsSummary()

Loads the job summary pickle files for each entry, aggregates them per schedd/collector pair, and return them. :return: A dictionary containing the needed information that looks like:

{

(‘schedd_name’,’collector_name’){

‘2994.000’: {‘condor_duration’: 1328, ‘glidein_duration’: 1334, ‘condor_started’: 1, ‘numjobs’: 0}, ‘2997.000’: {‘condor_duration’: 1328, ‘glidein_duration’: 1334, ‘condor_started’: 1, ‘numjobs’: 0}, …

}, (‘schedd_name’,’collector_name’) : {

‘2003.000’: {‘condor_duration’: 1328, ‘glidein_duration’: 1334, ‘condor_started’: 1, ‘numjobs’: 0}, ‘206.000’: {‘condor_duration’: 1328, ‘glidein_duration’: 1334, ‘condor_started’: 1, ‘numjobs’: 0}, …

}

}

glideinwms.factory.glideFactoryMonitorAggregator.aggregateLogSummary()

Create an aggregate of log summary files, write it in an aggregate log summary file and in the end return the values

glideinwms.factory.glideFactoryMonitorAggregator.aggregateRRDStats(log=None)

Create an aggregate of RRD stats, write it files

glideinwms.factory.glideFactoryMonitorAggregator.aggregateStatus(in_downtime)

Create an aggregate of status files, write it in an aggregate status file and in the end return the values

@type in_downtime: boolean @param in_downtime: Entry downtime information

@rtype: dict @return: Dictionary of status information

glideinwms.factory.glideFactoryMonitorAggregator.rrd_site(name)

glideinwms.factory.glideFactoryMonitorAggregator.sumDictInt(indict, outdict)

glideinwms.factory.glideFactoryMonitorAggregator.verifyHelper(filename, dict, fix_rrd=False)

Helper function for verifyRRD. Checks one file, prints out errors. if fix_rrd, will attempt to dump out rrd to xml, add the missing attributes, then restore. Original file is obliterated.

@param filename: filename of rrd to check @param dict: expected dictionary @param fix_rrd: if true, will attempt to add missing attrs

glideinwms.factory.glideFactoryMonitorAggregator.verifyRRD(fix_rrd=False)

Go through all known monitoring rrds and verify that they match existing schema (could be different if an upgrade happened) If fix_rrd is true, then also attempt to add any missing attributes.

glideinwms.factory.glideFactoryMonitorAggregator.writeLogSummaryRRDs(fe_dir, status_el)

glideinwms.factory.glideFactoryMonitoring module

class glideinwms.factory.glideFactoryMonitoring.Descript2XML(log=None)

Bases: object

create an XML file out of glidein.descript, frontend.descript, entry.descript, attributes.cfg, and params.cfg TODO: The XML is used by … “the monioring page”? The file created is descript.xml, w/ glideFactoryDescript and glideFactoryEntryDescript elements

entryDescript(e_dict)

frontendDescript(fe_dict)

getUpdated()

returns the time of last update

glideinDescript(g_dict)

writeFile(path, xml_str, singleEntry=False)

class glideinwms.factory.glideFactoryMonitoring.FactoryStatusData(log=None, base_dir=None)

Bases: object

this class handles the data obtained from the rrd files

average(input_list)

fetchData(rrd_file, pathway, res, start, end)

Uses rrdtool to fetch data from the clients. Returns a dictionary of lists of data. There is a list for each element.

rrdtool fetch returns 3 tuples: a[0], a[1], & a[2]. [0] lists the resolution, start and end time, which can be specified as arugments of fetchData. [1] returns the names of the datasets. These names are listed in the key. [2] is a list of tuples. each tuple contains data from every dataset. There is a tuple for each time data was collected.

getData(input_val, monitoringConfig=None)

Return the data fetched by rrdtool as a dictionary

This also modifies the rrd data dictionary for the client (input_val) in all RRD files and appends the client to the list of frontends

Where this side effect is used: - totals are updated in Entry.writeStats (writing the XML) - frontend data in check_and_perform_work

getUpdated()

returns the time of last update

getXMLData(rrd)

Return a XML formatted string the specific RRD file for the data fetched from a given site (all clients+total).

This also has side effects in the getData(self.total) invocation: - modifies the rrd data dictionary (all RRDs) for the total for this entry - and appends the total (self.total aka ‘total/’) to the list of clients (frontends)

@param rrd: @return: XML formatted string with stats data

writeFiles(monitoringConfig=None)

Write an xml file for the data fetched from a given site. Write rrd files

NOTE: writeFiles triggers the side effect of updating the rrd for totals (via getXMLData/getData)

@param monitoringConfig: @return: None

class glideinwms.factory.glideFactoryMonitoring.MonitoringConfig(log=None)

Bases: object

config_log(log_dir, max_days, min_days, max_mbs)

establish_dir(relative_dname)

logCompleted(client_name, entered_dict)

This function takes all newly completed glideins and logs them in logs/entry_Name/completed_jobs_date.log in an XML-like format.

It counts the jobs completed on a glidein but does not keep track of the cores received or used by the jobs

@type client_name: String @param client_name: the name of the frontend client @type entered_dict: Dictionary of dictionaries @param entered_dict: This is the dictionary of all jobs that have “Entered” the “Completed” states. It is indexed by job_id. Each data is an info dictionary containing the keys: username, jobs_duration (subkeys:total,goodput,terminated), wastemill (subkeys:validation,idle,nosuccess,badput) , duration, condor_started, condor_duration, jobsnr

rrd_obj

The name of the attribute that identifies the glidein

Type:

@ivar

write_completed_json(relative_fname, time, val_dict)

Write val_dict to a json file, creating if needed relative_fname: location of json relative to self.monitor_dir time: typically self.updated val_dict: dictionary object to be dumped to file

write_file(relative_fname, output_str)

Write out a string or bytes to a file

Parameters:

• relative_fname (AnyStr) – The relative path name to write out

• output_str (AnyStr) – the string (unicode str or bytes) to write to the file

write_rrd_multi(relative_fname, ds_type, time, val_dict, min_val=None, max_val=None)

Create a RRD file, using rrdtool.

write_rrd_multi_hetero(relative_fname, ds_desc_dict, time, val_dict)

Create a RRD file, using rrdtool. Like write_rrd_multi, but with each ds having each a specified type each element of ds_desc_dict is a dictionary with any of ds_type, min, max if ds_desc_dict[name] is not present, the defaults are {‘ds_type’:’GAUGE’, ‘min’:’U’, ‘max’:’U’}

class glideinwms.factory.glideFactoryMonitoring.condorLogSummary(log=None)

Bases: object

This class handles the data obtained from parsing the glidein log files

aggregate_frontend_data(updated, diff_summary)

This goes into each frontend in the current entry and aggregates the completed/stats/wastetime data into completed_data.json at the entry level

computeDiff()

This function takes the current_stats_data from the current iteration and the old_stats_data from the last iteration (see reset() function) to create a diff of the data in the stats_diff dictionary.

This stats_diff will be a dictionary with two entries for each status: “Entered” and “Exited” denoting which job ids have recently changed status, ie. stats_diff[frontend][username:client_int_name][“Completed”][“Entered”]

diffTimes(end_time, start_time)

get_completed_stats(entered_list)

get_data_summary()

Summarizes stats_diff data (computeDiff should have already been called) Sums over username in the dictionary stats_diff[frontend][username][entered/exited][status] to make stats_data[client_name][entered/exited][status]=count

@return: dictionary[client_name][entered/exited][status]=count

get_diff_summary()

Flattens stats_diff differential data.

@return: Dictionary of client_name with sub_keys Wait,Idle,Running,Held,Completed,Removed

get_diff_total()

get_stats_data_summary()

Summarizes current_stats_data: Adds up current_stats_data[frontend][user:client][status] across all username keys.

@return: returns dictionary stats_data[frontend][status]=count

get_stats_total()

@return: Dictionary with keys (wait,idle,running,held)

get_stats_total_summary()

get_total_summary()

get_xml_data(indent_tab='   ', leading_tab='')

get_xml_stats_data(indent_tab='   ', leading_tab='')

get_xml_stats_total(indent_tab='   ', leading_tab='')

get_xml_total(indent_tab='   ', leading_tab='')

get_xml_updated(indent_tab='   ', leading_tab='')

logSummary(client_name, stats)

log_stats taken during during an iteration of perform_work are added/merged into the condorLogSummary class here.

@type stats: dictionary of glideFactoryLogParser.dirSummaryTimingsOut @param stats: Dictionary keyed by “username:client_int_name” client_int_name is needed for frontends with multiple groups

reset()

Replaces old_stats_data with current_stats_data Sets current_stats_data to empty. This is called every iteration in order to later compare the diff of the previous iteration and current one to find any newly changed jobs (ie newly completed jobs)

summarize_completed_stats(entered_list)

write_file(monitoringConfig=None)

write_job_info(scheddName, collectorName)

The method itereates over the stats_diff dictionary looking for completed jobs and then fills out a dictionary that contains the monitoring information needed for this job. Those info looks like:

{

‘schedd_name’: ‘name’, ‘collector_name’: ‘name’, ‘joblist’ : {

‘2994.000’: {‘condor_duration’: 1328, ‘glidein_duration’: 1334, ‘condor_started’: 1, ‘numjobs’: 0, ‘2997.000’: {‘condor_duration’: 1328, ‘glidein_duration’: 1334, ‘condor_started’: 1, ‘numjobs’: 0 …

}

}

Parameters:

• scheddName – The schedd name to update the job

• collectorName – The collector name to update the job

class glideinwms.factory.glideFactoryMonitoring.condorQStats(log=None, cores=1)

Bases: object

aggregateStates(qc_status, el)

For each status in the condor_q count status dictionary (qc_status) add the count to the el dictionary (whose keys are state like ‘Idle’ instead of its number: 1)

finalizeClientMonitor()

static getEntryFromSubmitFile(submitFile)

Extract the entry name from submit files that look like: ‘entry_T2_CH_CERN/job.CMSHTPC_T2_CH_CERN_ce301.condor’

get_data()

get_total(history={'set_to_zero': False})

static get_xml_data(data, indent_tab='   ', leading_tab='')

Return a string with the XML formatted statistic data @param data: self.get_data() @param indent_tab: indentation space @param leading_tab: leading space @return: XML string

get_xml_downtime(leading_tab='   ')

static get_xml_total(total, indent_tab='   ', leading_tab='')

Return formatted XML for the total statistics @param total: self.get_total() @param indent_tab: indentation space @param leading_tab: leading space @return: XML string

get_xml_updated(indent_tab='   ', leading_tab='')

get_zero_data_element()

Return a dictionary with the keys defined in self.attributes, and all values to 0

Returns:

data element w/ all 0 values

logClientMonitor(client_name, client_monitor, client_internals, fraction=1.0)

client_monitor is a dictionary of monitoring info (GlideinMonitor… from glideclient ClassAd) client_internals is a dictionary of internals (from glideclient ClassAd) If fraction is specified it will be used to extract partial info

At the moment, it looks only for

‘Idle’ ‘Running’ ‘RunningHere’ ‘GlideinsIdle’, ‘GlideinsIdleCores’ ‘GlideinsRunning’, ‘GlideinsRunningCores’ ‘GlideinsTotal’, ‘GlideinsTotalCores’ ‘LastHeardFrom’

updates go in self.data (self.data[client_name][‘ClientMonitor’])

logRequest(client_name, requests)

requests is a dictinary of requests params is a dictinary of parameters

At the moment, it looks only for

‘IdleGlideins’ ‘MaxGlideins’

Request contains only that (no real cores info) It is eveluated using GLIDEIN_CPUS

logSchedd(client_name, qc_status, qc_status_sf)

Create or update a dictionary with aggregated HTCondor stats

client_name is the client requesting the glideins qc_status is a dictionary of condor_status:nr_jobs qc_status_sf is a dictionary of submit_file:qc_status OUTPUT: self.data[client_name][‘Status’] is the status for all Glideins

self.data[client_name][‘StatusEntries’] is the Glidein status by Entry

set_downtime(in_downtime)

write_file(monitoringConfig=None, alt_stats=None)

Calculate a summary for the entry and write statistics to files @param monitoringConfig: used to pass information from the Entry @param alt_stats: an alternative condorQStats object to use if self has no data @return:

glideinwms.factory.glideFactoryMonitoring.getAllJobRanges()

glideinwms.factory.glideFactoryMonitoring.getAllJobTypes()

glideinwms.factory.glideFactoryMonitoring.getAllMillRanges()

glideinwms.factory.glideFactoryMonitoring.getAllTimeRanges()

glideinwms.factory.glideFactoryMonitoring.getJobRange(absval)

glideinwms.factory.glideFactoryMonitoring.getLogCompletedDefaults()

glideinwms.factory.glideFactoryMonitoring.getMillRange(absval)

glideinwms.factory.glideFactoryMonitoring.getTimeRange(absval)

glideinwms.factory.glideFactoryMonitoring.get_completed_stats_xml_desc()

glideinwms.factory.glideFactoryPidLib module

class glideinwms.factory.glideFactoryPidLib.EntryGroupPidSupport(startup_dir, group_name)

Bases: PidWParentSupport

class glideinwms.factory.glideFactoryPidLib.EntryPidSupport(startup_dir, entry_name)

Bases: PidWParentSupport

class glideinwms.factory.glideFactoryPidLib.FactoryPidSupport(startup_dir)

Bases: PidSupport

glideinwms.factory.glideFactoryPidLib.get_entry_pid(startup_dir, entry_name)

glideinwms.factory.glideFactoryPidLib.get_entrygroup_pid(startup_dir, group_name)

glideinwms.factory.glideFactoryPidLib.get_factory_pid(startup_dir)

glideinwms.factory.glideFactorySelectionAlgorithms module

glideinwms.factory.glideFactorySelectionAlgorithms.selectionAlgoDefault(submit_files, status_sf, jobDescript, nr_glideins, log)

Given the list of sub entries (aka submit files), and the status of each sub entry (how many idle + running etc) figures out how many glideins to submit for each sub entry. 1) Shuffle the submit_files list 2) Try to “depth-wise” fill all the subentries untillimits are reached

@type submit_files: list @param submit_files: list of strings containing the name of the submit files for this entry set @type status_sf: dict @param status_sf: dictrionary where the keys are the submit files and the values is a condor states dict @type jobDescript: object @param jobDescript: will read here maximum number of idle/total fglideins for each sub entry @type nr_glideins: int @param nr_glideins: total number of glideins to submit to all the entries @type log: object @param log: logging object

Return a dictionary where keys are the submit files, and values are int indicating how many glideins to submit

glideinwms.factory.manageFactoryDowntimes module

glideinwms.factory.manageFactoryDowntimes.add(entry_name, opt_dict)

glideinwms.factory.manageFactoryDowntimes.check(entry_or_id, opt_dict)

glideinwms.factory.manageFactoryDowntimes.delay2time(delayStr)

glideinwms.factory.manageFactoryDowntimes.down(entry_name, opt_dict)

glideinwms.factory.manageFactoryDowntimes.get_args(argv)

glideinwms.factory.manageFactoryDowntimes.get_downtime_fd(entry_name, cmdname)

glideinwms.factory.manageFactoryDowntimes.get_downtime_fd_dict(entry_or_id, cmdname, opt_dict)

glideinwms.factory.manageFactoryDowntimes.get_entries(factory_dir)

glideinwms.factory.manageFactoryDowntimes.get_frontends(factory_dir)

glideinwms.factory.manageFactoryDowntimes.get_production_ress_entries(server, ref_dict_list)

glideinwms.factory.manageFactoryDowntimes.get_security_classes(factory_dir)

glideinwms.factory.manageFactoryDowntimes.infosys_based(entry_name, opt_dict, infosys_types)

glideinwms.factory.manageFactoryDowntimes.main(argv)

glideinwms.factory.manageFactoryDowntimes.printtimes(entry_or_id, opt_dict)

glideinwms.factory.manageFactoryDowntimes.str2time(timeStr)

glideinwms.factory.manageFactoryDowntimes.strtxt2time(timeStr)

glideinwms.factory.manageFactoryDowntimes.up(entry_name, opt_dict)

glideinwms.factory.manageFactoryDowntimes.usage()

glideinwms.factory.manageFactoryDowntimes.vacuum(entry_or_id, opt_dict)

glideinwms.factory.stopFactory module

glideinwms.factory.stopFactory.all_pids_in_pgid_dead(pgid)

glideinwms.factory.stopFactory.kill_and_check_pgid(pgid, signr=Signals.SIGTERM, retries=100, retry_interval=0.5)

glideinwms.factory.stopFactory.main(startup_dir, force=True)

Module contents