Glidein WMS Manual

VO Frontend

Description

The VO Frontend is, as the name suggests, the user interface of a glidein based WMS.

Overview

The main task of the VO frontend is to look for user jobs and ask the Glidein Factories to provide glideins, if needed. See the picture below for a general overview.

The VO frontend knows nothing about the glideins or Grid sites; the only information in has are the attributes that the various Glidein Factories publish. This makes the life of the frontend much easier, as it only has to worry about matching user job requirements to the factory attributes.

Implementation

The current implementation of the VO frontend is Python based. It uses a tree of process like the factory does.

The sub-process (group) code is composed of 4 logical pieces:

  • An element that holds the Glidein Factory Ads
  • An element that holds the user jobs Ads
  • A matchmaking procedure
  • A procedure that advertises the requests
  • ... and the glue that links the above together.

The top two elements simply contact the appropriate Condor Collector and store the data in memory.

The matchmaking procedure loops over all the factory ads and counts the number of idle user jobs that match the attributes of that particular factory entry point.

The matching expression is a Python expression, having as inputs

  • an object holding the job attributes, named job, and 

  • an object holding the entry point attributes, named glidein.

Both objects are dictionaries, with keys being attribute names, and values being attribute values.

A few examples:

job["CDFMaxHours"]<glidein["attrs"]["WMSMaxHours"]
job["HasData"] or (glidein["site"] in string.split(job["SitesWithRequiredData"]))
(job["Arch"] == glidein["Arch"]) and (job["gccVersion"]==glidein["attrs"]["gccVersion"])

Finally, the advertise procedure loops over the counts and converts them into appropriate Condor Class Ads.

The glue code is responsible of parsing the input parameters, loading the configuration values and loop over the above steps.

The frontend also can handle multiple proxies. The proxy selection is handled by a plugin mechanism.

The VO Frontend management

The VO Frontend configuration involves creating the configuration directory and files and then creating the daemons. As in the Glidein Factory set up, an XML file is converted into a configuration tree by a configuration tool.

For the installer to create the VO Frontend instance from the configuration directory and grid mapfile, the following objects can be defined:

  • <frontend frontend_name="your name" advertise_delay="seconds" loop_delay="nr" >
    The frontend_name is a combination of the frontend and instance names specified during installation. It is used to create VO Frontend instance directory and files. The delay parameters define how active the VO Frontend should be.

  • <frontend><match><factory><collectors><collector my_identity="joe@collector1.myorg.com" node="collector1.myorg.com" factory_identity="gfactory@gfactory1.my.org" DN="/DC=org/DC=doegrids/OU=Services/CN=collector/collectory1.my.org"/>
    This shows what WMS Collector the VO Frontend will map to. It is the mapped name for the identity of the classad. It alos tells what should be the idetnity used by the frontend itself and the expected identity of the factory

  • <frontend><match><job query_expr="expression" >
    A Condor constraint to be used with condor_q when looking for user jobs. (like 'JobUniverse=?=5', i.e. consider only Vanilla jobs). If you want to consider all user jobs, this can be set to TRUE.

  • <frontend><match><job><schedds><schedd fullname="schedd name" />
    When you provide the user pool collector to the installer, it will find all the available schedds. You can specify which schedds to monitor here for user jobs. The schedd fullname is the name under which schedd is registered with the user pool collector. 

  • <frontend><monitor base_dir="web_dir" flot_dir="web_dir" javascriptRRD_dir="web_dir" jquery_dir="web_dir" />
    The base_dir defines where the web monitoring is. The other entries point to where javascriptRRD, Flot and JQuery libraries are located.

  • <frontend><security proxy_DN="/DC=org/DC=doegrids/OU=Service/CN=frontend/frontend1.my.org" classad_proxy="proxy_dir" proxy_selection_plugin="ProxyAll" security_name="vofrontend1">
    Grid proxy to use is located in the classad_proxy directory and you must specify the full path to the proxy. security_name signifies the name under which the frontend is registered with the factory.

  • <frontend><security><proxies><proxy absfname="proxy" security_class="frontend"/>
    The location of the proxy the VO Frontend forwards to the factory for use in submitting glideins. You can have multiple proxy entries listed here.

  • <frontend><stage base_dir="web_dir" web_base_url="URL" />
    The location of the web server directories. It is the staging area where the files needed to run you job are located (i.e. condor). You may need to change this according to your requirements.

  • <frontend><work base_dir="directory" />
    This defines the path to the VO Frontend directory.

  • <frontend><files><file absfname="filepath" relfname="filename" executable="boolean" />
    The file parameters are used to specify the location of additional files. The grid mapfile location must be specified. This can be used to add custom scripts to the vo frontend. See below or the page dedicated to writing custom scripts for more information.

  • <frontend><attrs><attr name="attr_name" value="value" parameter="True" type="string" glidein_publish="boolean" job_publish="boolean" >
    The following three attributes are required to be set in the frontend requests and/or published for jobs. Others can be specified as well.

    • <attr name="GLIDEIN_Collector" >
      The contains the name of the pool collector, i.e. mymachine.mydomain.

    • <attr name="GLIDEIN_Expose_Grid_Env" >
      This determines if you want to expose the user to the grid environment.

    • <attr name="USE_MATCH_AUTH" >
      This determines whether or not you want to use match authentication. You specify the match expression in a group section of the config file.

    • <attr name="GLIDEIN_Glexec_Use" >
      This determines whether or not you want to mandate the use of GLEXEC. Possible values are NONE (do not use GLEXEC) or OPTIONAL (use GLEXEC if the site is configured with it) or REQUIRED (Always use GLEXEC). Mandating the use of GLEXEC also enforces the factory to submit jobs to sites that have GLEXEC configured.

Other attributes can be specified as well. They are used by the VO frontend matchmaking and job matchmaking. The format is similar to the attributes on the Factory config file. The table below describes the <attrs ... > tag in more detail.

Attribute Name

Attribute Description

name

Name of the attribute

value

Value of the attribute

const

If set to True, the attribute will be available in the constants file created in the staging area. Used only if parameter is True.

parameter

Set to True if the attribute should be passed as a parameter. Always set this to True unless you know what you are doing.

glidein_publish

If set to True, the attribute will be available in the condor_startd's classad. Used only if parameter is True.

job_publish

If set to True, the attribute will be available in the user job's environment. Used only if parameter is True.

comment

You can specify description of the attribute here.

type

Type of the attribute. Supported types are 'int', 'string' and 'expr'. Type expr is equivalent to condor constant/expression in condor_vars.lst

An example attribute would be:

<attrs><attr name="GLIDEIN_Collector" value="mymachine.mydomain" publish="True" parameter="True" const="True" glidein_publish="True" comment=”Just a test attribute”/>

The following group parameters are used to configure multiple frontends. If only one group is specified, they apply to all frontends. The objects specified are used for creating and monitoring glideins.

  • <frontend><groups><group name="name" enabled="boolean" >
    This specifies the name of the group and whether it is enabled.

  • <frontend><groups><config>
    The parameters listed here define how many glideins to create and maintain, such as idle_glideins_per_entry and running_glideins_per_entry.

  • <frontend><groups><match match_expr="expr" >i
    A Python boolean expression is used to match glideins to jobs. The glidein and job dictionaries are used in the expression, i.e. glidein["attrs"]["GLIDEIN_Site"] in job["DESIRED_Sites"].split(","). If you want to match all, just specify True.

  • <frontend><groups><group><match><factory query_expr="expr" >
    A Python expression that specifies a factory constraint.

  • <frontend><groups><group><match><factory><match attrs><match attr name="name" >
    A list of glidein factory attributes used in the factory match expression.

  • <frontend><groups><group><match><job query_expr="expr" >
    A Python expression that specifies a job constraint.

  • <frontend><groups><group><match><job><match attrs><match attr name="name" >
    A list of job attributes used in the factory match expression.

Adding Custom Code/Scripts to VO Frontend Glideins

You can add custom scripts to glideins created for this VO Frontend by adding scripts and files to the configuration in the files section:
<glidein>
 [<groups><group>]
 <files>
  <file absfname="script name" executable="True" comment="comment"/>

The script will be copied to the Web-accessible area and added to the glidein's file_list, and when a glidein starts, the glidein startup script will pull it and execute it. If any parameters are needed, they can be specified using <attr />.

For more detailed information, see the page dedicated to writing custom scripts.

You can also create wrapper scripts or tar-balls of files, see the factory entry page for syntax. (Use groups/group tags instead of the factory's entry tag).

Starting a VO Frontend Daemon

Once you have the desired configuration file, move to the VO frontend directory  and launch the command:

./frontend_startup start

All the activity messages will go into

group_*/log/frontend_info.<date>.log

while the warnings go into

group_*/log/frontend_err.<date>.log

The frontend logs are deleted after a week.

CVSROOT: cvsuser@cdcvs.fnal.gov:/cvs/cd

Packages: glideinWMS

glideinWMS Support: glideinwms-support@fnal.gov