Installation of a VO Frontend

4. Base tools installation instructions

4.1 Condor

The VO Frontend will need the the Condor binaries. You can reuse an existing installation, if available, like if you host the VO Frontend on the a submitter node.

Else you need to install a local copy. The suggested way is to install as the same non privileged VO Frontend user (see below).
The whole process is managed by a install script described below. You will need to provide a valid Condor tarball, so you may as well download it before starting the installer.

Move into

glideinWMS/install

and execute

./glideinWMS_install

You will be presented with this screen:

What do you want to install?
(May select several options at one, using a , separated list)
[1] glideinWMS Collector
[2] Glidein Factory
[3] GCB
[4] pool Collector
[5] Schedd node
[6] Condor for VO Frontend
[7] VO Frontend
[8] Components

Select 6.

Now follow the instructions. The installation is straightforward.

4.2 OSG VDT Client

The VO Frontend will need the OSG client provided by VDT. You can reuse an existing installation, if available, like if you host the VO Frontend on the a submitter node.

Else you need to install a local copy. The suggested way is to install as the same non privileged VO Frontend user (see below).
The whole process is managed by a install script described below.

Move into

glideinWMS/install

and execute

./glideinWMS_install

You will be presented with this screen:

What do you want to install?
(May select several options at one, using a , separated list)
[1] glideinWMS Collector
[2] Glidein Factory
[3] GCB
[4] pool Collector
[5] Schedd node
[6] Condor for VO Frontend
[7] VO Frontend
[8] Components

Select 8.

You will be given another menu:

What do you want to install?
(May select several options at one, using a , separated list)
[a] OSG VDT client
[b] Base Condor installation
[c] Web server
[d] rrdtool graphics package
[e] PostgreSQL database
[f] Quill-specific PostgreSQL database

Select a.

Now follow the instructions. The installation is straightforward.

4.3 HTTP server

The VO frontend needs a HTTP server, like Apache or TUX. The server should be installed on the same node, but a different node can be used as long as the web area is writable from this one.

Servers often come pre-installed with HTTP server software, so if you have one running, just reuse it.
Else, the installer can help you install one (as root).

To use the installer, move into

glideinWMS/install

and execute

./glideinWMS_install

You will be presented with this screen:

What do you want to install?
(May select several options at one, using a , separated list)
[1] glideinWMS Collector
[2] Glidein Factory
[3] GCB
[4] pool Collector
[5] Schedd node
[6] Condor for VO Frontend
[7] VO Frontend
[8] Components

Select 8.

You will be given another menu:

What do you want to install?
(May select several options at one, using a , separated list)
[a] OSG VDT client
[b] Base Condor installation
[c] Web server
[d] rrdtool graphics package
[e] PostgreSQL database
[f] Quill-specific PostgreSQL database

Select c.

Now follow the instructions. Most of the questions should be fairly straightforward.

4.4 RRDTool

You will also need the python module for RRDTool (v1.2.18 or later). Many systems come with packages for it; if possible use that.

RPMs for Redhat/Scientific Linux distributions can be found at

http://dag.wieers.com/rpm/packages/rrdtool/

Install python-rrdtool*.rpm and related dependencies.

If installing as root is not an option, you can build from the sources, as the vo frontend user (see below), and put the python libraries into the PYTHON_PATH.
The source code can be downloaded from:
http://oss.oetiker.ch/rrdtool/pub/

Beware: The build process has several dependencies that make it complicated. Use this option only if you cannot install from RPMs (or similar).

4.5 M2Crypto

If you are passing the proxy from the VO Frontend to the glidein Factory, you will need the M2Crypto python library. A few systems include it in the software distribution; if possible install the system one.

If your system does not come with m2crypto package, or you do not can/want install it as root, compile it from source as the VO frontend user (see below).
The source code can be downloaded from:
http://chandlerproject.org/Projects/MeTooCrypto#Downloads

The build process is very easy:

python setup.py build
python setup.py install --root <base dir>

Finally add M2Crypto into the PYTHONPATH, possibly in the .profile, .bashrc, and/or in the .cshrc:

export PYTHONPATH=$PYTHONPATH:<base dir>/usr/lib/python2.3/site-packages/

4.6 javascriptRRD

You will need the javascriptRRD package. It contains the javascript libraries needed by the monitoring.

Just download the tarball, and untar it. You will need to point the installer to this directory.

5. VO Frontend installation

The VO Frontend needs a x509 proxy to communicate with the glidein Factory. You need to create such proxy before starting a VO Frontend and then keep it valid for the life of the frontend. If used for job submission (i.e. if it is passed to the glidein Factory), this proxy must at any point in time have a validity of at least the longest expected job being run by the glideinWMS (and not less than 12 hours).
How you keep this proxy valid (via MyProxy, kx509, voms-proxy-init from a local certificate, scp from other nodes, or other methods), is beyond the scope of this document.

The VO frontend can also host the x509 proxies used for glidein submission. If you do use this (recommended) method, you need to keep these proxies valid at all time, as you do for the main frontend proxy.

The VO Frontend should be installed as a non privileged user. The provided installer can be used to create the configuration file, although some manual tunning will probably be needed.

Before starting the installation of the VO Frontend make sure that the WMS Collector is started and running.

Assuming VDT is installed in $VDT_LOCATION, source VDT's setup as follows

source $VDT_LOCATION/setup.sh

Move into

glideinWMS/install

and execute

./glideinWMS_install

You will be presented with this screen:

What do you want to install?
(May select several options at one, using a , separated list)
[1] glideinWMS Collector
[2] Glidein Factory
[3] GCB
[4] pool Collector
[5] Schedd node
[6] Condor for VO Frontend
[7] VO Frontend
[8] Components

Select 7.

Now follow the instructions.

You will need to specify the selection and matching criteria for the user jobs. The suggested values

jobs_constraint = (JobUniverse==5)&&(DESIRED_Sites=!=UNDEFINED)
match_string = glidein["attrs"]["GLIDEIN_Site"] in job["DESIRED_Sites"].split(",")

should be fine for some basic matchmatching, providing that your user jobs add

+DESIRED_Sites = “site1,site2,...siteN”
requirements=stringListMember(GLIDEIN_Site, DESIRED_Sites)

Here a possible set of answers is presented; your setup will probably be slightly different:

Do you have already a javascriptRRD installation?: (y/n) y
Where is javascriptRRD installed?: /home/frontend/javascriptrrd-0.4.1
To use the VO Frontend you need a valid GSI proxy.
VO Frontend will use this proxy to talk to the WMS Collector and User Schedd
Where is your proxy located?: [/home/frontend/.globus/x509_service_proxy] /home/frontend/.globus/x509_service_proxy

Using DN '/DC=org/DC=doegrids/OU=Service/CN=frontend/frontend1.my.org'
Make sure this DN is in the WMS collector grid-mapfile

For security reasons, we need to know what will the WMS collector map us to.
It will likely be something like joe@collector1.myorg.com
What is the mapped name?: myfe@gfactory1.my.org

Where will you host your config and log files?: [/home/frontend/frontstage] /home/frontend/frontstage
Where will the web data be hosted?: [/var/www/html/vofrontend] /var/www/html/vofrontend
Directory '/var/www/html/vofrontend' not empty.
Should I use it anyhow?: (y/n) y
What Web URL will you use?: [http://frontend1.my.org/vofrontend/] 
Give a name to this VO Frontend?: [myVO-frontend1] myVO1
Give a name to this VO Frontend instance?: [v1_0] v1
What node is the WMS collector running?: gfactory1.my.org
What is the classad identity of the glidein factory?: [gfactory@gfactory1.my.org] gfactory@gfactory1.my.org

What node is the pool collector running on?
If you have a hierarchy of collectors, list them all
(for example master1.my.org:9620-9624
          or master1.my.org:9619,master2.my.org:9620-9624,master3.my.org)
Collector name(s): [collector1.my.org:9618] collector1.my.org:9620-9624
The following schedds have been found:
 [1] schedd1.my.org
 [2] schedd_jobs1@schedd1.my.org
 [3] schedd_jobs2@schedd1.my.org
 [4] schedd_jobs3@schedd1.my.org
 [5] schedd_jobs4@schedd1.my.org
Do you want to monitor all of them?: (y/n) y

What kind of jobs do you want to monitor?: [(JobUniverse==5)&&(GLIDEIN_Is_Monitor =!= TRUE)&&(JOB_Is_Monitor =!= TRUE)] (JobUniverse==5)&&(GLIDEIN_Is_Monitor =!= TRUE)&&(JOB_Is_Monitor =!= TRUE)&&(DESIRED_Sites=!=UNDEFINED)
Give a name to the main group: [main] main
What expression do you want to use to match glideins to jobs?
It is an arbitrary python boolean expression using the dictionaries
  glidein and job
A simple example expression would be:
  glidein["attrs"]["GLIDEIN_Site"] in job["DESIRED_Sites"].split(",")
If you want to match all (OK for simple setups),
  just specify True (the default)
Match string: [True] glidein["attrs"]["GLIDEIN_Site"] in job["DESIRED_Sites"].split(",")
What job attributres are you using in the match expression?
I have computed my best estimate for your match string,
please verify and correct if needed.
Job attributes: [DESIRED_Sites] DESIRED_Sites
What glidein/factory attributres are you using in the match expression?
I have computed my best estimate for your match string,
please verify and correct if needed.
Factory attributes: [GLIDEIN_Site] GLIDEIN_Site
My DN = '/DC=org/DC=doegrids/OU=Service/CN=frontend/frontend1.my.org'

VO frontend proxy = '/home/sfiligoi/.globus/x509_service_proxy'
Do you want to use is to submit glideins: (y/n) [y] n
You may want to use other proxies
Please add all the proxies that this glidein will use

An empty entry means you are done.
proxy fname: /home/sfiligoi/.globus/x509_glidein_proxy1
proxy fname: /home/sfiligoi/.globus/x509_glidein_proxy2
proxy fname: /home/sfiligoi/.globus/x509_glidein_proxy3
proxy fname:
My DNs = ['/DC=org/DC=doegrids/OU=Service/CN=pilot1/myVO.my.org', '/DC=org/DC=doegrids/OU=Service/CN=pilot2/myVO.my.org', '/DC=org/DC=doegrids/OU=Service/CN=pilot3/myVO.my.org']

You will most probably need other DNs in the glidein grid mapfile
Please add all the DNs that this glidein will connect to
(usually the Pool Collector and the Submit nodes)

Please insert all such DNs, together with a user nickname.
An empty DN entry means you are done.

DN: /DC=org/DC=doegrids/OU=Services/CN-collector1.my.org
nickname: [condor001] collector

DN: /DC=org/DC=doegrids/OU=Services/CN=schedd1.my.org
nickname: [condor002] schedd1

DN (leave empty when finished):
Do you want to use the more efficient Match authentication (works for Condor v7.3.0 and later)?: (y/n) y

Make sure you use Match authentication in the schedds, too

Do you want to expose the Grid env. to the user jobs?: (y/n) y
Do you want to create the VO Frontend instance (as opposed to just the config file)?: (y/n) [n] n
To create the VO Frontend instance, you need to run
/home/frontend/glideinWMS/creation/create_frontend

Configuration files are located in /home/frontend/frontstage/instance_v1.cfg
Remember to set X509_USER_PROXY before starting the daemon