Installation of a GCB node


[

Description

|

Hardware requirements

|

Needed software

|

Installation instructions

]


1. Description

NOTE: It is strongly recommended to use CCB available in new version of condor over GCB. CCB provides same functionality as GCB and has performance benifits. Using GCB requires additional installation of condor daemon whereas this feature is integrated in newer versions of condor.

This node will serve as a Generic Connection Brokering (GCB) node. If you are working over firewalls or NATs, and are using an older version of condor (before v7.3.0) you will need one or more of these.
If in use, GCB is needed every time you have a firewall or a NAT; if this node dies, all the glideins relying on it will die with it. If possible use CCB instead.

2. Hardware requirements

This machine needs a reasonably recent CPU and a small amount of memory (256MB should be enough).
It must have
reliable network connectivity and must be on the public internet, with no firewalls; it will work as a router. It will use 20k IP ports, so it should not be collocated with other network intensive applications.
The machine must be
very stable; if the GCB dies, all the glideins relying on it will die with it (You can use multiple GCBs to minimize the damage, but you should still try to choose the stablest machine you can afford.)
The
disk needed is just for Condor binaries and log files (5GB should be enough)

BTW: As low spec machine could be difficult to find, please consider collocating it with a VO Frontend.

3. Needed software

A reasonably recent Linux OS (SL4 used at press time).
The Condor distribution.

4. Installation instructions

The GCB should be installed as a non privileged user.
The whole process is managed by a install script described below. You will need to provide a valid Condor tarball, so you may as well download it before starting the installer.

Move into

glideinWMS/install

and execute

./glideinWMS_install

You will be presented with this screen:

What do you want to install?
(May select several options at one, using a , separated list)
[1] glideinWMS Collector
[2] Glidein Factory
[3] GCB
[4] pool Collector
[5] Schedd node
[6] Condor for VO Frontend
[7] VO Frontend
[8] Components

Select 3.

Now follow the instructions. The installation is straightforward.

The installer will also start the Condor daemons. To stop the Condor daemons, issue

killall condor_master

To start them again:

cd <install dir>; ./start_condor.sh

4.1 Verify it is running

You can check that the processes are running:

ps -u `id -un` |grep gcb

You should see one gcb_broker and at least one gcb_relay_server.

You can also check that tey are working well, by pinging it with gcb_broker_query:

<install dir>/sbin/gcb_broker_query your_ip freesockets

4.2 Fine tunning

4.2.1 Increase the number of available ports

The default installation will set up GCB to handle up to 20k requests. Look in the <install dir>/etc/condor_config.local for

GCB_MAX_RELAY_SERVERS=200
GCB_MAX_CLIENTS_PER_RELAY_SERVER=100

This is enough for approx. 4k glideins (each glidein uses 5-6 ports).

If you want a single GCB to serve more glideins that that, you can increase those numbers. However, be aware that the OS also have its limits. On most Linux systems the limit is set in /proc/sys/net/ipv4/ip_local_port_range.
In this example:

$ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
system has only ~28k ports available. If you want to configure GCB/CCB to serve more than that, first change the system limit, then the GCB/CCB configuration.

4.2.1 Increase the number of available file descriptors

Note that every port used by the GCB/CCB also consumes available file descriptors. The default number of file descriptors per process is 1024 on most systems. Increase this limit to ~16k or value higher than number of ports GCB/CCB is allowed to open.

Back to the index


glideinWMS support: glideinwms-support@fnal.gov