Installation of a GCB node

1. Description

This node will serve as a Generic Connection Brokering (GCB) node. If you are working over firewalls or NATs, you will need one or more of these.
GCB is needed every time you have a firewall or a NAT;  if this node dies, all theglideins relying on it will die with it.

2. Hardware requirements

This machine needs a reasonably recent CPU and a small amount of memory (256MB should be enough).
It must have reliable network connectivity and must be on the public internet, with no firewalls; it will work as a router. It will use 20k IP ports, so it should not be collocated with other network intesive applications.
The machine must be very stable; if the GCB dies, all the glideins relying on it will die with it (You can use multiple GCBs to minimize the damage, but you should still try to choose the stablest machine you can afford.)
The disk needed is just for Condor binaries and log files (5GB should be enough)

BTW: So low spec machine could be difficult to find and using a virtual machine is out of question, due to the large amount of network ports used. Please consider collocationg it with a VO Frontend.

3. Needed software

A reasonably recent Linux OS (SL4 used at press time).
The Condor distribution.

4. Installation instructions

The installation will assume you install Condor v6.9.2 from tarballs as user gcbuser.
The install directory will be /home/gcbuser/condor/dist, the working directory is /home/gcbuser/condor/condor_local and the machine name is mymachine.fnal.gov and its IP 131.225.70.222
If you want to use a different setup, make the necessary changes.

P.S.: Please remmber to update the scheduler configuration every time you add a new GCB node.

4.1 Obtain Condor

The Condor binaries can be downloaded from http://www.cs.wisc.edu/condor/downloads/.
Choose the v6.9.2 link, read the license, insert your name and affiliation, and proceed to the real download page.
Download dynamic tarball for Red Hat Enterpise Linux 3 (you will get condor-6.9.2-linux-x86-rhel3-dynamic.tar.gz) and put it in /home/gcbuser/Downloads/.

4.2 Extract Condor binaries

GCB cannot be installed using the standard Condor installation procedure; you will need to install it by hand.

Create the target directory:
mkdir ~/condor
mkdir ~/condor/dist
and untar the content of the tarball:
cd ~/condor/dist
tar -xzf ~/Downloads/condor-6.9.2-linux-x86-rhel3-dynamic.tar.gz condor-6.9.2/release.tar
tar -xf condor-6.9.2/release.tar
rm -fr condor-6.9.2

4.3 Configure Condor

GCB configuration is also a special thing... it is very loosly integrated in the Condor infrastructure.
For convenience, it is best to let a condor_master handle the GCB binaries, so we will need to configure that, too.

The config file ~/condor/dist/etc/condor_config should be something like this:
##  What machine is your central manager?
CONDOR_HOST = mymachine.fnal.gov
COLLECTOR_HOST = $(CONDOR_HOST)

##--------------------------------------------------------------------
## Pathnames:
##--------------------------------------------------------------------
## Where have you installed the bin, sbin and lib condor directories?
RELEASE_DIR = /home/gcbuser/condor/dist

## Where is the local condor directory for each host?
LOCAL_DIR = /home/gcbuser/condor/condor_local

## Where is the machine-specific local config file for each host?
LOCAL_CONFIG_FILE =
LOCAL_ROOT_CONFIG_FILE = $(RELEASE_DIR)/etc/condor_config.root

######################################################################
## Daemon-wide settings:
######################################################################

## Pathnames
LOG = $(LOCAL_DIR)/log
SPOOL = $(LOCAL_DIR)/spool
EXECUTE = $(LOCAL_DIR)/execute
BIN = $(RELEASE_DIR)/bin
LIB = $(RELEASE_DIR)/lib
SBIN = $(RELEASE_DIR)/sbin

## Log files
MASTER_LOG = $(LOG)/MasterLog
COLLECTOR_LOG = $(LOG)/CollectorLog
GCB_BROKER_LOG = $(LOG)/GCBLog

## Lock file
LOCK = $(LOCAL_DIR)/condor.lock

## Where are the binaries for these daemons?
MASTER = $(SBIN)/condor_master
COLLECTOR = $(SBIN)/condor_collector


##--------------------------------------------------------------------
## Mail parameters:
##--------------------------------------------------------------------
## When something goes wrong with condor at your site, who should get
## the email?
CONDOR_ADMIN = me@fnal.gov

## Full path to a mail delivery program that understands that "-s"
## means you want to specify a subject:
MAIL = /bin/mailx

##--------------------------------------------------------------------
## Host/IP access levels
##--------------------------------------------------------------------
HOSTALLOW_ADMINISTRATOR = $(CONDOR_HOST)
HOSTALLOW_OWNER = $(FULL_HOSTNAME), $(HOSTALLOW_ADMINISTRATOR)
HOSTALLOW_READ = *
HOSTALLOW_WRITE = *
CREATE_CORE_FILES = True

##--------------------------------------------------------------------
## Settings that control the daemon's debugging output:
##--------------------------------------------------------------------

##
## The flags given in ALL_DEBUG are shared between all daemons.
##
ALL_DEBUG =

MAX_MASTER_LOG = 640000
MASTER_DEBUG = D_COMMAND

SEC_DEFAULT_AUTHENTICATION = REQUIRED
SEC_DEFAULT_AUTHENTICATION_METHODS = FS
SEC_DEFAULT_ENCRYPTION = OPTIONAL
SEC_DEFAULT_INTEGRITY = REQUIRED

SEC_READ_AUTHENTICATION = OPTIONAL
SEC_CLIENT_AUTHENTICATION = OPTIONAL
SEC_READ_ENCRYPTION = OPTIONAL
SEC_CLIENT_ENCRYPTION = OPTIONAL
SEC_READ_INTEGRITY = OPTIONAL
SEC_CLIENT_INTEGRITY = OPTIONAL

########################################################################

DAEMON_LIST = MASTER, COLLECTOR, GCB_BROKER

#####################################
# Here starts GCB configuration
#####################################

# Define the path to the broker binary for the master to spawn
GCB_BROKER = $(RELEASE_DIR)/libexec/gcb_broker

# Define the path to the release_server binary for the broker to use
GCB_RELAY = $(RELEASE_DIR)/libexec/gcb_relay_server

# Setup the gcb_broker's environment. We use a macro to build up the
# environment we want in pieces, and then finally define
# GCB_BROKER_ENVIRONMENT, the setting that condor_master uses.

# Initialize an empty macro
GCB_BROKER_ENV =

# (recommended) Provide the full path to the gcb_relay_server
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_RELAY_SERVER=$(GCB_RELAY)

# (recommended) Tell GCB to write all log files into the Condor log
# directory (the directory used by the condor_master itself)
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_LOG_DIR=$(LOG)
# Or, you can specify a log file seperately for each GCB daemon:
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_BROKER_LOG=$(LOG)/GCB_Broker_Log
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_RELAY_SERVER_LOG=$(LOG)/GCB_RS_Log

# (optional -- only set if true) Tell the GCB broker that it can
# directly connect to machines in the private network which it is
# handling communication for. This should only be enabled if the GCB
# broker is running directly on a network boundry and can open direct
# connections to the private nodes.
#GCB_BROKER_ENV = $(GCB_BROKER_ENV);ACTIVE_TO_CLIENT=yes

# (optional) turn on verbose logging for all of GCB
#GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_DEBUG_LEVEL=fulldebug
# Or, you can turn this on seperately for each GCB daemon:
#GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_BROKER_DEBUG=fulldebug
#GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_RELAY_SERVER_DEBUG=fulldebug

# (optional) specify the maximum log file size (in bytes)
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_MAX_LOG=6400000
# Or, you can define this seperately for each GCB daemon:
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_BROKER_MAX_LOG=64000000
#GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_RELAY_SERVER_MAX_LOG=640000

# How many connections should it serve
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_MAX_RELAY_SERVERS=200
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_MAX_CLIENTS_PER_RELAY_SERVER=100

# How tolerant should it be to blocked connections
GCB_BROKER_ENV = $(GCB_BROKER_ENV);GCB_COMMAND_TIMEOUT=1.5

# Finally, set the value the condor_master really uses
GCB_BROKER_ENVIRONMENT = $(GCB_BROKER_ENV)

# If your Condor installation on this host already has a public
# interface as the default (either because it is the first interface
# listed in this machine's host entry, or because you've already
# defined NETWORK_INTERFACE), you can just use Condor's special macro
# that holds the IP address for this.
GCB_BROKER_IP = $(ip_address)
# Otherwise, you could define it yourself with your real public IP:
#GCB_BROKER_IP = 131.225.70.222

# (required) define the command-line arguments for the broker
GCB_BROKER_ARGS = -i $(GCB_BROKER_IP)

The above config file can also be downloaded from example-config/glide-gcb/mymachine/condor_config.

4.4 Put Condor in the path

Put the following lines in ~/.bashrc:
if [ -z "$CONDOR_BASE_PATH" ]
then
 export CONDOR_BASE_PATH=/home/gcbuser/condor/dist
 export CONDOR_CONFIG=$CONDOR_BASE_PATH/etc/condor_config

 export PATH=$CONDOR_BASE_PATH/bin/:$PATH
 export MANPATH=$CONDOR_BASE_PATH/man:$MANPATH
fi
Reenter the shell to get the new setup.

4.5 Initialize Condor directories

Run condor_init:
~/condor/dist/sbin/condor_init

5. Start GCB

Run condor_master:
~/condor/dist/sbin/condor_master

5.1 Verify it is running

You can check that the processes are running:
ps -u gcbuser |grep gcb
You should see one gcb_broker and at least one gcb_relay_server.

You can also check that tey are working well, by pinging it with gcb_broker_query:
~/condor/dist/sbin/gcb_broker_query  131.225.70.222 freesockets


Back to the index