GlideinWMS The Glidein-based Workflow Management System

For scalability purposes, this section will describe the steps (configuration) necessary to add additional (secondary) HTCondor collectors for the WMS and/or User Collectors using or not, shared_port option.

Note: If you specified any of these options using the GlideinWMS configuration based installer, these files and initialization steps will already have been performed. These instructions are relevant to any post-installation changes you desire to make.

Important: When secondary (additional) collectors are added to either the WMS Collector or User Collector, changes must also be made to the Frontend configurations of all Frontends, so they are made aware of them.

HTCondor configuration changes

Individual Ports

For each secondary collector, the following Condor attributes are required:

COLLECTORnn = $(COLLECTOR)
COLLECTORnn_ENVIRONMENT = "_CONDOR_COLLECTOR_LOG=$(LOG)/CollectornnLog"
COLLECTORnn_ARGS = -f -p port_number
DAEMON_LIST = $(DAEMON_LIST), COLLECTORnn

In the above example, n is an arbitrary value to uniquely identify each secondary collector. Each secondary collector must also have a unique port_number.

After these changes have been made in your Condor configuration file, restart HTCondor to effect the change. You will see these collector processes running (example has 5 secondary collectors).

user 17732 1 0 13:34 ? 00:00:00 /usr/local/glideins/separate-no-privsep-7-6/condor-userpool/sbin/condor_master user 17735 17732 0 13:34 ? 00:00:00 condor_collector -f primary
user 17736 17732 0 13:34 ? 00:00:00 condor_negotiator -f
user 17737 17732 0 13:34 ? 00:00:00 condor_collector -f -p 9619 secondary
user 17738 17732 0 13:34 ? 00:00:00 condor_collector -f -p 9620 secondary
user 17739 17732 0 13:34 ? 00:00:00 condor_collector -f -p 9621 secondary
user 17740 17732 0 13:34 ? 00:00:00 condor_collector -f -p 9622 secondary
user 17741 17732 0 13:34 ? 00:00:00 condor_collector -f -p 9623 secondary

Separate Ports

Since GlideinWMS v3.4.1, shared_port is enabled by default for secondary collectors and CCBs, having all the collector communication behind a single TCP port. This helps to have a queue per daemon, instead of global queue. To carry out this, the following HTCondor attributes are required:

COLLECTOR_HOST = $(CONDOR_HOST):port_number USE_SHARED_PORT = True SHARED_PORT_MAX_WORKERS = 1000 SHARED_PORT_ARGS = -p port_number DAEMON_LIST = $(DAEMON_LIST), SHARED_PORT

For the secondary collectors configuration, in this example we show an arbitrary value to uniquely identify each secondary collector, but all of the collector host are behind of a single TCP port.

use Experimental:CollectorNode(nn) COLLECTORnn_ENVIRONMENT = "_CONDOR_COLLECTOR_LOG=$(LOG)/CollectornnLog"

After these changes have been made in your HTCondor configuration file, restart HTCondor to effect the change. You will see these collector processes running (example has 5 secondary collectors and the main one listening on the same port: 9618).

├─1675222 condor_shared_port -f -p 9618 TCP single port
├─1675223 condor_collector -f primary
├─1675227 condor_negotiator -f ├─1675229 condor_schedd -f ├─1675230 condor_collector -f -f -local-name COLLECTOR1 -sock collector1 secondary
├─1675232 condor_collector -f -f -local-name COLLECTOR2 -sock collector2 secondary
├─1675234 condor_collector -f -f -local-name COLLECTOR3 -sock collector3 secondary
├─1675237 condor_collector -f -f -local-name COLLECTOR4 -sock collector4 secondary
├─1675240 condor_collector -f -f -local-name COLLECTOR5 -sock collector5 secondary

Transition to Shared Port

This is a temporary configuration to switch from separate ports to shared_port In GlideinWMS v3.4.1, shared port only configuration is incompatible if talking to older Factories (v3.4 or older). Also, it requires the Frontend admin to drain the Frontend, change the configuration and restart it. To make it compatible and to allow a smother transition, the following configuration makes possible to support both, different ports and shared port and avoid the pitfalls mentioned. A secondary collector can both listen on a separate port and listen to the shared port daemon.

COLLECTORnn = $(COLLECTOR) COLLECTORnn_ARGS = -f -local-name COLLECTORnn -p port_number> -sock collectornn COLLECTORnn_ENVIRONMENT = "_CONDOR_COLLECTOR_LOG=$(LOG)/CollectornnLog" DAEMON_LIST=$(DAEMON_LIST), COLLECTORnn

After these changes have been made in your Condor configuration file, restart HTCondor to effect the change. You will see as many collector processes running as you defined.

---

---

Multiple Collectors for High Availability (HA)

For reliability purposes, you may want to utilize HTCondor's High Availability (HA) feature for collectors.

The HTCondor configuration of additional (secondary) collectors is the same as in the previous section, Multiple Collectors for Scalability. Refer to the HTCondor manual section on High Availability of the Central Manager for additional configuration requirements.

Important: When the HTCondor High Availability feature is used in the User Collector, changes must also be made to the Frontend configurations so it is made aware of them.

Installing Quill

The HTCondor manual section about Quill may have instructions more updated than this section.

Required software

• A reasonably recent Linux OS (SL4 used at press time).
• A PostgreSQL server.
• The HTCondor distribution.
  

Installation instructions

The installation will assume you have installed HTCondor v8+.

The install directory is /opt/glidecondor, the working directory is /opt/glidecondor/condor_local and the machine name is mymachine.fnal.gov. and its IP 131.225.70.222.

If you want to use a different setup, make the necessary changes.

Unless explicity mentioned, all operations are to be done as root.

Obtain and install PostgreSQL RPMs

Most Linux distributions come with very old versions of PostgreSQL, so you will want to download the latest version.

The RPMs can be found on http://www.postgresql.org/ftp/binary/

At the time of writing, the latest version is v8.2.4, and the RPM files to install are

postgresql-8.2.4-1PGDG.i686.rpm
postgresql-libs-8.2.4-1PGDG.i686.rpm
postgresql-server-8.2.4-1PGDG.i686.rpm

Initialize PostgreSQL

Switch to user postgres:

su - postgres

And initialize initialize the database with:

initdb -A "ident sameuser" -D /var/lib/pgsql/data

Configure PostgreSQL

PostgreSQL by default only accepts local connections., so you need to configure it in order for Quill to use it.

Please do it as user postgres.

To enable TCP/IP traffic, you need to change listen_addresses in /var/lib/pgsql/data/postgresql.conf to:

# Make it listen to TCP ports
listen_addresses = '*'

Moreover, you need to specify which machines will be able to access it.
Unless you have strict security policies forbiding this, I recommend enabling read access to the whole world by adding the following line to /var/lib/pgsql/data/pg_hba.conf:

host    all     quillreader     0.0.0.0/0        md5

On the other hand, we want only the local machine to be able to write the database. So, we will add to /var/lib/pgsql/data/pg_hba.conf:

host    all     quillwriter     131.225.70.222/32   md5

Start PostgreSQL

To start PostgreSQL, just run:

/etc/init.d/postgresql start

There should be no error messages.

Initalize Quill users

Switch to user postgres:

su - postgres

And initialize initialize the Quill users with:

createuser quillreader --no-createdb --no-adduser --no-createrole --pwprompt
# passwd reader
createuser quillwriter --createdb --no-adduser --no-createrole --pwprompt
# password <writer passwd>
psql -c "REVOKE CREATE ON SCHEMA public FROM PUBLIC;"
psql -d template1 -c "REVOKE CREATE ON SCHEMA public FROM PUBLIC;"
psql -d template1 -c "GRANT CREATE ON SCHEMA public TO quillwriter; GRANT USAGE ON SCHEMA public TO quillwriter;"

Configure Condor

Append the following lines to /opt/glidecondor/etc/condor_config:

#############################
# Quill settings
#############################
QUILL_ENABLED = TRUE
QUILL_NAME = quill@$(FULL_HOSTNAME)
QUILL_DB_NAME = $(HOSTNAME)
QUILL_DB_QUERY_PASSWORD = reader
QUILL_DB_IP_ADDR = $(HOSTNAME):5432
QUILL_MANAGE_VACUUM = TRUE

In /opt/glidecondor/condor_local/condor_config.local, add QUILL to DAEMON_LIST, getting something like:

DAEMON_LIST                     = MASTER, QUILL, SCHEDD

Finally, put the writer passwd into /opt/glidecondor/condor_local/spool/.quillwritepassword:

echo "<writer passwd>" > /opt/glidecondor/condor_local/spool/.quillwritepassword
chown condor /opt/glidecondor/condor_local/spool/.quillwritepassword
chmod go-rwx /opt/glidecondor/condor_local/spool/.quillwritepassword

Banner image by Madhusudan Katti used under Creative Commons license.
Original Home URL: http://glideinwms.fnal.gov. GlideinWMS email support: glideinwms-support at fnal.gov