Developer Q&A

Multi-threading or multi-processing?

Harvester has a multi-threading architecture. Each thread invokes plugin's functions synchronously. Plugins can spawn background processes and/or threads to do tasks asynchronously. However plugins must control the number of background processes or threads and must call wait() or join() by themselves.

How grid plugin API works?

Each worker is identified by a unique identifier in the batch system like batch jobid and condor jobid. Plugins take actions for the worker with the identifier.

Harvester handles heartbeats/status info back to panda?

Harvester's propagator agents send heartbeats every 30 min for running jobs or immediately for finished/failed jobs.

How Harvester uses external components like pilot, APF and aCT?

It uses external components as libraries. i.e. in the same process and same memory space.

How are pilots killed?

Normally pilots kill themselves once they get the kill command from PanDA through heartbeats. However, even if pilots stop sending heartbeats Harvester will be able to get the list of stuck pilots from PanDA to directly kill them using condor_rm etc.

Which harvesters handle which PQs?

Each harvester instance will have a unique identifier. Config files for harvester instances are stored on PanDA. A config file is downloaded with the identifier when the instance is up. The config file contains the list of PQs for which the instance works.

Multiple harvesters per PQ?

It is possible to have multiple harvester instances per PQ. For example, queue depth can be dynamically set by PanDA in an harvester instance. An easiest solution would be to set queue depth to 1000 when only one instance is running, then it would be reduced to 500 when another instance is up for the same PQ.

What exact job-specific attributes are desired from check_workers()?

In the pull model workflow, ultimately only status would be enough since the pilot directly reports other information to PanDA. In the push model workflow, all information which the pilot reports would be desirable.

Can we extend job attributes if needed?

Job attributes are stored in the harvester DB as a serialized dictionary, so that it is easy to add new attributes.

How harvester monitor works?

The idea is to periodically upload contents of harvester DB to Oracle. There will be a full or slimmed mirror table of the harvester DB in Oracle. BigPandaMon will show views on the table which will be harvester-based resource monitoring. There will be monitoring for harvester instances which shows aliveness of those instances.

API call arg docs / return docs

See Plugin API specifications

Development model

See Development workflow

Testing environment

See Testing and running

Dummy drivers

See dummy plugins like DummySubmitter and DummyMonitor.

How to play with jobSpec or workSpec objects

Two pickle files jobspec.pickle and workspec.pickle are available in the examples directory. For example

$ wget https://github.com/PanDAWMS/panda-harvester/raw/master/examples/workspec.pickle
$ python -i -c "import pickle;obj = pickle.load(open('workspec.pickle'))"

Note that you need to setup runtime beforehand. Otherwise, pickle cannot import XyzSpec classes.

What is harvester_id?

harvester_id is specified in panda_harvester.cfg. It is an arbitrary string (max 50 chars, w/o whitespace) to identify the harvester instance. Submit a JIRA ticket to register the string to the central database.