[9.0] Simplify and speed up the SiteDirector

[aldbr]

This PR aims at removing the extra logic checking the number of waiting jobs and pilots before submitting new pilots.
This logic does not work well in some cases:

• sites are always processing jobs (this is the case of LHCb, not sure it is the case for all VOs)
• multiple site directors run in parallel (two of them might look submit pilots at the same time for a same task queue, which means the logic of "preventing one to submit new pilots if another has already submitted the necessary number to process the waiting jobs" is not useful)

Here are the main changes:

• introduction of submission policies:

  • AggressiveFilling: will just fill up the available slots, no matter whether there are waiting jobs (good for sites that are always processing jobs)
  • WaitingSupportedJobs: will only fill up some of the available slots, according to the number of waiting jobs. It does not take into account the number of already submitted pilots for these jobs (might result in some unused pilots but should not be a big issue, as it should target sites that are just occasionally used).

• parallel submission: submission of pilots can now be done in parallel on the different CEs.

• parallel accounting in the monitoring phase: this was done sequentially before, and it was taking a lot of time.

Other suggestions to discuss:

• splitting the submission and the monitoring operations into two agents PilotSubmitter and PilotMonitor (could be merged with PilotStatusAgent):
  • Pros: both operations would be quickly executed (none of them would hamper the other one).
  • Cons: might result in mutiple instances of both PilotSubmitter and PilotMonitor if one of each is not enough for every site (this will surely be the case for LHCb).
• adding an intermediate submission policy SoftFilling: would check the N last Running (for some time), Done pilots and would get the average number of currentJobID. Then it would adapt the number of pilots to submit according to the value (e.g. for a given queue, 80% of the N pilots have a currentJobID, which means we have too much pilots, so only 80% of the MAX waiting pilots should be in the queue). This would be a intermediate policy that would fit with sites generally processing some jobs, but not enough to entirely fill up the queues with pilots.

BEGINRELEASENOTES
*WorkloadManagement
CHANGE: simplify and speed up the SiteDirector
ENDRELEASENOTES

[fstagni]

• sites are always processing jobs (this is the case of LHCb, not sure it is the case for all VOs)

This is not always the case. Apart from our certification system(s), also some production installations don't have that.

[fstagni]

splitting the submission and the monitoring operations into two agents PilotSubmitter and PilotMonitor (could be merged with PilotStatusAgent):

• Pros: both operations would be quickly executed (none of them would hamper the other one).
• Cons: might result in mutiple instances of both PilotSubmitter and PilotMonitor if one of each is not enough for every site (this will surely be the case for LHCb).

We might discuss this for DiracX, not for DIRAC.

[fstagni]

This deprecation was introduced for v9.

[aldbr]

Ah yes indeed.
How should we handle that?

Above you said:

The name of the method should then be changed to export_addPilotReference, and taskQueueID would not be needed anymore. But really it should be better to then deprecate the old method name (could be done in v8r0).

Should I make a PR targeting v8.0 to introduce addPilotReferences and deprecate addPilotTQReference (which would be removed from v9.0)?
In this case addPilotTQRef would also be removed from v9.0.

[fstagni]

The deprecation was introduced in #7157 because of the DB simplifications and remove of DN. Maybe what you are suggesting would work.

[aldbr]

I created #7358