plugin: add queue priority to priority calculation

[cmoussa1]

This might be a good time for me to start doing some initial design and making some progress on integrating per-queue priority into the multi-factor priority calculation.

In an offline discussion with @ryanday36, it is possible to implement Quality of Service (QoS for short) as queues, and since flux-accounting already contains some of the foundational work to define and configure queues in the database, I can leverage that work to continue integrating it into the priority plugin.

Initial Cleanup

One of the things I think I'll need to clean up is to remove the qos_table from the database and the qos commands from flux-account.py. Since we'll be defining a QoS as a queue, I don't think we'll need the table or the commands and I certainly wouldn't want to cause any confusion as to where QoS' should be defined.

Alter queue_table

Since queue information will be used to calculate a job's priority, a corresponding priority column will need to be added to the queue_table to be able to associate a priority with a queue.

CREATE TABLE IF NOT EXISTS queue_table (
        queue               tinytext                NOT NULL,
        min_nodes_per_job   int(11),
        max_nodes_per_job   int(11),
        max_time_per_job    int(11),
        priority            int(11),
        PRIMARY KEY (queue)
);

Sending queue information to the plugin as payloads

Since I already send user and bank information from the flux-accounting DB to the plugin, I can probably take the same approach with sending queue information; packaging the data into a JSON payload and sending an RPC to the plugin:

flux.Flux().rpc("job-manager.mf_priority.rec_update", json.dumps(queue_data)).get()

Storing queue priority information

What first comes to mind when storing queue names and their respective priorities is to define a map, perhaps with a queue_name as the key and a corresponding priority as a value.

I could then add two new members to the bank_info struct, called valid_queues and specified_queue, and use those members to 1) validate that the queue passed in is one the user is allowed to run jobs in, and 2) look up the associated priority value with the queue when calculating the job priority.

---

One question that's come up as I began thinking about this was how queues were planned to be specified when submitting a job. Will users be expected to pass in a queue using --setattr=system.queue=some_queue, similar to how they specify a bank? If so, then I think I can just extract the queue name out of the jobspec, similar to how the bank name is grabbed. If not, then a different method to grab a passed-in queue might be needed.

edit

#205 has now landed, so much of the cleanup of the queue commands and queue_table is now done. Now it is time to start integrating the data residing in the queue_table with the priority plugin. I'll update this issue as I go along with development.