support reading data from TOML files

[cmoussa1]

Feel free to add to this issue with any other ideas or suggestions.

The changes proposed in PR #207 introduce using queue information to help further calculate a job's priority. The data for these queues are defined in the queue_table in the flux-accounting database, but to my knowledge are also defined in a TOML file for fluxion. This leaves the potential for inconsistency between the two modules (flux-accounting and flux-sched) if one is updated but not the other. For example, the TOML file for fluxion has information concerning 3 queues (bronze, silver, gold), but the flux-accounting DB has 4 rows in its queue_table (bronze, silver, gold, platinum). So, as of now, it's up to system administrators or scheduler operators to keep track of these two sets of information and ensure they are consistent.

It might be worth changing the way flux-accounting retrieves queue information so that there is only one place that queues need to be defined and configured. Perhaps flux-accounting could be responsible for reading the TOML file that fluxion has to fetch queue information and import it into its queue_table, which would then be sent to the priority plugin via RPC. It would then remove the need for the add-queue, delete-queue, and edit-queue commands, since all of that configuration would happen with the TOML file.

Or maybe it could just skip the import into the flux-accounting DB and could just read the TOML file for queue information and send it directly to the priority plugin in the bulk-update script flux-account-priority-update.py.

I'm not too familiar with reading data from TOML files but it looks like Python has support for reading and converting it into a Python object. https://pypi.org/project/toml/

[cmoussa1]

Just wanted to update this issue. Looks like Python has some pretty strong and easy-to-learn support for reading TOML files and configurations.

import toml

data = toml.load("sample.toml")
print(toml.dumps(data)) 

[basics]
first_name = "christopher"
last_name = "moussa"
age = 25

We can either print the contents of the file itself, or access specific elements within this file by treating it like a Python dictionary (the more likely scenario):

first_name = data["basics"]["first_name"]
print("first name:", first_name)

first name: christopher

[cmoussa1]

Had a really helpful conversation during coffee hour today where this kind of support was brought up. Seems like this should be a little bit higher priority. Namely, being able to read data that is configured for multiple entities (like queue properties for schedulers) should only have to be set once (like in a TOML file) that can be read by flux-accounting, who is then responsible for initializing itself based on the data found in the configuration file.