[flux-core v0.13] example2 update: Update README.md, Python scripts

[cmoussa1]

This PR makes the following changes:

1. README.md

• Update format of instructions from bullet points to numbered steps
• Changed code instructions from bold font to in-line code
• Removed step to set FLUX_SCHED_OPTIONS environment variable

2. submitter.py and submitter2.py

• Updated scripts to utilize new Jobspec format, Python bindings (thanks a lot to @SteVwonder for all of his help to get these working! 🎉)

3. Lua scripts

• Since the Lua bindings have been deprecated, the Lua submission scripts, as well as compute and io-forwarding scripts have been removed.

---

The [WIP] tag is here because I'm not sure if we want to make changes to the new Python scripts so that they are Black compatible. If we want to do that, I'd be happy to update them, and then drop the [WIP] tag.

[grondo]

I wonder if we could try out using GitHub Actions to run black formatter on examples in this repo, rather than setting up Travis.

https://github.com/marketplace/actions/black-code-formatter

[cmoussa1]

@grondo. I like that suggestion; I will look into setting that up. In the meantime, I blackened the Python files manually in my forked repo and pushed those changes! I think I'll drop the [WIP] tag too, then.

[grondo]

I'm just commenting with a few ideas. I'm not sure if they are the right approach for these examples, or if we should save the more verbose descriptions for something more like a tutorial. So take or leave the comments, or discuss more here 😄

[grondo]

I wonder if it would be more instructive if a little more prose was included with these steps, i.e. make the style more tutorial-like.

e.g. instead of

1. salloc -N3 -p pdebug
2. srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out

Something like

1. Allocate 3 nodes from local resource manager:

$ salloc -N3 -p pdebug
salloc: granted job allocation 1234

2. Launch a flux instance on the current allocation by running flux start once per node, redirecting log messages to the file out in the current directory

$ srun --pty --mpi=none --ntasks-per-node=1 flux start -o,-S,log-filename=out
$

And so on..

[cmoussa1]

I like this idea! Since most of the beginning of the examples are identical (requesting an allocation from a resource manager, launching a Flux instance), what if we posted a note of the most common commands with their descriptions in a top-level README, along with the titles of each example?

We could have, for instance:

flux-workflow examples

<top level readme with links, descriptions of each example>
Some Notes About The Examples
Most of the examples contain the same commands. Descriptions for these commands are as follows:

• salloc -N3 -p pdebug: allocates 3 nodes from a local resource manager.
• srun --pty --mpi=none --ntasks-per-node=1 flux start -o,-S,log-filename=out: launch a Flux instance on the current allocation once per node, redirecting log messages to the file out in the current directory

[grondo]

My opinion would be that the explanations should be kept in-line with the examples to be most instructive. However, since as you say that will be very repetitive, maybe what we really want to work on next is to take the most interesting and general cases from these workflow examples and combine them into a single page or click-through Tutorial.

That might save us some work here, and having everything in a single place would make it easier for interested users.

[cmoussa1]

Good point. Maybe this could be a good starting point for that conversation about documentation with Elsa! 😄

[grondo]

Maybe we should consider pasting/including the source of these examples directly in the README text? Then readers could get the gist of the entire example on one page, without checking out the git repo, or using extra clicks to display the source via GitHub.

I'm not sure if you can include files in markdown, so that might be a point against this idea in practice.

[cmoussa1]

Just from a quick search of the internet, it looks like we can't include files directly in markdown, but I think that suggestion of manually including some of the source code directly in the README could be really helpful, like a "highlights of the code" section or something. Maybe the more significant portions of the source code?

In example2, for instance, including the declaration and submission of compute_jobreq:

compute_jobreq = JobspecV1.from_command(
    command=["./compute.py", "120"], num_tasks=4, num_nodes=2, cores_per_task=2
)
compute_jobreq.cwd = os.getcwd()
compute_jobreq.environment = dict(os.environ)
print(flux.job.submit(f, compute_jobreq))

We could also include a short explanation of some of the code as well.

[grondo]

Technically, flux job list isn't listing jobs in KVS. Maybe "List active jobs" instead?

[grondo]

Again, maybe more prose here, to explain what the value we're fetching from the KVS. Maybe with some background information, e.g.

5. Information about jobs, such as the submitted job specification, an eventlog, and the resource description format R are stored in the KVS. The data can be queried via the job-info module via the flux job info command. For example, to fetch R for a job which has been allocated resources:

$ flux job info 316703506432 R
{"version":1,"execution":{"R_lite":[{"rank":"0-1","children":{"core":"0-3"}}]}}

[cmoussa1]

Pushed some more changes thanks to the suggestions from @grondo. A couple changes were introduced in this last commit:

• Added more verbosity to the instructions in this walkthrough. I pretty much used @grondo's exact wording for a majority of the instructions since they were both concise and descriptive; since this specific example essentially contains two sets of the same instructions, part (b) has an abbreviated description of each instruction.
• Added a Notes section at the bottom of the README that tries to explain some of the more important parts of the source code used in this example.

[dongahn]

@cmoussa1: is this ready?

[cmoussa1]

This is also ready! 👍

[dongahn]

Do you need to rebase this?

[cmoussa1]

Oops, you're right - I forgot I had to rebase the two separate README update commits. I'm sorry about that. This should be good to go now.

[dongahn]

I just merged one of your PR. Please rebase once again.

[cmoussa1]

Just ran git rebase master and got this output:

Current branch example2-update-core-v0.13 is up to date.

Do I need to do something else instead?

[dongahn]

Maybe your master already have the up-to-date state... I typically do:

git fetch upstream
git rebase upstream/master
git push -f origin <the branch where you are working on this PR>

[cmoussa1]

Ah! I needed to add the remote upstream branch; then git rebase properly updated my branch with those changes. Thanks for clearing that up for me.

[cmoussa1]

Rebased and force pushed to get up to date with master after PR #46.

[cmoussa1]

Rebased and force pushed to get up to date with master after PR #47.

[cmoussa1]

Rebased and force pushed to get up to date with master after PR #49.

[dongahn]

LGTM. Just one minor enhancement request :-) I really like that you added the note section into the read me by the way @cmoussa1. Thanks.

[dongahn]

Maybe mark this with ```python so that the snippet will be color coded when rendered?

[cmoussa1]

Thanks! The Notes section was suggested by @grondo, and I'm really liking how the repo is turning out with the inclusion of additional documentation.

Made the changes to the Python code block in the Notes section, rebased, and force-pushed.

[cmoussa1]

Thanks!