Add a set-status RPC for marking vertices up or down

[jameshcorbett]

The motivation for this PR comes from this flux-coral2 issue. Rabbits can (and very often do) go down, and fluxion needs to avoid scheduling rabbit jobs on dead rabbits. Rabbit status is monitored by HPE software and reported in kubernetes objects, which flux-coral2 software checks. The flux-coral2 software needs a way to report to Fluxion that a rabbit has died (or resurrected).

This PR provides a way to do that. Flux-coral2 can now send a RPC like sched-fluxion-resource.set_status /path/to/rabbit down to mark a rabbit as down, and similarly to mark it up.

However, I did notice that marking all the nodes or racks in the resource graph didn't prevent jobs from running, hence the test failures. @milroy do you know whether this is expected? Is marking a resource as 'down' purely informational--does it have no effect on actual resource choices?

As an aside, one tricky aspect is that with this PR, flux-coral2 will need to know the resource graph layout. It will know the hostname of the rabbit that has gone down, but it won't necessarily know that the rabbit 'rzvernal201' is '/rzvernal/rack3/rzvernal201` in the resource graph. I think I can work around this though--the resource graph is already written out to a file (because it needs to contain information about rabbit locality) so the flux-coral2 software can read in that graph and figure things out.

[jameshcorbett]

@grondo can you describe what the core-sched interactions are when a node goes down?

[grondo]

When a node goes down the flux-core resource module sends a new resource.acquire response as detailed in RFC 28 with the down rank added to the down idset.

[jameshcorbett]

Thanks @grondo , that actually gave me the information I needed to fix the tests! I think this PR is a pretty harmless addition now.

[zekemorton]

I looked through all of the commits, and the code looks clean and all of the changes make sense! I couldn't find anything to complain about.

I think it's still a good idea for @milroy to give it a look over

[milroy]

I'll add my review today.

[milroy]

This PR is almost ready to merge. I'm requesting a quick change to remove duplicate code and it will be ready to merge.

[milroy]

You could save the repetition of the flux_respond_error and flux_log_error by setting errmsg and using goto error like this block:

flux-sched/resource/modules/resource_match.cpp

Line 1863 in 0b70113

error:

Suggested change

	if (flux_respond_error (h, msg, EINVAL, "malformed RPC") < 0)
	flux_log_error (h, "%s: flux_respond_error", __FUNCTION__);
	return;
	errmsg = "malformed RPC";
	goto error;

That would apply to the following blocks, too.

[jameshcorbett]

I originally wrote this function that way, but the compiler complained that I was jumping over the initialization of some variables. Do you have any recommendations to avoid that? Move all declarations and initializations to the top of the function? That also seemed clunky...

[milroy]

I've run into that a lot, too. You'll need to declare and initialize the variable before the if blocks. Sometimes that isn't feasible; if that's true in this case then we can merge this PR.

[jameshcorbett]

Done! And your spacing comments addressed.

[milroy]

LGTM! Approved.

Once you rebase against the current master you can set MWP.

[jameshcorbett]

Done, setting MWP. Thanks @milroy !

[codecov]

Codecov Report

Merging #1110 (1658f88) into master (0b70113) will decrease coverage by 0.1%.
The diff coverage is 70.9%.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #1110     +/-   ##
========================================
- Coverage    71.8%   71.8%   -0.1%     
========================================
  Files          88      88             
  Lines       11531   11562     +31     
========================================
+ Hits         8287    8309     +22     
- Misses       3244    3253      +9     

Files	Coverage Δ
resource/modules/resource_match.cpp	67.8% <70.9%> (+<0.1%)	⬆️