jobdesc check in job module would be useful, possibly implement job creation API?

[trws]

Two out of 4000 jobs so far have managed to run well past their walltime without being killed. This instance is not, unfortunately, in verbose mode, so all I know to get is the kvs output for one of them:

lwj.0.6.880.0.procdesc = {"command": "/p/gscratch1/splash/splash_semiproduction/041018/wo...
lwj.0.6.880.0.stdin -> lwj.0.6.880.input.files.stdin
lwj.0.6.880.cmdline = ["/p/gscratch1/splash/splash_semiproduction/041018/workspace/run_cr...
lwj.0.6.880.create-time = 1523337807.685208
lwj.0.6.880.cwd = /p/gscratch1/splash/splash_semiproduction/041018/workspace
lwj.0.6.880.environ = {"BASH_ENV": "/usr/share/lmod/lmod/init/bash", "BASH_FUNC_ml()": "(...
lwj.0.6.880.ncores = 24
lwj.0.6.880.nnodes = 1
lwj.0.6.880.ntasks = 1
lwj.0.6.880.options = {"stdio-delay-commit": 1}
lwj.0.6.880.opts = {"cores-per-task": 24, "nnodes": 1, "ntasks": 1, "tasks-per-node": 1}
lwj.0.6.880.pmi.PMI_process_mapping = (vector,(0,1,1))
lwj.0.6.880.rank.130.cores = 24
lwj.0.6.880.rdl = { "type": "cluster", "path": "\/cluster", "basename": "cluster", "name"...
lwj.0.6.880.running-time = 1523337821.050703
lwj.0.6.880.starting-time = 1523337820.911810
lwj.0.6.880.state = running
lwj.0.6.880.walltime = 08:00:00

The job is currently at about 11 hours. This particularly surprises me because running 25000 jobs through over the weekend I never saw this, and now it's twice. The only difference that immediately comes to mind between this and what I was running before is that this is using 8 hours where that was one.

[grondo]

lwj.0.6.880.walltime = 08:00:00

Does that format for walltime work? For the timeout handler installed in wrexecd it expects walltime in seconds, not HH:MM:SS. At least I thought it did....

[trws]

Now that might explain some things. The other jobs probably haven't needed to be killed, and I bet the walltime was set by the workflow manager, and it was never validated on the way through the system. Would it be reasonable to have a check for that kind of thing on submit do you think?

[grondo]

Yeah, it might make sense in the near term for the job module to do at least some verification of the contents of the submit/create RPC payload. Right now it writes whatever you have in the payload directly the kvs, and doesn't even examine the contents. 😜

[trws]

I will admit to shamelessly taking advantage of that for metadata… That said, I’m a bit surprised we didn’t hit more issues with the Jget_int calls on walltime for example. It’s a testament to well checked code on that end that the jobs ran correctly for sure.

…

On 10 Apr 2018, at 11:17, Mark Grondona wrote: Yeah, it might make sense in the near term for the job module to do at least some verification of the contents of the submit/create RPC payload. Right now it writes whatever you have in the payload directly the kvs, and doesn't even examine the contents. 😜 -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #1443 (comment)

[grondo]

Well the purpose was for rapid prototyping and so maybe it ended up helping somewhat. Unfortunately lack of any kind of reasonable schema and validation has caused some pain when trying to use this for real work as well! Sorry about that.

[trws]

Switching the title over to something closer to the actual issue here. The question is probably whether this should be called a dupe of #268?

[grondo]

I don't think this is a duplicate of #268.

IMO, this issue should address minimum required fixes for the current job module, which is known deficient and didn't even try to address any part of #268.

#268 will only be resolved when we've completely replaced the current job module with a true job ingest system. We should perhaps use what we learn here to feed in to #268.

[grondo]

closed by #1988