Add conformance tests for the cwltool issue 1330 (eager vs. lazy eval of reqs and hints)

[kinow]

Conformance tests for common-workflow-language/cwltool#1330

Used the example provided by @manabuishii, extending it for:

• Workflow
• Command-line tool
• Expression tool
• Using requirements
• Using hints
• With the default values
• With inputs

Running locally with the latest cwltool from PYPI:

(venv) kinow@ranma:~/Development/python/workspace/cwl-v1.2/tests/eager-eval-reqs-hints$ cwltest --test test-index.yaml --tool cwltool
Test [1/12] eager_reqs_wf_reqs_default_threads: Test a workflow with requirements using the default values
Test 1 failed: cwltool --outdir=/tmp/tmpw_4lbark --quiet wf-reqs.cwl empty-job.yml
Test a workflow with requirements using the default values
Returned non-zero
ERROR Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 393, in do_eval
    return interpolate(
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 297, in interpolate
    e = evaluator(
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 243, in evaluator
    raise JavascriptException(
cwltool.sandboxjs.JavascriptException: Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
ERROR Exception on step 'one'
ERROR [step one] Cannot make job: Expression evaluation error:
Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
WARNING Final process status is permanentFail

Test [2/12] eager_reqs_wf_reqs_10_threads: Test a workflow with requirements using 10 threads
Test 2 failed: cwltool --outdir=/tmp/tmp89rammpg --quiet wf-reqs.cwl 10-cpus-job.yml
Test a workflow with requirements using 10 threads
Returned non-zero
ERROR Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 393, in do_eval
    return interpolate(
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 297, in interpolate
    e = evaluator(
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 243, in evaluator
    raise JavascriptException(
cwltool.sandboxjs.JavascriptException: Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
ERROR Exception on step 'one'
ERROR [step one] Cannot make job: Expression evaluation error:
Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
WARNING Final process status is permanentFail

Test [3/12] eager_reqs_wf_hints_default_threads: Test a workflow with hints using the default values
Test 3 failed: cwltool --outdir=/tmp/tmprspevsub --quiet wf-hints.cwl empty-job.yml
Test a workflow with hints using the default values
Returned non-zero
ERROR Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 393, in do_eval
    return interpolate(
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 297, in interpolate
    e = evaluator(
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 243, in evaluator
    raise JavascriptException(
cwltool.sandboxjs.JavascriptException: Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
ERROR Exception on step 'one'
ERROR [step one] Cannot make job: Expression evaluation error:
Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
WARNING Final process status is permanentFail

Test [4/12] eager_reqs_wf_hints_10_threads: Test a workflow with hints using 10 threads
Test 4 failed: cwltool --outdir=/tmp/tmpa6cdoam2 --quiet wf-hints.cwl 10-cpus-job.yml
Test a workflow with hints using 10 threads
Returned non-zero
ERROR Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 393, in do_eval
    return interpolate(
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 297, in interpolate
    e = evaluator(
  File "/home/kinow/Development/python/workspace/cwl-v1.2/venv/lib/python3.9/site-packages/cwltool/expression.py", line 243, in evaluator
    raise JavascriptException(
cwltool.sandboxjs.JavascriptException: Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
ERROR Exception on step 'one'
ERROR [step one] Cannot make job: Expression evaluation error:
Syntax error in parameter reference 'inputs.threads_max': inputs does not contain key 'threads_max'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
WARNING Final process status is permanentFail

Test [5/12] eager_reqs_cmdtool_reqs_default_threads: Test a cmdtool with requirements using the default values
Test [6/12] eager_reqs_cmdtool_reqs_10_threads: Test a cmdtool with requirements using 10 threads
Test [7/12] eager_reqs_cmdtool_hints_default_threads: Test a cmdtool with hints using the default values
Test [8/12] eager_reqs_cmdtool_hints_10_threads: Test a cmdtool with hints using 10 threads
Test [9/12] eager_reqs_expr_reqs_default_threads: Test a expr with requirements using the default values
Test [10/12] eager_reqs_expr_reqs_10_threads: Test a expr with requirements using 10 threads
Test [11/12] eager_reqs_expr_hints_default_threads: Test a expr with hints using the default values
Test [12/12] eager_reqs_expr_hints_10_threads: Test a expr with hints using 10 threads
8 tests passed, 4 failures, 0 unsupported features

Running with the code from the branch of the PR common-workflow-language/cwltool#1703:

(venv) kinow@ranma:~/Development/python/workspace/cwl-v1.2/tests/eager-eval-reqs-hints$ cwltest --test test-index.yaml --tool cwltool
Test [1/12] eager_reqs_wf_reqs_default_threads: Test a workflow with requirements using the default values
Test [2/12] eager_reqs_wf_reqs_10_threads: Test a workflow with requirements using 10 threads
Test [3/12] eager_reqs_wf_hints_default_threads: Test a workflow with hints using the default values
Test [4/12] eager_reqs_wf_hints_10_threads: Test a workflow with hints using 10 threads
Test [5/12] eager_reqs_cmdtool_reqs_default_threads: Test a cmdtool with requirements using the default values
Test [6/12] eager_reqs_cmdtool_reqs_10_threads: Test a cmdtool with requirements using 10 threads
Test [7/12] eager_reqs_cmdtool_hints_default_threads: Test a cmdtool with hints using the default values
Test [8/12] eager_reqs_cmdtool_hints_10_threads: Test a cmdtool with hints using 10 threads
Test [9/12] eager_reqs_expr_reqs_default_threads: Test a expr with requirements using the default values
Test [10/12] eager_reqs_expr_reqs_10_threads: Test a expr with requirements using 10 threads
Test [11/12] eager_reqs_expr_hints_default_threads: Test a expr with hints using the default values
Test [12/12] eager_reqs_expr_hints_10_threads: Test a expr with hints using 10 threads
All tests passed

So 8 passed with the current code of cwltool, and 4 failed (I thought it was the other way round @mr-c, sorry). The problem appears to happen when class: Workflow.

-Bruno

[kinow]

Note, the file tests/eager-eval-reqs-hints/wf-reqs.cwl added in this PR was used to create a unit test in the cwltool PR common-workflow-language/cwltool#1703 (comment)

[mr-c]

Thanks!

I wouldn't frame it as eager versus lazy, but more about evaluating the expressions in requirements using the correct inputs object. Requirements specified at the Workflow level should have any CWL expressions evaluated with workflow-level inputs, even if the result is later inherited by a workflow step (or further).

[mr-c]

10 cores is much higher than the allotment you can get on any free CI system I've ever heard of; even 4 is more than the 2 vCPUs we typically get which is why the cmdline variants of the tests fail on the CI: https://github.com/common-workflow-language/cwl-v1.2/runs/7874857215?check_suite_focus=true#step:5:824 but pass for me locally on my 16 core system.

Try again, but use EnvVarRequirement to set a variable, and then echo out the contents later?

Also, please change the eager versus lazy wording to be more specific. Maybe context-eval-reqs-hints and so forth?