improve test running on mac-os

[mr-c]

@jfennick can you test this as well?

[codecov]

Codecov Report

Merging #1710 (92c4073) into main (92c4073) will not change coverage.
The diff coverage is n/a.

❗ Current head 92c4073 differs from pull request most recent head bbda773. Consider uploading reports for the commit bbda773 to get more accurate results

@@           Coverage Diff           @@
##             main    #1710   +/-   ##
=======================================
  Coverage   66.60%   66.60%           
=======================================
  Files          89       89           
  Lines       15850    15850           
  Branches     4188     4188           
=======================================
  Hits        10557    10557           
  Misses       4203     4203           
  Partials     1090     1090           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[lgtm-com]

This pull request fixes 6 alerts when merging e011b42 into 8763bed - view on LGTM.com

fixed alerts:

• 6 for Unused import

[jfennick]

I'm still getting a few test failures:

========================================================================================================================== FAILURES ==========================================================================================================================
________________________________________________________________________________________________________________ TestMpiRun.test_mpi_workflow ________________________________________________________________________________________________________________
[gw4] darwin -- Python 3.10.5 /Users/jakefennick/opt/anaconda3/envs/cwltool_test/bin/python3.10

self = <tests.test_mpi.TestMpiRun object at 0x10cc7d5a0>, fake_mpi_conf = '/Users/jakefennick/cwltool/tmp/popen-gw4/fake_mpi0/plat_mpi.yml', tmp_path = PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw4/test_mpi_workflow0')

    @needs_docker
    def test_mpi_workflow(self, fake_mpi_conf: str, tmp_path: Path) -> None:
        np = 3
        input_file = make_processes_input(np, tmp_path)
        stdout = StringIO()
        stderr = StringIO()
        with working_directory(tmp_path):
            rc = main(
                argsl=cwltool_args(fake_mpi_conf)
                + [get_data("tests/wf/mpi_simple_wf.cwl"), str(input_file)],
                stdout=stdout,
                stderr=stderr,
            )
>           assert rc == 0
E           assert 1 == 0

tests/test_mpi.py:189: AssertionError
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO     cwltool:main.py:1046  3.1.20220815105930
INFO     cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl' to 'file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl'
DEBUG    cwltool:workflow_job.py:503 [workflow _2] initialized from file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl
INFO     cwltool:workflow_job.py:791 [workflow _2] start
DEBUG    cwltool:workflow_job.py:803 [workflow _2] inputs {
    "processes": 3
}
DEBUG    cwltool:workflow_job.py:625 [workflow _2] job step file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#count not ready
INFO     cwltool:workflow_job.py:628 [workflow _2] starting step par_pids
DEBUG    cwltool:workflow_job.py:756 [step par_pids] job input {
    "file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#par_pids/processes": 3
}
DEBUG    cwltool:workflow_job.py:763 [step par_pids] evaluated job input to {
    "file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#par_pids/processes": 3
}
INFO     cwltool:workflow_job.py:74 [step par_pids] start
DEBUG    cwltool:command_line_tool.py:1022 [job par_pids] initializing from file:///Users/jakefennick/cwltool/tests/wf/mpi_expr.cwl as part of step par_pids
DEBUG    cwltool:command_line_tool.py:1030 [job par_pids] {
    "processes": 3
}
DEBUG    cwltool:command_line_tool.py:1046 [job par_pids] path mappings is {}
DEBUG    cwltool:command_line_tool.py:1101 [job par_pids] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "python"
    },
    {
        "position": [
            0,
            0
        ],
        "datum": "-c"
    },
    {
        "position": [
            0,
            1
        ],
        "datum": "import os; print(os.getpid())"
    }
]
DEBUG    cwltool:job.py:222 [job par_pids] initial work dir {}
INFO     cwltool:job.py:276 [job par_pids] /private/tmp/docker_tmpc0qvg7s_$ /Users/jakefennick/cwltool/tmp/popen-gw4/fake_mpi0/fake_mpirun \
    --num \
    3 \
    --no-fail \
    python \
    -c \
    'import os; print(os.getpid())' > /private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62
DEBUG    cwltool:job.py:561 Could not collect memory usage, job ended before monitoring began.
INFO     cwltool:job.py:433 [job par_pids] completed success
DEBUG    cwltool:job.py:436 [job par_pids] outputs {
    "pids": {
        "location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
        "size": 18,
        "http://commonwl.org/cwltool#generation": 0
    }
}
DEBUG    cwltool:workflow_job.py:577 [step par_pids] produced output {
    "file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#par_pids/pids": {
        "location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
        "size": 18,
        "http://commonwl.org/cwltool#generation": 0
    }
}
INFO     cwltool:workflow_job.py:587 [step par_pids] completed success
DEBUG    cwltool:job.py:464 [job par_pids] Removing input staging directory /private/tmp/docker_tmpvqgs9fql
DEBUG    cwltool:job.py:472 [job par_pids] Removing temporary directory /private/tmp/docker_tmpe7y7z9ia
INFO     cwltool:workflow_job.py:628 [workflow _2] starting step count
DEBUG    cwltool:workflow_job.py:756 [step count] job input {
    "file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#count/pid_file": {
        "location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
        "size": 18,
        "http://commonwl.org/cwltool#generation": 0
    }
}
DEBUG    cwltool:workflow_job.py:763 [step count] evaluated job input to {
    "file:///Users/jakefennick/cwltool/tests/wf/mpi_simple_wf.cwl#count/pid_file": {
        "location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
        "size": 18,
        "http://commonwl.org/cwltool#generation": 0
    }
}
INFO     cwltool:workflow_job.py:74 [step count] start
WARNING  cwltool:command_line_tool.py:431 MPIRequirement with containers is a beta feature
WARNING  cwltool:command_line_tool.py:443 MPI has been required while Docker is hinted, discarding Docker hint(s)
DEBUG    cwltool:command_line_tool.py:1022 [job count] initializing from file:///Users/jakefennick/cwltool/tests/wf/mpi_line_count.cwl as part of step count
DEBUG    cwltool:command_line_tool.py:1030 [job count] {
    "pid_file": {
        "location": "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "basename": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameroot": "f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$d9c35e837f89bb5223cec47343f517d931b94957",
        "size": 18,
        "http://commonwl.org/cwltool#generation": 0
    }
}
DEBUG    cwltool:command_line_tool.py:1046 [job count] path mappings is {
    "file:///private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62": [
        "/private/tmp/docker_tmpc0qvg7s_/f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "/var/lib/cwl/stg814c5222-3ba4-4f09-a3d8-54ec212d3faa/f02fc15c1a50421f8123149541a59ca53ef7ef62",
        "File",
        true
    ]
}
DEBUG    cwltool:command_line_tool.py:1101 [job count] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "wc"
    },
    {
        "position": [
            0,
            0
        ],
        "datum": "-l"
    }
]
ERROR    cwltool:executors.py:261 Got workflow error
Traceback (most recent call last):
  File "/Users/jakefennick/cwltool/cwltool/executors.py", line 251, in run_jobs
    job.run(runtime_context)
  File "/Users/jakefennick/cwltool/cwltool/job.py", line 581, in run
    self._setup(runtimeContext)
  File "/Users/jakefennick/cwltool/cwltool/job.py", line 191, in _setup
    os.makedirs(self.outdir)
  File "/Users/jakefennick/opt/anaconda3/envs/cwltool_test/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/ATmsgV'
ERROR    cwltool:main.py:1409 Workflow error, try again with --debug for more information:
[Errno 30] Read-only file system: '/ATmsgV'
________________________________________________________________________________________________________ test_single_process_subwf_subwf_inline_step _________________________________________________________________________________________________________
[gw5] darwin -- Python 3.10.5 /Users/jakefennick/opt/anaconda3/envs/cwltool_test/bin/python3.10

    @needs_docker
    def test_single_process_subwf_subwf_inline_step() -> None:
        """Test --single-process on an inline sub-sub-workflow step."""
        err_code, stdout, stderr = get_main_output(
            [
                "--single-process",
                "step1/stepX/stepY",
                get_data("tests/subgraph/count-lines17-wf.cwl.json"),
                get_data("tests/wf/wc-job.json"),
            ]
        )
>       assert err_code == 0
E       assert 1 == 0

tests/test_subgraph.py:254: AssertionError
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO     cwltool:main.py:1046  3.1.20220815105930
INFO     cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/subgraph/count-lines17-wf.cwl.json' to 'file:///Users/jakefennick/cwltool/tests/subgraph/count-lines17-wf.cwl.json'
ERROR    cwltool:main.py:1227 Tool definition failed validation:
while scanning for the next token
tests/subgraph/count-lines17-wf.cwl.json:116:1:   found character '\t' that cannot start any token
___________________________________________________________________________________________________________ test_print_targets_embedded_sub_subwfs ___________________________________________________________________________________________________________
[gw2] darwin -- Python 3.10.5 /Users/jakefennick/opt/anaconda3/envs/cwltool_test/bin/python3.10

    @needs_docker
    def test_print_targets_embedded_sub_subwfs() -> None:
        """Confirm that --print-targets works with inline sub-sub-workflows."""
        err_code, stdout, stderr = get_main_output(
            [
                "--print-targets",
                get_data("tests/subgraph/count-lines17-wf.cwl.json"),
            ]
        )
>       assert err_code == 0
E       assert 1 == 0

tests/test_subgraph.py:363: AssertionError
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO     cwltool:main.py:1046  3.1.20220815105930
INFO     cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/subgraph/count-lines17-wf.cwl.json' to 'file:///Users/jakefennick/cwltool/tests/subgraph/count-lines17-wf.cwl.json'
ERROR    cwltool:main.py:1227 Tool definition failed validation:
while scanning for the next token
tests/subgraph/count-lines17-wf.cwl.json:116:1:   found character '\t' that cannot start any token

[jfennick]

FYI the two errors about tabs can be fixed by replacing the tabs with spaces in tests/subgraph/count-lines17-wf.cwl.json.

The other error about Docker and MPI being mutually exclusive seems to be hardcoded in the decision tree starting at line 430 of command_line_tool.py I'm not exactly sure what you want to do to fix that. I tried swapping the hints and requirements in tests/wf/mpi_line_count.cwl which (as per the decision tree) ignores MPI. That 'fixes' that test, but note that you will also need to make the same writable=volume.staged change I made in #1708 to get test_iwdr_permutations to pass. WIth those changes, it works on my machine.

diff --git a/cwltool/docker.py b/cwltool/docker.py
index 1d7c9f97..98fe13ad 100644
--- a/cwltool/docker.py
+++ b/cwltool/docker.py
@@ -260,7 +260,9 @@ class DockerCommandLineJob(ContainerCommandLineJob):
         """Append volume a file/dir mapping to the runtime option list."""
         if not volume.resolved.startswith("_:"):
             _check_docker_machine_path(volume.resolved)
-            self.append_volume(runtime, volume.resolved, volume.target)
+            self.append_volume(
+                runtime, volume.resolved, volume.target, writable=volume.staged
+            )
 
     def add_writable_file_volume(
         self,
diff --git a/tests/subgraph/count-lines17-wf.cwl.json b/tests/subgraph/count-lines17-wf.cwl.json
index fa0905ef..279790b8 100644
--- a/tests/subgraph/count-lines17-wf.cwl.json
+++ b/tests/subgraph/count-lines17-wf.cwl.json
@@ -113,10 +113,10 @@
                       "-l"
                     ],
                     "class": "CommandLineTool",
-                   "hints": {
-                     "DockerRequirement": {
-                        "dockerPull": "docker.io/debian:stable-slim"
-                     }
+                    "hints": {
+                      "DockerRequirement": {
+                        "dockerPull": "docker.io/debian:stable-slim"
+                      }
                     },
                     "id": "count-lines17-wf.cwl@step_step1@run@step_stepX@run@step_stepY@run",
                     "inputs": [
diff --git a/tests/wf/mpi_line_count.cwl b/tests/wf/mpi_line_count.cwl
index a211506f..eb570280 100644
--- a/tests/wf/mpi_line_count.cwl
+++ b/tests/wf/mpi_line_count.cwl
@@ -3,7 +3,7 @@
 cwlVersion: v1.1
 class: CommandLineTool
 
-hints:
+requirements:
   DockerRequirement:
    dockerPull: docker.io/debian:stable-slim
 
@@ -15,7 +15,7 @@ doc: |
 
 baseCommand: wc
 
-requirements:
+hints:
   cwltool:MPIRequirement:
     processes: 1

[lgtm-com]

This pull request fixes 6 alerts when merging c047adb into f3a35f3 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[lgtm-com]

This pull request fixes 6 alerts when merging 64ed573 into f3a35f3 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[lgtm-com]

This pull request fixes 6 alerts when merging 2df4452 into f3a35f3 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[mr-c]

Thanks @jfennick for the testing; alas docker is not included on the GitHub hosted action runners due to licensing issues and the popular community provided action for install docker on macos is not reliably working right now. Can you test this branch one more time with docker on your macos system?

[lgtm-com]

This pull request fixes 6 alerts when merging 6e10ec6 into f3a35f3 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[jfennick]

pytest now works on my machine, but tox still gives 3 failures:

========================================================================================================================== FAILURES ==========================================================================================================================
___________________________________________________________________________________________________________________ test_iwdr_permutations ___________________________________________________________________________________________________________________
[gw3] darwin -- Python 3.10.5 /Users/jakefennick/cwltool/.tox/py310-unit/bin/python

tmp_path_factory = TempPathFactory(_given_basetemp=PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw3'), _trace=<pluggy._tracing.TagTracerSub object at 0x1047e80d0>, _basetemp=PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw3'))

    @needs_docker
    def test_iwdr_permutations(tmp_path_factory: Any) -> None:
        misc = tmp_path_factory.mktemp("misc")
        fifth = misc / "fifth"
        fifth.mkdir()
        sixth = misc / "sixth"
        sixth.mkdir()
        seventh = misc / "seventh"
        seventh.mkdir()
        eighth = misc / "eighth"
        eighth.mkdir()
        first = misc / "first"
        first.touch()
        second = misc / "second"
        second.touch()
        third = misc / "third"
        third.touch()
        fourth = misc / "fourth"
        fourth.touch()
        eleventh = misc / "eleventh"
        eleventh.touch()
        twelfth = misc / "twelfth"
        twelfth.touch()
        outdir = str(tmp_path_factory.mktemp("outdir"))
        err_code, stdout, _ = get_main_output(
            [
                "--outdir",
                outdir,
                "--debug",
                get_data("tests/wf/iwdr_permutations.cwl"),
                "--first",
                str(first),
                "--second",
                str(second),
                "--third",
                str(third),
                "--fourth",
                str(fourth),
                "--fifth",
                str(fifth),
                "--sixth",
                str(sixth),
                "--seventh",
                str(seventh),
                "--eighth",
                str(eighth),
                "--eleventh",
                str(eleventh),
                "--eleventh",
                str(twelfth),
            ]
        )
        assert err_code == 0
        log = json.loads(stdout)["log"]
>       assert log["checksum"] == "sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f", log
E       AssertionError: {'basename': 'log.txt', 'checksum': 'sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131', 'class': 'File', 'location': 'file:///Users/jakefennick/cwltool/tmp/popen-gw3/outdir0/docker_tmpiw1z7hc3/log.txt', ...}
E       assert 'sha1$9cf9aad...b8d751d2a5131' == 'sha1$bc51ebb...e9747d8abe75f'
E         - sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f
E         + sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131

tests/test_iwdr.py:134: AssertionError
-------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------
Using default tag: latest
latest: Pulling from library/debian
Digest: sha256:82bab30ed448b8e2509aabe21f40f0607d905b7fd0dec72802627a20274eba55
Status: Image is up to date for debian:latest
docker.io/library/debian:latest
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO     cwltool:main.py:1046  3.1.20220817095406
INFO     cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl' to 'file:///Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl'
DEBUG    cwltool:main.py:472 Parsed job order from command line: {
    "id": "/Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl",
    "eighth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth"
    },
    "eleventh": [
        {
            "class": "File",
            "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh"
        },
        {
            "class": "File",
            "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth"
        }
    ],
    "fifth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth"
    },
    "first": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first"
    },
    "fourth": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth"
    },
    "ninth": {
        "class": "Directory",
        "basename": "foo",
        "listing": []
    },
    "second": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second"
    },
    "seventh": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh"
    },
    "sixth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth"
    },
    "tenth": {
        "class": "Directory",
        "basename": "bar",
        "listing": []
    },
    "third": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third"
    }
}
DEBUG    cwltool:command_line_tool.py:1022 [job iwdr_permutations.cwl] initializing from file:///Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl
DEBUG    cwltool:command_line_tool.py:1030 [job iwdr_permutations.cwl] {
    "eighth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth",
        "basename": "eighth"
    },
    "eleventh": [
        {
            "class": "File",
            "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh",
            "size": 0,
            "basename": "eleventh",
            "nameroot": "eleventh",
            "nameext": ""
        },
        {
            "class": "File",
            "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth",
            "size": 0,
            "basename": "twelfth",
            "nameroot": "twelfth",
            "nameext": ""
        }
    ],
    "fifth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth",
        "basename": "fifth"
    },
    "first": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first",
        "size": 0,
        "basename": "first",
        "nameroot": "first",
        "nameext": ""
    },
    "fourth": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth",
        "size": 0,
        "basename": "fourth",
        "nameroot": "fourth",
        "nameext": ""
    },
    "ninth": {
        "class": "Directory",
        "basename": "foo",
        "listing": [],
        "location": "_:09f0ac62-f068-4c6c-b081-abec23241f5f"
    },
    "second": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second",
        "size": 0,
        "basename": "second",
        "nameroot": "second",
        "nameext": ""
    },
    "seventh": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh",
        "basename": "seventh"
    },
    "sixth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth",
        "basename": "sixth"
    },
    "tenth": {
        "class": "Directory",
        "basename": "bar",
        "listing": [],
        "location": "_:f6443482-d428-4988-9b8e-32aa9d4a546a"
    },
    "third": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third",
        "size": 0,
        "basename": "third",
        "nameroot": "third",
        "nameext": ""
    }
}
DEBUG    cwltool:command_line_tool.py:1046 [job iwdr_permutations.cwl] path mappings is {
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth",
        "/my_other_path/eighth_read_only_directory",
        "Directory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh",
        "/YWPMVO/eleventh",
        "File",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth",
        "/YWPMVO/twelfth",
        "File",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth",
        "/YWPMVO/fifth_writable_directory",
        "WritableDirectory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first",
        "/YWPMVO/first_writable_file",
        "WritableFile",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth",
        "/my_other_path/fourth_read_only_file",
        "File",
        false
    ],
    "_:09f0ac62-f068-4c6c-b081-abec23241f5f": [
        "_:09f0ac62-f068-4c6c-b081-abec23241f5f",
        "/YWPMVO/nineth_writable_directory_literal",
        "WritableDirectory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second",
        "/YWPMVO/second_read_only_file",
        "File",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh",
        "/my_path/seventh_writable_directory",
        "WritableDirectory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth",
        "/YWPMVO/sixth_read_only_directory",
        "Directory",
        false
    ],
    "_:f6443482-d428-4988-9b8e-32aa9d4a546a": [
        "_:f6443482-d428-4988-9b8e-32aa9d4a546a",
        "/my_path/tenth_writable_directory_literal",
        "WritableDirectory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third",
        "/my_path/third_writable_file",
        "WritableFile",
        false
    ]
}
DEBUG    cwltool:command_line_tool.py:1101 [job iwdr_permutations.cwl] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "bash"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "-c"
    },
    {
        "position": [
            0,
            0
        ],
        "datum": "find . | sort\nfind /my_path | sort\nfind /my_other_path | sort\necho \"a\" > first_writable_file\necho \"b\" > /my_path/third_writable_file\ntouch fifth_writable_directory/c\ntouch /my_path/seventh_writable_directory/d\nfind . | sort\nfind /my_path | sort\nfind /my_other_path | sort\n"
    }
]
INFO     cwltool:docker.py:155 ['docker', 'pull', 'docker.io/debian']
DEBUG    cwltool:job.py:222 [job iwdr_permutations.cwl] initial work dir {
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/first",
        "/YWPMVO/first_writable_file",
        "WritableFile",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second",
        "/YWPMVO/second_read_only_file",
        "File",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/third",
        "/my_path/third_writable_file",
        "WritableFile",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth",
        "/my_other_path/fourth_read_only_file",
        "File",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fifth",
        "/YWPMVO/fifth_writable_directory",
        "WritableDirectory",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth",
        "/YWPMVO/sixth_read_only_directory",
        "Directory",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/seventh",
        "/my_path/seventh_writable_directory",
        "WritableDirectory",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth",
        "/my_other_path/eighth_read_only_directory",
        "Directory",
        true
    ],
    "_:09f0ac62-f068-4c6c-b081-abec23241f5f": [
        "_:09f0ac62-f068-4c6c-b081-abec23241f5f",
        "/YWPMVO/nineth_writable_directory_literal",
        "WritableDirectory",
        true
    ],
    "_:f6443482-d428-4988-9b8e-32aa9d4a546a": [
        "_:f6443482-d428-4988-9b8e-32aa9d4a546a",
        "/my_path/tenth_writable_directory_literal",
        "WritableDirectory",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh",
        "/YWPMVO/eleventh",
        "File",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth",
        "/YWPMVO/twelfth",
        "File",
        true
    ],
    "_:165769e2-d381-49f7-ba3c-3a5321bd144b": [
        "baz",
        "/my_path/my_file_literal",
        "CreateFile",
        true
    ]
}
INFO     cwltool:job.py:276 [job iwdr_permutations.cwl] /private/tmp/docker_tmpiw1z7hc3$ docker \
    run \
    -i \
    --mount=type=bind,source=/private/tmp/docker_tmpiw1z7hc3,target=/YWPMVO \
    --mount=type=bind,source=/private/tmp/docker_tmpezipi7as,target=/tmp \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/second,target=/YWPMVO/second_read_only_file,readonly \
    --mount=type=bind,source=/private/tmp/docker_tmpp_cwer51/third,target=/my_path/third_writable_file \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/fourth,target=/my_other_path/fourth_read_only_file,readonly \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/sixth,target=/YWPMVO/sixth_read_only_directory,readonly \
    --mount=type=bind,source=/private/tmp/docker_tmpdec4fwng/seventh,target=/my_path/seventh_writable_directory \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eighth,target=/my_other_path/eighth_read_only_directory,readonly \
    --mount=type=bind,source=/private/tmp/docker_tmprov2ay4z/tenth_writable_directory_literal,target=/my_path/tenth_writable_directory_literal \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/eleventh,target=/YWPMVO/eleventh,readonly \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc0/twelfth,target=/YWPMVO/twelfth,readonly \
    --mount=type=bind,source=/private/tmp/docker_tmp01btz33i/my_file_literal,target=/my_path/my_file_literal,readonly \
    --workdir=/YWPMVO \
    --read-only=true \
    --net=none \
    --log-driver=none \
    --user=502:20 \
    --rm \
    --cidfile=/private/tmp/docker_tmpqxubkvvy/20220817090758-196751.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/YWPMVO \
    --env=LC_ALL=C \
    docker.io/debian \
    bash \
    -c \
    'find . | sort
find /my_path | sort
find /my_other_path | sort
echo "a" > first_writable_file
echo "b" > /my_path/third_writable_file
touch fifth_writable_directory/c
touch /my_path/seventh_writable_directory/d
find . | sort
find /my_path | sort
find /my_other_path | sort
' > /private/tmp/docker_tmpiw1z7hc3/log.txt
INFO     cwltool:job.py:943 [job iwdr_permutations.cwl] Max memory used: 0MiB
INFO     cwltool:job.py:433 [job iwdr_permutations.cwl] completed success
DEBUG    cwltool:job.py:436 [job iwdr_permutations.cwl] outputs {
    "log": {
        "location": "file:///private/tmp/docker_tmpiw1z7hc3/log.txt",
        "basename": "log.txt",
        "nameroot": "log",
        "nameext": ".txt",
        "class": "File",
        "checksum": "sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131",
        "size": 983,
        "http://commonwl.org/cwltool#generation": 0
    },
    "out": {
        "location": "file:///private/tmp/docker_tmpiw1z7hc3",
        "basename": "docker_tmpiw1z7hc3",
        "nameroot": "docker_tmpiw1z7hc3",
        "nameext": "",
        "class": "Directory"
    }
}
DEBUG    cwltool:job.py:464 [job iwdr_permutations.cwl] Removing input staging directory /private/tmp/docker_tmpx38wc7cw
DEBUG    cwltool:job.py:472 [job iwdr_permutations.cwl] Removing temporary directory /private/tmp/docker_tmpezipi7as
DEBUG    cwltool:process.py:371 Moving /private/tmp/docker_tmpiw1z7hc3 to /Users/jakefennick/cwltool/tmp/popen-gw3/outdir0/docker_tmpiw1z7hc3
INFO     cwltool:main.py:1393 Final process status is success
_______________________________________________________________________________________________________________ test_iwdr_permutations_inplace _______________________________________________________________________________________________________________
[gw3] darwin -- Python 3.10.5 /Users/jakefennick/cwltool/.tox/py310-unit/bin/python

tmp_path_factory = TempPathFactory(_given_basetemp=PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw3'), _trace=<pluggy._tracing.TagTracerSub object at 0x1047e80d0>, _basetemp=PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw3'))

    @needs_docker
    def test_iwdr_permutations_inplace(tmp_path_factory: Any) -> None:
        misc = tmp_path_factory.mktemp("misc")
        fifth = misc / "fifth"
        fifth.mkdir()
        sixth = misc / "sixth"
        sixth.mkdir()
        seventh = misc / "seventh"
        seventh.mkdir()
        eighth = misc / "eighth"
        eighth.mkdir()
        first = misc / "first"
        first.touch()
        second = misc / "second"
        second.touch()
        third = misc / "third"
        third.touch()
        fourth = misc / "fourth"
        fourth.touch()
        eleventh = misc / "eleventh"
        eleventh.touch()
        twelfth = misc / "twelfth"
        twelfth.touch()
        outdir = str(tmp_path_factory.mktemp("outdir"))
        err_code, stdout, _ = get_main_output(
            [
                "--outdir",
                outdir,
                "--enable-ext",
                "--overrides",
                get_data("tests/wf/iwdr_permutations_inplace.yml"),
                get_data("tests/wf/iwdr_permutations.cwl"),
                "--first",
                str(first),
                "--second",
                str(second),
                "--third",
                str(third),
                "--fourth",
                str(fourth),
                "--fifth",
                str(fifth),
                "--sixth",
                str(sixth),
                "--seventh",
                str(seventh),
                "--eighth",
                str(eighth),
                "--eleventh",
                str(eleventh),
                "--eleventh",
                str(twelfth),
            ]
        )
        assert err_code == 0
        log = json.loads(stdout)["log"]
>       assert log["checksum"] == "sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f", log
E       AssertionError: {'basename': 'log.txt', 'checksum': 'sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131', 'class': 'File', 'location': 'file:///Users/jakefennick/cwltool/tmp/popen-gw3/outdir2/docker_tmp864pxgrh/log.txt', ...}
E       assert 'sha1$9cf9aad...b8d751d2a5131' == 'sha1$bc51ebb...e9747d8abe75f'
E         - sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f
E         + sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131

tests/test_iwdr.py:243: AssertionError
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO     cwltool:main.py:1046  3.1.20220817095406
INFO     cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl' to 'file:///Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl'
DEBUG    cwltool:main.py:472 Parsed job order from command line: {
    "id": "/Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl",
    "eighth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth"
    },
    "eleventh": [
        {
            "class": "File",
            "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh"
        },
        {
            "class": "File",
            "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth"
        }
    ],
    "fifth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth"
    },
    "first": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first"
    },
    "fourth": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth"
    },
    "ninth": {
        "class": "Directory",
        "basename": "foo",
        "listing": []
    },
    "second": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second"
    },
    "seventh": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh"
    },
    "sixth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth"
    },
    "tenth": {
        "class": "Directory",
        "basename": "bar",
        "listing": []
    },
    "third": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third"
    }
}
DEBUG    cwltool:command_line_tool.py:1022 [job iwdr_permutations.cwl_2] initializing from file:///Users/jakefennick/cwltool/tests/wf/iwdr_permutations.cwl
DEBUG    cwltool:command_line_tool.py:1030 [job iwdr_permutations.cwl_2] {
    "eighth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth",
        "basename": "eighth"
    },
    "eleventh": [
        {
            "class": "File",
            "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh",
            "size": 0,
            "basename": "eleventh",
            "nameroot": "eleventh",
            "nameext": ""
        },
        {
            "class": "File",
            "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth",
            "size": 0,
            "basename": "twelfth",
            "nameroot": "twelfth",
            "nameext": ""
        }
    ],
    "fifth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth",
        "basename": "fifth"
    },
    "first": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first",
        "size": 0,
        "basename": "first",
        "nameroot": "first",
        "nameext": ""
    },
    "fourth": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth",
        "size": 0,
        "basename": "fourth",
        "nameroot": "fourth",
        "nameext": ""
    },
    "ninth": {
        "class": "Directory",
        "basename": "foo",
        "listing": [],
        "location": "_:d7c44520-b263-4ce1-977e-b0a55d92f303"
    },
    "second": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second",
        "size": 0,
        "basename": "second",
        "nameroot": "second",
        "nameext": ""
    },
    "seventh": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh",
        "basename": "seventh"
    },
    "sixth": {
        "class": "Directory",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth",
        "basename": "sixth"
    },
    "tenth": {
        "class": "Directory",
        "basename": "bar",
        "listing": [],
        "location": "_:1aed930b-811c-4a76-aec7-d4e347e3f5a6"
    },
    "third": {
        "class": "File",
        "location": "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third",
        "size": 0,
        "basename": "third",
        "nameroot": "third",
        "nameext": ""
    }
}
DEBUG    cwltool:command_line_tool.py:1046 [job iwdr_permutations.cwl_2] path mappings is {
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth",
        "/my_other_path/eighth_read_only_directory",
        "Directory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh",
        "/YWPMVO/eleventh",
        "File",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth",
        "/YWPMVO/twelfth",
        "File",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth",
        "/YWPMVO/fifth_writable_directory",
        "WritableDirectory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first",
        "/YWPMVO/first_writable_file",
        "WritableFile",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth",
        "/my_other_path/fourth_read_only_file",
        "File",
        false
    ],
    "_:d7c44520-b263-4ce1-977e-b0a55d92f303": [
        "_:d7c44520-b263-4ce1-977e-b0a55d92f303",
        "/YWPMVO/nineth_writable_directory_literal",
        "WritableDirectory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second",
        "/YWPMVO/second_read_only_file",
        "File",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh",
        "/my_path/seventh_writable_directory",
        "WritableDirectory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth",
        "/YWPMVO/sixth_read_only_directory",
        "Directory",
        false
    ],
    "_:1aed930b-811c-4a76-aec7-d4e347e3f5a6": [
        "_:1aed930b-811c-4a76-aec7-d4e347e3f5a6",
        "/my_path/tenth_writable_directory_literal",
        "WritableDirectory",
        false
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third",
        "/my_path/third_writable_file",
        "WritableFile",
        false
    ]
}
DEBUG    cwltool:command_line_tool.py:1101 [job iwdr_permutations.cwl_2] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "bash"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "-c"
    },
    {
        "position": [
            0,
            0
        ],
        "datum": "find . | sort\nfind /my_path | sort\nfind /my_other_path | sort\necho \"a\" > first_writable_file\necho \"b\" > /my_path/third_writable_file\ntouch fifth_writable_directory/c\ntouch /my_path/seventh_writable_directory/d\nfind . | sort\nfind /my_path | sort\nfind /my_other_path | sort\n"
    }
]
DEBUG    cwltool:job.py:222 [job iwdr_permutations.cwl_2] initial work dir {
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/first",
        "/YWPMVO/first_writable_file",
        "WritableFile",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second",
        "/YWPMVO/second_read_only_file",
        "File",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/third",
        "/my_path/third_writable_file",
        "WritableFile",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth",
        "/my_other_path/fourth_read_only_file",
        "File",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fifth",
        "/YWPMVO/fifth_writable_directory",
        "WritableDirectory",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth",
        "/YWPMVO/sixth_read_only_directory",
        "Directory",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/seventh",
        "/my_path/seventh_writable_directory",
        "WritableDirectory",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth",
        "/my_other_path/eighth_read_only_directory",
        "Directory",
        true
    ],
    "_:d7c44520-b263-4ce1-977e-b0a55d92f303": [
        "_:d7c44520-b263-4ce1-977e-b0a55d92f303",
        "/YWPMVO/nineth_writable_directory_literal",
        "WritableDirectory",
        true
    ],
    "_:1aed930b-811c-4a76-aec7-d4e347e3f5a6": [
        "_:1aed930b-811c-4a76-aec7-d4e347e3f5a6",
        "/my_path/tenth_writable_directory_literal",
        "WritableDirectory",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh",
        "/YWPMVO/eleventh",
        "File",
        true
    ],
    "file:///Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth": [
        "/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth",
        "/YWPMVO/twelfth",
        "File",
        true
    ],
    "_:1a3b779e-91c7-41d1-844c-c118a17aba55": [
        "baz",
        "/my_path/my_file_literal",
        "CreateFile",
        true
    ]
}
INFO     cwltool:job.py:276 [job iwdr_permutations.cwl_2] /private/tmp/docker_tmp864pxgrh$ docker \
    run \
    -i \
    --mount=type=bind,source=/private/tmp/docker_tmp864pxgrh,target=/YWPMVO \
    --mount=type=bind,source=/private/tmp/docker_tmp65c8saw7,target=/tmp \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/second,target=/YWPMVO/second_read_only_file,readonly \
    --mount=type=bind,source=/private/tmp/docker_tmp4cw3po0e/third,target=/my_path/third_writable_file \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/fourth,target=/my_other_path/fourth_read_only_file,readonly \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/sixth,target=/YWPMVO/sixth_read_only_directory,readonly \
    --mount=type=bind,source=/private/tmp/docker_tmphhcsd5f5/seventh,target=/my_path/seventh_writable_directory \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eighth,target=/my_other_path/eighth_read_only_directory,readonly \
    --mount=type=bind,source=/private/tmp/docker_tmpubgq0gqh/tenth_writable_directory_literal,target=/my_path/tenth_writable_directory_literal \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/eleventh,target=/YWPMVO/eleventh,readonly \
    --mount=type=bind,source=/Users/jakefennick/cwltool/tmp/popen-gw3/misc2/twelfth,target=/YWPMVO/twelfth,readonly \
    --mount=type=bind,source=/private/tmp/docker_tmpqgsqof9n/my_file_literal,target=/my_path/my_file_literal,readonly \
    --workdir=/YWPMVO \
    --read-only=true \
    --net=none \
    --log-driver=none \
    --user=502:20 \
    --rm \
    --cidfile=/private/tmp/docker_tmplagziroh/20220817090803-686776.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/YWPMVO \
    --env=LC_ALL=C \
    docker.io/debian \
    bash \
    -c \
    'find . | sort
find /my_path | sort
find /my_other_path | sort
echo "a" > first_writable_file
echo "b" > /my_path/third_writable_file
touch fifth_writable_directory/c
touch /my_path/seventh_writable_directory/d
find . | sort
find /my_path | sort
find /my_other_path | sort
' > /private/tmp/docker_tmp864pxgrh/log.txt
INFO     cwltool:job.py:943 [job iwdr_permutations.cwl_2] Max memory used: 0MiB
INFO     cwltool:job.py:433 [job iwdr_permutations.cwl_2] completed success
DEBUG    cwltool:job.py:436 [job iwdr_permutations.cwl_2] outputs {
    "log": {
        "location": "file:///private/tmp/docker_tmp864pxgrh/log.txt",
        "basename": "log.txt",
        "nameroot": "log",
        "nameext": ".txt",
        "class": "File",
        "checksum": "sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131",
        "size": 983,
        "http://commonwl.org/cwltool#generation": 0
    },
    "out": {
        "location": "file:///private/tmp/docker_tmp864pxgrh",
        "basename": "docker_tmp864pxgrh",
        "nameroot": "docker_tmp864pxgrh",
        "nameext": "",
        "class": "Directory"
    }
}
DEBUG    cwltool:job.py:464 [job iwdr_permutations.cwl_2] Removing input staging directory /private/tmp/docker_tmppdvy1ol_
DEBUG    cwltool:job.py:472 [job iwdr_permutations.cwl_2] Removing temporary directory /private/tmp/docker_tmp65c8saw7
DEBUG    cwltool:process.py:371 Moving /private/tmp/docker_tmp864pxgrh to /Users/jakefennick/cwltool/tmp/popen-gw3/outdir2/docker_tmp864pxgrh
INFO     cwltool:main.py:1393 Final process status is success
____________________________________________________________________________________________________________________ test_cid_file_dir[] _____________________________________________________________________________________________________________________
[gw1] darwin -- Python 3.10.5 /Users/jakefennick/cwltool/.tox/py310-unit/bin/python

tmp_path = PosixPath('/Users/jakefennick/cwltool/tmp/popen-gw1/test_cid_file_dir__0'), factor = ''

    @needs_docker
    @pytest.mark.parametrize("factor", test_factors)
    def test_cid_file_dir(tmp_path: Path, factor: str) -> None:
        """Test --cidfile-dir option works."""
        test_file = "cache_test_workflow.cwl"
        with working_directory(tmp_path):
            commands = factor.split()
            commands.extend(
                ["--cidfile-dir", str(tmp_path), get_data("tests/wf/" + test_file)]
            )
            error_code, stdout, stderr = get_main_output(commands)
            stderr = re.sub(r"\s\s+", " ", stderr)
            assert "completed success" in stderr
            assert error_code == 0
            cidfiles_count = sum(1 for _ in tmp_path.glob("**/*"))
>           assert cidfiles_count == 2
E           assert 5 == 2

tests/test_examples.py:1124: AssertionError
-------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
Status: Image is up to date for alpine:latest
docker.io/library/alpine:latest
--------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------
INFO     cwltool:main.py:1046  3.1.20220817095406
INFO     cwltool:load_tool.py:102 Resolved '/Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl' to 'file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl'
DEBUG    cwltool:main.py:472 Parsed job order from command line: {
    "id": "/Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl"
}
DEBUG    cwltool:workflow_job.py:503 [workflow _10] initialized from file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl
INFO     cwltool:workflow_job.py:791 [workflow _10] start
DEBUG    cwltool:workflow_job.py:803 [workflow _10] inputs {}
INFO     cwltool:workflow_job.py:628 [workflow _10] starting step task1
DEBUG    cwltool:workflow_job.py:756 [step task1] job input {
    "file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl#task1/message": "one"
}
DEBUG    cwltool:workflow_job.py:763 [step task1] evaluated job input to {
    "file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl#task1/message": "one"
}
INFO     cwltool:workflow_job.py:74 [step task1] start
DEBUG    cwltool:command_line_tool.py:1022 [job task1] initializing from file:///Users/jakefennick/cwltool/tests/wf/touch_tool.cwl as part of step task1
DEBUG    cwltool:command_line_tool.py:1030 [job task1] {
    "message": "one"
}
DEBUG    cwltool:command_line_tool.py:1046 [job task1] path mappings is {}
DEBUG    cwltool:command_line_tool.py:1101 [job task1] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "touch"
    },
    {
        "position": [
            1,
            "message"
        ],
        "datum": "one"
    }
]
INFO     cwltool:docker.py:155 ['docker', 'pull', 'docker.io/alpine']
DEBUG    cwltool:job.py:222 [job task1] initial work dir {}
INFO     cwltool:job.py:276 [job task1] /private/tmp/docker_tmp7uak_pw8$ docker \
    run \
    -i \
    --mount=type=bind,source=/private/tmp/docker_tmp7uak_pw8,target=/WXkvbN \
    --mount=type=bind,source=/private/tmp/docker_tmpmvbb_ve4,target=/tmp \
    --workdir=/WXkvbN \
    --read-only=true \
    --user=502:20 \
    --rm \
    --cidfile=/Users/jakefennick/cwltool/tmp/popen-gw1/test_cid_file_dir__0/20220817090807-856622.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/WXkvbN \
    docker.io/alpine \
    touch \
    one
INFO     cwltool:job.py:943 [job task1] Max memory used: 0MiB
INFO     cwltool:job.py:433 [job task1] completed success
DEBUG    cwltool:job.py:436 [job task1] outputs {
    "out": {
        "location": "file:///private/tmp/docker_tmp7uak_pw8/one",
        "basename": "one",
        "nameroot": "one",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
DEBUG    cwltool:workflow_job.py:577 [step task1] produced output {}
INFO     cwltool:workflow_job.py:587 [step task1] completed success
DEBUG    cwltool:job.py:464 [job task1] Removing input staging directory /private/tmp/docker_tmpp43ki28h
DEBUG    cwltool:job.py:472 [job task1] Removing temporary directory /private/tmp/docker_tmpmvbb_ve4
INFO     cwltool:workflow_job.py:628 [workflow _10] starting step task2
DEBUG    cwltool:workflow_job.py:756 [step task2] job input {
    "file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl#task2/message": "two"
}
DEBUG    cwltool:workflow_job.py:763 [step task2] evaluated job input to {
    "file:///Users/jakefennick/cwltool/tests/wf/cache_test_workflow.cwl#task2/message": "two"
}
INFO     cwltool:workflow_job.py:74 [step task2] start
DEBUG    cwltool:command_line_tool.py:1022 [job task2] initializing from file:///Users/jakefennick/cwltool/tests/wf/touch_tool.cwl as part of step task2
DEBUG    cwltool:command_line_tool.py:1030 [job task2] {
    "message": "two"
}
DEBUG    cwltool:command_line_tool.py:1046 [job task2] path mappings is {}
DEBUG    cwltool:command_line_tool.py:1101 [job task2] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "touch"
    },
    {
        "position": [
            1,
            "message"
        ],
        "datum": "two"
    }
]
DEBUG    cwltool:job.py:222 [job task2] initial work dir {}
INFO     cwltool:job.py:276 [job task2] /private/tmp/docker_tmpogdab0mc$ docker \
    run \
    -i \
    --mount=type=bind,source=/private/tmp/docker_tmpogdab0mc,target=/WXkvbN \
    --mount=type=bind,source=/private/tmp/docker_tmpiwl2mrok,target=/tmp \
    --workdir=/WXkvbN \
    --read-only=true \
    --user=502:20 \
    --rm \
    --cidfile=/Users/jakefennick/cwltool/tmp/popen-gw1/test_cid_file_dir__0/20220817090808-893250.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/WXkvbN \
    docker.io/alpine \
    touch \
    two
INFO     cwltool:job.py:943 [job task2] Max memory used: 0MiB
INFO     cwltool:job.py:433 [job task2] completed success
DEBUG    cwltool:job.py:436 [job task2] outputs {
    "out": {
        "location": "file:///private/tmp/docker_tmpogdab0mc/two",
        "basename": "two",
        "nameroot": "two",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
DEBUG    cwltool:workflow_job.py:577 [step task2] produced output {}
INFO     cwltool:workflow_job.py:587 [step task2] completed success
INFO     cwltool:workflow_job.py:549 [workflow _10] completed success
DEBUG    cwltool:workflow_job.py:551 [workflow _10] outputs {}
DEBUG    cwltool:job.py:464 [job task2] Removing input staging directory /private/tmp/docker_tmp_5x9uv03
DEBUG    cwltool:job.py:472 [job task2] Removing temporary directory /private/tmp/docker_tmpiwl2mrok
DEBUG    cwltool:process.py:424 Removing intermediate output directory /private/tmp/docker_tmp7uak_pw8
DEBUG    cwltool:process.py:424 Removing intermediate output directory /private/tmp/docker_tmpw197uwi0
DEBUG    cwltool:process.py:424 Removing intermediate output directory /private/tmp/docker_tmpogdab0mc
INFO     cwltool:main.py:1393 Final process status is success

[kinow]

Changes look OK, but I don't have a macos to test it 😞 . I tried testing the GH action, and left a comment in the code review 👍

I saw some tests failing to pull a Docker image with udocker. Didn't spend more time looking into that (Friday 5:30 pm here now, ⛱️ 🥳 ), but that error reminded me of this issue in udocker: indigo-dc/udocker#359

[lgtm-com]

This pull request fixes 6 alerts when merging 9a62868 into 79fd363 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[mr-c]

@jfennick @kinow I've improved test failure reporting for those tests; they should show the unexpected output

[lgtm-com]

This pull request fixes 6 alerts when merging 577cc29 into 79fd363 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[lgtm-com]

This pull request fixes 6 alerts when merging 369eddd into 49fc1c3 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[lgtm-com]

This pull request fixes 6 alerts when merging c308537 into 0e2ced5 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[jfennick]

FYI now I'm getting a pytest failure on test_bioconda:

UnavailableInvalidChannel: The channel is not accessible or is invalid.
  channel name: pkgs/main
  channel url: https://repo.anaconda.com/pkgs/main
  error code: 404

You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.


Traceback (most recent call last):
  File "/private/tmp/docker_tmpeuhhq6t1/run_job.py", line 101, in <module>
    sys.exit(main(sys.argv))
  File "/private/tmp/docker_tmpeuhhq6t1/run_job.py", line 78, in main
    sp = subprocess.Popen(  # nosec
  File "/Users/jakefennick/opt/anaconda3/envs/lol/lib/python3.10/subprocess.py", line 969, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/jakefennick/opt/anaconda3/envs/lol/lib/python3.10/subprocess.py", line 1845, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'seqtk'

The problem is my .condarc file was

channels:
  - conda-forge
  - omnia
  - defaults

The pytest failure goes away if I add

channels:
  - conda-forge
  - omnia
  - defaults
  - main

Perhaps a slightly better error message would be nice, considering the main channel is not added by default.

Here are the new and improved error messages for the tox failures:

E       AssertionError: .
E         ./.docker
E         ./.docker/contexts
E         ./.docker/contexts/meta
E         ./eleventh
E         ./fifth_writable_directory
E         ./first_writable_file
E         ./log.txt
E         ./nineth_writable_directory_literal
E         ./second_read_only_file
E         ./sixth_read_only_directory
E         ./twelfth
E         /my_path
E         /my_path/my_file_literal
E         /my_path/seventh_writable_directory
E         /my_path/tenth_writable_directory_literal
E         /my_path/third_writable_file
E         /my_other_path
E         /my_other_path/eighth_read_only_directory
E         /my_other_path/fourth_read_only_file
E         .
E         ./.docker
E         ./.docker/contexts
E         ./.docker/contexts/meta
E         ./eleventh
E         ./fifth_writable_directory
E         ./fifth_writable_directory/c
E         ./first_writable_file
E         ./log.txt
E         ./nineth_writable_directory_literal
E         ./second_read_only_file
E         ./sixth_read_only_directory
E         ./twelfth
E         /my_path
E         /my_path/my_file_literal
E         /my_path/seventh_writable_directory
E         /my_path/seventh_writable_directory/d
E         /my_path/tenth_writable_directory_literal
E         /my_path/third_writable_file
E         /my_other_path
E         /my_other_path/eighth_read_only_directory
E         /my_other_path/fourth_read_only_file
E         
E       assert 'sha1$9cf9aad...b8d751d2a5131' == 'sha1$bc51ebb...e9747d8abe75f'
E         - sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f
E         + sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131

tests/test_iwdr.py:136: AssertionError

E       AssertionError: .
E         ./.docker
E         ./.docker/contexts
E         ./.docker/contexts/meta
E         ./eleventh
E         ./fifth_writable_directory
E         ./first_writable_file
E         ./log.txt
E         ./nineth_writable_directory_literal
E         ./second_read_only_file
E         ./sixth_read_only_directory
E         ./twelfth
E         /my_path
E         /my_path/my_file_literal
E         /my_path/seventh_writable_directory
E         /my_path/tenth_writable_directory_literal
E         /my_path/third_writable_file
E         /my_other_path
E         /my_other_path/eighth_read_only_directory
E         /my_other_path/fourth_read_only_file
E         .
E         ./.docker
E         ./.docker/contexts
E         ./.docker/contexts/meta
E         ./eleventh
E         ./fifth_writable_directory
E         ./fifth_writable_directory/c
E         ./first_writable_file
E         ./log.txt
E         ./nineth_writable_directory_literal
E         ./second_read_only_file
E         ./sixth_read_only_directory
E         ./twelfth
E         /my_path
E         /my_path/my_file_literal
E         /my_path/seventh_writable_directory
E         /my_path/seventh_writable_directory/d
E         /my_path/tenth_writable_directory_literal
E         /my_path/third_writable_file
E         /my_other_path
E         /my_other_path/eighth_read_only_directory
E         /my_other_path/fourth_read_only_file
E         
E       assert 'sha1$9cf9aad...b8d751d2a5131' == 'sha1$bc51ebb...e9747d8abe75f'
E         - sha1$bc51ebb3f65ca44282789dd1e6de9747d8abe75f
E         + sha1$9cf9aad015b3dcc9cc9daa368dbb8d751d2a5131

tests/test_iwdr.py:247: AssertionError

E           AssertionError: Should be 2 cidfiles, but got [PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/.docker'), PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/20220822073109-807385.cid'), PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/20220822073110-839889.cid'), PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/.docker/contexts'), PosixPath('/Users/jakefennick/bbq/cwltool/tmp/popen-gw1/test_cid_file_dir__0/.docker/contexts/meta')]
E           assert 5 == 2

tests/test_examples.py:1125: AssertionError

Looks like the .docker/ directory is messing thing up.

[mr-c]

@jfennick Thanks for the information!

I can't find any documentation on why Docker for Desktop, Mac edition adds that directory (and how to turn that off), so I've just hacked around it for now.

[lgtm-com]

This pull request fixes 6 alerts when merging 22cd86d into 0e2ced5 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[jfennick]

Looks like we're just about done. With this change, both pytest and tox work on my machine.

diff --git a/tests/test_examples.py b/tests/test_examples.py
index b6a7edab..0516a6df 100644
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -1120,7 +1120,7 @@ def test_cid_file_dir(tmp_path: Path, factor: str) -> None:
         stderr = re.sub(r"\s\s+", " ", stderr)
         assert "completed success" in stderr
         assert error_code == 0
-        cidfiles = list(tmp_path.glob("**/*"))
+        cidfiles = list(tmp_path.glob("**/*.cid"))
         cidfiles_count = len(cidfiles)
         assert cidfiles_count == 2, f"Should be 2 cidfiles, but got {cidfiles}"

[lgtm-com]

This pull request fixes 6 alerts when merging 4b59481 into 0e2ced5 - view on LGTM.com

fixed alerts:

• 6 for Unused import

[jfennick]

pytest and tox are now both working on my machine. However, I'm still running into #1694

[lgtm-com]

This pull request fixes 6 alerts when merging bbda773 into 92c4073 - view on LGTM.com

fixed alerts:

• 6 for Unused import