WIP: Add guide for running CWL workflows on Puhti

[mr-c]

No description provided.

[joonas-somero]

I tried following the tutorial and

• Looks like by default in Puhti only python3 is in $PATH, so I used python3 -m venv venv. (I noted that python is available inside the venv.)
• Pip informed me that toil requires Python '>=3.7' but the running Python is 3.6.8.

[mr-c]

I tried following the tutorial and

* Looks like by default in Puhti only `python3` is in $PATH, so I used `python3 -m venv venv`. (I noted that `python` _is_ available inside the `venv`.)

Fixed, thank you.

* Pip informed me that _toil requires Python '>=3.7' but the running Python is 3.6.8_.

@teemukataja I think you did a module load command to get a newer Python version, yes? What was it?

Looking at the other docs, I presume module load python-data/3.10-23.07 would work, but I don't have an account so I can't test that myself.

[joonas-somero]

Looking at the other docs, I presume module load python-data/3.10-23.07 would work, but I don't have an account so I can't test that myself.

ATM, without specifying a version, the command module load python-data loads python-data/3.10-22.09. I can't imagine a sudden downgrade to <3.7, so perhaps that would be a simple command to include in the instructions. With the module loaded, though, we get

$ python --version
Python 3.10.6

that makes my previous comment about python3 outdated. With this, I'll see if I can follow the instructions a bit further.

[teemukataja]

I tried following the tutorial and

* Looks like by default in Puhti only `python3` is in $PATH, so I used `python3 -m venv venv`. (I noted that `python` _is_ available inside the `venv`.)

Fixed, thank you.

* Pip informed me that _toil requires Python '>=3.7' but the running Python is 3.6.8_.

@teemukataja I think you did a module load command to get a newer Python version, yes? What was it?

Looking at the other docs, I presume module load python-data/3.10-23.07 would work, but I don't have an account so I can't test that myself.

I checked the history, we used python3.9 -m venv venv

[joonas-somero]

I successfully ran a "hello world" on Puhti after containerizing toil[cwl] installation with Tykky. I'll post the details next week, if not today.

[mr-c]

I successfully ran a "hello world" on Puhti after containerizing toil[cwl] installation with Tykky. I'll post the details next week, if not today.

Cool!

Did that "hello world" CWL description reference its own container?

[joonas-somero]

Did that "hello world" CWL description reference its own container?

I just used this .cwl file, but instead of example-job.yaml, message was defined with toil-cwl-runner ... --message "Hello world!".

[mr-c]

Did that "hello world" CWL description reference its own container?

I just used this .cwl file, but instead of example-job.yaml, message was defined with toil-cwl-runner ... --message "Hello world!".

I'm curious as to how that will work with calling another container from that containerized toil-cwl-runner app is working; as @teemukataja and I didn't test that method.

Can you test again using toil-cwl-runner --singularity --default-container=docker.io/debian:stable-slim ... ?

I'm also concerned about file paths, and access to nodejs for evaluating CWL expressions. If the Tykky version is promoted, I would be more comfortable if the entire CWL conformance test suite was run using that method.

To run the CWL conformance tests, do the following

1. Ensure that cwltest is installed in the virtualenv.

   source /projappl/<project_nnnnnnn>/venv/bin/activate
   pip install cwltest

2. Checkout the CWL v1.2 conformance tests:

   cd /projappl/<project_nnnnnnn>
   git checkout https://github.com/common-workflow-language/cwl-v1.2.git
   

3. Take the example sbatch script in this PR and replace the toil-cwl-runner invocation with the following:

   cd /projappl/<project_nnnnnnn>/cwl-v1.2
   ./run_test.sh RUNNER=toil-cwl-runner EXTRA="--batchSystem slurm"
   

(Or pass to RUNNER whatever the name/path to the Tykky version of toil-cwl-runner is)

[joonas-somero]

I'm curious as to how that will work with calling another container from that containerized toil-cwl-runner app is working; as @teemukataja and I didn't test that method.

Yes, that's what I've also been worrying curious about. 😅

Can you test again using toil-cwl-runner --singularity --default-container=docker.io/debian:stable-slim ... ?

I sure can!

I'm also concerned about file paths, and access to nodejs for evaluating CWL expressions. If the Tykky version is promoted, I would be more comfortable if the entire CWL conformance test suite was run using that method.

...

I'll have to get back to you next week, though. In the mean time, I modified the instructions to reflect what I did with Tykky and pushed them here.

[joonas-somero]

Can you test again using toil-cwl-runner --singularity --default-container=docker.io/debian:stable-slim ... ?

So, yeah, doesn't I can't get it to work. The project I tried the Pip installation under earlier has it's disk quota nearly filled, and I thought I could containerize the installation with Tykky to get the number of files down. Well, I did containerize the installation, but that isn't very helpful if the installation can't be used! 😃

[mr-c]

Can you test again using toil-cwl-runner --singularity --default-container=docker.io/debian:stable-slim ... ?

So, yeah, doesn't I can't get it to work. The project I tried the Pip installation under earlier has it's disk quota nearly filled, and I thought I could containerize the installation with Tykky to get the number of files down. Well, I did containerize the installation, but that isn't very helpful if the installation can't be used! 😃

I saw that in your setup, you are placing the temporary files in the same directory as the inputs/outputs instead of on /scratch:

f1e9991#diff-d62e70e58f6d3e0d7db596a87253f37c907b53a07f5a5f17c8edbbea5d42eb1aL98

Ah, nevermind, you put everything in /scratch. Maybe we can schedule a video chat with @teemukataja to debug in realtime?