job-manager: job submission fails with FileNotFoundError

[mdonadoni]

When running heavily parallelised workflows, it can happen that job submission fails with FileNotFoundError.

This is a sample traceback:

2024-01-15 10:09:33,244 | root | Thread-4710 | ERROR | Job submission failed. 
[Errno 2] No such file or directory: '/var/reana/users/[...]/workflows/[...]/.snakemake/incomplete/@[...]'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/reana_job_controller/rest.py", line 231, in create_job
    backend_jod_id = job_obj.execute()
  File "/usr/local/lib/python3.8/dist-packages/reana_job_controller/job_manager.py", line 64, in wrapper
    inst.cache_job()
  File "/usr/local/lib/python3.8/dist-packages/reana_job_controller/job_manager.py", line 132, in cache_job
    access_times = calculate_file_access_time(workflow.workspace_path)
  File "/usr/local/lib/python3.8/dist-packages/reana_commons/utils.py", line 122, in calculate_file_access_time
    access_times[file_path] = os.stat(file_path).st_atime
FileNotFoundError: [Errno 2] No such file or directory: '/var/reana/users/[...]/workflows/[...]/.snakemake/incomplete/@[...]'
2024-01-15 10:09:33,245 | werkzeug | Thread-4710 | INFO | 127.0.0.1 - - [15/Jan/2024 10:09:33] "POST /jobs HTTP/1.1" 500 -

[mdonadoni]

This workflow occasionally fails also locally (otherwise it runs forever):

Snakefile

rule all:
    input:
        "r1.txt",
        "r2.txt",
        "r3.txt"

# keep moving the file back and forth
rule r1:
    output:
        "r1.txt"
    container:
        "docker://docker.io/library/python:3.8-slim"
    shell:
        "touch a.txt && while : ; do mv a.txt b.txt && mv b.txt a.txt; done"

# dummy rule to delay the submission of rule 3
rule r2:
    output:
        "r2.txt"
    container:
        "docker://docker.io/library/python:3.8-slim"
    shell:
        "sleep 5 && touch r2.txt"

rule r3:
    input:
        "r2.txt"
    output:
        "r3.txt"
    container:
        "docker://docker.io/library/python:3.8-slim"
    shell:
        "touch r3.txt"

reana.yaml

version: 0.9.0
inputs:
  files:
    - Snakefile
workflow:
  type: snakemake
  file: Snakefile