[BUG] Improper escape of curly bracketed variable when used as a default value for a file argument.

[DriesSchaumont]

What happened?

Using curly brackets as a value for the default: field of a argument that is of type file leads to unboud variable error in bash.

Steps to reproduce

mkdir -p /tmp/test_bug
cd /tmp/test_bug
cat > _viash.yaml << 'HERE'
viash_version: 0.8.6

source: src
target: target
HERE

mkdir -p src/dummycomponent

cat > src/dummycomponent/config.vsh.yaml << 'HERE'
functionality:
  name: myscript
  namespace: "dummycomponent"
  arguments:
    - name: "--output"
      type: file
      direction: output
      default: "${id}"
  resources:
    - type: python_script
      path: script.py
      text: |
        print("OK!", flush=True)
platforms:
  - type: nextflow
HERE

mkdir -p src/dummyworkflow
cat > src/dummyworkflow/config.vsh.yaml << 'HERE'
functionality:
  name: myyworkflow
  namespace: "dummyworkflow"
  arguments:
    - name: "--output"
      type: file
      direction: output
      default: "${id}"
  resources:
    - type: nextflow_script
      path: main.nf
      text: |
        workflow run_wf {
          take:
            input_ch

          main:
            output_ch = input_ch
              | myscript

          emit:
            output_ch
        }
      entrypoint: run_wf
  dependencies:
    - name: dummycomponent/myscript
platforms:
  - type: nextflow
HERE

viash ns build
nextflow run . -main-script ./target/nextflow/dummyworkflow/myyworkflow/main.nf --publish_dir .

Expected behavior

Both ${id} and $id are correct syntax for groovy variables. Sometimes, using the ${} form is preferred because it clearly delineates where the variable name ends and the rest of the string begins when doing string interpolation: $id_foo vs ${id}_foo.

I would expect to use ${id} without it being interpreted a s a bash variable, but still being to use it as a proper default value for type: file

Relevant log output

ERROR ~ Error executing process > 'myyworkflow:run_wf:myscript:processWf:myscript_process (run)'

Caused by:
  Process `myyworkflow:run_wf:myscript:processWf:myscript_process (run)` terminated with an error exit status (1)

Command executed:

  # meta exports
  export VIASH_META_RESOURCES_DIR=".viash_meta_resources"
  export VIASH_META_TEMP_DIR="/var/folders/fv/p0mc5zps48v15zw2q667t6gw0000gn/T"
  export VIASH_META_FUNCTIONALITY_NAME="myscript"
  # export VIASH_META_EXECUTABLE="$VIASH_META_RESOURCES_DIR/$VIASH_META_FUNCTIONALITY_NAME"
  export VIASH_META_CONFIG="$VIASH_META_RESOURCES_DIR/.config.vsh.yaml"
  export VIASH_META_CPUS=1

  if [ ! -z ${VIASH_META_MEMORY_B+x} ]; then
    export VIASH_META_MEMORY_KB=$(( ($VIASH_META_MEMORY_B+1023) / 1024 ))
    export VIASH_META_MEMORY_MB=$(( ($VIASH_META_MEMORY_KB+1023) / 1024 ))
    export VIASH_META_MEMORY_GB=$(( ($VIASH_META_MEMORY_MB+1023) / 1024 ))
    export VIASH_META_MEMORY_TB=$(( ($VIASH_META_MEMORY_GB+1023) / 1024 ))
    export VIASH_META_MEMORY_PB=$(( ($VIASH_META_MEMORY_TB+1023) / 1024 ))
  fi

  # meta synonyms
  export VIASH_TEMP="$VIASH_META_TEMP_DIR"
  export TEMP_DIR="$VIASH_META_TEMP_DIR"

  # create output dirs if need be
  function mkdir_parent {
    for file in "$@"; do
      mkdir -p "$(dirname "$file")"
    done
  }
  mkdir_parent "${id}"

  # argument exports
  export VIASH_PAR_OUTPUT="${id}"

  # process script
  set -e
  tempscript=".viash_script.sh"
  cat > "$tempscript" << VIASHMAIN
  ## VIASH START
  # The following code has been auto-generated by Viash.
  par = {
    'output': $( if [ ! -z ${VIASH_PAR_OUTPUT+x} ]; then echo "r'${VIASH_PAR_OUTPUT//\'/\'\"\'\"r\'}'"; else echo None; fi )
  }
  meta = {
    'functionality_name': $( if [ ! -z ${VIASH_META_FUNCTIONALITY_NAME+x} ]; then echo "r'${VIASH_META_FUNCTIONALITY_NAME//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'resources_dir': $( if [ ! -z ${VIASH_META_RESOURCES_DIR+x} ]; then echo "r'${VIASH_META_RESOURCES_DIR//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'executable': $( if [ ! -z ${VIASH_META_EXECUTABLE+x} ]; then echo "r'${VIASH_META_EXECUTABLE//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'config': $( if [ ! -z ${VIASH_META_CONFIG+x} ]; then echo "r'${VIASH_META_CONFIG//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'temp_dir': $( if [ ! -z ${VIASH_META_TEMP_DIR+x} ]; then echo "r'${VIASH_META_TEMP_DIR//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'cpus': $( if [ ! -z ${VIASH_META_CPUS+x} ]; then echo "int(r'${VIASH_META_CPUS//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_b': $( if [ ! -z ${VIASH_META_MEMORY_B+x} ]; then echo "int(r'${VIASH_META_MEMORY_B//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_kb': $( if [ ! -z ${VIASH_META_MEMORY_KB+x} ]; then echo "int(r'${VIASH_META_MEMORY_KB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_mb': $( if [ ! -z ${VIASH_META_MEMORY_MB+x} ]; then echo "int(r'${VIASH_META_MEMORY_MB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_gb': $( if [ ! -z ${VIASH_META_MEMORY_GB+x} ]; then echo "int(r'${VIASH_META_MEMORY_GB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_tb': $( if [ ! -z ${VIASH_META_MEMORY_TB+x} ]; then echo "int(r'${VIASH_META_MEMORY_TB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_pb': $( if [ ! -z ${VIASH_META_MEMORY_PB+x} ]; then echo "int(r'${VIASH_META_MEMORY_PB//\'/\'\"\'\"r\'}')"; else echo None; fi )
  }
  dep = {

  }

  ## VIASH END
  print("OK!", flush=True)
  VIASHMAIN
  python -B "$tempscript"

Command exit status:
  1

Command output:
  (empty)

Command error:
  .command.sh: line 28: id: unbound variable

Work dir:
  /private/tmp/test_bug/work/f4/ae306dde928adacf0665e3cab1c7c6

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details

Version

• OS: MacOS 14.4.1 (23E224)
• openjdk 21.0.2
• Nextflow version 23.10.1 build 5891

Possible solution

No response

Confirmation

• I have searched the existing issues to make sure this is not a duplicate.
• I have provided clear and concise information about the bug.

Additional context

No response

[rcannood]

After having discussed a little bit with @Grifs, this seems to be multiple bugs happening at once. The bugs we can fix, Hendrik is working on in #756. As part of #756, ${id} will be detected in the same way $id is already being detected.

However, there seems to be a bug in Nextflow also not properly escaping input files with dollar signs in them.

Reprex:

main.nf:

process copy_file {
  input:
    tuple path(input), val(output)
  output:
    path{output}
  script:
  """
  cp '${input}' '${output}'
  """
}

workflow {
  Channel.fromList([
    [file(params.input), params.output]
  ])
    | view{"in: $it"}
    | copy_file
    | view{"out: $it"}
}

This works when I provide a regular filename as input path:

$ touch foo
$ nextflow run main.nf --input foo --output output.txt
Nextflow 24.04.4 is available - Please consider updating your version to it
N E X T F L O W  ~  version 23.10.0
Launching `main.nf` [backstabbing_lattes] DSL2 - revision: d31f18c0ba
executor >  local (1)
[27/b8d1c1] process > copy_file (1) [100%] 1 of 1 ✔
in: [/home/rcannood/workspace/viash-io/viash/foo, output.txt]
out: /home/rcannood/workspace/viash-io/viash/work/27/b8d1c112cdd266134cb703eb422145/output.txt

However when I add a '$' to the filename:

$ touch '$foo'
$ nextflow run main.nf --input \$foo --output output.txt

Nextflow 24.04.4 is available - Please consider updating your version to it
N E X T F L O W  ~  version 23.10.0
Launching `main.nf` [cheesy_banach] DSL2 - revision: d31f18c0ba
executor >  local (1)
[04/23b0d9] process > copy_file (1) [100%] 1 of 1, failed: 1 ✘
in: [/home/rcannood/workspace/viash-io/viash/$foo, output.txt]
ERROR ~ Error executing process > 'copy_file (1)'

Caused by:
  Process `copy_file (1)` terminated with an error exit status (1)

Command executed:

  cp '$foo' 'output.txt'

Command exit status:
  1

Command output:
  (empty)

Command error:
  .command.run: line 98: foo: unbound variable

Work dir:
  /home/rcannood/workspace/viash-io/viash/work/04/23b0d9fcafe235eaa6d11f85ee6dd5

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

The error occurs during the staging of the input files:

nxf_stage() {
    true
    # stage input files
    rm -f $foo
    ln -s /home/rcannood/workspace/viash-io/viash/$foo $foo
}

We should create an issue for this in nextflow-io/nextflow.

[Grifs]

Fixed in Viash 0.9.0-RC8