SSE flag not supported when creating flagfiles or storing exit codes for s3 buckets with enforced encryption from source

[onaiv22]

Bug report

**The SSE flag is used when staging out, but not when creating flagfiles or storing exit codes on bucket policy that enforces strict encryption from source/client **

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy

Line 138 in 89c04d3

"echo start | $aws s3 cp --only-show-errors - s3:/${Escape.path(file)}"

https://github.com/nextflow-io/nextflow/blob/89c04d3b2d0ae0762f889b0dcbb845158e503ca0/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy

Expected behavior and actual behavior

Expected behaviour - I expect when sse is enabled on s3 bucket, all PutObject operations (in the script) have the --sse flag.

Actual behaviour - Some PutObject Operation are failing when sse is enabled on the bucket. Scenerio1 - Used s3 bucket without SSE enabled on s3 bucket and it worked perfectly fine

Steps to reproduce the problem

Scenerio - Used s3 bucket as workdir and SSE enabled on s3 bucket and it failed

Both workshop produced thesame error

I have implemented this Nextflow workshop https://docs.opendata.aws/genomics-workflows/orchestration/nextflow/nextflow-overview.html

Attached is my Dockerfile and Entrypoint.sh

Dockerfile.txt
entrypoint.txt

I have also tried this workshop https://github.com/seqeralabs/nextflow-tutorial#Step-10---Run-in-the-cloud-using-AWS-Batch
with my nextflow.config file looking like this: and got same error.
profiles {
batch {
process.container = 'nextflow/rnaseq-nf:latest'
process.executor = 'awsbatch'
process.queue = 'nextflow-dev-ondemand'
workDir = 's3://bucket_with_encryption/_nextflow/runs'
aws.region = 'eu-west-2'
aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws'
aws.client.storageEncryption = 'AES256'
}
}

(Provide a test case that reproduce the problem either with a self-contained script or GitHub repository)

Program output

**Caused by:
Essential container in task exited

Command executed:

salmon index --threads 1 -t ggal_1_48850000_49020000.Ggal71.500bpflank.fa -i index

Command exit status:

Command output:
(empty)

Command error:
upload failed: - to s3://bucket_name_with_encyption/_nextflow/logs/44/f755baaec4feb6851d90841d87d329/.command.begin An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
upload failed: - to s3://bucket_name_with_encyption/_nextflow/logs/44/f755baaec4feb6851d90841d87d329/.exitcode An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
upload failed: ./.command.log to s3://bucket_name_with_encyption/_nextflow/logs/44/f755baaec4feb6851d90841d87d329/.command.log An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Work dir:
s3://bucket_name_with_encyption/_nextflow/logs/44/f755baaec4feb6851d90841d87d329
(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)**

Environment

• Nextflow version: [?] Latest
• Java version: [?]
• Operating system: [macOS, Linux, etc] Linux - (Docker container)
• Bash version: (use the command $SHELL --version) GNU bash

Additional context

(Add any other context about the problem here)
nextflow.logs.txt
Could the -sse flag be added for cases of where s3 bucket policy enforces encryption from clients. So it doesnt drop the connection should a PutObject is attempted and it detects its not encrypted

[pontus]

(https://nfcore.slack.com/archives/CE7FBAMRP/p1637139063092000 and onwards.)

[pditommaso]

This has been solved and included in version 21.10.2. Thanks for reporting the problem.