Enhance the azcopy options configuration scope

[abhi18av]

This PR builds upon the available settings with the core azcopy tool which is used for uploads and downloads in the context of Azure Blob storage service.

[abhi18av]

@pditommaso this PR is ready from my side - but depends on a sister PR nextflow-io/azcopy-tool#1 where I've mentioned the major benefit of the update since we source the azcopy tool from that repo.

UPDATE: Keeping this as draft so as not to merge accidentally before the sister PR.

[abhi18av]

Quick update:

I've had to remove the options which are not compatible with the older 10.8.0 version of azcopy which we use and now the behavior is as expected on the Azure side

The following config with custom azcopy opts are reflected on the generated .command.run files ✅

      azcopy {
          blockSize        = "17"
          blobTier         = "Cool"
          putMD5           = true
          overwrite        = false
          checkMD5         = "NoCheck"
          copyToolInstallMode= 'task'
      }

The only problem remains to be solved is the last test in AzBashLibTest , for which @jorgeaguileraseqera has graciously offered to help (thanks!)

nextflow/plugins/nf-azure/src/test/nextflow/cloud/azure/file/AzBashLibTest.groovy

Line 163 in db89af1

def 'should return script with config, with custom azcopy opts'() {

[abhi18av]

@pditommaso , shall we merge the PR with this edge release cycle?

[pditommaso]

Abi, there's the usual problem your PRs do not run the tests and I've tested locally and it fails

[abhi18av]

Abi, there's the usual problem your PRs do not run the tests and I've tested locally and it fails

🤔 this is curious. Let me take a look why there's no automated testing here.

[pditommaso]

We lost the momentum on this. Closing it because I don't know the status of this PR

[abhi18av]

True 😞

But quick question, do you think that this effort would be a useful addition since fusion is now available for Azure as well?

[pditommaso]

That's another good point. The ideal goal is to not rely anymore on azcopy and use instead fusion in the mid-term

[abhi18av]

I see, in that case let me know if there's a need for the retouches on this PR and then I can make the iteration and circle back. Otherwise, I agree, good opportunity to showcase fusionfs for this 👍