[WIP] Adding Cloud Pipelines

[vsoch]

do not merge this is a draft pull request!

This PR will encompass early work to add Google Cloud Pipelines to Snakemake. Specifically this means that:

• the user selects --google-life-sciences provider (boolean) when running snakemake
• GOOGLE_APPLICATION_CREDENTIALS and GOOGLE_PROJECT should be exported
• a content hash named tarball of the working directory is uploaded to Google Storage
• the worker uses a custom image to download and extract the working directory, and then run the script for the step in question.
• the storage tarballs are deleted after the workflow completes, only if the user hasn't specified a flag to keep the cache.

We have several technical questions that still need answering, which I'll summarize here:

• upload of environment variables vs. secrets
• another method to discover application credentials and project (without exporting the envar)
• how to provide an entrypoint + command with && (we've tried almost all variations of this and the second command seems to run outside of the container, or the entire thing is parsed together and the result is command not found.
• privileged containers OR custom addition of permissions (to allow running Singularity)
• difference between action and resource environment
• it would be cool if Google provided a function to choose a machine type based on these criteria, seems like it would be a common thing to need!

A few notes for some choices I made for the implementation:

• As discussed in our meeting, I was able to use the Google Discovery Build function against the manifest endpoint to create a client.
• regions is another variable that could be derived from some call to the API (to get a default region) but most likely the requester for storage objects would want to choose a region that is closest to the objects, which might not be where he or she is. The default regions are all zones in the US, which I chose based on guessing that the majority of Snakemake / GCP users are in the US. This might be entirely wrong, or a bad default.
• the machine type is selected based on comparing the requested memory and cores to those available, and first eliminating the ones that are too small, and then choosing the smallest of those remaining. If the user provides a machine type prefix, we further filter down the choices to that.
• Google Storage (for public objects) often has a 503 error (yesterday it popped up several times for me in testing) and if this happens again it might be a separate issue that the GS remote should look into handling - for most Google Cloud API stuffs retrying is essential.
• speaking of retrying - the functions here are run by a client function I wrote to retry three times, and only then error. This is because BrokenPipeError (and other random failures) are common enough to warrant it.

I suspect this will fail linting and I'll push again with fixes :)

[vsoch]

@johanneskoester this is a little confusing with black, because I installed the exact version that is run in the test suites, and there seems to be a difference in result. Here is what I'm doing locally:

$ black --check snakemake tests/*.py
All done! ✨ 🍰 ✨
51 files would be left unchanged.
$ black --version
black, version 19.3b0

Note that the version is the same installed there. I also tried installing with conda in case it made a difference (it didn't). What am I missing?

[vsoch]

That's so weird! The black formatting passed this time. I don't think I made any changes to that file. Must be ghosties.

[johanneskoester]

@vsoch did you select a draft PR? I still have a merge button here and the icon is green instead of grey in the list.

[vsoch]

oops I just totally forgot :/ Merging is still blocked so is it okay? I'll add WIP to the heading so others know.

[johanneskoester]

Yeah, no problem ;-).

[johanneskoester]

xref #91

[vsoch]

@johanneskoester I'll want to ask for your suggestions for how to reduce "cognitive complexity" of functions - I made them modular at a level I thought made sense, and most of the length for the ones flagged is due to really long WorkflowError messages. I'm happy to refactor in any way you think is better.

[vsoch]

Not sure what is going on with the build container image step, but it's been going for over 2 hours.

[johanneskoester]

That issue is resolved now.

[johanneskoester]

@johanneskoester I'll want to ask for your suggestions for how to reduce "cognitive complexity" of functions - I made them modular at a level I thought made sense, and most of the length for the ones flagged is due to really long WorkflowError messages. I'm happy to refactor in any way you think is better.

I have looked at them again, and I think they are indeed ok to remain like this. Sorry for the noise.

[johanneskoester]

GCloud credentials are now in place. I think you should be able to activate your test case here. In line with the kubernetes and tibanna tests, please move your test case in a separate file tests/test_google_life_sciences.py which shall be called in a separate job step.

[vsoch]

@johanneskoester what is the name of the envar secret - did you just call it GOOGLE_APPLICATION_CREDENTIALS? I'm adding the step in main.yml now, should be able to push to test as soon as I know that name!

[vsoch]

E.g., this is what I'd guess, let me know if I'm off

    env:
      AWS_AVAILABLE: ${{ secrets.AWS_ACCESS_KEY_ID }}
      GCP_AVAILABLE: ${{ secrets.GCP_SA_KEY }}
      GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}

[vsoch]

Dearest sonarcloud - you've had a change of heart! Because before you said my code was smelly. Maybe it's good smelly now :)

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 2 Security Hotspots to review)
0 Code Smells

No Coverage information
0.1% Duplication

[vsoch]

@johanneskoester the test did not run still - I can't see settings/secrets so I'll need your guidance about what needs to be checked.

[johanneskoester]

You are on a fork. github does not expose secrets to forks.

[johanneskoester]

Let us go on in PR #384