Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for several process directives #51

Merged
merged 1 commit into from
Aug 4, 2023

Conversation

bentsherman
Copy link
Contributor

Close #27

This PR adds support for the following process directives:

  • disk
  • machineType
  • resourceLabels
  • time

While the CLI options can already be specified via float.commonExtra, the process directives are more user-friendly and can also be applied on a per-process level.

Also adds custom config options for --vmPolicy and --migratePolicy, since these options are some of the most valuable parts of Float but also have the most complicated syntax, we should try to make them as user-friendly as possible.

I also added a few default labels for things like process name, task name, session ID, etc. But feel free to remove them if you don't think they are needed. I copied them from the K8s executor.

I added unit tests for these options but haven't tested them end to end. Let me know if you need any help with testing.

Not urgent or groundbreaking but these settings would be nice to have for the initial blog post.

@jealous
Copy link
Contributor

jealous commented Aug 4, 2023

Thanks for the contribution @bentsherman.

@jealous jealous merged commit 70672ba into MemVerge:master Aug 4, 2023
@jealous
Copy link
Contributor

jealous commented Aug 4, 2023

For the custom tag names, some platforms such as GCP does not support upper case or characters like ., / or space. For now, I will try to replace them with -

@jealous
Copy link
Contributor

jealous commented Aug 4, 2023

I update the rootVolSize to imageVolSize because image volume is where we run the container where the actual work load happens. The root volume only contains the OS for VM.

@bentsherman bentsherman deleted the 27-float-cli-options branch August 4, 2023 13:27
@bentsherman
Copy link
Contributor Author

Both fair points. I realized also that we probably need to replace : as well, at least while nf-float uses the CLI, since that character is used to separate the key/value.

jealous added a commit to jealous/plugins that referenced this pull request Aug 8, 2023
* What's Changed
  * Add Fusion support by @jealous in MemVerge/nf-float#48
  * Add support for several process directives by @bentsherman in MemVerge/nf-float#51
  * [nextflow-ioGH-50] Input or output on s3 is not available. by @jealous in MemVerge/nf-float#52

* New Contributors
  * @bentsherman made their first contribution in MemVerge/nf-float#51

Full Changelog: MemVerge/nf-float@0.2.0...v0.2.1
jealous added a commit to jealous/plugins that referenced this pull request Aug 8, 2023
* What's Changed
  * Add Fusion support by @jealous in MemVerge/nf-float#48
  * Add support for several process directives by @bentsherman in MemVerge/nf-float#51
  * [nextflow-ioGH-50] Input or output on s3 is not available. by @jealous in MemVerge/nf-float#52
  * Upgrade version to 0.3.0 by @jealous in MemVerge/nf-float#53

* New Contributors
  * @bentsherman made their first contribution in MemVerge/nf-float#51

Full Changelog: MemVerge/nf-float@0.2.0...v0.3.0
pditommaso pushed a commit to nextflow-io/plugins that referenced this pull request Aug 10, 2023
* Add nf-float release 0.3.0

* What's Changed
  * Add Fusion support by @jealous in MemVerge/nf-float#48
  * Add support for several process directives by @bentsherman in MemVerge/nf-float#51
  * [GH-50] Input or output on s3 is not available. by @jealous in MemVerge/nf-float#52
  * Upgrade version to 0.3.0 by @jealous in MemVerge/nf-float#53

* New Contributors
  * @bentsherman made their first contribution in MemVerge/nf-float#51

Full Changelog: MemVerge/nf-float@0.2.0...v0.3.0

* Update to include the fix for nf-float GH-54

Include a fix for GH-54.  Support extra options.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add config options for float CLI options
2 participants