-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add the options
argument to all get_builder_from_protocol
#846
Conversation
Often a user will need to use the same options for all the jobs that will be run in a nested workflow, such as specifying the `queue_name` or the `account`, etc. Currently, the user will have to manually specify those on the builder, or in the `overrides`, but this means that the user knows exactly where in the nesting the jobs are. The `options` argument is added to all `get_builder_from_protocol` implementations, which takes a dictionary just as one would pass to the `metadata.options` input for a `CalcJob`. The implementation ensures that these inputs are recursively set on all the inputs of all the `CalcJob`s that are part of the nested namespace.
@ltalirz quick implementation of the first feature request. If you could verify this satisfies your requirements and works in the wild, that would be great. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot @sphuber , this is exactly what I was looking for!
I will put a note here when I've tested it, but it looks fine to me.
I can definitely see the use case, have thought about implementing something similar in the past. I had some reservations at the time, since there are many inputs one might want to set for all calculation jobs (e.g. That said, I don't want to block this feature, it can definitely be useful for people that just begin to use the work chain, and the options are one of the inputs you pretty much always have to override. |
The points you raise are indeed why we at the time didn't implement this feature. There simply doesn't seem to be a single solution that fits all use-cases, which is what I mentioned to @ltalirz as to why this didn't exist yet. That being said though, currently it is quite difficult to use the Regarding the parallel with the |
@mbercx what is the final verdict here? |
Verdict is that it is not a 100% sure that this is the best solution, but let's merge it for now as it brings considerable advantages at the risk we may have to change things forcefully in the future. |
Fixes #844
Often a user will need to use the same options for all the jobs that will be run in a nested workflow, such as specifying the
queue_name
or theaccount
, etc. Currently, the user will have to manually specify those on the builder, or in theoverrides
, but this means that the user knows exactly where in the nesting the jobs are.The
options
argument is added to allget_builder_from_protocol
implementations, which takes a dictionary just as one would pass to themetadata.options
input for aCalcJob
. The implementation ensures that these inputs are recursively set on all the inputs of all theCalcJob
s that are part of the nested namespace.