Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run multiple recipes, add tmp_dir, disable stack trace for RecipeErrors #265

Closed
wants to merge 1 commit into from

Conversation

bouweandela
Copy link
Member

This pull request

  • Adds the possibility to run more than one recipe
  • Adds an option to specify a tmp_dir in config-user.yml, for use on cluster nodes with local storage attached
  • Hides the stack trace if a RecipeError occurs
  • Improves the code for reading config-user.yml

Closes {Link to corresponding issue}

Tasks

  • Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
  • Add unit tests
  • Public functions should have a numpy-style docstring so they appear properly in the API documentation. For all other functions a one line docstring is sufficient.
  • If writing a new/modified preprocessor function, please update the documentation
  • Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
  • Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
  • Please use yamllint to check that your YAML files do not contain mistakes
  • If you make backward incompatible changes to the recipe format, make a new pull request in the ESMValTool repository and add the link below

@mattiarighi mattiarighi added the enhancement New feature or request label Sep 20, 2019
@valeriupredoi
Copy link
Contributor

ooh this is cool and dangerous at the same time! I am very concerned that, the implementation not actually using a scheduler but rather pool, will be abused by users and we'll end up with user X hammering the head node on an HPC like Jasmin -> can we limit the usage somehow? Note that running multiple recipes is possible now with the rose suite I wrote last year so we should use these both ie use this implementation for small numbers of recipes and jobs and use the rose suite for a lot of analysis

@bouweandela
Copy link
Member Author

I do not foresee finishing this anytime soon.

@bouweandela bouweandela closed this Sep 7, 2021
@bouweandela bouweandela deleted the tmp_dir branch September 7, 2021 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants