Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide option to raise error explicitly for find_pipelines #2910

Closed
noklam opened this issue Aug 8, 2023 · 1 comment · Fixed by #3823
Closed

Provide option to raise error explicitly for find_pipelines #2910

noklam opened this issue Aug 8, 2023 · 1 comment · Fixed by #3823
Assignees
Labels
Issue: Feature Request New feature or improvement to existing feature

Comments

@noklam
Copy link
Contributor

noklam commented Aug 8, 2023

Description

find_pipelines is added since 0.18.3 as a pipeline autodiscovery feature. While this is useful, it is sometimes hard to debug because it raises warning but not error so the program will keep running. Cc @deepyaman

Context

For library developers, they may need strict control and want the pipeline fails explicitly. This features was designed for development, so any half-developed pipeline won't prevent other pipeline from running (incomplete pipeline, not installed library etc)

Some quotes from users:
User 1 (library developer)

We recently have users reporting issues when running tensorflow on M1. The error was hidden in some kedro pipelines and it took a while to dig out the root cause from the user. In general, it can be difficult to predict when and where we will encounter these issue. Therefore, we are planning to build some CI workflows for checking our packages and pipelines for M1 machines in general.

User 2 (general user)

I'm noticing that warnings like SyntaxErrors and type errors are considered "warnings" by Kedro, which continues to try to run the pipeline. Is there a setting to escalate these to Errors so they can abort the pipeline run?
15 days ago
Is there a way to mark a pipeline as a work in progress and then tell find_pipelines to error on warnings? Or is the answer to just not use pipeline autodiscovery

Possible Implementation

find_pipelines(raise_errors=False) - default unchanged, optionally fail eagerly.

Possible Alternatives

@noklam noklam added the Issue: Feature Request New feature or improvement to existing feature label Aug 8, 2023
@datajoely
Copy link
Contributor

Yes! I've heard this too!

@merelcht merelcht added this to the Improve Developer Experience milestone Jan 12, 2024
@merelcht merelcht moved this to To Do in Kedro Framework Mar 18, 2024
@lrcouto lrcouto moved this from To Do to In Progress in Kedro Framework Mar 27, 2024
@merelcht merelcht moved this from In Progress to To Do in Kedro Framework Apr 15, 2024
@deepyaman deepyaman moved this from To Do to In Progress in Kedro Framework Apr 18, 2024
@deepyaman deepyaman moved this from In Progress to Done in Kedro Framework Apr 18, 2024
@deepyaman deepyaman moved this from Done to In Review in Kedro Framework Apr 18, 2024
@github-project-automation github-project-automation bot moved this from In Review to Done in Kedro Framework May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: Feature Request New feature or improvement to existing feature
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

5 participants