Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update kedro pipeline create to use new /conf file structure #2752

Closed
3 tasks done
amandakys opened this issue Jul 3, 2023 · 3 comments · Fixed by #2856
Closed
3 tasks done

Update kedro pipeline create to use new /conf file structure #2752

amandakys opened this issue Jul 3, 2023 · 3 comments · Fixed by #2856
Assignees
Labels
Issue: Feature Request New feature or improvement to existing feature

Comments

@amandakys
Copy link

amandakys commented Jul 3, 2023

Description

Currently, the kedro pipeline create command creates a new /parameters folder and adds a {pipeline-name}.yml inside (or just adds the file to the parameters folder if it aready exists.

Context

To make this more consistent and so the global parameters.yml doesn't just randomly sit outside the directories, we want to maintian all the pipeline specific config files directly in the /conf with a specific naming convention.

Before

├── conf
    └── base
        └── parameters         
	    ├── data_science.yml     #blank files
        ├── catalog.yml              #blank files   
	└── parameters.yml           #blank files

After

├── conf
    └── base
	├── catalog_data_science.yml          #blank files
	└── parameters_data_science.yml       #blank files
        ├── catalog.yml          #blank files   
	└── parameters.yml       #blank files

(This is outdated - edited by Nok)

Possible Implementation

calling the updated kedro pipeline create should:

  1. add parameters_{pipeline_name}.yml to /conf
  2. add catalog_{pipeline_name}.yml to /conf
  3. add a pipeline diretory called {pipeline_name} to the /pipelines with empty nodes.py and pipeline.py files

Currently, it also adds a /tests directory, but we want to remove this from the command because of work on #2149

Final decision - see comments for the full details

Following discussion with @merelcht @deepyaman @astrojuanlu @DimedS we decided to move forward with this work with the following updates:

calling kedro pipeline create will:

  • add parameters_{pipeline_name}.yml to /conf
  • add a pipeline directory called {pipeline_name} to the /pipelines with empty nodes.py and pipeline.py files
  • No catalog_pipeline_name.yml
    it will no longer add a /tests directory it will also no longer add a catalog_{pipeline_name}.yml to /conf, this will instead be delegated to the kedro catalog create command which will also need updating as a follow up to this. Update kedro catalog create to use new /conf structure  #2844
@amandakys amandakys added the Issue: Feature Request New feature or improvement to existing feature label Jul 3, 2023
@merelcht merelcht moved this to To Do in Kedro Framework Jul 11, 2023
@DimedS DimedS self-assigned this Jul 19, 2023
@DimedS DimedS moved this from To Do to In Progress in Kedro Framework Jul 19, 2023
@amandakys
Copy link
Author

amandakys commented Jul 26, 2023

Following discussion with @merelcht @deepyaman @astrojuanlu @DimedS we decided to move forward with this work with the following updates:

calling kedro pipeline create will:

  • add parameters_{pipeline_name}.yml to /conf
  • add a pipeline directory called {pipeline_name} to the /pipelines with empty nodes.py and pipeline.py files

it will no longer add a /tests directory
it will also no longer add a catalog_{pipeline_name}.yml to /conf, this will instead be delegated to the kedro catalog create command which will also need updating as a follow up to this. #2844

@merelcht
Copy link
Member

Following discussion with @merelcht @deepyaman @astrojuanlu @DmitriiDeriabinQB we decided to move forward with this work with the following updates:

I think you meant to tag @DimedS 😄

@noklam
Copy link
Contributor

noklam commented Aug 11, 2023

I update the description to reflect the final decision - I also make it a parent ticket so it's easy to track the rest. This is done already but I think it will be useful to keep the record for future reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: Feature Request New feature or improvement to existing feature
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants