Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REFACTOR][UAI] Refactor uai with DataProcessor + update MENJ resource to use the latest one #441

Merged
merged 4 commits into from
Dec 23, 2024
Merged

[REFACTOR][UAI] Refactor uai with DataProcessor + update MENJ resource to use the latest one #441

merged 4 commits into from
Dec 23, 2024

Conversation

hacherix
Copy link
Contributor

@hacherix hacherix commented Dec 16, 2024 •
edited
Loading

Related to #405

This PR implements the UAI processor class, configuration settings and update the related DAG (see second commit).

This PR change the MENJ resource from 2019 to the latest one. The file only include open établissements scolaires.

Unlike previous DAGs, UAI use multiple resources, so the following method had to be modifief (see first commit):

  • change the DataProcessor.save_date_last_modified() method to handle multiple data.gouv resources and data.gouv datasets
  • change the DataProcessor.download_data() method to handle multiple data.gouv resources and data.gouv datasets
  • update existing DAGs with the new configuration style

@hacherix hacherix self-assigned this Dec 16, 2024
@hacherix hacherix requested a review from HAEKADI as a code owner December 16, 2024 11:08
@hacherix hacherix marked this pull request as draft December 16, 2024 11:08
@hacherix hacherix force-pushed the refactor-uai-dag branch 2 times, most recently from 1fa9ee5 to 4f5cd4e Compare December 16, 2024 17:13
@hacherix hacherix marked this pull request as ready for review December 16, 2024 17:14
@hacherix hacherix changed the title [REFACTOR][UAI] Refactor uai dag [REFACTOR][UAI] Refactor uai dag + update MENJ resource to the latest Dec 16, 2024
@hacherix hacherix changed the title [REFACTOR][UAI] Refactor uai dag + update MENJ resource to the latest [REFACTOR][UAI] Refactor uai with DataProcessor + update MENJ resource to use the latest one Dec 16, 2024
@hacherix hacherix force-pushed the refactor-uai-dag branch 2 times, most recently from 52a6b4c to 83224c1 Compare December 17, 2024 23:26
HAEKADI
HAEKADI approved these changes Dec 23, 2024
View reviewed changes
config.py Show resolved Hide resolved
helpers/data_processor.py Outdated Show resolved Hide resolved
helpers/data_processor.py Outdated Show resolved Hide resolved
helpers/datagouv.py Show resolved Hide resolved
workflows/data_pipelines/egapro/config.py Show resolved Hide resolved
workflows/data_pipelines/ess_france/config.py Show resolved Hide resolved
workflows/data_pipelines/finess/config.py Show resolved Hide resolved
workflows/data_pipelines/uai/uai_config.py Show resolved Hide resolved
workflows/data_pipelines/uai/uai_dag.py Outdated Show resolved Hide resolved
workflows/data_pipelines/uai/uai_processor.py Show resolved Hide resolved
HAEKADI
HAEKADI reviewed Dec 23, 2024
View reviewed changes
config.py Outdated Show resolved Hide resolved
HAEKADI
HAEKADI reviewed Dec 23, 2024
View reviewed changes
config.py Outdated Show resolved Hide resolved
HAEKADI
HAEKADI reviewed Dec 23, 2024
View reviewed changes
config.py Outdated Show resolved Hide resolved
HAEKADI
HAEKADI approved these changes Dec 23, 2024
View reviewed changes
Copy link
Contributor

@HAEKADI HAEKADI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@hacherix
refactor: make DataProcessor handle multiple resources
7d25520 
Refactor DataProcessor.download_data()
Refactor DataProcessor.save_date_last_modified()

Make it so it can download multiple files and handle data.gouv datasets
@hacherix
refactor(uai): refactor dag with DataProcessor
33b508d
@hacherix
feat(uai): change menj resource to the latest
2601dd6
@hacherix
refactor: use consistent task name
e8f0312
@hacherix
Copy link
Contributor Author

Merging!

image image image image image

image

@hacherix hacherix merged commit 1969e83 into main Dec 23, 2024
4 checks passed
@hacherix hacherix deleted the refactor-uai-dag branch December 23, 2024 15:21
@HAEKADI HAEKADI mentioned this pull request Dec 23, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HAEKADI HAEKADI HAEKADI approved these changes

Assignees

@hacherix hacherix

Labels
refactoring/clean code
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hacherix @HAEKADI