Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove retired DAGs from the codebase #4867

Merged
merged 2 commits into from
Sep 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/new_source_suggestion.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@ body:
- The script must use the ProviderDataIngester class in `catalog/dags/providers/provider_api_scripts/provider_data_ingester.py`.
- The script must use the `ImageStore` or `AudioStore` class (Import this from `catalog/dags/provider_api_scripts/common/storage/image.py`).
- The script should use the `DelayedRequester` class (Import this from `catalog/dags/provider_api_scripts/common/requester.py`).
- The script must not use anything from `catalog/dags/retired/providers/provider_api_scripts/modules/etlMods.py`, since that module is deprecated.
- If the provider API has can be queried by 'upload date' or something similar, the script should take a `--date` parameter when run as a script, giving the date for which we should collect images. The form should be `YYYY-MM-DD` (so, the script can be run via `python my_favorite_provider.py --date 2018-01-01`).
- The script must provide a main function that takes the same parameters as from the CLI. In our example from above, we'd then have a main function `my_favorite_provider.main(date)`. The main should do the same thing calling from the CLI would do.
- The script *must* conform to [PEP8][pep8]. Please use `ov just lint` before committing.
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
exclude: Pipfile\.lock|migrations|\.idea|node_modules|retired|\.snap
exclude: Pipfile\.lock|migrations|\.idea|node_modules|\.snap

repos:
- repo: local # More local hooks are defined at the bottom.
Expand Down
1 change: 0 additions & 1 deletion catalog/dags/.airflowignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# Ignore all non-DAG files
common/
providers/provider_api_scripts
retired
6 changes: 0 additions & 6 deletions catalog/dags/retired/README.md

This file was deleted.

Empty file removed catalog/dags/retired/__init__.py
Empty file.
72 changes: 0 additions & 72 deletions catalog/dags/retired/cleaner_workflow.py

This file was deleted.

Empty file.
8 changes: 0 additions & 8 deletions catalog/dags/retired/common/etl/bootstrap/config-py27.sh

This file was deleted.

Loading