Skip to content

Commit

Permalink
Remove retired DAGs from the codebase (#4867)
Browse files Browse the repository at this point in the history
* Remove retired DAGs

* Remove or stabilise references to retired DAGs
  • Loading branch information
sarayourfriend authored Sep 5, 2024
1 parent cea8050 commit 67e3869
Show file tree
Hide file tree
Showing 38 changed files with 5 additions and 3,067 deletions.
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/new_source_suggestion.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@ body:
- The script must use the ProviderDataIngester class in `catalog/dags/providers/provider_api_scripts/provider_data_ingester.py`.
- The script must use the `ImageStore` or `AudioStore` class (Import this from `catalog/dags/provider_api_scripts/common/storage/image.py`).
- The script should use the `DelayedRequester` class (Import this from `catalog/dags/provider_api_scripts/common/requester.py`).
- The script must not use anything from `catalog/dags/retired/providers/provider_api_scripts/modules/etlMods.py`, since that module is deprecated.
- If the provider API has can be queried by 'upload date' or something similar, the script should take a `--date` parameter when run as a script, giving the date for which we should collect images. The form should be `YYYY-MM-DD` (so, the script can be run via `python my_favorite_provider.py --date 2018-01-01`).
- The script must provide a main function that takes the same parameters as from the CLI. In our example from above, we'd then have a main function `my_favorite_provider.main(date)`. The main should do the same thing calling from the CLI would do.
- The script *must* conform to [PEP8][pep8]. Please use `ov just lint` before committing.
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
exclude: Pipfile\.lock|migrations|\.idea|node_modules|retired|\.snap
exclude: Pipfile\.lock|migrations|\.idea|node_modules|\.snap

repos:
- repo: local # More local hooks are defined at the bottom.
Expand Down
1 change: 0 additions & 1 deletion catalog/dags/.airflowignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# Ignore all non-DAG files
common/
providers/provider_api_scripts
retired
6 changes: 0 additions & 6 deletions catalog/dags/retired/README.md

This file was deleted.

Empty file removed catalog/dags/retired/__init__.py
Empty file.
72 changes: 0 additions & 72 deletions catalog/dags/retired/cleaner_workflow.py

This file was deleted.

Empty file.
8 changes: 0 additions & 8 deletions catalog/dags/retired/common/etl/bootstrap/config-py27.sh

This file was deleted.

Loading

0 comments on commit 67e3869

Please sign in to comment.