Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use bellybutton to encode various Airflow best practices #1317

Open
sarayourfriend opened this issue Dec 15, 2022 · 3 comments
Open

Use bellybutton to encode various Airflow best practices #1317

sarayourfriend opened this issue Dec 15, 2022 · 3 comments
Labels
🤖 aspect: dx Concerns developers' experience with the codebase 🧰 goal: internal improvement Improvement that benefits maintainers, not users 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs 🧹 status: ticket work required Needs more details before it can be worked on

Comments

@sarayourfriend
Copy link
Collaborator

sarayourfriend commented Dec 15, 2022

Problem

There are some subtle rules/conventions that our repository follows that could be automatically checked using https://github.com/hchasestevens/bellybutton.

Description

Add https://github.com/hchasestevens/bellybutton and create two new rules:

  1. Prevent top level Variable.get (see Move API key Variable access to provider ingester init function #1457)
  2. Disallow pendulum and to instruct the use of datetime instead (see Standardize on datetime over pendulum openverse-catalog#678)
  3. Set sensor modes to rescheduled (see this review comment explaining why: Fix various issues that existed in the rotate_db_snapshots DAG #2158 (comment))
@sarayourfriend sarayourfriend added help wanted Open to participation from the community 🟩 priority: low Low priority and doesn't need to be rushed 🤖 aspect: dx Concerns developers' experience with the codebase 🧰 goal: internal improvement Improvement that benefits maintainers, not users labels Dec 15, 2022
@AetherUnbound
Copy link
Collaborator

This is fantastic! 🤩 What a find! I love it, it'll be perfect for the conventions you point out (and probably others yet-to-be-identified).

@obulat obulat added the 🧱 stack: catalog Related to the catalog and Airflow DAGs label Feb 23, 2023
@github-project-automation github-project-automation bot moved this to 📋 Backlog in Openverse Backlog Apr 17, 2023
@obulat obulat transferred this issue from WordPress/openverse-catalog Apr 17, 2023
@sarayourfriend sarayourfriend changed the title Use bellybutton to advise against using pendulum or top-level Variable.get Use bellybutton to encode various Airflow best practices May 25, 2023
@sarayourfriend sarayourfriend self-assigned this May 25, 2023
@sarayourfriend
Copy link
Collaborator Author

sarayourfriend commented May 25, 2023

I'm playing around with astpath, the library underneath Bellybutton, and I can't get it to select anything, not even basic nodes that I know exist like "Module" or "ClassDef".

I could be doing this wrong, of course. Bellybutton and astpath haven't been updated in a long time though, so I wouldn't be surprised if they're working off of old information.

@sarayourfriend
Copy link
Collaborator Author

Rather than bellybutton, we could use https://fixit.readthedocs.io/en/latest/ and only enable it for custom rules. It's custom rules even support fixes, which the sensor changes, for example, would be decent at providing in some cases.

Writing the rules along with tests in Python seems a lot easier and more straightforward to me than bellybutton and LibCST looks to be a bit of a better AST to work with as well.

@sarayourfriend sarayourfriend removed their assignment Jun 8, 2023
@sarayourfriend sarayourfriend added 🧹 status: ticket work required Needs more details before it can be worked on and removed help wanted Open to participation from the community labels Jun 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🤖 aspect: dx Concerns developers' experience with the codebase 🧰 goal: internal improvement Improvement that benefits maintainers, not users 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs 🧹 status: ticket work required Needs more details before it can be worked on
Projects
Status: 📋 Backlog
Development

No branches or pull requests

3 participants