Implementation Plan: Move Airflow and API to openverse.org #3650

sarayourfriend · 2024-01-10T04:22:37Z

Fixes

Fixes #2038 by @sarayourfriend

Part of #2037

Description

I've requested review from @AetherUnbound and @dhruvkb:

Madison, you are the person (I believe) most familiar with Airflow deployment details, so your input the Airflow configuration in particular is critical. You are also deeply familiar with our infrastructure configuration overall.
Dhruv, you are extremely perceptive about the interconnected dependencies of our API and frontend, and our services in general, and I think you'll be a great person to review the plan. Reviewing this plan may also increase your exposure to any areas of our infrastructure with which you are not already familiar. There are also potential changes to our Terraform configuration for GitHub, and I suspect you'll have interest in those as well.

This discussion is following the Openverse decision-making process. Information about this process can be found on the Openverse documentation site. Requested reviewers or participants will be following this process. If you are being asked to give input on a specific detail, you do not need to familiarise yourself with the process and follow it.

Current round

This discussion is currently in the Decision round.

The deadline for review of this round is 2 February 2024.

Testing Instructions

Read the plan and leave any questions.

Checklist

My pull request has a descriptive title (not a vague title likeUpdate index.md).
My pull request targets the default branch of the repository (main) or a parent feature branch.
My commit messages follow best practices.
My code follows the established code style of the repository.
[N/A] I added or updated tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no visible errors.
[N/A] I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

sarayourfriend · 2024-01-10T04:35:41Z

@dhruvkb and @AetherUnbound, I haven't fully proofread this, but I plan to do that and then undraft it tomorrow before I go AFK until the end of the month. As mentioned in the PR description, there's no big rush to review this until February, because I'm gone until then anyway and wouldn't be able to respond to questions or start revision. However, it is a rather long plan, so starting to review it sooner than later is a good idea.

- Clarify redirects - Replace step to coordinate with Gutenberg and Jetpack with a step to open PRs ourselves to update the URLs

github-actions · 2024-01-10T23:24:23Z

Full-stack documentation: https://docs.openverse.org/_preview/3650

Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again.

You can check the GitHub pages deployment action list to see the current status of the deployments.

New files ➕:

AetherUnbound

This is excellent and wonderfully thorough! Here's my feedback for the clarification round 🙂

AetherUnbound · 2024-01-19T00:14:21Z

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

+### 1. Move Airflow to `airflow.openverse.org`
+
+1. Add `airflow.openverse.org` to Cloudflare access.
+1. Create a new `airflow` module in the `next` root module. Use `ec2-service` as


With this plan, it seems like we'd have two instances of Airflow running (and accessing the metadata database) at the same time. With the modern version of Airflow we're running, I don't think this should be an issue. The webserver is just a webserver and the scheduler can now by default have multiple instances running. Just wanted to clarify that this was intended at this step, because at present we remove the current EC2 instance before deploying a new one and only have 1 instance running at any given time.

Oh, I should add a step to just spin down the existing one to simplify things. In my mind I definitely meant to just spin down the existing one in the meantime to keep things simple.

Sounds good!

AetherUnbound · 2024-01-19T00:20:55Z

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

+   1. Open PRs in Gutenberg and Jetpack to update the Openverse integration URL
+      to `api.openverse.org`


Thank you for including this step explicitly! Maybe we should mention other integrations more broadly? This isn't expansive but we could also submit PRs for folks using the old URL in GitHub: https://github.com/search?type=code&q=api.openverse.engineering

That's a nice bonus but I am hesitant to add it as a requirement to the project. At most I'd accept the idea of opening issues in those repositories, with Openverse maintainers optionally opening PRs if they feel personally motivated to do so for one reason or another.

I am hesitant to add anything to this project that we don't have control over or at least very good, clear relationahips with the other maintainers to be able to set expectations about completion.

That's a great point, I think making issues is good but also agree it would not be a requirement.

AetherUnbound · 2024-01-19T00:22:31Z

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

+      remove the special header condition of the redirect listeners so that all
+      requests to the old production API domains redirect to
+      `api.openverse.org`.
+      - I suggest 2 weeks from the day we send the emails.


This seems reasonable!

AetherUnbound · 2024-01-19T00:24:54Z

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

+Later on, in the Airflow and API work, we'll expand the Cloudflare configuration
+in `cloudflare` to include the page and WAF rules related to these services that
+are currently configured by hand in the `openverse.engineering` zone.


A wonderful side-effect made possible by this change! 🥳

AetherUnbound · 2024-01-19T00:27:01Z

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

+A potential alternative to using a new `cloudflare` root module to include these
+in the `org` root module, which is separated from the `next` environment root
+modules for similar reasons: the resources it configures are almost entirely
+independent of our AWS infrastructure. That would be fine, in fact, except that
+aside from the Cloudflare Access configuration, `org` is a strange name for the
+place to store Cloudflare configurations, because they aren't related to
+Openverse's organisational practices (again, aside from Cf Access). We could
+rename `org` to something else, but I'm not sure what would include the GitHub
+repository configuration and our Cloudflare configuration. I've thought about
+this quite a bit and can't come up with a name that would be _more_ intuitive
+than a separate `cloudflare` root module. If we do that, I would also like to
+rename the `org` root module to `github`, and move the configuration in its
+`github` child module into the root module.


Agreed that what you're describing doesn't belong in the org root module, and I would be fine with renaming org to github!

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

openverse-bot · 2024-01-24T00:00:09Z

Based on the low urgency of this PR, the following reviewers are being gently reminded to review this PR:

@dhruvkb
This reminder is being automatically generated due to the urgency configuration.

Excluding weekend¹ days, this PR was ready for review 9 day(s) ago. PRs labelled with low urgency are expected to be reviewed within 5 weekday(s)².

@sarayourfriend, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings.

Specifically, Saturday and Sunday. ↩
For the purpose of these reminders we treat Monday - Friday as weekdays. Please note that the operation that generates these reminders runs at midnight UTC on Monday - Friday. This means that depending on your timezone, you may be pinged outside of the expected range. ↩

dhruvkb

All of this makes sense to me. LGTM!

dhruvkb · 2024-01-24T11:06:12Z

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

+A potential alternative to using a new `cloudflare` root module to include these
+in the `org` root module, which is separated from the `next` environment root
+modules for similar reasons: the resources it configures are almost entirely
+independent of our AWS infrastructure. That would be fine, in fact, except that
+aside from the Cloudflare Access configuration, `org` is a strange name for the
+place to store Cloudflare configurations, because they aren't related to
+Openverse's organisational practices (again, aside from Cf Access). We could
+rename `org` to something else, but I'm not sure what would include the GitHub
+repository configuration and our Cloudflare configuration. I've thought about
+this quite a bit and can't come up with a name that would be _more_ intuitive
+than a separate `cloudflare` root module. If we do that, I would also like to
+rename the `org` root module to `github`, and move the configuration in its
+`github` child module into the root module.


AetherUnbound · 2024-01-25T23:27:24Z

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

+environment root modules, because there is no specific environment related to
+these settings, and it's best to keep these all together in a single place.
+Later on, in the Airflow and API work, we'll expand the Cloudflare configuration
+in `cloudflare` to include the page and WAF rules related to these services that


Noting also that I've added some Cache Rules which mirror the Page Rules in an attempt to more accurately use Cloudflare's intended settings for caching, so those will need to be moved over as well!

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

Co-authored-by: Madison Swain-Bowden <[email protected]>

sarayourfriend · 2024-01-29T04:05:42Z

@AetherUnbound @dhruvkb Thank you for your reviews. I've made minor revisions for editorial changes (thanks, Madison) and added a new explicit step to spin down the existing Airflow instance before deploying the new one, to avoid any potential (though apparently unlikely) issues with multiple instances running on the same databases.

As you are both in agreement with the usage of a new root module, I'll also add a single new step to rename the existing org root module to github, without any changes to the code in the module (which are more of a hassle than the benefit they'd provide of shorter resource paths in the terraform state).

sarayourfriend · 2024-01-29T04:21:38Z

@AetherUnbound @dhruvkb I've moved this into the decision round. I kept the deadline to 2 February because that's still five days. Dhruv has already approved, and the changes are minor (just adding steps we've agreed to) so I am not worried about an explicit re-approval. The decision rests with you, Madison! Please let me know if there are any blockers to resolve.

AetherUnbound

One last small addition to make sure we're explicit about what we're carrying over, but otherwise this looks great! I'm excited for it!

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md

Co-authored-by: Madison Swain-Bowden <[email protected]>

sarayourfriend · 2024-02-01T22:07:42Z

Creating issues for this plan now!

sarayourfriend requested review from AetherUnbound and dhruvkb January 10, 2024 04:22

krysal added the skip-changelog label Jan 10, 2024

IP: Move Airflow and API to openverse.org

ee60ac2

sarayourfriend force-pushed the add/openverse_org-all-in branch from e9c4841 to ee60ac2 Compare January 10, 2024 22:51

sarayourfriend added 2 commits January 11, 2024 10:14

Clarify and simplify the plan

d2cda27

- Clarify redirects - Replace step to coordinate with Gutenberg and Jetpack with a step to open PRs ourselves to update the URLs

Fix orphaned document

dc9bfde

sarayourfriend marked this pull request as ready for review January 10, 2024 23:29

sarayourfriend requested a review from a team as a code owner January 10, 2024 23:29

AetherUnbound added 🟩 priority: low Low priority and doesn't need to be rushed and removed 🟧 priority: high Stalls work on the project or its dependents labels Jan 15, 2024

AetherUnbound reviewed Jan 19, 2024

View reviewed changes

dhruvkb approved these changes Jan 24, 2024

View reviewed changes

AetherUnbound reviewed Jan 25, 2024

View reviewed changes

sarayourfriend commented Jan 29, 2024

View reviewed changes

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md Show resolved Hide resolved

Editorial changes; add explicit step to spin down existing Airflow

ec641b5

Co-authored-by: Madison Swain-Bowden <[email protected]>

Add step to rename org root module

d03cec5

AetherUnbound approved these changes Jan 30, 2024

View reviewed changes

sarayourfriend commented Jan 31, 2024

View reviewed changes

...ucture_improvements/20240105_implementation_plan_move_remaining_services_to_openverse_org.md Outdated Show resolved Hide resolved

Add approvals and list all Cloudflare resources

bcd69a3

Co-authored-by: Madison Swain-Bowden <[email protected]>

sarayourfriend force-pushed the add/openverse_org-all-in branch from f4f6632 to bcd69a3 Compare January 31, 2024 00:23

sarayourfriend merged commit ff7802b into main Feb 1, 2024
37 checks passed

sarayourfriend deleted the add/openverse_org-all-in branch February 1, 2024 22:07

krysal mentioned this pull request Mar 28, 2024

Add implementation plan for documenting the API media properties #3965

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation Plan: Move Airflow and API to openverse.org #3650

Implementation Plan: Move Airflow and API to openverse.org #3650

sarayourfriend commented Jan 10, 2024 •

edited

Loading

sarayourfriend commented Jan 10, 2024

github-actions bot commented Jan 10, 2024 •

edited

Loading

AetherUnbound left a comment

AetherUnbound Jan 19, 2024

sarayourfriend Jan 29, 2024

AetherUnbound Jan 30, 2024

AetherUnbound Jan 19, 2024

sarayourfriend Jan 29, 2024

AetherUnbound Jan 30, 2024

AetherUnbound Jan 19, 2024

AetherUnbound Jan 19, 2024

AetherUnbound Jan 19, 2024

dhruvkb Jan 24, 2024

openverse-bot commented Jan 24, 2024

dhruvkb left a comment

dhruvkb Jan 24, 2024

AetherUnbound Jan 25, 2024

sarayourfriend commented Jan 29, 2024 •

edited

Loading

sarayourfriend commented Jan 29, 2024

AetherUnbound left a comment

sarayourfriend commented Feb 1, 2024

		1. Open PRs in Gutenberg and Jetpack to update the Openverse integration URL
		to `api.openverse.org`

Implementation Plan: Move Airflow and API to openverse.org #3650

Implementation Plan: Move Airflow and API to openverse.org #3650

Conversation

sarayourfriend commented Jan 10, 2024 • edited Loading

Fixes

Description

Current round

Testing Instructions

Checklist

Developer Certificate of Origin

sarayourfriend commented Jan 10, 2024

github-actions bot commented Jan 10, 2024 • edited Loading

AetherUnbound left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openverse-bot commented Jan 24, 2024

Footnotes

dhruvkb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarayourfriend commented Jan 29, 2024 • edited Loading

sarayourfriend commented Jan 29, 2024

AetherUnbound left a comment

Choose a reason for hiding this comment

sarayourfriend commented Feb 1, 2024

sarayourfriend commented Jan 10, 2024 •

edited

Loading

github-actions bot commented Jan 10, 2024 •

edited

Loading

sarayourfriend commented Jan 29, 2024 •

edited

Loading