Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run migrations in order across repos #4466

Merged
merged 21 commits into from
Sep 9, 2024
Merged

Run migrations in order across repos #4466

merged 21 commits into from
Sep 9, 2024

Conversation

ruslandoga
Copy link
Contributor

@ruslandoga ruslandoga commented Aug 24, 2024

Changes

This PR makes /entrypoint.sh db migrate and /entrypoint.sh db pending-migrations order all pending migrations and run them in a single queue instead of per-repo.

Related card: https://3.basecamp.com/5308029/buckets/29267832/card_tables/cards/7508896723

Contents of that card

Some migrations in PostgreSQL implicitly depend on migrations in ClickHouse (there is at least one such migration), and because PostgreSQL migrations run first, they fail because from their point of view ClickHouse schemas are now "outdated":

How migrations were added (example):

  • add table to PostgreSQL db
  • add column to ClickHouse table
  • run PostgreSQL data migration that reads from new column in ClickHouse

How migrations are run:

  • run all Plausible.Repo migrations
  • run all Plausible.IngestRepo migrations

Which results in this broken order:

  • add table to PostgreSQL db
  • run PostgreSQL data migration that reads from new column in ClickHouse <- fails
  • add column to ClickHouse table <- never runs

It might be possible to run them in a single queue / order in db migrate command:

  • collect all migrations to be run, order them by version / date
  • start all repos (Repo, IngestRepo, ClickhouseRepo)
  • execute them one by one

Tests

  • Automated tests have been added (only for pending migrations listings, I'll try to add an actual db migrate test later)

Changelog

  • Entry has been added to changelog

Documentation

  • This change does not need a documentation update

Dark mode

  • This PR does not change the UI

@ruslandoga ruslandoga changed the title Order migrations in CE Order migrations Aug 24, 2024
@ruslandoga ruslandoga changed the title Order migrations Run migrations in order across repos Aug 26, 2024
@ruslandoga ruslandoga added the self-hosting Anything self-hosted label Aug 27, 2024
lib/plausible_release.ex Outdated Show resolved Hide resolved
lib/plausible_release.ex Outdated Show resolved Hide resolved
@zoldar zoldar added the preview label Sep 6, 2024
Copy link

github-actions bot commented Sep 6, 2024

Preview environment👷🏼‍♀️🏗️
PR-4466

@zoldar zoldar merged commit 8ee4827 into master Sep 9, 2024
10 checks passed
@zoldar zoldar deleted the order-migrations branch September 9, 2024 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview self-hosting Anything self-hosted
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants