Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove initialize_docker: Integrate data initialization to make up #22663

Merged
merged 4 commits into from
Oct 11, 2024

Conversation

KevinMind
Copy link
Contributor

@KevinMind KevinMind commented Sep 12, 2024

Fixes: mozilla/addons#15020

Description

Integrate data initialization to make up by integrating the 3 types of data initiailization commands to management commands, making them idempotent, and hooking them into make up.

With this, we no longer need initialize_docker as make up will do everything it does if the project is pristine.

Context

We haven't really needed the initiailzation command for a while, but now that everyone is fairly used to rely on make up for "doing everything" we should make it actually do everything.

This approach integrates data initialization as another part of the make up pipeline. What is mostly different here is that now make up will ensure:

  • database and other service level dependencies are up and running
  • database is migrated
  • database is seeded (unless skipped)
  • database is indexed in ES (unless skipped)

This has a pleasant side effect that CI now verifies database migrations more explicitly. We could consider migrating our testDB to use the regular DB in CI, but that is outside of this PRs scope.

Testing

You should run make up in the following scenarios.

  1. completely fresh project, Run make down && make docker_mysqld_volume_remove).
  2. Run make up to start from a "clean" data perspective. Expect: create DB, seed DB, index.
  3. Run make up again with running container and seeded db Expect: it does nothing
  4. Run make down && make up Expect: It should skip seeding and only reindex the DB.
  5. Enter a shell with make shell and run ./manage.py flush --noinput. Then run make down && make up Expect: it should reseed DB and reindex again.

Expect the flags to work as expected:

  • INIT_CLEAN: will wipe DB and index and reseed and reindex regardless of the existing DB state
  • INIT_LOAD: will load the given snapshot name and reindex.
  • DATA_BACKUP_SKIP: This will skip the initialize command entirely. No migration/seed/load/reindex, nothing.

The command itself is tested for various scenarios specified above. But you can manually inspect the commands run as the tests do not test the results of the commands but the execution order.

Checklist

  • Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
  • Successfully verified the change locally.
  • The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
  • Add before and after screenshots (Only for changes that impact the UI).
  • Add or update relevant docs reflecting the changes made.

@KevinMind KevinMind force-pushed the clean-init branch 15 times, most recently from 0810482 to 5737991 Compare September 16, 2024 17:51
@KevinMind KevinMind marked this pull request as ready for review September 17, 2024 08:29
@KevinMind KevinMind requested review from a team, eviljeff and diox and removed request for a team September 17, 2024 08:29
@KevinMind KevinMind force-pushed the clean-init branch 5 times, most recently from 44d0a4f to 6b9dbdc Compare September 19, 2024 12:07
@eviljeff
Copy link
Member

I tried following the testing steps and seemed to be a state where the script thought I had a database ("Database already exists") and needed to run all the migrations, which seems wrong.

log
andy@Andy-x1-2023:~/addons-server$ make up
for path in src/olympia.egg-info supervisord.pid version.json logs buildx-bake-metadata.json deps ; do rm -rf "/home/andy/addons-server/$path" && echo "$path removed"; done
src/olympia.egg-info removed
supervisord.pid removed
version.json removed
logs removed
buildx-bake-metadata.json removed
deps removed
./scripts/setup.py
Docker tag:  mozilla/addons-server:local
version:  local
digest:  None
Environment:
COMPOSE_FILE=docker-compose.yml
DOCKER_TAG=mozilla/addons-server:local
DOCKER_TARGET=development
HOST_UID=1000
image: mozilla/addons-server:local
make[1]: Entering directory '/home/andy/addons-server'
docker buildx bake --file docker-bake.hcl --file .env --progress auto --metadata-file buildx-bake-metadata.json
[+] Building 295.2s (29/29) FINISHED                                                          docker:default
 => [internal] load .dockerignore                                                                       0.1s
 => => transferring context: 195B                                                                       0.0s
 => [internal] load build definition from Dockerfile                                                    0.1s
 => => transferring dockerfile: 7.31kB                                                                  0.0s
 => [internal] load metadata for docker.io/library/python:3.11-slim-bookworm                            1.7s
 => [olympia 1/4] FROM docker.io/library/python:3.11-slim-bookworm@sha256:669bbd08353610485a94d5d0c976  0.3s
 => => resolve docker.io/library/python:3.11-slim-bookworm@sha256:669bbd08353610485a94d5d0c976b4b6498c  0.0s
 => => sha256:669bbd08353610485a94d5d0c976b4b6498c55280fe42c00f7581f85ee9f3121 9.13kB / 9.13kB          0.0s
 => => sha256:f236e0cc9c83d8cfe15e88297fe18f82287ff9acc55a22e647d81457faa0673c 1.75kB / 1.75kB          0.0s
 => => sha256:efacf1a70bae0321507f39c70be419fbb10362b82a786abbec2243e2e0c102d5 5.24kB / 5.24kB          0.0s
 => [internal] load build context                                                                       2.4s
 => => transferring context: 126.75MB                                                                   2.3s
 => [internal] settings cache mount permissions                                                         0.1s
 => [olympia 2/4] RUN <<EOF (groupadd -g 9500 olympia...)                                               0.7s
 => [olympia 3/4] WORKDIR /data/olympia                                                                 0.1s
 => [olympia 4/4] RUN chown -R olympia:olympia /data/olympia                                            0.6s
 => [base 1/7] COPY docker/*.gpg.asc /etc/apt/trusted.gpg.d/                                            0.1s
 => [base 2/7] COPY docker/*.list /etc/apt/sources.list.d/                                              0.1s
 => [base 3/7] RUN <<EOF (# Add support for https apt repos and gpg signed repos...)                    9.8s
 => [base 4/7] RUN --mount=type=bind,source=docker/debian_packages.txt,target=/debian_packages.txt <<  94.1s
 => [base 5/7] RUN localedef -i en_US -f UTF-8 en_US.UTF-8                                              1.7s
 => [base 6/7] RUN <<EOF (# Create directory for dependencies...)                                       0.5s
 => [base 7/7] RUN     --mount=type=bind,source=./requirements/pip.txt,target=/data/olympia/requiremen  4.1s
 => [locales 1/2] COPY --chown=olympia:olympia locale /data/olympia/locale                              0.2s
 => [pip_production 1/1] RUN     --mount=type=bind,source=./requirements/prod.txt,target=/data/olymp  134.2s
 => [pip_development 1/1] RUN     --mount=type=bind,source=./requirements/prod.txt,target=/data/olym  147.7s
 => [sources 1/5] COPY docker/etc/mime.types /etc/mime.types                                            0.2s
 => [sources 2/5] COPY --chown=olympia:olympia . /data/olympia                                          1.0s
 => [locales 2/2] RUN     --mount=type=bind,source=requirements/locale.txt,target=/data/olympia/requi  52.6s
 => [sources 3/5] COPY --from=locales --chown=olympia:olympia /data/olympia/locale /data/olympia/local  0.4s
 => [assets 1/2] COPY --chown=olympia:olympia static/ /data/olympia/static/                             0.3s
 => [assets 2/2] RUN     --mount=type=bind,src=src,target=/data/olympia/src     --mount=type=bind,src  31.8s
 => [sources 4/5] COPY --from=assets --chown=olympia:olympia /data/olympia/site-static /data/olympia/s  1.4s
 => [sources 5/5] COPY --from=assets --chown=olympia:olympia /data/olympia/static-build /data/olympia/  0.1s
 => [development 1/1] COPY --from=pip_development --chown=olympia:olympia /deps /deps                   3.4s
 => exporting to image                                                                                  8.0s
 => => exporting layers                                                                                 8.0s
 => => writing image sha256:81d0784fcacd800f26d6aad28d154e95268c58327dcda21a9204c4dd743bbb12            0.0s
 => => naming to docker.io/mozilla/addons-server:local                                                  0.0s
make[1]: Leaving directory '/home/andy/addons-server'
docker volume create addons-server_data_mysqld
addons-server_data_mysqld
docker compose up  -d --wait --remove-orphans --quiet-pull
[+] Building 0.0s (0/0)                                                                       docker:default
[+] Running 16/16
 ✔ Network addons-server_default              Created                                                   0.1s
 ✔ Volume "addons-server_storage"             Created                                                   0.0s
 ✔ Volume "addons-server_data_elasticsearch"  Created                                                   0.0s
 ✔ Volume "addons-server_data_redis"          Created                                                   0.0s
 ✔ Volume "addons-server_data_rabbitmq"       Created                                                   0.0s
 ✔ Volume "addons-server_data_olympia"        Created                                                   0.0s
 ✔ Container addons-server-addons-frontend-1  Healthy                                                   0.3s
 ✔ Container addons-server-mysqld-1           Healthy                                                   0.3s
 ✔ Container addons-server-redis-1            Healthy                                                   0.3s
 ✔ Container addons-server-rabbitmq-1         Healthy                                                   0.3s
 ✔ Container addons-server-autograph-1        Healthy                                                   0.3s
 ✔ Container addons-server-memcached-1        Healthy                                                   0.2s
 ✔ Container addons-server-elasticsearch-1    Healthy                                                   0.3s
 ✔ Container addons-server-web-1              Healthy                                                   4.7s
 ✔ Container addons-server-worker-1           Healthy                                                   4.7s
 ✔ Container addons-server-nginx-1            Healthy                                                   0.1s
docker image prune --filter "dangling=true" --filter "label!=buildx.cache" --force
Deleted Images:
deleted: sha256:8184e3224235053906f4dcdd0ac857d15ca53f800a318db3e17723f83f8d90e8
deleted: sha256:837e18180d68cd46ec0d0108ba8de2472b7fc96c68396671dcbd889c49004fb0
deleted: sha256:851860beeb69060eebf03b1df062ae0665407f7e851c71d7b5daed710f045e15
deleted: sha256:7d72dfd93e73893d3ead0d0a9024226f1e2f24611ab826d66a89f9b3a1f33890
deleted: sha256:fd63788128135658bd0e789f8c7ed75ad262303dc063706191cf6d09d39a1495
deleted: sha256:82a3e3c66e995947cdc14542525962ee61c84ec46eeacb2b4ece5521e5b4a3e2
deleted: sha256:62d467f4c9dd226f4a772270bb36cb87b55b3c98c7f61c9d2abb8f580a00b7f4
deleted: sha256:05a070550219175735e30f94be58992836bbeaaea4e9f476f1b56e3f0cdb5140
deleted: sha256:174079b29ec573440e162296f44b0059ae2c46c7c158fbf6c0f309ef942dcc12
deleted: sha256:f3d6625ddf483d14c4896b39da697a6e086e5b645e5d3e456f40c8a57651b9a0
deleted: sha256:ab96dba19fb9ee696bf47a2838ac10a9d0a315f9944761b41fd8593e227a6c3f
deleted: sha256:d80d99bd977f80d134414e44134c9d799c595d7e8d90c98aa4bb376db4315787
deleted: sha256:2abb35c95cfbee1c3f3b96f164aac0fb94e3a9f8f9a0a6475db48a430f24a603
deleted: sha256:aa6864eb0eeb88b50fb2901bf93cecb015fda00ee9269a0c21476583a144cd19
deleted: sha256:4038b0afe8f3c064c0b19f5d580c9104ef4e6302a26aa1c0ffec03695a921bb5
deleted: sha256:ad764c51df9a16d8dc3523cca4336b824e934befac612763147e5cf0dc795464
deleted: sha256:64ff2db1ee4df96e150e4376b134845ce0e7e21705d3e855d4693a2f391469d5

Total reclaimed space: 1.693GB
docker volume prune --force
Total reclaimed space: 0B
# Explicitly run initialize via the web container as make can get confused
# both routing the command to the web container and
# routing the command to the proper target.
docker compose exec --user olympia web make -f Makefile-docker initialize_data ARGS=''
Initializing data...
args:
python3 ./manage.py initialize_data
{"Timestamp": 1726751304921650176, "Type": "root", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "connecting to db"}, "severity": 200}
{"Timestamp": 1726751310378783744, "Type": "root", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "options: {'verbosity': 1, 'settings': None, 'pythonpath': None, 'traceback': False, 'no_color': False, 'force_color': False, 'skip_checks': False, 'force_db': False, 'skip_seed': False, 'skip_index': False}"}, "severity": 200}
{"Timestamp": 1726751310381134336, "Type": "root", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "Database already exists."}, "severity": 200}
{"Timestamp": 1726751310382600448, "Type": "root", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "Migrating..."}, "severity": 200}
Operations to perform:
  Apply all migrations: abuse, access, accounts, activity, addons, admin, akismet, amo, api, applications, auth, bandwagon, blocklist, contenttypes, devhub, discovery, es, files, git, hero, promoted, ratings, reviewers, scanners, search, sessions, shelves, stats, tags, translations, users, versions, waffle, yara, zadmin
Running migrations:
  Applying users.0001_initial... OK
  Applying translations.0001_initial... OK
  Applying applications.0001_initial... OK
  Applying addons.0001_initial... OK
  Applying versions.0001_initial... OK
  Applying ratings.0001_initial... OK
  Applying addons.0002_addon_fk... OK
  Applying addons.0003_addonreviewerflags_auto_approval_disabled_until... OK
  Applying addons.0004_auto_20191126_1712... OK
  Applying addons.0005_auto_20200527_0513.../deps/lib/python3.11/site-packages/multidb/__init__.py:65: UserWarning: [multidb] No replica databases are configured! You can configure them with the REPLICA_DATABASES setting.
  warnings.warn(
 OK
  Applying addons.0006_auto_20200527_0843... OK
  Applying addons.0007_addonreviewerflags_notified_about_auto_approval_delay... OK
  Applying addons.0008_auto_20200604_0928... OK
  Applying addons.0009_auto_20200603_1251... OK
  Applying addons.0010_auto_20200604_1319... OK
  Applying addons.0011_auto_20200610_0553... OK
  Applying addons.0012_remove_addon_public_stats... OK
  Applying addons.0013_auto_20200625_1139... OK
  Applying addons.0014_remove_addon_view_source... OK
  Applying addons.0015_auto_20200710_0859... OK
  Applying addons.0016_addonreviewerflags_auto_approval_disabled_until_next_approval... OK
  Applying addons.0017_addonreviewerflags_notified_about_expiring_delayed_rejections... OK
  Applying addons.0018_auto_20200803_1311... OK
  Applying addons.0019_auto_20200901_1459... OK
  Applying addons.0020_auto_20200805_1350... OK
  Applying addons.0021_auto_20200909_1302... OK
  Applying addons.0022_auto_20200916_0802... OK
  Applying addons.0023_auto_20200916_0749... OK
  Applying addons.0024_auto_20200923_1808... OK
  Applying addons.0025_auto_20201028_1647... OK
  Applying addons.0026_addonregionalrestrictions... OK
  Applying addons.0027_auto_20201119_1028... OK
  Applying addons.0028_auto_20210114_1347... OK
  Applying addons.0029_delete_appsupport... OK
  Applying addons.0030_auto_20210118_1527... OK
  Applying addons.0031_delete_category... OK
  Applying ratings.0002_ratingaggregate... OK
  Applying ratings.0003_auto_20210813_0941... OK
  Applying ratings.0004_auto_20210823_1255... OK
  Applying ratings.0005_alter_rating_body... OK
  Applying ratings.0006_alter_rating_body... OK
  Applying ratings.0007_auto_20230324_1053... OK
  Applying ratings.0008_alter_deniedratingword_options_alter_ratingflag_flag... OK
  Applying ratings.0009_alter_deniedratingword_word... OK
  Applying users.0002_auto_20200612_1604... OK
  Applying users.0003_auto_20200624_0225... OK
  Applying users.0004_auto_20201002_1006... OK
  Applying users.0005_auto_20210601_1421... OK
  Applying users.0006_auto_20210823_1454... OK
  Applying users.0007_auto_20220125_1636... OK
  Applying users.0008_alter_emailuserrestriction_email_pattern... OK
  Applying versions.0002_version_needs_human_review... OK
  Applying versions.0003_drop_version_int_column... OK
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying waffle.0001_initial... OK
  Applying waffle.0002_auto_20161201_0958... OK
  Applying waffle.0003_update_strings_for_i18n... OK
  Applying files.0001_initial... OK
  Applying yara.0001_initial... OK
  Applying yara.0002_auto_20191009_1239... OK
  Applying yara.0003_auto_20191010_1446... OK
  Applying scanners.0001_initial... OK
  Applying scanners.0002_auto_20191017_1257... OK
  Applying scanners.0003_auto_20191017_1514... OK
  Applying scanners.0004_auto_20191018_0740... OK
  Applying scanners.0005_scannerrule... OK
  Applying scanners.0006_auto_20191021_1704... OK
  Applying scanners.0007_remove_scannerresult_matches... OK
  Applying scanners.0008_auto_20191021_1718... OK
  Applying scanners.0009_auto_20191023_0906... OK
  Applying scanners.0010_auto_20191023_0908... OK
  Applying scanners.0011_auto_20191023_0908... OK
  Applying scanners.0012_auto_20191105_1037... OK
  Applying scanners.0013_auto_20191105_1522... OK
  Applying scanners.0014_alter_action_choices_on_scannerrule... OK
  Applying scanners.0015_auto_20191113_1126... OK
  Applying scanners.0016_scannerrule_definition... OK
  Applying scanners.0017_auto_20200107_1650... OK
  Applying scanners.0018_auto_20200114_1359... OK
  Applying scanners.0019_auto_20200113_1258... OK
  Applying scanners.0020_auto_20200116_1250... OK
  Applying scanners.0021_auto_20200122_1347... OK
  Applying scanners.0022_auto_20200122_1644... OK
  Applying scanners.0023_auto_20200122_1649... OK
  Applying scanners.0024_auto_20200128_1343... OK
  Applying scanners.0025_switch_enable_call_ml_api_task... OK
  Applying scanners.0026_auto_20200217_1528... OK
  Applying scanners.0027_add_group_result_id_and_task_count_to_scannerqueryrule... OK
  Applying scanners.0028_auto_20200309_1230... OK
  Applying scanners.0029_rename_ml_api_switch... OK
  Applying scanners.0030_scannerresult_score... OK
  Applying scanners.0031_versionscannerflags... OK
  Applying scanners.0032_auto_20200519_1438... OK
  Applying scanners.0033_delete_waffle_switch... OK
  Applying scanners.0034_delete_version_flags_again... OK
  Applying scanners.0035_delete_versionscannerflags... OK
  Applying scanners.0036_scannerresult_model_version... OK
  Applying scanners.0037_auto_20200717_1233... OK
  Applying reviewers.0001_initial... OK
  Applying reviewers.0002_autoapprovalsummary_help_text... OK
  Applying reviewers.0003_autoapprovalsummary_is_listing_disabled... OK
  Applying reviewers.0004_remove_autoapprovalsummary_is_listing_disabled... OK
  Applying reviewers.0005_remove_code_manager_waffle_flag... OK
  Applying reviewers.0006_auto_20200609_1655... OK
  Applying reviewers.0007_auto_20200730_1335... OK
  Applying reviewers.0008_auto_20200730_1335... OK
  Applying reviewers.0009_auto_20200923_1808... OK
  Applying reviewers.0010_add_subscription_channel... OK
  Applying reviewers.0011_auto_20201028_1647... OK
  Applying reviewers.0012_autoapprovalsummary_score... OK
  Applying reviewers.0013_backfill_autoapprovalsummary_score... OK
  Applying reviewers.0014_auto_20210120_1232... OK
  Applying reviewers.0015_auto_20210511_1256... OK
  Applying reviewers.0016_reviewactionreason... OK
  Applying blocklist.0001_initial... OK
  Applying blocklist.0002_auto_20191107_1302... OK
  Applying blocklist.0003_remove_block_addon... OK
  Applying access.0001_initial... OK
  Applying activity.0001_initial... OK
  Applying activity.0002_auto_20191107_1539... OK
  Applying activity.0003_auto_20191115_1151... OK
  Applying activity.0004_auto_20191125_1659... OK
  Applying activity.0005_auto_20200625_1139... OK
  Applying activity.0006_auto_20200731_1212... OK
  Applying activity.0007_auto_20210114_1347... OK
  Applying activity.0008_iplog... OK
  Applying activity.0009_auto_20210325_1320... OK
  Applying activity.0010_auto_20210813_0941... OK
  Applying activity.0011_iplog_log_activity_ip_ip_address_ba36172a... OK
  Applying activity.0012_reviewactionreasonlog... OK
  Applying activity.0013_auto_20211104_1748... OK
  Applying activity.0014_auto_20220222_1359... OK
  Applying activity.0015_auto_20220826_0956... OK
  Applying activity.0016_auto_20220826_1055... OK
  Applying activity.0017_auto_20220905_1541... OK
  Applying activity.0018_auto_20221128_1706... OK
  Applying activity.0018_auto_20221122_1313... OK
  Applying activity.0019_merge_0018_auto_20221122_1313_0018_auto_20221128_1706... OK
  Applying activity.0020_auto_20221214_1331... OK
  Applying activity.0021_alter_iplog_activity_log... OK
  Applying activity.0022_alter_activitylog_user... OK
  Applying activity.0023_remove_draftcomment_canned_response... OK
  Applying activity.0024_alter_activitylog_action... OK
  Applying ratings.0010_auto_20231106_1533... OK
  Applying ratings.0011_remove_rating_one_review_per_user_and_more... OK
  Applying bandwagon.0001_initial... OK
  Applying bandwagon.0002_delete_featuredcollection... OK
  Applying bandwagon.0003_auto_20210415_1620... OK
  Applying bandwagon.0004_auto_20210511_1256... OK
  Applying bandwagon.0005_auto_20210813_0941... OK
  Applying bandwagon.0006_auto_20221122_1312... OK
  Applying bandwagon.0007_alter_collection_description_alter_collection_name_and_more... OK
  Applying bandwagon.0008_alter_collection_options_alter_collection_managers_and_more... OK
  Applying blocklist.0004_multiblocksubmit... OK
  Applying blocklist.0005_multiblocksubmit_processed_guids... OK
  Applying blocklist.0006_multiblocksubmit_version_max_length_increase... OK
  Applying blocklist.0007_block_kinto_id... OK
  Applying blocklist.0008_kintoimport... OK
  Applying blocklist.0008_blocksubmission... OK
  Applying blocklist.0009_merge_20200117_0302... OK
  Applying blocklist.0009_blocksumission_rm_processed_guids... OK
  Applying blocklist.0010_merge_20200120_0426... OK
  Applying blocklist.0011_blocksubmission_submission_type... OK
  Applying blocklist.0012_auto_20200318_0442... OK
  Applying blocklist.0013_kintoimport_timestamp... OK
  Applying blocklist.0014_auto_20200513_1212... OK
  Applying blocklist.0015_auto_20200520_1725... OK
  Applying blocklist.0016_auto_20200521_1710... OK
  Applying blocklist.0017_auto_20200723_0906... OK
  Applying blocklist.0018_auto_20200731_1212... OK
  Applying blocklist.0019_block_average_daily_users... OK
  Applying blocklist.0020_auto_20200923_1808... OK
  Applying blocklist.0021_auto_20201028_1647... OK
  Applying blocklist.0022_auto_20201103_1107... OK
  Applying blocklist.0023_auto_20201103_1108... OK
  Applying blocklist.0024_delete_legacyimport... OK
  Applying blocklist.0025_alter_block_legacy_id... OK
  Applying blocklist.0026_auto_20211214_1555... OK
  Applying blocklist.0027_auto_20220530_1639... OK
  Applying addons.0032_addonreviewerflags_auto_approval_disabled_unlisted... OK
  Applying addons.0033_auto_20210511_1256... OK
  Applying addons.0034_addonreviewerflags_auto_approval_disabled_until_next_approval_unlisted... OK
  Applying addons.0035_auto_20210812_1726... OK
  Applying addons.0036_add_allow_deleted_guid_reuse_switch... OK
  Applying addons.0037_auto_20220321_1545... OK
  Applying addons.0038_auto_20220530_1639... OK
  Applying addons.0039_auto_20220531_0953... OK
  Applying addons.0040_auto_20221011_1104... OK
  Applying addons.0041_auto_20221122_1313... OK
  Applying addons.0042_auto_20221205_1328... OK
  Applying addons.0043_addonreviewerflags_auto_approval_delayed_until_unlisted... OK
  Applying addons.0044_remove_addonreviewerflags_notified_about_auto_approval_delay_and_more... OK
  Applying addons.0045_addonbrowsermapping... OK
  Applying addons.0046_auto_20230612_1057... OK
  Applying addons.0047_addonuser_original_role_alter_addon_type... OK
  Applying addons.0048_deletedpreviewfile_alter_addonuser_original_role_and_more... OK
  Applying addons.0049_clear_bad_url_data... OK
  Applying abuse.0001_initial... OK
  Applying abuse.0002_add_amo_entrypoint... OK
  Applying abuse.0003_abusereport_addon_install_source_url... OK
  Applying abuse.0004_auto_20210506_1232... OK
  Applying abuse.0005_auto_20210729_1339... OK
  Applying abuse.0006_auto_20210813_0941... OK
  Applying abuse.0007_auto_20220803_0948... OK
  Applying abuse.0008_alter_abusereport_report_entry_point... OK
  Applying abuse.0009_abusereport_reporter_email_name_reasons... OK
  Applying abuse.0010_cinderreport... OK
  Applying abuse.0011_auto_20231006_1035... OK
  Applying abuse.0012_cinderreport_appeal_id_cinderreport_decision_date_and_more... OK
  Applying abuse.0013_alter_abusereport_reason... OK
  Applying abuse.0014_abusereport_location... OK
  Applying abuse.0015_remove_abusereport_just_one_of_guid_and_user_must_be_set_and_more... OK
  Applying abuse.0016_remove_abusereport_just_one_of_guid_user_rating_must_be_set_and_more... OK
  Applying abuse.0017_cinderjob_cinderjobappeal_delete_cinderreport_and_more... OK
  Applying abuse.0018_cinderpolicy_cinderjob_policies... OK
  Applying abuse.0019_alter_abusereport_drop_state_options_alter_abusereport_managers... OK
  Applying abuse.0020_alter_abusereport_addon_install_method_and_more... OK
  Applying abuse.0021_remove_abusereport_state_abusereport_appeal_date_and_more... OK
  Applying abuse.0022_rename_appeal_date_abusereport_reporter_appeal_date_and_more... OK
  Applying abuse.0023_cinderjob_decision_notes_alter_cinderjob_appeal_job_and_more... OK
  Applying abuse.0024_cinderpolicy_parent_alter_cinderpolicy_uuid... OK
  Applying abuse.0025_cinderjob_resolvable_in_reviewer_tools_and_more... OK
  Applying abuse.0026_add_cinderjob_decision_cinder_id... OK
  Applying abuse.0027_add_cinderdecision... OK
  Applying abuse.0028_cinderdecision_backfill... OK
  Applying abuse.0029_alter_cinderpolicy_name_length... OK
  Applying abuse.0030_rename_cinder_waffle_switch... OK
  Applying abuse.0031_remove_cinderjob_appeal_job_and_more... OK
  Applying abuse.0032_cinderpolicy_default_cinder_action_and_more... OK
  Applying abuse.0033_abusereport_illegal_category... OK
  Applying abuse.0034_abusereport_illegal_subcategory... OK
  Applying abuse.0035_alter_abusereport_addon_signature... OK
  Applying abuse.0036_alter_abusereport_appellant_job_cinderappealtext... OK
  Applying abuse.0037_auto_20240823_1236... OK
  Applying abuse.0038_cinderjob_forwarded_to_job_and_more... OK
  Applying abuse.0039_auto_20240829_1154... OK
  Applying access.0002_give_api_bypass_throttling_permission... OK
  Applying access.0003_add_force_send_mail_group... OK
  Applying accounts.0001_remove_old_2fa_waffle_switch... OK
  Applying activity.0025_alter_activitylog_action_cinderpolicylog... OK
  Applying activity.0026_alter_activitylog_action_attachmentlog... OK
  Applying addons.0050_remove_www_of_buymeacoffee... OK
  Applying addons.0051_remove_accessibility... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying akismet.0001_initial... OK
  Applying akismet.0002_delete_akismetreport... OK
  Applying amo.0001_initial... OK
  Applying api.0001_initial... OK
  Applying api.0002_auto_20210511_1256... OK
  Applying applications.0002_auto_20191101_1222... OK
  Applying applications.0003_auto_20201014_1333... OK
  Applying applications.0004_create_appversions_for_fenix_ga... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying blocklist.0028_auto_20230316_1317... OK
  Applying blocklist.0029_alter_blocklistsubmission_delayed_until_and_more... OK
  Applying blocklist.0030_alter_blocklistsubmission_signoff_state_blockversion... OK
  Applying blocklist.0031_alter_blocklistsubmission_block_add_changed_versions... OK
  Applying blocklist.0032_alter_blocklistsubmission_reason_url_null_true... OK
  Applying devhub.0001_initial... OK
  Applying devhub.0002_auto_20220209_1845... OK
  Applying devhub.0003_auto_20220413_1016... OK
  Applying devhub.0004_alter_blogpost_options... OK
  Applying devhub.0005_auto_20220531_1043... OK
  Applying devhub.0006_auto_20240109_1118... OK
  Applying shelves.0001_initial... OK
  Applying shelves.0002_auto_20200716_1254... OK
  Applying shelves.0003_auto_20200720_1509... OK
  Applying shelves.0004_auto_20200805_1851... OK
  Applying shelves.0005_auto_20200807_1732... OK
  Applying shelves.0006_shelfmanagement... OK
  Applying shelves.0007_auto_20200817_1828... OK
  Applying shelves.0008_auto_20201028_1647... OK
  Applying shelves.0009_auto_20210119_1505... OK
  Applying shelves.0010_shelf_addon_count... OK
  Applying shelves.0011_auto_20210215_1216... OK
  Applying shelves.0012_auto_20210218_1517... OK
  Applying shelves.0013_auto_20210325_1320... OK
  Applying shelves.0014_auto_20210416_2225... OK
  Applying shelves.0015_auto_20210422_1032... OK
  Applying shelves.0016_auto_20210617_1113... OK
  Applying shelves.0017_auto_20210726_1422... OK
  Applying shelves.0018_delete_shelfmanagement... OK
  Applying versions.0004_auto_20200512_1718... OK
  Applying versions.0005_auto_20200518_1259... OK
  Applying versions.0006_remove_version_source_git_hash... OK
  Applying versions.0007_versionreviewerflags... OK
  Applying versions.0008_auto_20200625_1114... OK
  Applying promoted.0001_initial... OK
  Applying promoted.0002_promotedaddon_application_id... OK
  Applying promoted.0003_auto_20200721_1201... OK
  Applying promoted.0004_auto_20200731_1212... OK
  Applying promoted.0005_auto_20200803_1214... OK
  Applying discovery.0001_initial... OK
  Applying hero.0001_initial... OK
  Applying hero.0002_auto_20190930_1557... OK
  Applying hero.0003_auto_20191001_0756... OK
  Applying hero.0004_auto_20191021_0831... OK
  Applying hero.0005_auto_20191115_1123... OK
  Applying hero.0006_auto_20191127_1246... OK
  Applying hero.0007_auto_20200603_0207_squashed_0010_primaryhero_select_image... OK
  Applying discovery.0002_secondary_hero_shelf... OK
  Applying discovery.0003_primaryheroimageupload... OK
  Applying discovery.0004_shelfmodule... OK
  Applying discovery.0005_auto_20200717_1256... OK
  Applying discovery.0006_auto_20200807_2051... OK
  Applying discovery.0007_auto_20200811_1319... OK
  Applying discovery.0008_remove_discoveryitem_recommendable... OK
  Applying discovery.0009_auto_20201027_1903... OK
  Applying discovery.0010_auto_20201104_1424... OK
  Applying discovery.0011_auto_20210325_1320... OK
  Applying discovery.0012_delete_shelfmodule... OK
  Applying discovery.0013_homepageshelves... OK
  Applying discovery.0014_auto_20230227_1144... OK
  Applying es.0001_initial... OK
  Applying es.0002_alter_reindexing_site... OK
  Applying es.0003_remove_reindexing_site... OK
  Applying es.0004_delete_reindexing... OK
  Applying versions.0009_auto_20200717_1137... OK
  Applying versions.0010_auto_20200901_1458... OK
  Applying versions.0011_auto_20200923_1808... OK
  Applying versions.0012_auto_20201020_1147... OK
  Applying versions.0013_auto_20201028_1647... OK
  Applying versions.0014_auto_20201030_0949... OK
  Applying versions.0015_auto_20201105_1039... OK
  Applying versions.0016_auto_20210325_1320... OK
  Applying files.0002_add_remote_addr_and_source_to_fileupload... OK
  Applying files.0003_webextpermission_optional_permissions... OK
  Applying files.0004_auto_20200923_1808... OK
  Applying files.0005_auto_20201120_0926... OK
  Applying files.0006_auto_20210223_1215... OK
  Applying files.0007_auto_20210315_1220... OK
  Applying files.0008_remove_file_platform... OK
  Applying files.0009_add_enable_manifest_normalization_switch... OK
  Applying files.0010_enable_mv3_switch... OK
  Applying files.0011_file_manifest_version... OK
  Applying files.0012_alter_file_is_restart_required... OK
  Applying files.0013_auto_20210830_1424... OK
  Applying files.0014_auto_20210824_1722... OK
  Applying files.0015_auto_20210906_1356... OK
  Applying files.0016_auto_20210929_1203... OK
  Applying files.0017_auto_20211128_0808... OK
  Applying files.0018_auto_20211130_1539... OK
  Applying files.0019_alter_fileupload_user... OK
  Applying files.0020_auto_20220124_1457... OK
  Applying files.0021_auto_20220503_1855... OK
  Applying files.0022_alter_fileupload_source... OK
  Applying files.0023_auto_20220803_1240... OK
  Applying files.0024_auto_20220819_1332... OK
  Applying files.0025_auto_20221104_1311... OK
  Applying files.0026_auto_20221104_1312... OK
  Applying files.0027_auto_20230131_1818... OK
  Applying files.0028_set_approval_date... OK
  Applying files.0029_auto_20230301_1843... OK
  Applying files.0030_remove-add-guid-to-manifest-waffle-switch... OK
  Applying files.0031_file_status_disabled_reason... OK
  Applying files.0032_auto_20231107_1546... OK
  Applying files.0033_filemanifest... OK
  Applying git.0001_initial... OK
  Applying git.0002_auto_20200420_1525... OK
  Applying git.0003_auto_20200427_1211... OK
  Applying git.0004_auto_20210511_1256... OK
  Applying git.0005_auto_20220225_1141... OK
  Applying git.0006_auto_20220412_1144... OK
  Applying hero.0008_auto_20200616_2044... OK
  Applying hero.0009_auto_20200618_1421... OK
  Applying hero.0010_auto_20200623_1949... OK
  Applying hero.0011_auto_20200624_1809... OK
  Applying hero.0012_auto_20200709_0316... OK
  Applying hero.0013_auto_20200715_1751... OK
  Applying hero.0014_auto_20200717_1120... OK
  Applying hero.0015_auto_20200728_0154... OK
  Applying hero.0016_auto_20201028_1647... OK
  Applying hero.0017_auto_20210511_1256... OK
  Applying hero.0018_alter_primaryheroimage_custom_image... OK
  Applying hero.0019_alter_secondaryheromodule_icon... OK
  Applying promoted.0006_auto_20200819_0951... OK
  Applying promoted.0007_auto_20200924_1328... OK
  Applying promoted.0008_auto_20200924_1329... OK
  Applying promoted.0009_promotedtheme... OK
  Applying promoted.0010_promotedsubscription... OK
  Applying promoted.0011_create_switch_for_subscription... OK
  Applying promoted.0012_auto_20201022_0903... OK
  Applying promoted.0013_auto_20201022_0906... OK
  Applying promoted.0014_promotedsubscription_onboarding_rate... OK
  Applying promoted.0015_promotedsubscription_stripe_subscription_id... OK
  Applying promoted.0016_auto_20201106_1355... OK
  Applying promoted.0017_auto_20201106_1507... OK
  Applying promoted.0018_auto_20201106_1507... OK
  Applying promoted.0019_delete_promotedsubscription... OK
  Applying promoted.0020_auto_20221214_1331... OK
  Applying versions.0017_installorigin... OK
  Applying versions.0018_alter_license_some_rights... OK
  Applying versions.0019_auto_20211123_1229... OK
  Applying versions.0020_auto_20211124_1050... OK
  Applying versions.0021_auto_20211130_1539... OK
  Applying versions.0022_auto_20220203_2057... OK
  Applying versions.0023_auto_20220204_1646... OK
  Applying versions.0024_auto_20220530_1639... OK
  Applying versions.0025_auto_20220614_1653... OK
  Applying versions.0026_auto_20220722_1704... OK
  Applying versions.0027_auto_20220729_0932... OK
  Applying versions.0028_alter_license_on_form... OK
  Applying versions.0029_auto_20221114_1747... OK
  Applying versions.0030_auto_20221122_1312... OK
  Applying versions.0031_add_due_date... OK
  Applying versions.0032_auto_20230107_0048... OK
  Applying versions.0033_auto_20230118_1301... OK
  Applying versions.0034_auto_20230131_1821... OK
  Applying versions.0035_set_human_review_date... OK
  Applying versions.0036_backfill-human-review-date-more-agressively... OK
  Applying versions.0037_auto_20230502_1107... OK
  Applying versions.0038_auto_20230511_1416... OK
  Applying reviewers.0017_alter_reviewactionreason_options... OK
  Applying reviewers.0018_alter_autoapprovalsummary_is_promoted_prereview... OK
  Applying reviewers.0019_reviewactionreason_canned_response... OK
  Applying reviewers.0020_reviewactionreason_addon_type... OK
  Applying reviewers.0020_auto_20221114_1835... OK
  Applying reviewers.0021_merge_20221205_1147... OK
  Applying reviewers.0022_alter_reviewactionreason_canned_response... OK
  Applying reviewers.0023_delete_reviewerscore... OK
  Applying reviewers.0024_delete_cannedresponse... OK
  Applying reviewers.0025_needshumanreviewhistory... OK
  Applying reviewers.0026_auto_20230516_0949... OK
  Applying reviewers.0027_backfill_needshumanreview... OK
  Applying reviewers.0028_usagetier... OK
  Applying reviewers.0029_auto_20230606_1708... OK
  Applying reviewers.0030_auto_20230622_1231... OK
  Applying reviewers.0031_reviewactionreason_canned_block_reason... OK
  Applying reviewers.0032_alter_needshumanreview_reason... OK
  Applying reviewers.0033_reviewactionreason_cinder_policy... OK
  Applying reviewers.0034_create-specific-cinder-switch... OK
  Applying reviewers.0035_alter_reviewactionreason_cinder_policy... OK
  Applying reviewers.0036_alter_needshumanreview_reason_and_more... OK
  Applying reviewers.0037_autoapprovalsummary_is_pending_rejection... OK
  Applying reviewers.0038_auto_20240909_1647... OK
  Applying versions.0039_alter_version_needs_human_review... OK
  Applying scanners.0038_scannerqueryrule_completed... OK
  Applying scanners.0039_auto_20200923_1808... OK
  Applying scanners.0040_scannerqueryresult_was_blocked... OK
  Applying scanners.0041_auto_20201104_1043... OK
  Applying scanners.0042_remove_waffle_switch... OK
  Applying scanners.0043_auto_20210325_1320... OK
  Applying scanners.0044_auto_20210511_1256... OK
  Applying scanners.0045_auto_20210604_1340... OK
  Applying scanners.0046_delete_waffle_switch... OK
  Applying scanners.0047_delete_wat_scanner_results... OK
  Applying scanners.0048_auto_20221017_1900... OK
  Applying scanners.0049_auto_20221018_1635... OK
  Applying scanners.0050_auto_20221107_1721... OK
  Applying scanners.0051_auto_20221108_1701... OK
  Applying scanners.0052_alter_scannerqueryresult_version_and_more... OK
  Applying search.0001_remove-es-use-classic-similarity-waffle-switch... OK
  Applying search.0002_auto_20210906_0958... OK
  Applying search.0003_reindexing... OK
  Applying sessions.0001_initial... OK
  Applying shelves.0019_auto_20210812_1412... OK
  Applying shelves.0020_alter_shelf_footer_pathname... OK
  Applying shelves.0021_alter_shelf_addon_type... OK
  Applying shelves.0022_alter_shelf_addon_type... OK
  Applying stats.0001_initial... OK
  Applying stats.0002_beta_stats_flag... OK
  Applying stats.0003_delete_beta_stats_flag... OK
  Applying stats.0004_delete_updatecount... OK
  Applying stats.0005_create_switch_bigquery_download_stats_cron_tasks... OK
  Applying stats.0006_create_switch_for_fenix_build_ids... OK
  Applying stats.0007_delete_downloadcount... OK
  Applying stats.0008_auto_20201007_0839... OK
  Applying stats.0009_auto_20201013_1417... OK
  Applying stats.0010_enable_bigquery_switch... OK
  Applying tags.0001_initial... OK
  Applying tags.0002_auto_20210713_1131... OK
  Applying tags.0003_auto_20210721_1146... OK
  Applying tags.0004_auto_20210721_1206... OK
  Applying tags.0005_auto_20210728_1724... OK
  Applying tags.0006_auto_20210813_0941... OK
  Applying tags.0007_tag_enable_for_random_shelf... OK
  Applying users.0009_alter_userprofile_biography... OK
  Applying users.0010_auto_20230209_1546... OK
  Applying users.0011_alter_disposableemaildomainrestriction_restriction_type_and_more... OK
  Applying users.0012_bannedusercontent... OK
  Applying users.0013_bannedusercontent_picture_backup_name_and_more... OK
  Applying users.0014_suppressedemail... OK
  Applying users.0015_clear_bad_homepage_url_data... OK
  Applying users.0016_suppressedemailverification... OK
  Applying users.0017_suppressedemailverification_status... OK
  Applying versions.0040_alter_license_name_and_more... OK
  Applying versions.0041_remove_static_themes_android_compatibility... OK
  Applying versions.0042_auto_20230531_0926... OK
  Applying versions.0043_remove_version_needs_human_review_and_more... OK
  Applying versions.0044_auto_20240409_1358... OK
  Applying versions.0045_versionprovenance... OK
  Applying versions.0046_auto_20240916_1240... OK
  Applying waffle.0004_update_everyone_nullbooleanfield... OK
  Applying yara.0004_delete_yararesult... OK
  Applying zadmin.0001_initial... OK
  Applying zadmin.0002_auto_20230522_0938... OK
{"Timestamp": 1726751620256027904, "Type": "root", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "Skipping load initial data."}, "severity": 200}
{"Timestamp": 1726751620256429568, "Type": "root", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "Skipping seeding the database."}, "severity": 200}
Starting the reindexation for addons.
{"Timestamp": 1726751620428064512, "Type": "z.elasticsearch", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Unflagging the database"}, "severity": 200}
Building the task chain
{"Timestamp": 1726751620478544384, "Type": "z.elasticsearch", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Returning reindexing group for addons-20240919131340"}, "severity": 200}
{"Timestamp": 1726751620487556096, "Type": "z.task", "Logger": "http_app_addons", "Hostname": "f40c5c8f1018", "EnvVersion": "2.0", "Severity": 6, "Pid": 131, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Created a group of 0 tasks for task \"olympia.addons.tasks.index_addons\"."}, "severity": 200}
Running all indexation tasks


*** Reindexation done ***

Current Aliases configuration:

{
    "addons-20240919131340": {
        "aliases": {
            "addons": {}
        }
    }
}

@KevinMind
Copy link
Contributor Author

@eviljeff This looks like expected behavior. If you follow the tests and the code the migrations always run.

I'd be curious what happens if you run make up again. It should simply say all migrations have been applied.

Look at the test cases for more sophisticated scenarios.

@eviljeff
Copy link
Member

@eviljeff This looks like expected behavior. If you follow the tests and the code the migrations always run.

I'm trying to test the scenario of a clean install - that all the migrations had to run indicates the database was empty, but initialize_data seemingly didn't do that creation, because the logging indicates the database already existed at that point.

@KevinMind
Copy link
Contributor Author

KevinMind commented Sep 19, 2024

@eviljeff This looks like expected behavior. If you follow the tests and the code the migrations always run.

I'm trying to test the scenario of a clean install - that all the migrations had to run indicates the database was empty, but initialize_data seemingly didn't do that creation, because the logging indicates the database already existed at that point.

Can you show me exactly what you ran to do a fresh install? For that you'd have to drop the mysql volume and run make up. or run with INIT_FORCE_DB.

It looks like the database existsed, but was not migrated in this case. I'm not totally sure how that happened so I would need more context on exactly what you ran.

@KevinMind
Copy link
Contributor Author

Can you show me exactly what you ran to do a fresh install? For that you'd have to drop the mysql volume and run make up. or run with INIT_FORCE_DB.
It looks like the database existsed, but was not migrated in this case. I'm not totally sure how that happened so I would need more context on exactly what you ran.

I ran exactly what the testing procedure specified: make down && make clean_docker

I'm very glad you caught that. That was a bug and in squashing it I realized that this is still a lot more complicated than it needs to be. I'm going to switch back to draft and will re-request review once I've updated the code/tests.

@KevinMind KevinMind marked this pull request as draft September 20, 2024 14:35
@KevinMind KevinMind force-pushed the clean-init branch 10 times, most recently from 66c6a6c to cbb9f0d Compare September 26, 2024 17:16
@KevinMind KevinMind marked this pull request as ready for review September 26, 2024 17:18
@KevinMind
Copy link
Contributor Author

@eviljeff now that #22693 is merged we can continue here.

@KevinMind KevinMind force-pushed the clean-init branch 2 times, most recently from 99e4844 to a4b5460 Compare September 27, 2024 21:26
@diox
Copy link
Member

diox commented Oct 7, 2024

  1. Run make down && make up Expect: It should skip seeding and only reindex the DB.

That didn't work for me, reindex did not occur, and I was left with a broken frontend. I can see these initialization steps (but no reindex) in the output:

(...)
# Explicitly run initialize via the web container as make can get confused
# both routing the command to the web container and
# routing the command to the proper target.
docker compose exec --user olympia web make -f Makefile-docker initialize ARGS=''
Initializing data...
args: 
python3 ./manage.py initialize 
{"Timestamp": 1728292235158532864, "Type": "root", "Logger": "http_app_addons", "Hostname": "d6fb82f6c34b", "EnvVersion": "2.0", "Severity": 6, "Pid": 47, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "options: {'verbosity': 1, 'settings': None, 'pythonpath': None, 'traceback': False, 'no_color': False, 'force_color': False, 'skip_checks': False, 'clean': False, 'load': None}"}, "severity": 200}
{"Timestamp": 1728292235159064832, "Type": "root", "Logger": "http_app_addons", "Hostname": "d6fb82f6c34b", "EnvVersion": "2.0", "Severity": 6, "Pid": 47, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "Migrating..."}, "severity": 200}
(...)

.github/actions/run-docker/action.yml Show resolved Hide resolved
Makefile-docker Show resolved Hide resolved
Makefile-docker Show resolved Hide resolved
@KevinMind KevinMind mentioned this pull request Oct 8, 2024
5 tasks
@KevinMind KevinMind requested a review from diox October 9, 2024 12:53
@KevinMind
Copy link
Contributor Author

  1. Run make down && make up Expect: It should skip seeding and only reindex the DB.

That didn't work for me, reindex did not occur, and I was left with a broken frontend. I can see these initialization steps (but no reindex) in the output:

(...)
# Explicitly run initialize via the web container as make can get confused
# both routing the command to the web container and
# routing the command to the proper target.
docker compose exec --user olympia web make -f Makefile-docker initialize ARGS=''
Initializing data...
args: 
python3 ./manage.py initialize 
{"Timestamp": 1728292235158532864, "Type": "root", "Logger": "http_app_addons", "Hostname": "d6fb82f6c34b", "EnvVersion": "2.0", "Severity": 6, "Pid": 47, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "options: {'verbosity': 1, 'settings': None, 'pythonpath': None, 'traceback': False, 'no_color': False, 'force_color': False, 'skip_checks': False, 'clean': False, 'load': None}"}, "severity": 200}
{"Timestamp": 1728292235159064832, "Type": "root", "Logger": "http_app_addons", "Hostname": "d6fb82f6c34b", "EnvVersion": "2.0", "Severity": 6, "Pid": 47, "Fields": {"uid": "", "remoteAddressChain": "", "msg": "Migrating..."}, "severity": 200}
(...)

@diox this commit fixes that issue.

Was introduced by a different PR that landed in between this one opening and now.

Rename "initialize_data" to "initialize"

Remove DOCKER_SERVICES as dangerous option given hard dependency on mysqld and elasticsearch.

Better timing for mysql healthcheck
@KevinMind KevinMind merged commit daee988 into master Oct 11, 2024
31 checks passed
@KevinMind KevinMind deleted the clean-init branch October 11, 2024 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task]: Make up should handle data initialization and migrations
3 participants