Database update testing script and tests (#9064)

.github/workflows/upgrade-tests.yml

@@ -0,0 +1,49 @@
+name: Upgrade Tests
+
+on:
+  schedule:
+    - cron: '0 4 * * *'
+  workflow_dispatch:
+
+jobs:
+  upgrade-test:
+    name: Run migration tests
+    runs-on: ubuntu-20.04
+
+    steps:
+      - name: Install Docker
+        run: |
+          sudo apt-get -y update
+          sudo apt-get -y install ca-certificates curl gnupg lsb-release
+          sudo mkdir -p /etc/apt/keyrings
+          curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+          echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+          sudo apt-get update
+          sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
+
+      - name: Install prerequisites
+        run: |
+          sudo apt-get -y install jq
+
+      - name: Install gojira
+        run: |
+          cd $RUNNER_WORKSPACE
+          git clone https://github.com/Kong/gojira
+          mkdir -p $HOME/.local/bin
+          ln -s $(pwd)/gojira/gojira.sh $HOME/.local/bin/gojira
+
+      - name: Clone Kong source code
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+
+      - name: Run upgrade tests (Postgres)
+        run: |
+          export GOJIRA_KONG_REPO_URL=$GITHUB_WORKSPACE
+          bash -x ./scripts/test-upgrade-path.sh -d postgres 2.8.0 $GITHUB_REF_NAME
+
+      - name: Run upgrade tests (Cassandra)
+        run: |
+          export GOJIRA_KONG_REPO_URL=$GITHUB_WORKSPACE
+          gojira nuke
+          bash -x ./scripts/test-upgrade-path.sh -d postgres 2.8.0 $GITHUB_REF_NAME

.gitignore

@@ -26,3 +26,5 @@ bin/grpcurl
 
 *.so
 *.bak
+
+upgrade-test-log

.luacheckrc

@@ -64,6 +64,10 @@ files["spec/**/*.lua"] = {
     std = "ngx_lua+busted",
 }
 
+files["**/*_test.lua"] = {
+    std = "ngx_lua+busted",
+}
+
 files["spec-old-api/**/*.lua"] = {
     std = "ngx_lua+busted",
 }

CHANGELOG.md

@@ -222,6 +222,15 @@
 - Change the default of `lua_ssl_trusted_certificate` to `system`
   [#8602](https://github.com/Kong/kong/pull/8602) to automatically load trusted CA list from system CA store.
 
+#### Migrations
+
+- Postgres migrations can now have an `up_f` part like Cassandra
+  migrations, designating a function to call.  The `up_f` part is
+  invoked after the `up` part has been executed against the database
+  for both Postgres and Cassandra.
+
+- A new CLI command, `kong migrations status`, generates the status on a JSON file.
+
 ### Dependencies
 
 - Bumped OpenResty from 1.19.9.1 to [1.21.4.1](https://openresty.org/en/changelog-1021004.html)

DEVELOPER.md

@@ -177,6 +177,70 @@ languages) is performing static linting of your code. You can use [luacheck]
 make lint
 ```
 
+#### Upgrade tests
+
+Kong Gateway supports no-downtime upgrades through its database schema
+migration mechanism (see [UPGRADE.md](./UPGRADE.md)).  Each schema
+migration needs to be written in a way that allows the previous and
+the current version of Kong Gateway run against the same database
+during upgrades.  Once all nodes have been upgraded to the current
+version of Kong Gateway, additional changes to the database can be
+made that are incompatible with the previous version.  To support
+that, each migration is split into two parts, an `up` part that can
+only make backwards-compatible changes, and a `teardown` part that
+runs after all nodes have been upgraded to the current version.
+
+Each migration that is contained in Kong Gateway needs to be
+accompanied with a test that verifies the correct operation of both
+the previous and the current version during an upgrade.  These tests
+are located in the [spec/05-migration/](spec/05-migration/) directory
+and must be named after the migration they test such that the
+migration `kong/**/*.lua` has a test in
+`spec/05-migration/**/*_spec.lua`.  The presence of a test is enforced
+by the [upgrade testing](scripts/test-upgrade-path.sh) shell script
+which is [automatically run](.github/workflows/upgrade-tests.yml)
+through a GitHub Action.
+
+The [upgrade testing](scripts/test-upgrade-path.sh) shell script works
+as follows:
+
+ * A new Kong Gateway installation is brought up using
+   [Gojira](https://github.com/Kong/gojira), consisting of one node
+   containing the previous version of Kong Gateway ("OLD"), one node
+   containing the current version of Kong Gateway ("NEW") and a shared
+   database server (PostgreSQL or Cassandra).
+ * NEW: The database is initialized using `kong migrations bootstrap`.
+ * OLD: The `setup` phase of all applicable migration tests is run.
+ * NEW: `kong migrations up` is run to run the `up` part of all
+   applicable migrations.
+ * OLD: The `old_after_up` phase of all applicable migration tests is
+   run.
+ * NEW: The `new_after_up` phase of all applicable migration tests is
+   run.
+ * NEW: `kong migrations finish` is run to invoke the `teardown` part
+   of all applicable migrations.
+ * NEW: The `new_after_finish` phase of all applicable migration tests
+   is run.
+
+Upgrade tests are run using [busted].  To support the specific testing
+method of upgrade testing, a number of helper functions are defined in
+the [spec/upgrade_helpers.lua](spec/upgrade_helpers.lua) module.
+Migration tests use functions from this module to define test cases
+and associate them with phases of the upgrade testing process.
+Consequently, they are named `setup`, `old_after_up`, `new_after_up`
+and `new_after_finish`.  Additonally, the function `all_phases` can be
+used to run a certain test in the three phases `old_after_up`,
+`new_after_up` and `new_after_finish`.  These functions replace the
+use of busted's `it` function and accept a descriptive string and a
+function as argument.
+
+It is important to note that upgrade tests need to run on both the old
+and the new version of Kong.  Thus, they can only use features that
+are available in both versions (i.e. from helpers.lua).  The module
+[spec/upgrade_helpers.lua](spec/upgrade_helpers.lua) is copied from
+the new version into the container of the old version and it can be
+used to make new library functionality available to migration tests.
+
 #### Makefile
 
 When developing, you can use the `Makefile` for doing the following operations:

kong/cmd/migrations.lua

@@ -26,6 +26,8 @@ The available commands are:
 
   reset                             Reset the database. The `reset` command erases all of the data in Kong's database and deletes all of the schemas.
 
+  status                            Dump the database migration status in JSON format
+
 Options:
  -y,--yes                           Assume "yes" to prompts and run
                                     non-interactively.
@@ -154,6 +156,27 @@ local function execute(args)
 
     -- exit(0)
 
+  elseif args.command == "status" then
+
+    -- Clean up the schema_state data structure so that it can be
+    -- serialized as json.
+    local function cleanup (namespace_migrations)
+      if namespace_migrations then
+        for _, namespace_migration in pairs(namespace_migrations) do
+          for i = 1, #namespace_migration.migrations do
+            namespace_migration.migrations[i] = namespace_migration.migrations[i].name
+          end
+        end
+      end
+    end
+
+    cleanup(schema_state.new_migrations)
+    cleanup(schema_state.pending_migrations)
+    cleanup(schema_state.executed_migrations)
+
+    local cjson = require "cjson"
+    print(cjson.encode(schema_state))
+
   elseif args.command == "bootstrap" then
     if args.force then
       migrations_utils.reset(schema_state, db, args.lock_timeout)
@@ -208,5 +231,6 @@ return {
     finish = true,
     list = true,
     reset = true,
+    status = true
   }
 }