[Upgrade Assistant] Server-side batch reindexing

[jloleysens]

Summary

Add the ability for Upgrade Assistant (UA) to handle reindexing in batches. This contribution focusses on the server-side code.

Two new endpoints are exposed to users:

POST /api/upgrade_assistant/reindex/batch

This new endpoint accepts an array of index names to be reindexed.

Body args

{
    "indexNames": []
}

How to test

1. Start Kibana with yarn start --verbose & ES with default configuration
2. Run the cURL commands below
3. Should see something like:

{
  // Top level keys are stable
  "enqueued": [
    {
      // The values in here are subject to change and should not be considered stable!
      "indexName": "test7",
      "newIndexName": "reindexed-v8-test7",
      "status": 3,
      "lastCompletedStep": 0,
      "locked": null,
      "reindexTaskId": null,
      "reindexTaskPercComplete": null,
      "errorMessage": null,
      "runningReindexCount": null,
      "reindexOptions": {
        "queueSettings": {
          "queuedAt": 1583406985489
        }
      }
    },
    {
      "indexName": "test8",
      "newIndexName": "reindexed-v8-test8",
      "status": 3,
      "lastCompletedStep": 0,
      "locked": null,
      "reindexTaskId": null,
      "reindexTaskPercComplete": null,
      "errorMessage": null,
      "runningReindexCount": null,
      "reindexOptions": {
        "queueSettings": {
          "queuedAt": 1583406987334
        }
      }
    }
  ],
  "errors": []
}

4. Keep an eye on the server logs for output like this:

server    log   [11:34:14.933] [debug][plugins][reindex_worker][upgradeAssistant] Queue detected; current length 2, current item ReindexOperation(id: 9ab22360-5e03-11ea-9d2d-1ff9c3164643, indexName: test12)

This is an indication that each job is being processed in a queue when a batch submission is made.

5. (Optional) Make sure that multiple batch requests can be submitted at the same time (submit a batch request to reindex test3 and test4 followed by a batch request to reindex test5 and test6 (after you have created each).

6. (Optional) Submit a request to reindex ["test7", "test7", "test8"]. The batch endpoint should report that it could not create a new item for the second "test7".

cURL commands

# Change test1 and test2 to the name of any non-existing index
curl -XPUT http://elastic:changeme@localhost:9200/test1
curl -XPUT http://elastic:changeme@localhost:9200/test2

# swap out "pku" with your local path prefix
curl --request POST \
  --url http://localhost:5603/pku/api/upgrade_assistant/reindex/batch \
  --header 'authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
  --header 'content-type: application/json' \
  --header 'kbn-xsrf: xxxx' \
  --data '{
        "indexNames": [
                "test1",
                "test2"
        ]
}'

GET /api/upgrade_assistant/reindex/batch/queue

This endpoint provides visibility into the current batch queue.

How to test

With Kibana started (does not require --verbose flag), hit the previous endpoint followed by this endpoint to get a current view of the reindex batch queue. Order in which operations will occur is indicated by the order in the array.

{
  "queue": [
    {
      "indexName": "test8",
      "newIndexName": "reindexed-v8-test8",
      "status": 3,
      "lastCompletedStep": 0,
      "locked": null,
      "reindexTaskId": null,
      "reindexTaskPercComplete": null,
      "errorMessage": null,
      "runningReindexCount": null,
      "reindexOptions": {
        "queueSettings": {
          "queuedAt": 1583406987334
        }
      }
    }
]
}

Release Note

Upgrade Assistant has a new batch endpoint that enables submitting multiple indices for reindexing in one network request. Indices are processed one-by-one to minimize cluster resource usage. This also makes it much easier for users to upgrade indices when a new version of the Elastic stack is released.

Checklist

• Manual testing
• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[jloleysens]

@elasticmachine merge upstream

[jloleysens]

@elasticmachine merge upstream

[alisonelizabeth]

Nice job @jloleysens!

Batch reindexing works great. I may have found a small regression with reindexing a single index. If I create an index, POST /api/upgrade_assistant/reindex/<my_index>, then execute that request again, I get an Internal Error error message. However, on master, I get A reindex operation already in-progress for <my_index>, which is what I would expect. Can you take a look?

[jloleysens]

@alisonelizabeth I noticed the error handling was a bit problematic with the reindex service return Boom errors. There was an error here with differences between the reindex endpoints.

I created this PR #58715 to fix those. Do you think we can merge that one first, then I will resolve the conflicts and we can return to this one?

[alisonelizabeth]

@jloleysens 👍sounds good. I'll review that one next.

[jloleysens]

@alisonelizabeth pointed out that we probably don't want to, by default, kick off reindex jobs in parallel for batches as these could be really resource intensive. This will require a queue mechanism and probably some other revisions to the current contribution.

[alisonelizabeth]

Nice job @jloleysens! Code works as described. The regression I found earlier with the error handling has also been resolved 🎉

Would you mind updating the "How to test" section with the updated response (step 3)? Also, I happened to notice there are existing docs for the UA APIs; we should work with Gail to update these whenever we’re ready (https://www.elastic.co/guide/en/kibana/current/upgrade-assistant-api.html).

Other thoughts - maybe this will become more clear once we start working on the UI (and I get more familiar with UA 😄) but, would it be useful at all to know where in the queue the index is and how many total are in the queue in the response? Also, did you test this with a large number of indices (I only did a handful)? Should we impose any sort of limit?

[joshdover]

This took me some time to figure out what this was supposed to do. I think this could benefit from a comment that explains the algorithm that decides which reindex operations should be processed.

Alternatively, this could benefit from extracting the logic into a named function that explains what it does:

const inProgressOps = await this.reindexService.findAllByStatus(ReindexStatus.inProgress);
const firstOpInQueue = getFirstInQueue(inProgressOps);
const unqueuedOps = inProgressOps
  .filter(op => !!op.attributes.reindexOptions?.queueSettings);
this.inProgressOps = [...unqueuedOps, ...([firstOpInQueue] ? firstOpInQueue : [])];

[jloleysens]

Thanks for taking a look @joshdover ! I'm happy the implementation (on the whole) makes sense.

I'll clarify the context in which this code runs with a comment because it's not clear from just looking at it here that it runs on a schedule and what exactly "parallel" means.

[joshdover]

Overall this makes sense to me! This will definitely make upgrading less tedious 😄

[jloleysens]

@alisonelizabeth thanks for taking another look!

Would you mind updating the "How to test" section with the updated response (step 3)

Will do!

would it be useful at all to know where in the queue the index is and how many total are in the queue in the response?

I was not entirely clear on which response you were referring to here. The response from the POST BASE_PATH/reindex/batch endpoint does indicate the enqueued items in the order they are enqueued. But if we want to, on an ongoing basis, know what the queue looks like I think it does make sense to create a GET BASE_PATH/reindex/batch/queue endpoint that returns an array indicating the current operations in the queue. WDYT?

I've also realised we need to cater better for restarting a failed/cancelled operation in a batch (the queuedAt time needs to updated again) and we probably want a batch/cancel. The latter can wait until we get to the UI I think!

[jloleysens]

@elasticmachine merge upstream

[alisonelizabeth]

I was not entirely clear on which response you were referring to here. The response from the POST BASE_PATH/reindex/batch endpoint does indicate the enqueued items in the order they are enqueued. But if we want to, on an ongoing basis, know what the queue looks like I think it does make sense to create a GET BASE_PATH/reindex/batch/queue endpoint that returns an array indicating the current operations in the queue. WDYT?

Sorry I wasn't vert clear. Yeah, that's similar to what I had in mind. I think this makes sense. Thanks!

I've also realised we need to cater better for restarting a failed/cancelled operation in a batch (the queuedAt time needs to updated again) and we probably want a batch/cancel. The latter can wait until we get to the UI I think!

👍

[alisonelizabeth]

Latest LGTM. Nice work @jloleysens!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 63f6659

History

• 💚 Build #31079 succeeded 63f6659
• 💔 Build #31045 failed 5cd2270
• 💚 Build #30699 succeeded 02ae1b2
• 💚 Build #30014 succeeded 6407b6a
• 💚 Build #30004 succeeded 17f39e7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.