Delegate Flint index vacuum operation to Spark

[dai-chen]

Description

Currently, Flint index vacuum operation was handled by the SQL plugin to avoid launching a Spark job. This was introduced in #2557 to:

1. Minimize the latency and cost of submitting to Spark.
2. Offer flexibility to vacuum refreshing/active indexes directly (though this decision was ultimately reversed).

As a result, the vacuum logic in the SQL plugin became a duplicate of that in Spark. With the need for checkpoint cleanup, handling this in the SQL plugin became impractical. This PR removes the redundant logic from the SQL plugin and fully delegates the vacuum operation to Flint Spark.

Testing

Deployed OpenSearch and SQL 3.0 to an EC2 instance and tested with Flint Spark 0.6 in EMR-S as below:

$ curl --request POST \
  --url localhost:9200/_plugins/_async_query \
  --header 'content-type: application/x-ndjson' \
  --data '{"datasource": "mys3","lang": "sql","query": "CREATE MATERIALIZED VIEW mys3.default.vacuum_checkpoint_test_1 AS SELECT clientip FROM mys3.default.http_logs WITH (auto_refresh = true, checkpoint_location = '\''s3://checkpoint/vacuum-checkpoint-test-1'\'')"}'

$ curl localhost:9200/_cat/indices
yellow open flint_mys3_default_vacuum_checkpoint_test_1 iZQ9igSzS8SC5rfRcCnGvA 1 1 12291133 0  644mb  644mb

$ curl --request POST \
  --url localhost:9200/_plugins/_async_query \
  --header 'content-type: application/x-ndjson' \
  --data '{"datasource": "mys3","lang": "sql","query": "DROP MATERIALIZED VIEW mys3.default.vacuum_checkpoint_test_1"}'

$ aws s3 ls s3://checkpoint/ | grep vacuum-checkpoint-test
                           PRE vacuum-checkpoint-test-1/
2024-09-04 18:14:43          0 vacuum-checkpoint-test-1_$folder$

$ curl --request POST \
  --url localhost:9200/_plugins/_async_query \
  --header 'content-type: application/x-ndjson' \
  --data '{"datasource": "mys3","lang": "sql","query": "VACUUM MATERIALIZED VIEW mys3.default.vacuum_checkpoint_test_1"}'

$ aws s3 ls s3://checkpoint/ | grep vacuum-checkpoint-test
(empty)

Related Issues

Resolves opensearch-project/opensearch-spark#580

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Confirmed with @noCharger that this will be added to Flint Spark later.

[noCharger]

LGTM, let's make sure all code path related to VACUUM removed

[seankao-az]

What are some use case for FlintIndexClient and its deleteIndex now that we don't have vacuum handled over here

[dai-chen]

I'm not sure when we added this FlintIndexClient abstraction. It seems useful in future. Let me know if we want to remove it now. cc: @ykmr1224

[ykmr1224]

I think we can remove it now.

[dai-chen]

LGTM, let's make sure all code path related to VACUUM removed

Sure, will double check. Thanks!

[dai-chen]

Only BWC failed. Same as main branch.

[opensearch-trigger-bot]

The backport to 2.17 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.17 2.17
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.17
# Create a new branch
git switch --create backport/backport-2985-to-2.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 83e89fb0f6a659b6cf5877d14ef260438b459c61
# Push it to GitHub
git push --set-upstream origin backport/backport-2985-to-2.17
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.17

Then, create a pull request where the base branch is 2.17 and the compare/head branch is backport/backport-2985-to-2.17.