[Backport 2.11] Add Flint Index Purging Logic #2389
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![opensearch-trigger-bot[bot]](https://avatars.githubusercontent.com/u/80134844?s=60&v=4){kind=small}
* Add Flint Index Purging Logic - Introduce dynamic settings for enabling/disabling purging and controlling index TTL. - Reuse default result index name as a common prefix for all result indices. - Change result index to a non-hidden index for better user experience. - Allow custom result index specification in the data source. - Move default result index name from spark to core package to avoid cross-package references. - Add validation for provided result index name in the data source. - Use pattern prefix + data source name for default result index naming. Testing: - Verified old documents are purged in a cluster setup. - Checked result index naming with and without custom names, ensuring validation is applied. Note: Tests will be added in a subsequent PR. Signed-off-by: Kaituo Li <[email protected]> * address comments Signed-off-by: Kaituo Li <[email protected]> --------- Signed-off-by: Kaituo Li <[email protected]> (cherry picked from commit 1bcacd1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot
bot
requested review from
pjfitzgibbons,
ps48,
kavithacm,
derek-ho,
joshuali925,
dai-chen,
YANG-DB,
rupal-bq,
mengweieric,
vmmusings,
Swiddis,
penghuo,
seankao-az,
MaxKsyunz,
Yury-Fridlyand,
anirudha,
forestmvey,
acarbonetto and
GumpacG
as code owners
ps48
approved these changes
Oct 27, 2023
Codecov Report
@@ Coverage Diff @@
## 2.11 #2389 +/- ##
============================================
- Coverage 96.50% 95.61% -0.90%
Complexity 4955 4955
============================================
Files 466 469 +3
Lines 13689 13835 +146
Branches 916 918 +2
============================================
+ Hits 13211 13228 +17
- Misses 458 587 +129
Partials 20 20
Flags with carried forward coverage won't be shown. Click here to find out more.
|
penghuo
approved these changes
Oct 28, 2023
mengweieric
added a commit
to mengweieric/sql
that referenced
this pull request
Nov 8, 2023
…search-project#2389)" This reverts commit dd48b9b. Signed-off-by: Eric <[email protected]>
mengweieric
added a commit
to mengweieric/sql
that referenced
this pull request
Nov 8, 2023
…search-project#2389)" This reverts commit dd48b9b. Signed-off-by: Eric <[email protected]>
vmmusings
added a commit
to mengweieric/sql
that referenced
this pull request
Nov 13, 2023
…search-project#2389)" This reverts commit dd48b9b.
vmmusings
added a commit
that referenced
this pull request
Nov 13, 2023
* Revert "Add more metrics and handle emr exception message (#2422) (#2426)" This reverts commit b57f7cc. * Revert "Block settings in sql query settings API and add more unit tests (#2407) (#2412)" This reverts commit 3024737. * Revert "Added session, statement, emrjob metrics to sql stats api (#2398) (#2400)" This reverts commit 6e17ae6. * Revert "Redefine Drop Index as logical delete (#2386) (#2397)" This reverts commit e939bb6. * Revert "add concurrent limit on datasource and sessions (#2390) (#2395)" This reverts commit deb3ccf. * Revert "Add Flint Index Purging Logic (#2372) (#2389)" This reverts commit dd48b9b. * Revert "Refactoring for tags usage in test files and also added explicit denly list setting. (#2383) (#2385)" This reverts commit 37e010f. * Revert "Enable session by default (#2373) (#2375)" This reverts commit 7d95e4c. * Revert "Create new session if client provided session is invalid (#2368) (#2371)" This reverts commit 5ab7858. * Revert "Add where clause support in create statement (#2366) (#2370)" This reverts commit b620a56. * Revert "create new session if current session not ready (#2363) (#2365)" This reverts commit 5d07281. * Revert "Handle Describe,Refresh and Show Queries Properly (#2357) (#2362)" This reverts commit 16e2f30. * Revert "Add Session limitation (#2354) (#2359)" This reverts commit 0f334f8. * Revert "Bug Fix, support cancel query in running state (#2351) (#2353)" This reverts commit 9a40591. * Revert "Fix bug, using basic instead of basicauth (#2342) (#2355)" This reverts commit e4827a5. * Revert "Add missing tags and MV support (#2336) (#2346)" This reverts commit 8791bb0. * Revert "[Backport 2.x] deprecated job-metadata-index (#2340) (#2343)" This reverts commit bea432c. * Revert "Integration with REPL Spark job (#2327) (#2338)" This reverts commit 58a5ae5. * Revert "Implement patch API for datasources (#2273) (#2329)" This reverts commit 4c151fe. * Revert "Add sessionId parameters for create async query API (#2312) (#2324)" This reverts commit 3d1a376. * Revert "Add Statement (#2294) (#2318) (#2319)" This reverts commit b3c2e94. * Revert "Upgrade json (#2307) (#2314)" This reverts commit 6c65bb4. * Revert "Minor Refactoring (#2308) (#2317)" This reverts commit 051cc4f. * Revert "add InteractiveSession and SessionManager (#2290) (#2293) (#2315)" This reverts commit 6ac197b. --------- Co-authored-by: Vamsi Manohar <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 1bcacd1 from #2372.