Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LUCENE-9662: CheckIndex should be concurrent - parallelizing index check across segments #128

Merged
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
bbd0250
LUCENE-9662: Parallelize checking each index part within each segment
zacharymorn May 6, 2021
1ac1305
Support passing in executorService through checkIndex, and awaits shu…
zacharymorn May 7, 2021
9679cea
Callable to supplier wrapper.
dweiss May 7, 2021
e87d891
Tidy.
dweiss May 7, 2021
2009c70
Use callable and function to reduce the repetitive section of code
zacharymorn May 8, 2021
ed04964
Add segment and part ids
zacharymorn May 18, 2021
b7f3c0f
"Simplify" failFast logic a bit as RuntimeException will be thrown re…
zacharymorn May 18, 2021
22c87c2
Make soft deletes check follows the same style with others
zacharymorn May 18, 2021
99cea46
Remove no longer used failFast
zacharymorn May 18, 2021
c5c1e56
Remove nocommit
zacharymorn May 18, 2021
6ab669c
Address comments
zacharymorn May 19, 2021
a97de22
Add happy path test case
zacharymorn May 25, 2021
9883192
LUCENE-9974: The test-framework module should apply the test ruleset …
dweiss May 25, 2021
1bc1f8a
Use RandomizedTest.randomIntBetween for reproducibility
zacharymorn May 26, 2021
dadfe80
Fix bugs for segment file name and -threadCount, and cap default and …
zacharymorn May 26, 2021
57f542f
Use per part buffer for synchronization
zacharymorn May 26, 2021
71b5279
Remove no longer needed constants (they didnt seems to fail precommit)
zacharymorn May 26, 2021
cad119e
Remove unneeded newline
zacharymorn May 27, 2021
9c88f2f
fix typo
zacharymorn May 27, 2021
8f949dc
Move 11 to constant
zacharymorn May 28, 2021
51bcaa2
Revert changes for concurrency within segment
zacharymorn May 29, 2021
b39a856
Small refactoring to prep for method extraction
zacharymorn May 29, 2021
b892b21
IDE automatic method extraction
zacharymorn May 29, 2021
09ae809
Refactor
zacharymorn May 29, 2021
138b72e
Add concurrency across segments, reducing total time from 300+ second…
zacharymorn May 30, 2021
6cd6a46
Add back spacing between segment output
zacharymorn May 30, 2021
d1b19ea
sync on NumberFormat
zacharymorn May 30, 2021
bae7e7a
Fix bug to work with -segment flag
zacharymorn May 31, 2021
70dc71c
Enhance console output so that result from smaller segments get print…
zacharymorn May 31, 2021
817c050
Start larger segment earlier
zacharymorn Jun 1, 2021
2ef3e8d
Remove uneeded assignment
zacharymorn Aug 26, 2021
034f209
Fix typo
zacharymorn Aug 26, 2021
c7e3d4d
Merge branch 'main' into LUCENE-9662-CheckIndex-Concurrent
zacharymorn Aug 27, 2021
68fe4f5
Fix test to use the latest vector format
zacharymorn Aug 27, 2021
afea6a7
Change threadCount handling per feedback
zacharymorn Aug 29, 2021
6400304
Add change entry
zacharymorn Aug 30, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions gradle/validation/forbidden-apis.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -136,5 +136,15 @@ allprojects { prj ->
'jdk-system-out'
]
}

// the test-framework module is special in that its main sources are exported for all tests.
// include the test signature file (defaults.tests.txt) to the main sourceset.
if (prj.path in [
":lucene:test-framework"
]) {
forbiddenApisMain.signaturesFiles += files(
file("${resources}/defaults.tests.txt")
)
}
})
}
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ public void testCheckOldIndex() throws IOException {
dir.setCheckIndexOnClose(false);

// ... because we check ourselves here:
TestUtil.checkIndex(dir, false, true, null);
TestUtil.checkIndex(dir, false, true, true, null);
dir.close();
}
}
Loading