-
Notifications
You must be signed in to change notification settings - Fork 429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added bloom filter support. Improve concurrency and fix some issues. #93
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Conflicts: # leveldb/src/main/java/org/iq80/leveldb/impl/TableCache.java
# Conflicts: # leveldb/pom.xml
… and reduce code duplication. Same more work can be done to enable one to externally change file access implementation.
…g DB#compactRange(null,null)
# Conflicts: # .travis.yml # leveldb-benchmark/src/main/java/org/iq80/leveldb/benchmark/DbBenchmark.java # leveldb/src/main/java/org/iq80/leveldb/impl/DbImpl.java # leveldb/src/main/java/org/iq80/leveldb/impl/FileChannelLogWriter.java # leveldb/src/main/java/org/iq80/leveldb/impl/Level.java # leveldb/src/main/java/org/iq80/leveldb/impl/Level0.java # leveldb/src/main/java/org/iq80/leveldb/impl/MMapLogWriter.java # leveldb/src/main/java/org/iq80/leveldb/impl/TableCache.java # leveldb/src/main/java/org/iq80/leveldb/impl/Version.java # leveldb/src/main/java/org/iq80/leveldb/impl/WriteBatchImpl.java # leveldb/src/main/java/org/iq80/leveldb/table/FileChannelTable.java # leveldb/src/main/java/org/iq80/leveldb/table/MMapTable.java # leveldb/src/main/java/org/iq80/leveldb/table/Table.java # leveldb/src/main/java/org/iq80/leveldb/table/TableBuilder.java # leveldb/src/test/java/org/iq80/leveldb/table/TableTest.java
SnapshotSeekingIterator could return deleted entries (it should't) depending on the order they where disposed in source iterator. File selection in compaction process was not collecting all overlapping files in level 0. This could cause the db to forget that one entry was deleted after a compaction (tombstone removed, but not the earlier entry).
…el if the files are overlapping
…ompaction as complete without executing it. A ManualCompaction could be notified as complete, but not executed, if it was set during the execution of an other compaction execution work.
Track snapshots number (and not full version). This change enable to release "version" instance and compaction operation to only delete entries older that older snapshot. This change fix issue dain#48.
Similar to original implementation and reduce complexity
…k access Reduce lock access while doing background compaction.
Add git attributes
Added stats to compaction
If you have interest in this merge contact me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I did port some missing feature and fix some issues, I hope to be able to contribute back to main repo.
I know it's bigger pull request than anticipated(for that I am sorry), but hope to be useful to everybody.
Most this work was done following original implementation without to much refactoring to simplify comparison.
Work in PR
Removed some JVM global lock
Removed synchronized block done on class object. A synchronized block like
synchronized (MMapTable.class) {...}
will use a mutex that is shared acrossthe entire JVM, and any other database instance would block on same mutex instance.
More concurrent friendly read/write
Previous implementation was based on full locking during read/write.
Now, only one writer can write to database at a time, but the first to enter the
write lock may help some other writers to finish at the same time it write its own data.
Read will release the lock when accessing immutable table and files to enable other
reads to enter.
Added bloom filter support
Added bloom filter policy implementation as the one implemented in original implementation.
Added LRU block cache to Tables
Added an LRU block cache to cache uncompressed read blocks. Existing, but
unused, Options.cacheSize can now be used to configure the cache size.
Improved abstraction over file access
Tables and log files read/write classes had no abstraction over file access, this lead to
a confusing code duplication. Interface to abstract file access where added and
duplicated classes where merged. Besides a cleaned design, it enabled easier
testing.
As memory mapped files in JVM can have there issues, Options.allowMmapWrites and
Options.allowMmapWrites where added to configure if memory mapped files
are to be used for read or/and write.
Compaction
Implement
DbImpl#compactRange(byte[] begin, byte[] end)
requested in by issue #30, with supportfor full compaction by using
compactRange(null, null)
.Fix issue with
VersionSet#getOverlappingInputs(int level, InternalKey begin, InternalKey end)
that could return wrong file set when used with level = 0 (ranges may overlap in level 0).
Fix issue #85 correcting Level$findFile(InternalKey) by enabling to return index
greater that last possible index.
Fix SnapshotSeekingIterator
Fix SnapshotIterator that could, in some case, return deleted entries.
A simple test to show this incorrect behavior would be:
Failing with:
Other notes
DbimplTest was modified to enable to run all test using distinct configuration set
and more tests where ported and created.
DbBenchmark was modified to enable concurrent benchmark and set more configurations. Added histogram support.