Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/string compression #7803

Draft
wants to merge 23 commits into
base: next-major
Choose a base branch
from
Draft

Conversation

jedelbo
Copy link
Contributor

@jedelbo jedelbo commented Jun 12, 2024

What, How & Why?

☑️ ToDos

  • 📝 Changelog update
  • 🚦 Tests (or not relevant)
  • C-API, if public C++ API changed
  • bindgen/spec.yml, if public C++ API changed

Copy link

There is an existing patch(es) for this commit SHA:

Please note that the status that is posted is not in the context of this PR but rather the (latest) existing patch and that may affect some tests that may depend on the particular PR. If your tests do not rely on any PR-specific values (like base or head branch name) then your tests will report the same status. If you would like a patch to run in the context of this PR and abort the other(s), comment 'evergreen retry'.

nicola-cab and others added 2 commits June 17, 2024 12:42
* No unique ptrs for string interner + limit number of interners

* point fix client-reset test

* code review
@jedelbo jedelbo force-pushed the feature/string-compression branch from 8e8356c to 8f1d472 Compare June 17, 2024 13:05
Copy link

coveralls-official bot commented Jul 1, 2024

Pull Request Test Coverage Report for Build jorgen.edelbo_334

Details

  • 1141 of 1267 (90.06%) changed or added relevant lines in 33 files are covered.
  • 172 unchanged lines in 24 files lost coverage.
  • Overall coverage decreased (-0.04%) to 90.829%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/realm/node.hpp 3 4 75.0%
src/realm/table.cpp 105 106 99.06%
src/realm/array_backlink.cpp 0 2 0.0%
src/realm/array_unsigned.cpp 10 12 83.33%
src/realm/cluster.cpp 23 25 92.0%
src/realm/string_compressor.hpp 15 18 83.33%
src/realm/obj.cpp 38 44 86.36%
src/realm/array_string.cpp 68 84 80.95%
src/realm/string_compressor.cpp 234 267 87.64%
src/realm/string_interner.cpp 441 501 88.02%
Files with Coverage Reduction New Missed Lines %
src/realm/array_mixed.cpp 1 92.39%
src/realm/array_string_short.cpp 1 86.27%
src/realm/query_expression.hpp 1 93.82%
src/realm/sync/instructions.hpp 1 76.03%
src/realm/util/encrypted_file_mapping.cpp 1 90.99%
src/realm/uuid.cpp 1 98.48%
test/test_query2.cpp 1 98.73%
src/realm/array_string.cpp 2 87.97%
test/test_sync.cpp 2 93.04%
src/realm/alloc.cpp 3 79.03%
Totals Coverage Status
Change from base Build jorgen.edelbo_333: -0.04%
Covered Lines: 218291
Relevant Lines: 240332

💛 - Coveralls

Copy link

coveralls-official bot commented Jul 3, 2024

Pull Request Test Coverage Report for Build jorgen.edelbo_338

Details

  • 1144 of 1267 (90.29%) changed or added relevant lines in 33 files are covered.
  • 155 unchanged lines in 24 files lost coverage.
  • Overall coverage decreased (-0.008%) to 90.856%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/realm/node.hpp 3 4 75.0%
src/realm/table.cpp 105 106 99.06%
src/realm/array_backlink.cpp 0 2 0.0%
src/realm/array_unsigned.cpp 10 12 83.33%
src/realm/cluster.cpp 23 25 92.0%
src/realm/string_compressor.hpp 15 18 83.33%
src/realm/obj.cpp 38 44 86.36%
src/realm/array_string.cpp 68 84 80.95%
src/realm/string_compressor.cpp 234 267 87.64%
src/realm/string_interner.cpp 444 501 88.62%
Files with Coverage Reduction New Missed Lines %
src/realm/array_mixed.cpp 1 94.44%
src/realm/array_string_short.cpp 1 86.27%
src/realm/sync/noinst/server/server_history.cpp 1 63.7%
src/realm/util/encrypted_file_mapping.cpp 1 90.99%
src/realm/array_blobs_big.cpp 2 98.58%
src/realm/array_string.cpp 2 88.64%
src/realm/sync/transform.cpp 2 61.05%
test/test_index_string.cpp 2 93.41%
src/realm/alloc.cpp 3 79.03%
src/realm/alloc.hpp 3 86.57%
Totals Coverage Status
Change from base Build jorgen.edelbo_337: -0.008%
Covered Lines: 218583
Relevant Lines: 240583

💛 - Coveralls

Copy link

coveralls-official bot commented Jul 12, 2024

Pull Request Test Coverage Report for Build jorgen.edelbo_402

Details

  • 1929 of 1999 (96.5%) changed or added relevant lines in 46 files are covered.
  • 695 unchanged lines in 51 files lost coverage.
  • Overall coverage decreased (-0.03%) to 91.054%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/realm/node.hpp 0 1 0.0%
src/realm/table.cpp 120 121 99.17%
src/realm/array_backlink.cpp 0 2 0.0%
src/realm/array_unsigned.cpp 10 12 83.33%
src/realm/cluster.cpp 20 22 90.91%
src/realm/obj.hpp 0 3 0.0%
src/realm/string_compressor.hpp 15 18 83.33%
src/realm/array_string.cpp 58 62 93.55%
src/realm/sort_descriptor.hpp 6 12 50.0%
src/realm/obj.cpp 48 55 87.27%
Files with Coverage Reduction New Missed Lines %
src/realm/array_mixed.cpp 1 94.58%
src/realm/array_string_short.cpp 1 86.27%
src/realm/dictionary.cpp 1 85.52%
src/realm/util/encrypted_file_mapping.cpp 1 90.99%
src/realm/util/file.cpp 1 84.84%
test/test_index_string.cpp 1 93.4%
test/test_query.cpp 1 96.47%
src/realm/query_engine.cpp 2 92.1%
src/realm/query_expression.cpp 2 86.62%
src/realm/sync/changeset.cpp 2 39.15%
Totals Coverage Status
Change from base Build jorgen.edelbo_389: -0.03%
Covered Lines: 220142
Relevant Lines: 241772

💛 - Coveralls

nicola-cab and others added 9 commits July 16, 2024 10:25
…f is compressed (#7880)

* use CompressedViewString cmp for compressed leaves

* fix asan

* code review

* fix redudant code
…e fast comparison provided by the interner (#7892)

* initial test

* add logic for fetching stringID to Descriptors (partially working)

* fixes core tests

* ops I cannot use C++20 :-)

* fix test

* fix handling for array mixed when it holds strings

* fix improper use of mixed when it is not string

* Compare only if both string ids are available

* works for strings

* enable mixed and fix bug in cmp function

* code cleanup

* remove alias dup

* fix cmp function + simplify comparison function for sorting

* lint

* more readeable cmp function

* test refactory, still missing cmp over links + mixed of diff type cmp

* tests

* code review

* more tests for utf8

* code review

* code review
finnschiermer and others added 2 commits August 9, 2024 13:26
RCORE-2233 Reduce locking for StringInterner lookup and compare methods
When writing the realm to a new file from a write transaction,
the Table may be COW so that the top ref is changed. So don't
use the ref that is present in the group when the operation starts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants