Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leverage inverted indexes on array in CRDB 20.1 to remove joins tables #304

Merged
merged 6 commits into from
May 22, 2020

Conversation

steeling
Copy link
Contributor

This PR greatly simplifies the DSS internal logic and structure by leveraging CRDB's new inverted indexes on arrays.

Instead of a joins table from the payload data to the cells, we can now store the cells directly and each entity, and query for intersection with other cells.

Migrations are intended to be handled by the operator.

@steeling steeling requested review from vosst and charlie-pisuraj May 21, 2020 23:30
@steeling steeling force-pushed the feature/inverted branch from debc0f0 to 4973d77 Compare May 21, 2020 23:55
@steeling
Copy link
Contributor Author

Note that we don't actually get to use the indexes yet due to lack of disjunction on Inverted indexes in CRBD cockroachdb/cockroach#47340. This feature is slated for 20.2 which should arrive in ~6 months.

Additionally with the current queries we use, we have the index on end_time and owner for Subscription Search, and the index on end_time for ISA's, so this wouldn't become an issue until we hit ~100k live ISA's or subscriptions. Until then, I anticipate an overall speed up of queries since we are dealing in single tables, and able to reduce the amount of transactions (and therefore contention) we have.

Required future improvements, if the disjunction does not make it through, can leverage the existing inverted index by "unfurling" the array like so:

SELECT * FROM my_table WHERE array_column && ARRAY['sky', 'bird']
becomes ->

SELECT * FROM my_table WHERE array_column @> ARRAY['sky'] UNION SELECT * FROM my_table WHERE array_column @> ARRAY['bird']

@steeling steeling merged commit 8f2687c into master May 22, 2020
@BenjaminPelletier BenjaminPelletier deleted the feature/inverted branch March 4, 2022 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants