-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sql: Add support for specifying window frames for window functions #26666
Conversation
(nit: As this is a user-visible change you'll need to populate a release note in the commit message(s) before this merges.) |
7fa6cf0
to
939b7f5
Compare
Review status: complete! 0 of 0 LGTMs obtained pkg/sql/sem/tree/walk.go, line 217 at r1 (raw file):
this if is problematic. I think you mean this: if windowDef.Frame.Bounds != nil {
windowFrameBoundsCopy := *windowDef.Frame.Bounds
windowDef.Frame.Bounds = &windowFrameBoundsCopy
} Comments from Reviewable |
Reviewed 9 of 10 files at r1. pkg/sql/window.go, line 170 at r1 (raw file):
Do this instead: no need for a separate field. Ditto below. pkg/sql/window.go, line 280 at r1 (raw file):
You're making a copy of WindowDef here. If you are making this copy before the calls to analyzeExpr() in Remember that
I believe you need to analyze the WindowDef just once (and all its sub-expressions) and then memoize the result somehow. pkg/sql/window.go, line 357 at r1 (raw file):
Make a copy of both the frame and bounds object because you are mutating them below. As per my comment above the original syntax tree must remain immutable. pkg/sql/parser/sql.y, line 7177 at r1 (raw file):
This needs both positive and negative tests in pkg/sql/sem/tree/window_funcs.go, line 54 at r1 (raw file):
Don't make a separate field for the typed expression. Reuse Comments from Reviewable |
939b7f5
to
e6141bb
Compare
Review status: complete! 0 of 0 LGTMs obtained pkg/sql/window.go, line 170 at r1 (raw file): Previously, knz (kena) wrote…
Done. pkg/sql/window.go, line 280 at r1 (raw file): Previously, knz (kena) wrote…
Thanks for the detailed explanation - it helped me find a bug (I called pkg/sql/window.go, line 357 at r1 (raw file): Previously, knz (kena) wrote…
Done. pkg/sql/parser/sql.y, line 7177 at r1 (raw file): Previously, knz (kena) wrote…
Done. pkg/sql/sem/tree/walk.go, line 217 at r1 (raw file): Previously, knz (kena) wrote…
Done. pkg/sql/sem/tree/window_funcs.go, line 54 at r1 (raw file): Previously, knz (kena) wrote…
Done. Comments from Reviewable |
Reviewed 6 of 10 files at r1, 1 of 6 files at r2. pkg/sql/window.go, line 165 at r1 (raw file):
Where are you enforcing this? pkg/sql/window.go, line 169 at r1 (raw file):
This is leaking an abstraction. Either make pkg/sql/logictest/testdata/logic_test/window, line 1518 at r1 (raw file):
I'm assuming you tested all of this against PG as well? pkg/sql/logictest/testdata/logic_test/window, line 1543 at r1 (raw file):
Add tests where you're given a non-int start and end bounds. pkg/sql/parser/sql.y, line 7183 at r1 (raw file):
Are these errors straight from PG? pkg/sql/sem/builtins/window_builtins.go, line 181 at r2 (raw file):
Will this abstraction be useful when we look to break the current quadratic behavior by performing some memoization using a segment tree? pkg/sql/sem/tree/window_funcs.go, line 1 at r1 (raw file):
For the sake of reviewability, I'd strongly suggest you go back and make the filename fix in a commit where you touch nothing else. pkg/sql/sem/tree/window_funcs.go, line 52 at r1 (raw file):
I'm also not sure we need this. OffsetExpr should be all we need. pkg/sql/sem/tree/window_funcs.go, line 60 at r1 (raw file):
Does this need to be nullable? Looks like it's always non-nil if pkg/sql/sem/tree/window_funcs.go, line 65 at r1 (raw file):
I think we're misconstruing two concepts here. pkg/sql/sem/tree/window_funcs.go, line 86 at r1 (raw file):
Replace here and below with
pkg/sql/sem/tree/window_funcs.go, line 93 at r1 (raw file):
Go doesn't statically enforce that switch statements are exhaustive. It's generally good practice to do so with a default case like:
I'd add those throughout. Comments from Reviewable |
e6141bb
to
0dda580
Compare
Review status: complete! 0 of 0 LGTMs obtained pkg/sql/window.go, line 165 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
pkg/sql/window.go, line 169 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/logictest/testdata/logic_test/window, line 1518 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Yes, the only differences I found in the output are the number of zeros that PG and CRDB return in number like 200.00 and printing out NULL values (PG leaves them empty), but I think that these are expected behavior, right? pkg/sql/logictest/testdata/logic_test/window, line 1543 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/parser/sql.y, line 7183 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Yes, I manually tried all these combination to get their error messages so that ours would be consistent with them. pkg/sql/sem/builtins/window_builtins.go, line 181 at r2 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
I'm not sure, to be honest. I plan on adding a method that would replace current aggregator with faster implementation (via pkg/sql/sem/tree/window_funcs.go, line 1 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Good idea. pkg/sql/sem/tree/window_funcs.go, line 52 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
As we discussed, pkg/sql/sem/tree/window_funcs.go, line 60 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Correct, pkg/sql/sem/tree/window_funcs.go, line 65 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Ok, I divided it into two structs. Please let me know if further refinements are desirable (and give suggestions if so). pkg/sql/sem/tree/window_funcs.go, line 86 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/sem/tree/window_funcs.go, line 93 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. Comments from Reviewable |
b34d891
to
91476b0
Compare
Generally looks good, and nice tests. once comments get resolved. Looks like you'll need to rebase this against master as there's some merge conflicts. Reviewed 2 of 10 files at r1, 1 of 6 files at r2, 8 of 9 files at r3. pkg/sql/window.go, line 428 at r3 (raw file):
you should use a pgerror here, to match Postgres. In
pkg/sql/sem/builtins/window_builtins.go, line 181 at r2 (raw file): Previously, yuzefovich wrote…
I think it'll be useful to keep around as well for testing purposes. THis is a straightforward implementation that will be useful as a testing baseline, in case it's difficult to implement the more efficient algorithms. pkg/sql/sem/tree/window_funs.go, line 113 at r3 (raw file):
minor: It seems odd that the receiver is not a pointer but it is for Format. I don't know if it matters much, but for consistency I'd make the receiver a pointer as well. pkg/sql/sem/tree/window_funs.go, line 132 at r3 (raw file):
Add a comment above this new field explaining what this is. If it can be nil, say so - something like pkg/sql/sem/tree/window_funs.go, line 153 at r3 (raw file):
minor: I would just panic in these situations instead of returning Comments from Reviewable |
91476b0
to
377f57e
Compare
Review status: complete! 0 of 0 LGTMs obtained (and 1 stale) pkg/sql/window.go, line 428 at r3 (raw file): Previously, jordanlewis (Jordan Lewis) wrote…
Done. pkg/sql/sem/builtins/window_builtins.go, line 181 at r2 (raw file): Previously, jordanlewis (Jordan Lewis) wrote…
I agree. pkg/sql/sem/tree/window_funs.go, line 113 at r3 (raw file): Previously, jordanlewis (Jordan Lewis) wrote…
Done. pkg/sql/sem/tree/window_funs.go, line 132 at r3 (raw file): Previously, jordanlewis (Jordan Lewis) wrote…
Done. Comments from Reviewable |
377f57e
to
25abf99
Compare
25abf99
to
f0a04a1
Compare
This is looking better and better. Nice job so far. Reviewed 10 of 11 files at r4. pkg/sql/window.go, line 357 at r4 (raw file):
Why are we type checking again? We shouldn't need to perform any type checking in this file. pkg/sql/logictest/testdata/logic_test/window, line 1518 at r1 (raw file): Previously, yuzefovich wrote…
Yep, both of those are expected divergences. pkg/sql/sem/builtins/window_builtins.go, line 206 at r4 (raw file):
Don't reallocate this, just reassign the value: pkg/sql/sem/builtins/window_builtins.go, line 234 at r4 (raw file):
Is this modifying the builtin itself? Is that safe? Aren't these static objects? pkg/sql/sem/builtins/window_builtins.go, line 239 at r4 (raw file):
It's ok if pkg/sql/sem/tree/type_check.go, line 765 at r4 (raw file):
I'm still not sure how we're getting away with not replacing this line with
pkg/sql/sem/tree/window_funs.go, line 268 at r4 (raw file):
Add a comment to this method to explain it's meaning. pkg/sql/sem/tree/window_funcs.go, line 1 at r1 (raw file): Previously, yuzefovich wrote…
Thanks, that helps a lot. pkg/sql/sem/tree/window_funcs.go, line 52 at r1 (raw file): Previously, yuzefovich wrote…
But then pkg/sql/sem/tree/window_funcs.go, line 60 at r1 (raw file): Previously, yuzefovich wrote…
Ok, that makes sense. pkg/sql/sem/tree/window_funcs.go, line 65 at r1 (raw file): Previously, yuzefovich wrote…
Thanks! I would suggest that you collocate Comments from Reviewable |
f0a04a1
to
de95f05
Compare
Review status: complete! 0 of 0 LGTMs obtained (and 1 stale) pkg/sql/window.go, line 357 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
You're right - calling pkg/sql/sem/builtins/window_builtins.go, line 206 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/sem/builtins/window_builtins.go, line 234 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
The builtins are not modified, and I believe this to be safe - we simply create a new instance of a builtin (if necessary) to achieve "resetting" behavior. As far as I understand, these objects are not static, i.e. we can create as many as we want without new ones influencing the old ones. pkg/sql/sem/builtins/window_builtins.go, line 239 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Yes, it's ok since I've amended all builtins specific to window functions (i.e. non-aggregates, but functions like pkg/sql/sem/tree/type_check.go, line 765 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Thanks for pointing it out. Indeed, type checking and requiring pkg/sql/sem/tree/window_funs.go, line 268 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/sem/tree/window_funcs.go, line 1 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/sem/tree/window_funcs.go, line 52 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/sem/tree/window_funcs.go, line 65 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
I moved everything into Comments from Reviewable |
b0ad1c7
to
433d30c
Compare
Reviewed 4 of 7 files at r5. pkg/sql/sem/tree/type_check.go, line 765 at r5 (raw file):
I'm not sure when Comments from Reviewable |
433d30c
to
408f3e0
Compare
TFTRs. Review status: complete! 0 of 0 LGTMs obtained (and 2 stale) pkg/sql/sem/tree/type_check.go, line 765 at r5 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. Comments from Reviewable |
bors r+ |
Merge conflict (retrying...) |
WIP on cockroachdb#26464: ROWS mode is fully supported whereas RANGE works only with UNBOUNDED PRECEDING/CURRENT ROW/UNBOUNDED FOLLOWING boundaries (same as in PostgreSQL). Current implementation of aggregate functions is naive (it simply computes the value of aggregate directly and discards all the previous computations which results in quadratic time). Release note (sql change): CockroachDB now supports custom frame specification for window functions using ROWS (fully-supported) and RANGE ('value' PRECEDING and 'value' FOLLOWING are not supported) modes.
408f3e0
to
b007aa2
Compare
bors r+ |
26666: sql: Add support for specifying window frames for window functions r=yuzefovich a=yuzefovich WIP on #26464: ROWS mode is fully supported whereas RANGE works only with UNBOUNDED PRECEDING/CURRENT ROW/UNBOUNDED FOLLOWING boundaries (same as in PostgreSQL). Current implementation of aggregate functions is naive (it simply computes the value of aggregate directly and discards all the previous computations which results in quadratic time). Release note: None Co-authored-by: yuzefovich <[email protected]>
Build succeeded |
There is actually no such notion as "no peers." For some reason, I couldn't match behavior of PG on some logic tests while working on cockroachdb#26666, so I thought PG had "no peers" in RANGE mode. I don't know whether I had a bug at some point and fixed it later or this problem was resolved with making all tests produce deterministic results (see cockroachdb#27635). Release note: None
28155: sql: add create_regfoo builtins r=jordanlewis a=jordanlewis These builtins permit explicitly creating the various regfoo types (regtype, regproc, regclass, etc) with both an OID and a name. This is required to properly disambiguate the formatting of OID types, without forcing a re-execution of an introspection query to determine the name or oid of the regfoo type given only the oid or name. Release note: None 28166: sql: remove noPeers peerGroupChecker r=yuzefovich a=yuzefovich There is actually no such notion as "no peers." For some reason, I couldn't match behavior of PG on some logic tests while working on #26666, so I thought PG had "no peers" in RANGE mode. I don't know whether I had a bug at some point and fixed it later or this problem was resolved with making all tests produce deterministic results (see #27635). Release note: None 28172: storageccl: don't skip row after deleted row r=dt a=dt We don't need to manually advance the iterator before calling `continue` since the for loop advances it as well, and doing so means the row _following_ a deleted row is also skipped. Fixes #28171. Release note (bug fix): Fix bug that could skip the row following a deleted row during BACKUP. Co-authored-by: Jordan Lewis <[email protected]> Co-authored-by: yuzefovich <[email protected]> Co-authored-by: David Taylor <[email protected]>
WIP on #26464:
ROWS mode is fully supported whereas RANGE works only with
UNBOUNDED PRECEDING/CURRENT ROW/UNBOUNDED FOLLOWING boundaries
(same as in PostgreSQL). Current implementation of aggregate functions
is naive (it simply computes the value of aggregate directly and
discards all the previous computations which results in quadratic time).
Release note: None