Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

colexec: miscellaneous cleanup #63371

Merged
merged 2 commits into from
Apr 9, 2021
Merged

Conversation

yuzefovich
Copy link
Member

colexec: clean up external distinct test a bit

Release note: None

colbuilder: enforce unique memory monitor names

Our disk-spilling infrastructure (namely, diskSpillerBase) uses the
name of the limited memory monitor of its in-memory operator in order to
distinguish the "memory budget exceeded" errors between the ones it
needs to catch and the ones it needs to propagate further. In
particular, previously it was possible that the in-memory chains within
two disk-backed sorters created for the external distinct (one in the
fallback strategy and another when the output ordering is needed, on top
of the external distinct) would have the same memory monitor names. This
is now fixed by appending a unique suffix to all names and is enforced
via an invariants assertion in the test setup.

Note that even with the previous behavior no actual bug would occur
because the monitors with the same name were on "different levels of the
tree" and there was the correct catcher in-between. So this commit
simply restores the assumption that we had implicitly without fixing any
production bugs.

Release note: None

@yuzefovich yuzefovich requested review from michae2 and a team April 9, 2021 05:31
@cockroach-teamcity
Copy link
Member

This change is Reviewable

Copy link
Collaborator

@mgartner mgartner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 2 of 2 files at r1, 4 of 4 files at r2.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @michae2 and @yuzefovich)


pkg/sql/colexec/colbuilder/execplan.go, line 1537 at r2 (raw file):

	ctx context.Context, flowCtx *execinfra.FlowCtx, name string,
) (*mon.BoundAccount, string) {
	monitorName := r.getMemMonitorName(name + "-limited")

nit: it's a bit confusing that the name (which appears to be the monitor name) is an argument, but a modified monitor name is also returned. Not sure if it'd necessarily be better, but you could change the name string argument here and in the function below to opName string, pid execinfrapb.ProcessorID, and pass both and a suffix string to a getMemMonitorName(opName string, pid execinfrapb.ProcessorID, suffix string). (Not sure opName is the right term, maybe there's something better). This would eliminate some duplicat fmt.Sprintf's I think.

Copy link
Member Author

@yuzefovich yuzefovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @mgartner and @michae2)


pkg/sql/colexec/colbuilder/execplan.go, line 1537 at r2 (raw file):

Previously, mgartner (Marcus Gartner) wrote…

nit: it's a bit confusing that the name (which appears to be the monitor name) is an argument, but a modified monitor name is also returned. Not sure if it'd necessarily be better, but you could change the name string argument here and in the function below to opName string, pid execinfrapb.ProcessorID, and pass both and a suffix string to a getMemMonitorName(opName string, pid execinfrapb.ProcessorID, suffix string). (Not sure opName is the right term, maybe there's something better). This would eliminate some duplicat fmt.Sprintf's I think.

I like this idea, done. Overall, this mechanism of monitor name matching is quite brittle, and the code is not very pretty, but at least it does the job :)

@yuzefovich
Copy link
Member Author

TFTR!

bors r+

Copy link
Collaborator

@michae2 michae2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 2 files at r1, 3 of 4 files at r2, 1 of 1 files at r3.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @mgartner and @yuzefovich)


pkg/sql/colexec/colbuilder/execplan.go, line 1537 at r2 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

I like this idea, done. Overall, this mechanism of monitor name matching is quite brittle, and the code is not very pretty, but at least it does the job :)

Nit: Now that you've changed it, I don't think you need the hyphen on "-limited" any more.

Copy link
Member Author

@yuzefovich yuzefovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bors r-

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @mgartner and @michae2)


pkg/sql/colexec/colbuilder/execplan.go, line 1537 at r2 (raw file):

Previously, michae2 (Michael Erickson) wrote…

Nit: Now that you've changed it, I don't think you need the hyphen on "-limited" any more.

Indeed, nice catch!

@craig
Copy link
Contributor

craig bot commented Apr 9, 2021

Canceled.

Our disk-spilling infrastructure (namely, `diskSpillerBase`) uses the
name of the limited memory monitor of its in-memory operator in order to
distinguish the "memory budget exceeded" errors between the ones it
needs to catch and the ones it needs to propagate further. In
particular, previously it was possible that the in-memory chains within
two disk-backed sorters created for the external distinct (one in the
fallback strategy and another when the output ordering is needed, on top
of the external distinct) would have the same memory monitor names. This
is now fixed by appending a unique suffix to all names and is enforced
via an invariants assertion in the test setup.

Note that even with the previous behavior no actual bug would occur
because the monitors with the same name were on "different levels of the
tree" and there was the correct catcher in-between. So this commit
simply restores the assumption that we had implicitly without fixing any
production bugs.

Release note: None
@yuzefovich
Copy link
Member Author

bors r+

@craig
Copy link
Contributor

craig bot commented Apr 9, 2021

Build succeeded:

@craig craig bot merged commit 71a023f into cockroachdb:master Apr 9, 2021
@yuzefovich yuzefovich deleted the vec-misc-cleanup branch April 9, 2021 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants