Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-23.1: sql: enable resumption of a flow for pausable portals #99237

Merged

Conversation

ZhouXing19
Copy link
Collaborator

@ZhouXing19 ZhouXing19 commented Mar 22, 2023

Backports 3/3 commits from #99173
/cc @cockroachdb/release


This PR is part of the implementation of multiple active portals. (Extracted from #96358)

We now introduce a Resume() method for flow, and when a pausable portal is being re-executed, rather than generating a new flow, we resume the persisted flow to continue the previous execution.

sql: add telemetry MultipleActivePortalCounter
This commit added a telemetry counter MultipleActivePortalCounter that would
be incremented each time the cluster setting
sql.pgwire.multiple_active_portals.enabled is set to true

sql: add Resume method for flowinfra.Flow and execinfra.Processor
For pausable portals, each execution needs to resume the processor with new
output receiver. We don't need to restart the processors, and this Resume()
step can be called many times after Run() is called.

sql: reuse flow for pausable portal
To execute portals in an interleaving manner, we need to persist the flow and
queryID so that we can continue fetching the result when we come back to the same
portal.

We now introduce pauseInfo field in sql.PreparedPortal that stores this
metadata. It's set during the first-time execution of an engine, and all
following executions will reuse the flow and the queryID. This also implies that
these resources should not be cleaned up with the end of each execution.
Implementation for the clean-up steps is included in the next commit.

Also, in this commit we hang a *PreparedPortal to the planner, and how it is
set can be seen in the next commit as well.

Release note: None

Epic: CRDB-17622

Release justification: part of implementation for high-priority functionality

This commit added a telemetry counter `MultipleActivePortalCounter` that would
be incremented each time the cluster setting
`sql.pgwire.multiple_active_portals.enabled` is set to true

Release note: None
For pausable portals, each execution needs to resume the processor with new
output receiver. We don't need to restart the processors, and this `Resume()`
step can be called many times after `Run()` is called.

Release note: None
To execute portals in an interleaving manner, we need to persist the flow and
queryID so that we can _continue_ fetching the result when we come back to the same
portal.

We now introduce `pauseInfo` field in `sql.PreparedPortal` that stores this
metadata. It's set during the first-time execution of an engine, and all
following executions will reuse the flow and the queryID. This also implies that
these resources should not be cleaned up with the end of each execution.
Implementation for the clean-up steps is included in the next commit.

Also, in this commit we hang a `*PreparedPortal` to the planner, and how it is
set can be seen in the next commit as well.

Release note: None
@ZhouXing19 ZhouXing19 requested a review from rafiss March 22, 2023 15:15
@ZhouXing19 ZhouXing19 requested a review from a team as a code owner March 22, 2023 15:15
@ZhouXing19 ZhouXing19 requested a review from a team March 22, 2023 15:15
@ZhouXing19 ZhouXing19 requested review from a team as code owners March 22, 2023 15:15
@ZhouXing19 ZhouXing19 requested a review from rharding6373 March 22, 2023 15:15
@blathers-crl
Copy link

blathers-crl bot commented Mar 22, 2023

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Patches should only be created for serious issues or test-only changes.
  • Patches should not break backwards-compatibility.
  • Patches should change as little code as possible.
  • Patches should not change on-disk formats or node communication protocols.
  • Patches should not add new functionality.
  • Patches must not add, edit, or otherwise modify cluster versions; or add version gates.
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

  • What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
  • Will this work in a cluster of mixed patch versions? Did we test that?
  • If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

@cockroach-teamcity
Copy link
Member

This change is Reviewable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants