release-23.2: colexec: fix incorrect eager cancellation in parallel unordered sync

[yuzefovich]

Backport 1/1 commits from #127076.

/cc @cockroachdb/release

---

As of 2a928d0 we have eager cancellation of work in the parallel unordered sync for local plans. This is an optimization to cancel outstanding work in some inputs whenever the sync is being drained as a whole (and is needed to power the locality-optimized search feature). The cancellation is achieved by cancelling the contexts of all inputs and then ignoring all errors from those inputs.

However, if we have multiple parallel unordered syncs in the plan, then eager cancellation in the ancestor can "poison" the execution of the descendant because the latter cannot distinguish between benign errors (that happened because of the eager cancellation) and true query errors. (The sync that performs the eager cancellation distinguishes between the two by checking whether its "root" context is canceled or not.) Other operators don't have this issue since they all run within a single goroutine, with the exception of the lookup and index joins that use the streamer, but there the main "read" goroutine is the same one as for other operators that use a single goroutine, so the parallel sync can correctly filter out benign errors.

This commit fixes an issue of spurious context cancellation errors during valid query execution by allowing eager cancellation on drain only in "leaf" parallel syncs. This should be sufficient for the locality-optimized search feature where we have multiple table readers at the same level combined by the sync (and table readers themselves are always leaf operators).

Fixes: #127043.

Release note (bug fix): CockroachDB previously could encounter spurious (error encountered after some results were delivered) ERROR: context canceled errors when evaluating some queries in rare cases. Most likely conditions are:

• the query has to be executed locally
• have a LIMIT
• have at least two UNION clauses
• have some lookup / index joins in the UNION branches.

The bug is present since 22.2 and is now fixed.

Release justification: bug fix.

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Backports should only be created for serious
  issues or test-only changes.
• Backports should not break backwards-compatibility.
• Backports should change as little code as possible.
• Backports should not change on-disk formats or node communication protocols.
• Backports should not add new functionality (except as defined
  here).
• Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
• All backports must be reviewed by the owning areas TL and one additional
  TL. For more information as to how that review should be conducted, please consult the backport
  policy.

If your backport adds new functionality, please ensure that the following additional criteria are satisfied:

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
• Your backport must be accompanied by a post to the appropriate Slack
  channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

[cockroach-teamcity]

This change is 

[michae2]

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @mgartner)

[yuzefovich]

@mgartner as we agreed, not merging without your approval - please merge this when you take a look

[rafiss]

if this gets merged, please also include this test change to make the test less flaky: #131680

[yuzefovich]

will be backporting together with the fix for #127942