Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: add extra information to protocol errors in bind #106130

Merged
merged 1 commit into from
Aug 8, 2023

Conversation

cucaroach
Copy link
Contributor

A user is running into mysterious protocol errors when using prepared
statements. Add some extra information to the error message to help
guide the investigation.

Informs: https://github.com/cockroachlabs/support/issues/2184
Release note: none
Epic: none

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@cucaroach cucaroach marked this pull request as ready for review July 5, 2023 16:41
@cucaroach cucaroach requested review from a team as code owners July 5, 2023 16:41
@cucaroach cucaroach requested a review from rafiss July 5, 2023 16:42
retErr := func(err error) (fsm.Event, fsm.EventPayload) {
retErr1 := func(err error) (fsm.Event, fsm.EventPayload) {
if bindCmd.PreparedStatementName != "" {
err = errors.Wrapf(err, "bind prepared statement %q failed", bindCmd.PreparedStatementName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: pls consider errors.WithDetail instead of copying the full SQL syntax as a prefix.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍


retErr := func(err error) (fsm.Event, fsm.EventPayload) {
if ps.StatementSummary != "" {
err = errors.Wrapf(err, "bind statement (%q) failed", ps.StatementSummary)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@cucaroach cucaroach requested review from a team and rytaft July 6, 2023 13:38
@cucaroach
Copy link
Contributor Author

@rafiss @knz what I can't figure out is how I get this error message into the logs. Nothing I've tried gets this error message into the crdb logs. I'm running a PG test with the server running and I tried sql.trace.session_eventlog.enabled, server.auth_log.sql_sessions.enabled, server.auth_log.sql_connections.enabled, sql.trace.log_statement_execute and nothing seems to work. Any help/ideas would be most welcome!

Running test like this:

./dev test pkg/sql/pgwire -f TestPGTest/bind_proto_error --test-args "-addr localhost:26257 -user root" -v

Where bind_proto_error just contains the new test I added to errors to hit this error path. Ideally we'd just roll this patch into an upgrade and error message would show up in logs the next time it gets hit.

Copy link
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Detail field of an ErrorResponse is separate from the Message field. See:

if errmsg, ok := recv.(*pgproto3.ErrorResponse); ok {
if typ != typErrorResponse {
return nil, errors.Errorf("waiting for %T, got %#v", typs[0], errmsg)
}
var message, detail, hint string
if keepErrMsg {
message = errmsg.Message
detail = errmsg.Detail
hint = errmsg.Hint
}
// ErrorResponse doesn't encode/decode correctly, so
// manually append it here.
msgs = append(msgs, &pgproto3.ErrorResponse{
Code: errmsg.Code,
Message: message,
ConstraintName: errmsg.ConstraintName,
Detail: detail,
Hint: hint,
})

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @cucaroach, @knz, and @rytaft)


pkg/sql/conn_executor_prepare.go line 349 at r1 (raw file):

Previously, cucaroach (Tommy Reilly) wrote…

👍

nit: the word "failed" is probably redundant in the detail message


pkg/sql/conn_executor_prepare.go line 364 at r1 (raw file):

Previously, cucaroach (Tommy Reilly) wrote…

👍

nit: i would say statement summary instead of statement, so that this message is distinguished from the one above


pkg/sql/conn_executor_prepare.go line 362 at r2 (raw file):

	}

	retErr := func(err error) (fsm.Event, fsm.EventPayload) {

nit: i'd prefer not to have two retErr functions. could we just add this to the first one above:

		if ps != nil && ps.StatementSummary != "" {
			err = errors.WithDetailf(err, "statement (%q) failed", ps.StatementSummary)
		}

(and add a var ps *PreparedStatement declaration above)

@cucaroach cucaroach force-pushed the proto-err-detail branch 3 times, most recently from d8e157b to 6f9a637 Compare July 12, 2023 13:07
A user is running into mysterious protocol errors when using prepared
statements. Add some extra information to the error message to help
guide the investigation.

Informs: https://github.com/cockroachlabs/support/issues/2184
Release note: none
Epic: none
@cucaroach
Copy link
Contributor Author

cucaroach commented Jul 17, 2023

This is RFAL, I added a log statement where the mysterious error is occurring, these errors are logged by default and I wanted to be able to search for these errors in logs. Maybe there's a better way? Ping @rafiss @knz hoping to get this into next 23.1 release to finally get to bottom of lingering customer issue.

@yuzefovich
Copy link
Member

We got the same support ticket again over in https://github.com/cockroachlabs/support/issues/2510 - should we push this over the finish line and backport? @cucaroach @rafiss @knz

@cucaroach
Copy link
Contributor Author

I think so. I think I addressed all the review feedback so just waiting on review approval.

Copy link
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 7 files at r1, 4 of 5 files at r3, 2 of 2 files at r4, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @cucaroach, @knz, and @rytaft)

Copy link
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i think we should try to get this out. thanks for your improvements @cucaroach, and @yuzefovich for following up

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @cucaroach, @knz, and @rytaft)

@rafiss
Copy link
Collaborator

rafiss commented Aug 8, 2023

general tip: if you click the "re-request review" button, then Github reminders would kick in to prod me to review this so you don't have to :)

image

@cucaroach
Copy link
Contributor Author

general tip: if you click the "re-request review" button, then Github reminders would kick in to prod me to review this so you don't have to :)

TIL! Thanks!

@cucaroach
Copy link
Contributor Author

bors r=rafiss

@cucaroach cucaroach added the backport-23.1.x Flags PRs that need to be backported to 23.1 label Aug 8, 2023
@craig
Copy link
Contributor

craig bot commented Aug 8, 2023

Build failed (retrying...):

@craig craig bot merged commit 0d110cd into cockroachdb:master Aug 8, 2023
@craig
Copy link
Contributor

craig bot commented Aug 8, 2023

Build succeeded:

@blathers-crl
Copy link

blathers-crl bot commented Aug 8, 2023

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from ee3a9ab to blathers/backport-release-23.1-106130: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1.x failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-23.1.x Flags PRs that need to be backported to 23.1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants