Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tracing: assorted tracing fixes #103225

Merged
merged 6 commits into from
May 24, 2023
Merged

tracing: assorted tracing fixes #103225

merged 6 commits into from
May 24, 2023

Conversation

knz
Copy link
Contributor

@knz knz commented May 12, 2023

Fixes #103052.
Fixes #103227.
Epic: CRDB-27642

Prior to this patch, structured items added to verbose traces were
improperly converted to an unsafe string prior to recording.
This was making the entire structured payload redactable in the
recording representation.

This patch fixes the situation by delegating the representation
to the structured item itself, which may implement a SafeFormat
method.

This patch also ensures that the JSON copy of structured events
is properly marked as redactable.

@knz knz requested review from nvanbenschoten, abarganier and a team May 12, 2023 20:19
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@knz knz mentioned this pull request May 12, 2023
@knz knz changed the title tracing: preserve redactability of structured items in verbose traces tracing: assorted tracing fixes May 12, 2023
@knz knz force-pushed the 20230512-tracing branch from 203014a to aa052b9 Compare May 13, 2023 04:53
@knz knz requested a review from a team as a code owner May 13, 2023 04:53
Copy link
Collaborator

@dhartunian dhartunian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: thx for splitting up the PRs nicely.

Reviewed 1 of 1 files at r1, 3 of 3 files at r2, 3 of 3 files at r3, 3 of 3 files at r4, 6 of 6 files at r5, 4 of 4 files at r6.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @abarganier and @nvanbenschoten)

knz added 3 commits May 24, 2023 14:47
`require.NoError` calls `.Error` which strips redaction markers. For
clarity, this patch changes the calls to pass the strings to `t.Fatal`
directly.

Release note: None
Prior to this patch, the value part of span tags was included
in the redactable representation of traces as a safe string.
This is generally incorrect: the value part can be populated
from arbitrary objects in the code including sensitive values.

To fix this, this commit considers the value part to always be a
sensitive string.

A next iteration of this change would be to define a new protobuf
field of type `RedactableString` (new field to ensure cross-version
compatibility when traces are transferred over the network) and use
that to propagate more fine grained redactability in tag values. The
treatment of log tags in the logging package can be used for
inspiration.

Release note: None
@knz knz force-pushed the 20230512-tracing branch from aa052b9 to b9ab9e8 Compare May 24, 2023 13:07
knz added 3 commits May 24, 2023 17:18
This picks up a fix for a bug that was marking unsafe data incorrectly
as safe in trace recordings.

Release note: None
This intermediate commit removes the extra white space trim that was
performed on structured tracing events, and demonstrates (via unit
test success) that it was unnecessary.

Release note: None
Prior to this patch, structured items added to verbose traces were
improperly converted to an unsafe string prior to recording.
This was making the entire structured payload redactable in the
recording representation.

This patch fixes the situation by delegating the representation
to the structured item itself, which may implement a SafeFormat
method.

Release note: None
@knz knz force-pushed the 20230512-tracing branch from b9ab9e8 to 89a2012 Compare May 24, 2023 15:19
@knz knz requested a review from a team as a code owner May 24, 2023 15:19
@knz knz requested a review from RaduBerinde May 24, 2023 15:19
Copy link
Member

@RaduBerinde RaduBerinde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (and 1 stale) (waiting on @abarganier, @dhartunian, @knz, and @nvanbenschoten)


-- commits line 24 at r10:
Does this mean we'll no longer see these values in eg SHOW TRACES? How about in query bundles or a collector like Jaeger?

@knz
Copy link
Contributor Author

knz commented May 24, 2023

Does this mean we'll no longer see these values in eg SHOW TRACES? How about in query bundles or a collector like Jaeger?

It depends. If trace redaction is enabled, they would be removed yes. But it's not enabled by default.

@knz
Copy link
Contributor Author

knz commented May 24, 2023

FYI:

  • when trace redaction is enabled (cluster setting), the redaction of trace data happens at the KV/SQL boundary
  • it's enabled in Serverless. The reasoning is that unstructured data in trace can contain sensitive data about other tenants than the one the trace is flowing to, and should be removed.
  • trace data emitted within the SQL layer is thus not removed, regardless of whether redaction is enabled.

@RaduBerinde
Copy link
Member

Very cool, makes sense!

@knz
Copy link
Contributor Author

knz commented May 24, 2023

TFYR

bors r=dhartunian,RaduBerinde

@craig
Copy link
Contributor

craig bot commented May 24, 2023

This PR was included in a batch that timed out, it will be automatically retried

@craig
Copy link
Contributor

craig bot commented May 24, 2023

Build succeeded:

@craig craig bot merged commit bd6f8f5 into cockroachdb:master May 24, 2023
@knz knz deleted the 20230512-tracing branch May 25, 2023 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants