Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ui: fix errors thrown by 0.toNumber() #108752

Merged
merged 1 commit into from
Aug 15, 2023

Conversation

sjbarag
Copy link
Contributor

@sjbarag sjbarag commented Aug 14, 2023

When generating JS protobuf bindings, protobufjs's pbjs CLI accepts a --strict-long (or the more modern --force-long) flag with the following description:

Enfores the use of 'Long' for s-/u-/int64 and s-/fixed64 fields.

This uses the long package on NPM1, which it expects to be present when one of those clients is used at runtime. If long isn't available when pbjs is executed, it falls back to generating number-based fields where possible. Long is available in all consumers of crdb-protobuf-client, and protobufjs declares long as one of its few build-time and run-time dependencies, so nothing looked alarming.

For each field with a type that might be represented by a Long (a heuristic influenced by --strict-long), that field is marked as "long" if and only if the protobufjs-internal util.Long is non-null2. That util.Long value is set via the use of the @protobufjs/inquire package instead of a standard require3 though, since the former silently returns null instead of throwing an error4. This, again, is not alarming on its own because protobufjs depends directly on long.

The complication comes in with rules_js' use of the pnpm visibility rules, which are very strict when it comes to JS package dependencies. Put simply: packages only have access to the dependencies they directly declare, not transitive dependencies ("phantom dependencies")5. Since @protobufjs/inquire is a separate package with no direct dependencies of its own6, Bazel lays out a node_modules tree where @protobufjs/inquire has no access to the long package. Since inaccessible packages are indistinguishable from nonexistent packages and inquire is intended to be silent, it returns null. The pbjs CLI then assumes Long isn't installed and falls back to number-based fields where a Long isn't strictly required.

When combined with protobuf's zero-value, this caused several cases of '0.toNumber is not a function' JS errors at runtime7. We requested a Long, we support Longs, but a client was silently generated without respecting --strict-long.

Like with most other phantom dependencies, use pnpm.packageOverrides to declare that @protobufjs/inquire does in fact depend on Long.

Fixes: #106318
Release note: Observability pages no longer crash when they encounter zeros (e.g. a session with no memory allocated).

Footnotes

  1. https://www.npmjs.com/package/long

  2. https://github.com/protobufjs/protobuf.js/blob/918ff014efe19f3eb43195ae3d71f7aeb3fcdd73/src/field.js#L153-L157

  3. https://github.com/protobufjs/protobuf.js/blob/918ff014efe19f3eb43195ae3d71f7aeb3fcdd73/src/util/minimal.js#L163-L167

  4. https://github.com/protobufjs/protobuf.js/blob/918ff014efe19f3eb43195ae3d71f7aeb3fcdd73/lib/inquire/index.js#L4-L17

  5. https://docs.aspect.build/rules/aspect_rules_js/docs/pnpm/#hoisting

  6. https://github.com/protobufjs/protobuf.js/blob/918ff014efe19f3eb43195ae3d71f7aeb3fcdd73/lib/inquire/package.json

  7. https://github.com/cockroachdb/cockroach/issues/106318

@sjbarag sjbarag requested review from a team and zachlite August 14, 2023 23:46
@cockroach-teamcity
Copy link
Member

This change is Reviewable

Copy link
Contributor

@zachlite zachlite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Looks like there was test fallout.

When generating JS protobuf bindings, protobufjs's `pbjs` CLI accepts a
--strict-long (or the more modern --force-long) flag with the following
description:

  > Enfores the use of 'Long' for s-/u-/int64 and s-/fixed64 fields.

This uses the `long` package on NPM[^1], which it expects to be present
when one of those clients is used at runtime. If `long` isn't
available when `pbjs` is executed, it falls back to generating
number-based fields where possible. Long is available in all consumers
of `crdb-protobuf-client`, and `protobufjs` declares `long` as one
of its few build-time and run-time dependencies, so nothing looked
alarming.

For each field with a type that might be represented by a Long (a
heuristic influenced by --strict-long), that field is marked as "long"
if and only if the protobufjs-internal util.Long is non-null[^2]. That
util.Long value is set via the use of the `@protobufjs/inquire` package
instead of a standard `require`[^3] though, since the former silently
returns `null` instead of throwing an error[^4]. This, again, is not
alarming on its own because `protobufjs` depends directly on `long`.

The complication comes in with rules_js' use of the pnpm visibility
rules, which are very strict when it comes to JS package dependencies.
Put simply: packages only have access to the dependencies they directly
declare, not transitive dependencies ("phantom dependencies")[^5]. Since
`@protobufjs/inquire` is a separate package with no direct dependencies
of its own[^6], Bazel lays out a node_modules tree where
`@protobufjs/inquire` has no access to the `long` package. Since
inaccessible packages are indistinguishable from nonexistent packages
and `inquire` is intended to be silent, it returns `null`. The `pbjs`
CLI then assumes Long isn't installed and falls back to number-based
fields where a Long isn't strictly required.

When combined with protobuf's zero-value, this caused several cases of
'0.toNumber is not a function' JS errors at runtime[^7]. We requested a
Long, we support Longs, but a client was silently generated without
respecting --strict-long.

Like with most other phantom dependencies, use pnpm.packageOverrides to
declare that @protobufjs/inquire does in fact depend on Long.

[^1]: https://www.npmjs.com/package/long
[^2]: https://github.com/protobufjs/protobuf.js/blob/918ff014efe19f3eb43195ae3d71f7aeb3fcdd73/src/field.js#L153-L157
[^3]: https://github.com/protobufjs/protobuf.js/blob/918ff014efe19f3eb43195ae3d71f7aeb3fcdd73/src/util/minimal.js#L163-L167
[^4]: https://github.com/protobufjs/protobuf.js/blob/918ff014efe19f3eb43195ae3d71f7aeb3fcdd73/lib/inquire/index.js#L4-L17
[^5]: https://docs.aspect.build/rules/aspect_rules_js/docs/pnpm/#hoisting
[^6]: https://github.com/protobufjs/protobuf.js/blob/918ff014efe19f3eb43195ae3d71f7aeb3fcdd73/lib/inquire/package.json
[^7]: cockroachdb#106318

Fixes: cockroachdb#106318
Release note: Observability pages no longer crash when they encounter
zeros (e.g. a session with no memory allocated).
@sjbarag sjbarag force-pushed the fix-pbjs-strict-long-behavior branch from c118c1a to a656a81 Compare August 15, 2023 16:34
@sjbarag sjbarag requested a review from maryliag August 15, 2023 16:37
@sjbarag
Copy link
Contributor Author

sjbarag commented Aug 15, 2023

@maryliag can you or someone on your team gut-check the test change here? Now that Longs are always produced, looks like there was one small test update required.

Copy link
Contributor

@maryliag maryliag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @sjbarag)

@sjbarag
Copy link
Contributor Author

sjbarag commented Aug 15, 2023

Thanks for the reviews y'all!

bors r+

@sjbarag sjbarag added the backport-23.1.x Flags PRs that need to be backported to 23.1 label Aug 15, 2023
@craig
Copy link
Contributor

craig bot commented Aug 15, 2023

Build succeeded:

@craig craig bot merged commit 8ebc6f1 into cockroachdb:master Aug 15, 2023
@sjbarag sjbarag deleted the fix-pbjs-strict-long-behavior branch August 15, 2023 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-23.1.x Flags PRs that need to be backported to 23.1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

key visualizer not loading
4 participants