Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

protoc: support inf, -inf, nan, and -nan in option values #15017

Closed

Conversation

jhump
Copy link
Contributor

@jhump jhump commented Dec 8, 2023

The parser already supports these values for the special "default" field option. But, inconsistently, does not support them for other options whose type is float or double. This addresses that inconsistency.

Fixes #15010.

@jhump jhump requested a review from a team as a code owner December 8, 2023 14:27
@jhump jhump requested review from deannagarcia and removed request for a team December 8, 2023 14:27
@fowles fowles requested review from acozzette and removed request for deannagarcia December 8, 2023 15:05
@hlopko hlopko added protoc 🅰️ safe for tests Mark a commit as safe to run presubmits over labels Dec 12, 2023
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Dec 12, 2023
@acozzette acozzette added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jan 2, 2024
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jan 2, 2024
Copy link
Member

@acozzette acozzette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix and sorry for taking so long to get to this. This looks great to me but would you mind also adding some examples to unittest_custom_options.proto and another short unit test case in parser_unittest.cc that checks that we can read the correct values for those options? I think it would be good to just have test coverage that validates the options after they're interpreted.

src/google/protobuf/compiler/parser.cc Show resolved Hide resolved
Copy link

github-actions bot commented Apr 2, 2024

We triage inactive PRs and issues in order to make it easier to find active work. If this PR should remain active, please add a comment.

This PR is labeled inactive because the last activity was over 90 days ago. This PR will be closed and archived after 14 additional days without activity.

@github-actions github-actions bot added the inactive Denotes the issue/PR has not seen activity in the last 90 days. label Apr 2, 2024
@jhump jhump force-pushed the jh/inf-and-nan-in-option-values branch from f130ac9 to 475cd68 Compare April 2, 2024 15:43
@jhump
Copy link
Contributor Author

jhump commented Apr 2, 2024

I think it would be good to just have test coverage that validates the options after they're interpreted.

@acozzette, sorry for the long hiatus from the branch!

This was a good idea since I totally messed up my original change and forgot to actually update the options interpreting step to allow identifiers "inf" and "nan" for double and float values 🤦.

I've done as requested... but the new test case in parser_unittest.cc is really complicated. We can't easily extract values from a parsed/linked descriptor since interpreting options stores them in the unrecognized field set. So we have to serialize and de-serialize them. I'm using a dynamic message to avoid issues where the FileOptions descriptor from the compiler output != the descriptor returned by the generated type. (Though I think I could also get around that by instead creating a new pool that uses the generated pool as its fallback database, so it wouldn't try to rebuild a descriptor from the FileDescriptorProto for google/protobuf/descriptor.proto.)

Anyhow. I wonder if this new test is really necessary? There is an existing test (ParseDescriptorDebugTestTestCustomOptions) that indirectly exercises much of the same, by effectively compiling unittest_custom_options.proto and making sure the output matches the embedded descriptor of the generated code for that file. Is that sufficient? If not, is there a way to at least make the new test a little more readable? I'm not as familiar with all of the helpers in this test file, but it doesn't look like much else is actually going through the full pool.BuildFile step to link the descriptors and interpret options 🤷.

@acozzette acozzette added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Apr 2, 2024
@github-actions github-actions bot removed 🅰️ safe for tests Mark a commit as safe to run presubmits over inactive Denotes the issue/PR has not seen activity in the last 90 days. labels Apr 2, 2024
@acozzette
Copy link
Member

@jhump How about a test that just looks at the generated descriptors from the protos you added in unittest_custom_options.proto and verifies that the options have the expected values? That should be a lot simpler because then you can just rely on the generated descriptor pool instead of building your own.

@jhump
Copy link
Contributor Author

jhump commented Apr 5, 2024

How about a test that just looks at the generated descriptors from the protos you added in unittest_custom_options.proto and verifies that the options have the expected values?

@acozzette, okay, I've replaced the complicated test with that. Definitely simpler.

@acozzette acozzette added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Apr 5, 2024
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Apr 5, 2024
@jhump
Copy link
Contributor Author

jhump commented Apr 11, 2024

@acozzette, where does this PR go from here? Does it need to be tested internally before it can actually be merged?

@acozzette
Copy link
Member

@jhump Sorry for the delay, I was out for a few days last week and lost track of this. Would you mind merging main into your PR (or rebasing)? I'm hitting an issue syncing it into our internal system which I think should be fixed by that. After that it should be straightforward to test it internally and land the change.

@jhump jhump force-pushed the jh/inf-and-nan-in-option-values branch from 49ec659 to 1f8217c Compare April 16, 2024 12:14
@jhump
Copy link
Contributor Author

jhump commented Apr 16, 2024

Would you mind merging main into your PR (or rebasing)?

@acozzette, done.

@acozzette acozzette added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Apr 16, 2024
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Apr 16, 2024
@acozzette
Copy link
Member

@jhump The only test failure I ran into is this one, which makes sense because it expects to see the current protoc behavior of rejecting values like -inf. I assume the right fix is just to delete the line with expectedDiffWithProtoc: true?

@jhump
Copy link
Contributor Author

jhump commented Apr 22, 2024

@acozzette, yes, that would be the right fix (and what I will do in that repo as soon as this fix is released and that repo updated to use that latest protoc release).

thomasvl added a commit to thomasvl/swift-protobuf that referenced this pull request May 9, 2024
As odd as it sounds, upstream supports this and there is a unittest that
ensure it parses:

https://github.com/protocolbuffers/protobuf/blob/3c03e9351c57081d0dffae120ed37497017f105c/src/google/protobuf/compiler/parser_unittest.cc#L464

It seems to have come from:

protocolbuffers/protobuf#15017

So make sure Swift is also able to parse it.
thomasvl added a commit to thomasvl/swift-protobuf that referenced this pull request May 9, 2024
As odd as it sounds, upstream supports this and there is a unittest that
ensure it parses:

  https://github.com/protocolbuffers/protobuf/blob/3c03e9351c57081d0dffae120ed37497017f105c/src/google/protobuf/compiler/parser_unittest.cc#L464

It seems to have come from:

  protocolbuffers/protobuf#15017

So make sure Swift is also able to parse it.

Also reflow some of the unknown field parsing as inf/nan don't need to be
special cased with how the flow now works.
thomasvl added a commit to apple/swift-protobuf that referenced this pull request May 9, 2024
As odd as it sounds, upstream supports this and there is a unittest that
ensure it parses:

  https://github.com/protocolbuffers/protobuf/blob/3c03e9351c57081d0dffae120ed37497017f105c/src/google/protobuf/compiler/parser_unittest.cc#L464

It seems to have come from:

  protocolbuffers/protobuf#15017

So make sure Swift is also able to parse it.

Also reflow some of the unknown field parsing as inf/nan don't need to be
special cased with how the flow now works.
deannagarcia pushed a commit to deannagarcia/protobuf that referenced this pull request Jun 20, 2024
…ffers#15017)

The parser already supports these values for the special "default" field option. But, inconsistently, does not support them for other options whose type is float or double. This addresses that inconsistency.

Fixes protocolbuffers#15010.

Closes protocolbuffers#15017

COPYBARA_INTEGRATE_REVIEW=protocolbuffers#15017 from jhump:jh/inf-and-nan-in-option-values 1f8217c
PiperOrigin-RevId: 627399259
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

protoc: parsing of float/double numeric values for options is inconsistent
3 participants