-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix/luaproto encode hooks #8297
base: master
Are you sure you want to change the base?
Conversation
Pavol Ostertag seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
83c03cc
to
8668a3e
Compare
8668a3e
to
ecfb556
Compare
c1ae42c
to
2db3e76
Compare
21efbd4
to
e29c17d
Compare
2e426d4
to
eb9b26c
Compare
10bc45f
to
a02a226
Compare
e5d8eeb
to
beaa482
Compare
Can I be of any help? I would still gladly contribute to unit-tests (if I know how to do them :) ) |
We want to rework the PR and before the implementation is written we don't know how to unit test them. The best help we could expect from you should be a list of the improvements that we want in detail like the list we made before. To be honest, I don't fully understand the PR's intention, and the list I made is not very in-depth. |
Google provides automated GRPC-REST translation in their own API gateway solution. Kong provides its own, but lacks many of the translation features required by Google. The goal of this PR is to bring google-compliant translation into Kong. For this purpose Google offers several libraries (go, c++, C#, python), but none for lua. The rest is to use protojson library in grpc-gateway plugin. Does this bring more light into this PR? |
fix: deco.lua extra newline at the end deco.lua: remove unnecessary whitespaces deco.lua: fix uneccessary table creation deco.lua: unused parameter of load_json_names_info replaced with _ deco.lua: fix metatable of upstream payload deco.lua: adopt early return style opentelemetry: fix casual style of grpc instantiation protojson.lua: move to tools tools/grpc: rename parse_file() to traverse_proto_file() Update kong/plugins/grpc-gateway/deco.lua Co-authored-by: Xumin <[email protected]> Update spec/01-unit/22-grpc-utils_spec.lua Remove indent Co-authored-by: Xumin <[email protected]> deco.lua: remove extra spaces fix: conflict cleanup fix: add extra line at the end of tools/protojson.lua add: grpc-gateway schema changes to removed_fields.lua
fix: return to good old optimization fix: grow tail test moved upward to emit_defaults=true zone add: default are emitted by default in protojson
45f29aa
to
c06beec
Compare
5628688
to
d7f7fa8
Compare
@git-torrent 3.5 is delayed and we have planned so many things for 3.6. I see little hope that we can fit this improvement into the foreseeable future... In fact, we have many other possible improvements in the queue. Sorry again for this message. |
Oh, that could be. Is there any guide to write modules or extension to lua-protobuf? Or some example on the web (google did not help me much :) ). I am not the C guy, so I have to learn that first. |
By module, I mean "Lua module". There's no such mechanism of lua-protobuf to support plugins. |
And in my very personal POV, there's no need to write such a library in C given LuaJIT emits quite efficient binary and cooperates with C types well. |
Ok, let's write it in Lua. Do you suggest I should agree with author of lua-protobuf to add here https://github.com/starwing/lua-protobuf a Btw, I've already reported a missing support for hooking data-types and it was rejected as too laborious. I would not expect anything more :( (it seems nobody wants to be compatible with google). |
That would be ideal. If they do not like the idea, you could try make it a library which imports lua-protobuf.
Lua is not a mainstream user of the protobuf and gRPC and people seem to tolerate those glitches in their use. After all, with some workarounds, they can still use it. |
Hi, any chance this PR will solve the issue where |
yes, this PR solves this by enabling int64_as_string option https://github.com/starwing/lua-protobuf?tab=readme-ov-file#options |
Great news! Any chance this PR will be merged soon? Without this fix, the gRPC-gateway plugin is not really much usable for anyone who uses 64-bit integers like identifiers... I think I will try to rebase this PR against the latest master and test if it works as expected... |
FYI I rebased that branch onto the current master and it builds fine, unfortunately, the int64 types don't really transcode well - they are now wrapped as strings but still processed as floats.
but almost there! |
Made it working if anyone is interested...
|
Unfortunately, there is another problem in the opposite direction (JSON to proto for instance when transcoding a POST body) - 64-bit integers are transcoded wrong without warning! An example: This is especially dangerous if this is an ID of something because these IDs get silently modified without any warning. Troubleshooting this could be quite a nightmare. Any idea where to look to fix this one? |
Thank you for your input. I am leaving for a short vacation, let me fix it upon arrival. |
Any ETA for the fix? |
Related issues/PRs to the problem I have reported above: |
|
Summary
Transcoding between GRPC and JSON requires some types to be handled exceptionally.
Hooks provided by lua-protobuf are insufficient as they:
bytes
)Any
)Further, gateway must read custom JSON names from proto definitions. Conversion must also allow to keep camel_case names, enum names (instead of numbers) and to emit also JSON with default values.
Full changelog
plugins/grpc-gateway/protojson.lua
handling proto->JSON formattingIn
tools/grpc.lua
:add_path
andparse_file
methods are now exposed instead offor_each_method
in
plugins/grpc-gateway/deco.lua
include_imports
directive wasfalse
.tools/grpc.lua
protojson.lua
in
plugins/grpc-web
tools/grpc.lua
google.protobuf.Any
conversions)Also fixed bug when shorter binding path matched longer one (e.g.
/test
matched also/test/something/else
).Issue reference
Fix #7469
KAG-2262(internal ticket)
Todo: more test coverage