Add logging grpc request and response content with grpc-proxy mode #14266
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ahrtr
reviewed
Jul 26, 2022
| grpc_zap.PayloadStreamServerInterceptor(lg, alwaysLoggingDeciderServer), | ||
| ) | ||
| grpcChainUnaryList = append(grpcChainUnaryList, | ||
| grpc_ctxtags.UnaryServerInterceptor(), |
There was a problem hiding this comment.
There is no any option being provided to function grpc_ctxtags.UnaryServerInterceptor, so the o.requestFieldsFunc will be always nil. Accordingly it will do nothing.
Did I miss anything?
There was a problem hiding this comment.
ahrtr
reviewed
Jul 28, 2022
server/etcdmain/grpc_proxy.go
Outdated
| @@ -159,6 +163,7 @@ func newGRPCProxyStartCommand() *cobra.Command { | |||
| // experimental flags | |||
| cmd.Flags().BoolVar(&grpcProxyEnableOrdering, "experimental-serializable-ordering", false, "Ensure serializable reads have monotonically increasing store revisions across endpoints.") | |||
| cmd.Flags().StringVar(&grpcProxyLeasing, "experimental-leasing-prefix", "", "leasing metadata prefix for disconnected linearized reads.") | |||
| cmd.Flags().BoolVar(&grpcProxyEnableLogging, "experimental-grpc-logging", false, "logging all grpc request and response") | |||
There was a problem hiding this comment.
--experimental-enable-grpc-debug ?
|
Two comments:
|
Environmentetcd server: 192.168.56.11:23790 in vagrant ubuntu Start scripts
export REGISTRY=quay.io/coreos/etcd
docker run -d \
-p 23790:23790 \
-p 23800:23800 \
--name etcd ${REGISTRY}:latest \
/usr/local/bin/etcd \
--data-dir=/etcd-data --name node1 \
--initial-advertise-peer-urls http://0.0.0.0:23800 --listen-peer-urls http://0.0.0.0:23800 \
--advertise-client-urls http://0.0.0.0:23790 --listen-client-urls http://0.0.0.0:23790 \
--initial-cluster node1=http://0.0.0.0:23800
./etcd grpc-proxy start \
--debug \
--experimental-enable-grpc-logging=true \
--endpoints=192.168.56.11:23790 \
--listen-addr=127.0.0.1:2379Test process./etcdctl put greeting "Hello, world"
# logging content
{"level":"info","ts":"2022-07-29T10:14:41.890+0800","caller":"zap/payload_interceptors.go:131","msg":"server request payload logged as grpc.request.content field","system":"grpc","span.kind":"server","grpc.service":"etcdserverpb.KV","grpc.method":"Put","peer.address":"127.0.0.1:54276","grpc.request.content":{"msg":{"key":"Z3JlZXRpbmc=","value":"SGVsbG8sIHdvcmxk"}}}
{"level":"info","ts":"2022-07-29T10:14:41.891+0800","caller":"zap/payload_interceptors.go:131","msg":"server response payload logged as grpc.response.content field","system":"grpc","span.kind":"server","grpc.service":"etcdserverpb.KV","grpc.method":"Put","peer.address":"127.0.0.1:54276","grpc.response.content":{"msg":{"header":{"clusterId":"12526192129151561082","memberId":"3877455311180318034","revision":"2","raftTerm":"2"}}}}
./etcdctl get greeting
# logging content
{"level":"info","ts":"2022-07-29T10:16:24.250+0800","caller":"zap/payload_interceptors.go:131","msg":"server request payload logged as grpc.request.content field","system":"grpc","span.kind":"server","grpc.service":"etcdserverpb.KV","grpc.method":"Range","peer.address":"127.0.0.1:54297","grpc.request.content":{"msg":{"key":"Z3JlZXRpbmc="}}}
{"level":"info","ts":"2022-07-29T10:16:24.251+0800","caller":"zap/payload_interceptors.go:131","msg":"server response payload logged as grpc.response.content field","system":"grpc","span.kind":"server","grpc.service":"etcdserverpb.KV","grpc.method":"Range","peer.address":"127.0.0.1:54297","grpc.response.content":{"msg":{"header":{"clusterId":"12526192129151561082","memberId":"3877455311180318034","revision":"2","raftTerm":"2"},"kvs":[{"key":"Z3JlZXRpbmc=","createRevision":"2","modRevision":"2","version":"1","value":"SGVsbG8sIHdvcmxk"}],"count":"1"}}}summaryIn Test deployment, with grpc-proxy before etcdserver, |
stefanmonkey
force-pushed
the
feat/grpc-logging
branch
from
4ab7e77 to
d261b93
Compare
ahrtr
reviewed
Jul 29, 2022
CHANGELOG/CHANGELOG-3.6.md
Outdated
| @@ -56,6 +56,7 @@ See [code changes](https://github.com/etcd-io/etcd/compare/v3.5.0...v3.6.0). | |||
| - Add [v3 discovery](https://github.com/etcd-io/etcd/pull/13635) to bootstrap a new etcd cluster. | |||
| - Add [field `storage`](https://github.com/etcd-io/etcd/pull/13772) into the response body of endpoint `/version`. | |||
| - Add [`etcd --max-concurrent-streams`](https://github.com/etcd-io/etcd/pull/14169) flag to configure the max concurrent streams each client can open at a time, and defaults to math.MaxUint32. | |||
| - Add [`etcd grpc-proxy --experimental-enable-grpc-debug`](https://github.com/etcd-io/etcd/pull/14266) flag to logging all grpc request and response | |||
There was a problem hiding this comment.
minor comments, requests and responses
stefanmonkey
force-pushed
the
feat/grpc-logging
branch
from
d261b93 to
7f4fd56
Compare
spzala
requested changes
Aug 1, 2022
There was a problem hiding this comment.
lgtm but please see inline comment related to naming duplication cc @ahrtr . Thanks @stefanmonkey
…ponse content in grpc-proxy mode In our test environment, it may be very useful to debug who delete etcd's key with grpc-proxy mode inspired by https://github.com/grpc-ecosystem/go-grpc-middleware Signed-off-by: stefanbo <[email protected]> Update flag name 1. add changelog 2. update flag name to `experimental-enable-grpc-debug` Signed-off-by: stefan bo <[email protected]> Update CHANGELOG-3.6.md Signed-off-by: stefan bo <[email protected]> change flag name Signed-off-by: stefan bo <[email protected]>
stefanmonkey
force-pushed
the
feat/grpc-logging
branch
from
7f4fd56 to
6a04f7f
Compare
Signed-off-by: stefan bo <[email protected]>
|
All comments resolved, please @spzala take another look, thx |
spzala
approved these changes
Aug 18, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In our test environment, it may be very useful to debug who delete etcd's key with grpc-proxy mode
inspired by https://github.com/grpc-ecosystem/go-grpc-middleware
Signed-off-by: stefanbo [email protected]