Add configgrpc and confighttp with hotfix, update golang to 1.22.4 #221
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Uses our own version of the zstd compressor for gRPC servers. The code for it is based on the gzip compressor that comes built-in with gRPC. Benchmarks before this PR: ``` Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkCompressors$ go.opentelemetry.io/collector/config/configgrpc sm_log_requestgoos: linux goarch: amd64 pkg: go.opentelemetry.io/collector/config/configgrpc cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz BenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_162/compressor_gzip-16 71594 19066 ns/op 615 B/op 4 allocs/op sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_159/compressor_zstd-16 151503 8544 ns/op 640 B/op 6 allocs/op sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_178/compressor_snappy-16 3632570 303.8 ns/op 304 B/op 3 allocs/op md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_219/compressor_gzip-16 68114 16938 ns/op 748 B/op 4 allocs/op md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_209/compressor_zstd-16 138091 8047 ns/op 896 B/op 6 allocs/op md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_260/compressor_snappy-16 3081198 402.5 ns/op 400 B/op 3 allocs/op lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_253/compressor_gzip-16 43414 27174 ns/op 386 B/op 3 allocs/op lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_216/compressor_zstd-16 117534 9903 ns/op 10112 B/op 6 allocs/op lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_454/compressor_snappy-16 1000000 1190 ns/op 528 B/op 2 allocs/op sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_203/compressor_gzip-16 67275 17508 ns/op 700 B/op 4 allocs/op sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_201/compressor_zstd-16 196862 6137 ns/op 848 B/op 6 allocs/op sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_220/compressor_snappy-16 3595815 331.7 ns/op 272 B/op 2 allocs/op md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_249/compressor_gzip-16 64105 19104 ns/op 844 B/op 4 allocs/op md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_256/compressor_zstd-16 169221 6929 ns/op 1120 B/op 6 allocs/op md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_279/compressor_snappy-16 2602239 473.0 ns/op 336 B/op 2 allocs/op lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_303/compressor_gzip-16 33861 36473 ns/op 904 B/op 4 allocs/op lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_258/compressor_zstd-16 107828 10596 ns/op 16832 B/op 6 allocs/op lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_591/compressor_snappy-16 725080 1540 ns/op 689 B/op 2 allocs/op sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_140/compressor_gzip-16 76315 16394 ns/op 496 B/op 4 allocs/op sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_137/compressor_zstd-16 193314 5957 ns/op 688 B/op 6 allocs/op sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_152/compressor_snappy-16 3558649 345.2 ns/op 208 B/op 2 allocs/op md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_194/compressor_gzip-16 68497 18413 ns/op 699 B/op 4 allocs/op md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_198/compressor_zstd-16 177841 6520 ns/op 1136 B/op 6 allocs/op md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_222/compressor_snappy-16 2354102 497.4 ns/op 272 B/op 2 allocs/op lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_601/compressor_gzip-16 21943 54603 ns/op 1941 B/op 5 allocs/op lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_559/compressor_zstd-16 71260 16077 ns/op 25312 B/op 6 allocs/op lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_1055/compressor_snappy-16 335415 3026 ns/op 1200 B/op 2 allocs/op PASS ok go.opentelemetry.io/collector/config/configgrpc 37.766s ``` After this version: ``` Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkCompressors$ go.opentelemetry.io/collector/config/configgrpc sm_log_requestgoos: linux goarch: amd64 pkg: go.opentelemetry.io/collector/config/configgrpc cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz BenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_162/compressor_gzip-16 74952 15710 ns/op 603 B/op 4 allocs/op sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_159/compressor_zstd-16 156784 6966 ns/op 208 B/op 2 allocs/op sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_178/compressor_snappy-16 2216174 510.4 ns/op 308 B/op 3 allocs/op md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_219/compressor_gzip-16 68095 18569 ns/op 736 B/op 4 allocs/op md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_209/compressor_zstd-16 150705 8849 ns/op 294 B/op 2 allocs/op md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_260/compressor_snappy-16 2149710 556.8 ns/op 406 B/op 3 allocs/op lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_253/compressor_gzip-16 40040 26159 ns/op 368 B/op 3 allocs/op lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_216/compressor_zstd-16 123043 10254 ns/op 299 B/op 2 allocs/op lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_454/compressor_snappy-16 726780 1457 ns/op 533 B/op 2 allocs/op sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_203/compressor_gzip-16 64660 18186 ns/op 701 B/op 4 allocs/op sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_201/compressor_zstd-16 193225 6267 ns/op 273 B/op 2 allocs/op sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_220/compressor_snappy-16 2925073 418.2 ns/op 276 B/op 2 allocs/op md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_249/compressor_gzip-16 61320 20641 ns/op 846 B/op 4 allocs/op md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_256/compressor_zstd-16 190965 6440 ns/op 321 B/op 2 allocs/op md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_279/compressor_snappy-16 2051575 656.8 ns/op 341 B/op 2 allocs/op lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_303/compressor_gzip-16 30097 40680 ns/op 907 B/op 4 allocs/op lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_258/compressor_zstd-16 127027 8437 ns/op 363 B/op 2 allocs/op lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_591/compressor_snappy-16 716541 1803 ns/op 694 B/op 2 allocs/op sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_140/compressor_gzip-16 82287 15054 ns/op 496 B/op 4 allocs/op sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_137/compressor_zstd-16 230558 5470 ns/op 221 B/op 2 allocs/op sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_152/compressor_snappy-16 2759403 417.1 ns/op 211 B/op 2 allocs/op md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_194/compressor_gzip-16 58208 18925 ns/op 702 B/op 4 allocs/op md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_198/compressor_zstd-16 199226 6247 ns/op 256 B/op 2 allocs/op md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_222/compressor_snappy-16 2065202 609.8 ns/op 276 B/op 2 allocs/op lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_601/compressor_gzip-16 20583 59762 ns/op 1945 B/op 5 allocs/op lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_559/compressor_zstd-16 98254 13152 ns/op 728 B/op 2 allocs/op lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_1055/compressor_snappy-16 389401 3976 ns/op 1209 B/op 2 allocs/op PASS ok go.opentelemetry.io/collector/config/configgrpc 40.394s ``` Signed-off-by: Juraci Paixão Kröhling <[email protected]> --------- Signed-off-by: Juraci Paixão Kröhling <[email protected]>
… body (open-telemetry#10289) This change applies a restriction on the size of the decompressed payload. Before this change, this is the result of the test added in this PR: ``` --- FAIL: TestServerWithDecompression (0.03s) /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/confighttp_test.go:1327: Error Trace: /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/confighttp_test.go:1327 /usr/lib/golang/src/net/http/server.go:2166 /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/compression.go:163 /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/confighttp.go:455 /usr/lib/golang/src/net/http/server.go:2166 /home/jpkroehling/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:212 /home/jpkroehling/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:73 /usr/lib/golang/src/net/http/server.go:2166 /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/clientinfohandler.go:26 /usr/lib/golang/src/net/http/server.go:3137 /usr/lib/golang/src/net/http/server.go:2039 /usr/lib/golang/src/runtime/asm_amd64.s:1695 Error: An error is expected but got nil. Test: TestServerWithDecompression /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/confighttp_test.go:1328: Error Trace: /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/confighttp_test.go:1328 /usr/lib/golang/src/net/http/server.go:2166 /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/compression.go:163 /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/confighttp.go:455 /usr/lib/golang/src/net/http/server.go:2166 /home/jpkroehling/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:212 /home/jpkroehling/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:73 /usr/lib/golang/src/net/http/server.go:2166 /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/clientinfohandler.go:26 /usr/lib/golang/src/net/http/server.go:3137 /usr/lib/golang/src/net/http/server.go:2039 /usr/lib/golang/src/runtime/asm_amd64.s:1695 Error: Test: TestServerWithDecompression /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/confighttp_test.go:1357: Error Trace: /home/jpkroehling/Projects/src/github.com/open-telemetry/opentelemetry-collector/config/confighttp/confighttp_test.go:1357 Error: Not equal: expected: 200 actual : 400 Test: TestServerWithDecompression FAIL FAIL go.opentelemetry.io/collector/config/confighttp 0.036s FAIL ``` Signed-off-by: Juraci Paixão Kröhling <[email protected]> --------- Signed-off-by: Juraci Paixão Kröhling <[email protected]>
jefchien
approved these changes
Jul 9, 2024
| go 1.22.0 | ||
|
|
||
| toolchain go1.22.2 | ||
| go 1.22.4 |
There was a problem hiding this comment.
There are some inconsistencies in the go.mods. Not sure why this one removed the toolchain. The confighttp/configgrpc use 1.21 too.
okankoAMZ
approved these changes
Jul 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Took the 0.98.0 version of configgrpc and confighttp and applied the following cherry picked commits:
24f2df8
890f046
Updated go to 1.22.4
Link to tracking Issue: N/A
Testing:
Added change to agent and ran integ tests: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/9851056277
Documentation: N/A