Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/splunk_hec] Apply compression if it's enabled in the config #22969

Merged

Conversation

dmitryax
Copy link
Member

@dmitryax dmitryax commented May 31, 2023

The compression used to be enabled only if the payload size was greater than 1.5KB which significantly complicated the logic and made it hard to test. This change makes the compression unconditionally applied to the payload if it's enabled in the config. The benchmarking shows improvements in the throughput and CPU usage for large payloads and expected degradation for small payloads, which is acceptable given that it's not a common case.

Before:

Benchmark_pushLogData_compressed_10_10_1024
Benchmark_pushLogData_compressed_10_10_1024-10    	   10000	    105616 ns/op	   84893 B/op	    1285 allocs/op
Benchmark_pushLogData_compressed_10_10_8K
Benchmark_pushLogData_compressed_10_10_8K-10      	    4635	    260261 ns/op	  899893 B/op	    1130 allocs/op
Benchmark_pushLogData_compressed_10_10_2M
Benchmark_pushLogData_compressed_10_10_2M-10      	    3620	    334618 ns/op	 2993165 B/op	    1131 allocs/op
Benchmark_pushLogData_compressed_10_200_2M
Benchmark_pushLogData_compressed_10_200_2M-10     	     308	   3884943 ns/op	 4376605 B/op	   22031 allocs/op
Benchmark_pushLogData_compressed_100_200_2M
Benchmark_pushLogData_compressed_100_200_2M-10    	      32	  35486009 ns/op	17481330 B/op	  220032 allocs/op
Benchmark_pushLogData_compressed_100_200_5M
Benchmark_pushLogData_compressed_100_200_5M-10    	      31	  35574226 ns/op	20626891 B/op	  220032 allocs/op

After:

Benchmark_pushLogData_compressed_10_10_1024
Benchmark_pushLogData_compressed_10_10_1024-10    	    1850	    650097 ns/op	 3106085 B/op	    1192 allocs/op
Benchmark_pushLogData_compressed_10_10_8K
Benchmark_pushLogData_compressed_10_10_8K-10      	    7755	    158985 ns/op	   74715 B/op	    1109 allocs/op
Benchmark_pushLogData_compressed_10_10_2M
Benchmark_pushLogData_compressed_10_10_2M-10      	    7790	    152750 ns/op	   74079 B/op	    1109 allocs/op
Benchmark_pushLogData_compressed_10_200_2M
Benchmark_pushLogData_compressed_10_200_2M-10     	     332	   3607309 ns/op	 1459565 B/op	   22010 allocs/op
Benchmark_pushLogData_compressed_100_200_2M
Benchmark_pushLogData_compressed_100_200_2M-10    	      32	  34716503 ns/op	14595388 B/op	  220012 allocs/op
Benchmark_pushLogData_compressed_100_200_5M
Benchmark_pushLogData_compressed_100_200_5M-10    	      31	  34683440 ns/op	14596493 B/op	  220012 allocs/op

The compression used to be enabled only if the payload size was greater than 1.5KB which significantly complicated the logic and made it hard to test. This change makes the compression unconditionally applied to the payload if it's enabled in the config. The benchmarking shows improvements in the throughput and CPU usage for large payloads and expected degradation for small payloads which is acceptable given that it's not a common case.
@dmitryax dmitryax force-pushed the hecexporter-always-compress-if-enabled branch from 83f311a to 78b5a7c Compare May 31, 2023 00:39
@dmitryax dmitryax marked this pull request as ready for review May 31, 2023 00:39
@dmitryax dmitryax requested a review from a team May 31, 2023 00:39
@dmitryax dmitryax merged commit 6c71da2 into open-telemetry:main May 31, 2023
@github-actions github-actions bot added this to the next release milestone May 31, 2023
@dmitryax dmitryax deleted the hecexporter-always-compress-if-enabled branch May 31, 2023 21:12
Caleb-Hurshman pushed a commit to observIQ/opentelemetry-collector-contrib that referenced this pull request Jul 6, 2023
…open-telemetry#22969)

The compression used to be enabled only if the payload size was greater than 1.5KB which significantly complicated the logic and made it hard to test. This change makes the compression unconditionally applied to the payload if it's enabled in the config. The benchmarking shows improvements in the throughput and CPU usage for large payloads and expected degradation for small payloads which is acceptable given that it's not a common case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants