Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chore] [exporter/splunk_hec] Remove redundant bufState struct #22986

Merged
merged 1 commit into from
Jun 1, 2023

Conversation

dmitryax
Copy link
Member

@dmitryax dmitryax commented Jun 1, 2023

Most of the stuff was moved out from this struct, so it's not needed anymore.

Before:

pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter
Benchmark_pushLogData_10_10_1024
Benchmark_pushLogData_10_10_1024-10               	   10000	    107221 ns/op	   84895 B/op	    1285 allocs/op
Benchmark_pushLogData_10_10_8K
Benchmark_pushLogData_10_10_8K-10                 	   13437	     90507 ns/op	   73997 B/op	    1120 allocs/op
Benchmark_pushLogData_10_10_2M
Benchmark_pushLogData_10_10_2M-10                 	   13738	     88993 ns/op	   73353 B/op	    1109 allocs/op
Benchmark_pushLogData_10_200_2M
Benchmark_pushLogData_10_200_2M-10                	     648	   1894437 ns/op	 1458670 B/op	   22010 allocs/op
Benchmark_pushLogData_100_200_2M
Benchmark_pushLogData_100_200_2M-10               	      64	  17309091 ns/op	14628003 B/op	  220022 allocs/op
Benchmark_pushLogData_100_200_5M
Benchmark_pushLogData_100_200_5M-10               	      61	  17405197 ns/op	14699228 B/op	  220011 allocs/op
Benchmark_pushLogData_compressed_10_10_1024
Benchmark_pushLogData_compressed_10_10_1024-10    	    1788	    693214 ns/op	 3083792 B/op	    1191 allocs/op
Benchmark_pushLogData_compressed_10_10_8K
Benchmark_pushLogData_compressed_10_10_8K-10      	    7432	    158729 ns/op	   74559 B/op	    1109 allocs/op
Benchmark_pushLogData_compressed_10_10_2M
Benchmark_pushLogData_compressed_10_10_2M-10      	    7195	    158115 ns/op	   75396 B/op	    1109 allocs/op
Benchmark_pushLogData_compressed_10_200_2M
Benchmark_pushLogData_compressed_10_200_2M-10     	     326	   3711475 ns/op	 1459604 B/op	   22010 allocs/op
Benchmark_pushLogData_compressed_100_200_2M
Benchmark_pushLogData_compressed_100_200_2M-10    	      31	  36415599 ns/op	14596465 B/op	  220012 allocs/op
Benchmark_pushLogData_compressed_100_200_5M
Benchmark_pushLogData_compressed_100_200_5M-10    	      31	  35341629 ns/op	14596468 B/op	  220012 allocs/op
BenchmarkConsumeLogsRejected
BenchmarkConsumeLogsRejected-10                   	    1262	    944553 ns/op	  730226 B/op	   11012 allocs/op

After:

Benchmark_pushLogData_10_10_1024
Benchmark_pushLogData_10_10_1024-10               	   11834	     99971 ns/op	   84935 B/op	    1285 allocs/op
Benchmark_pushLogData_10_10_8K
Benchmark_pushLogData_10_10_8K-10                 	   13904	     86312 ns/op	   74047 B/op	    1120 allocs/op
Benchmark_pushLogData_10_10_2M
Benchmark_pushLogData_10_10_2M-10                 	   13968	     86110 ns/op	   73364 B/op	    1109 allocs/op
Benchmark_pushLogData_10_200_2M
Benchmark_pushLogData_10_200_2M-10                	     654	   1831256 ns/op	 1460156 B/op	   22011 allocs/op
Benchmark_pushLogData_100_200_2M
Benchmark_pushLogData_100_200_2M-10               	      66	  16905388 ns/op	14659144 B/op	  220024 allocs/op
Benchmark_pushLogData_100_200_5M
Benchmark_pushLogData_100_200_5M-10               	      70	  16717557 ns/op	14742392 B/op	  220013 allocs/op
Benchmark_pushLogData_compressed_10_10_1024
Benchmark_pushLogData_compressed_10_10_1024-10    	    1810	    652715 ns/op	 3098670 B/op	    1193 allocs/op
Benchmark_pushLogData_compressed_10_10_8K
Benchmark_pushLogData_compressed_10_10_8K-10      	    7550	    154149 ns/op	   74909 B/op	    1109 allocs/op
Benchmark_pushLogData_compressed_10_10_2M
Benchmark_pushLogData_compressed_10_10_2M-10      	    7508	    155652 ns/op	   74373 B/op	    1109 allocs/op
Benchmark_pushLogData_compressed_10_200_2M
Benchmark_pushLogData_compressed_10_200_2M-10     	     333	   3595814 ns/op	 1462712 B/op	   22011 allocs/op
Benchmark_pushLogData_compressed_100_200_2M
Benchmark_pushLogData_compressed_100_200_2M-10    	      31	  34835004 ns/op	14597835 B/op	  220014 allocs/op
Benchmark_pushLogData_compressed_100_200_5M
Benchmark_pushLogData_compressed_100_200_5M-10    	      31	  34560606 ns/op	14597822 B/op	  220014 allocs/op
BenchmarkConsumeLogsRejected
BenchmarkConsumeLogsRejected-10                   	    1242	    932774 ns/op	  730061 B/op	   11013 allocs/op

@dmitryax dmitryax requested a review from a team June 1, 2023 05:29
@dmitryax dmitryax requested a review from atoulme as a code owner June 1, 2023 05:29
@dmitryax dmitryax force-pushed the hec-exp-simplify-buffer branch from 0a7d8c3 to f33d85b Compare June 1, 2023 05:30
@dmitryax dmitryax changed the title [chore] [exporter/splunk/hec] Remove redundant bufState struct [chore] [exporter/splunk_hec] Remove redundant bufState struct Jun 1, 2023
maxCapacity: bufCap,
}
}
return &bufferState{
buf: buf,
jsonStream: jsoniter.NewStream(jsoniter.ConfigDefault, nil, initBufferCap),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you changed the jsoniter config with this change - I can't remember if there is a reason for how it was set, just pointing that out. If tests pass, I guess we're ok.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, you're right I changed it to jsoniter.ConfigFastest unintentionally. Let me revert as it's not related

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You still have my approval either way. Since you're reverting that change, maybe worth updating the benchmark numbers real quick as well. I guess you'd want to also run a separate PR with the jsoniter changes.

Copy link
Member Author

@dmitryax dmitryax Jun 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the benchmarks appeared to be not as good without with the same jsoniter config. Maybe because of moving the jsoniter stream pool. Let me look into it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I changed it to fetch the stream from the pool once per batch. Benchmarks are good now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome. Do you want to file an issue for the jsoniter changes? @crobert-1 can make those if he's interested

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The jsoniter.ConfigFastest doesn't escape html and drops precision after 6 decimal places. Do we want to accept that? If so, we can use this issue #22018

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Splunk timestamps have a precision to the millisecond. I'm unsure about any other field that uses numbers specifically. I don't understand if escaping html is needed. I would be a cautious yes, maybe a feature gate.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say we need to check what UF is doing and keep the exporter behavior close to that

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good conclusion. Let's get this one in and we can table the jsoniter changes until we have more tests against the UF.

@dmitryax dmitryax force-pushed the hec-exp-simplify-buffer branch from f33d85b to d3f5fa1 Compare June 1, 2023 15:45
@dmitryax dmitryax force-pushed the hec-exp-simplify-buffer branch from d3f5fa1 to 7eec47e Compare June 1, 2023 16:04
@dmitryax dmitryax merged commit a25c147 into open-telemetry:main Jun 1, 2023
@github-actions github-actions bot added this to the next release milestone Jun 1, 2023
@dmitryax dmitryax deleted the hec-exp-simplify-buffer branch June 1, 2023 20:57
Caleb-Hurshman pushed a commit to observIQ/opentelemetry-collector-contrib that referenced this pull request Jul 6, 2023
…telemetry#22986)

Most of the stuff was moved out from this struct, so it's not needed anymore.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants