Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encoding buffer resizing #58

Merged
merged 5 commits into from
Jul 28, 2020
Merged

Encoding buffer resizing #58

merged 5 commits into from
Jul 28, 2020

Conversation

camdencheek
Copy link
Contributor

Description of Changes

Fixes the short destination buffer error introduced by the multiple encoding support by doubling its capacity each time we hit that error.

Additionally, made a bunch of incremental refactors:

  • Pulled out duplicated code into the emit closure
  • Moved the read to end logic out of the defer statement for proper error handling and more linear logic flow
  • Simplified channel operations and goroutine wait logic, fixing a bug with unclean Stop() in the process (which should help stabilize our tests a bit)
  • Pulled the nasty scanner stuff into its own object, which isolates the confusing parts

Please check that the PR fulfills these requirements

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • Add a changelog entry (for non-trivial bug fixes / features)
  • CI passes

@codecov
Copy link

codecov bot commented Jul 27, 2020

Codecov Report

Merging #58 into master will decrease coverage by 0.96%.
The diff coverage is 79.72%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #58      +/-   ##
==========================================
- Coverage   75.94%   74.98%   -0.96%     
==========================================
  Files          61       63       +2     
  Lines        3691     3833     +142     
==========================================
+ Hits         2803     2874      +71     
- Misses        665      716      +51     
- Partials      223      243      +20     
Impacted Files Coverage Δ
operator/buffer/memory_buffer.go 91.53% <ø> (ø)
operator/builtin/output/elastic.go 17.35% <0.00%> (ø)
operator/duration.go 76.47% <ø> (ø)
operator/helper/expr_string.go 87.88% <ø> (ø)
operator/helper/namespace.go 100.00% <ø> (ø)
operator/helper/severity.go 96.97% <ø> (ø)
operator/builtin/output/google_cloud.go 43.58% <27.03%> (ø)
entry/nil_field.go 40.00% <40.00%> (ø)
operator/builtin/input/udp.go 77.36% <44.44%> (ø)
operator/builtin/transformer/restructure.go 73.81% <50.00%> (ø)
... and 55 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1e749e...4e0b353. Read the comment docs.

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Average (%) Memory Average (MB)
1 1000 2.9139338 30.354795
1 5000 6.2760353 33.39359
1 10000 11.362376 44.68494
1 50000 40.432327 99.75296
1 100000 70.62374 189.81236
10 100 5.0001464 27.50633
10 500 8.258801 33.395878
10 1000 12.552064 45.369072
10 5000 45.82837 92.670525
10 10000 81.502075 187.86732

@camdencheek camdencheek merged commit e137723 into master Jul 28, 2020
@camdencheek camdencheek deleted the encoding-buffer-resizing branch July 28, 2020 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants