Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved coverage #110

Merged
merged 14 commits into from
Sep 9, 2020
Merged

Improved coverage #110

merged 14 commits into from
Sep 9, 2020

Conversation

jmwilliams89
Copy link
Contributor

Description of Changes

Please check that the PR fulfills these requirements

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • Add a changelog entry (for non-trivial bug fixes / features)
  • CI passes

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 3.6207368 -0.29312015 34.21794 -0.85762405
1 5000 8.379478 -1.034524 44.731144 -0.29956818
1 10000 13.638176 -2.3448896 57.2368 +1.2249489
1 50000 62.326756 -9.762539 186.7764 -30.522903
1 100000 119.23953 -24.31697 282.16016 -24.172424
10 100 5.034678 -0.55170155 32.11557 -1.6441269
10 500 10.827811 -1.086196 44.28556 +1.6973343
10 1000 17.327984 -1.7758656 58.172684 +2.7943153
10 5000 69.45129 -7.4224243 191.09483 +13.0763855
10 10000 132.45947 -23.653458 331.5182 -0.71066284

@codecov
Copy link

codecov bot commented Sep 8, 2020

Codecov Report

Merging #110 into submod-diskbuff will increase coverage by 4.92%.
The diff coverage is 97.72%.

Impacted file tree graph

@@                 Coverage Diff                 @@
##           submod-diskbuff     #110      +/-   ##
===================================================
+ Coverage            65.36%   70.28%   +4.92%     
===================================================
  Files                   77       79       +2     
  Lines                 5006     5013       +7     
===================================================
+ Hits                  3272     3523     +251     
+ Misses                1390     1174     -216     
+ Partials               344      316      -28     
Impacted Files Coverage Δ
operator/buffer/memory.go 81.13% <ø> (ø)
...erator/builtin/transformer/ratelimit/rate_limit.go 69.57% <ø> (ø)
operator/config.go 100.00% <ø> (+7.55%) ⬆️
operator/helper/duration.go 64.86% <ø> (ø)
cmd/stanza/service.go 67.65% <28.57%> (-10.92%) ⬇️
agent/agent.go 100.00% <100.00%> (+42.11%) ⬆️
agent/builder.go 100.00% <100.00%> (ø)
cmd/stanza/graph.go 55.26% <100.00%> (-1.15%) ⬇️
cmd/stanza/offsets.go 64.71% <100.00%> (ø)
database/database.go 100.00% <100.00%> (ø)
... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f8965c2...6b6e34b. Read the comment docs.

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 2.741477 -1.17238 36.019936 +0.94437027
1 5000 8.465718 -0.94828415 44.38847 -0.64224243
1 10000 15.172726 -0.8103399 59.02775 +3.0158958
1 50000 60.91491 -11.174385 173.29378 -44.005524
1 100000 121.27192 -22.284584 278.0396 -28.292969
10 100 5.258753 -0.3276267 31.911772 -1.8479252
10 500 9.414136 -2.4998713 42.999866 +0.41164017
10 1000 17.62111 -1.4827385 57.840115 +2.4617462
10 5000 68.79513 -8.078583 191.1809 +13.162445
10 10000 128.78233 -27.330597 302.46634 -29.762512

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 3.7758875 -0.1379695 33.00081 -2.0747566
1 5000 8.931363 -0.4826393 45.60237 +0.5716591
1 10000 13.707166 -2.2759 57.402344 +1.3904915
1 50000 60.829323 -11.259972 179.87553 -37.423767
1 100000 115.32958 -28.226921 306.33514 +0.0025634766
10 100 5.4655976 -0.1207819 31.531115 -2.2285824
10 500 9.3105545 -2.6034527 42.853313 +0.26508713
10 1000 16.655888 -2.4479618 55.975754 +0.5973854
10 5000 68.77778 -8.095932 183.01292 +4.9944763
10 10000 132.30779 -23.805145 336.90866 +4.6798096

@jmwilliams89 jmwilliams89 changed the base branch from master to submod-diskbuff September 8, 2020 14:25
@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.7759027 -2.1379542 125.63941 +90.56384
1 5000 6.3104486 -3.1035538 135.3878 +90.357086
1 10000 12.776152 -3.206914 143.33702 +87.325165
1 50000 64.3306 -7.7586975 209.64682 -7.652481
1 100000 121.18548 -22.371025 379.70163 +73.36905
10 100 2.4828057 -3.1035738 130.14804 +96.38834
10 500 7.4484525 -4.4655547 135.2985 +92.710266
10 1000 14.448613 -4.6552362 146.01913 +90.64076
10 5000 68.1876 -8.686111 211.46754 +33.449097
10 10000 127.57127 -28.541656 359.91647 +27.687622

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.9310716 -1.9827853 126.115036 +91.039474
1 5000 6.776013 -2.6379895 135.92941 +90.8987
1 10000 12.672634 -3.3104315 143.58702 +87.575165
1 50000 65.61699 -6.4723053 215.74919 -1.5501099
1 100000 123.54545 -20.011055 400.8044 +94.47183
10 100 2.4483302 -3.1380494 131.14816 +97.388466
10 500 7.379429 -4.5345783 135.8455 +93.25728
10 1000 14.05202 -5.0518293 142.37851 +87.00014
10 5000 68.553604 -8.3201065 219.61166 +41.593216
10 10000 131.47931 -24.633621 380.0023 +47.773438

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.7586442 -2.1552129 125.21484 +90.13928
1 5000 6.844978 -2.5690246 136.8556 +91.82489
1 10000 12.655398 -3.3276672 143.03233 +87.02048
1 50000 64.31493 -7.7743607 222.93494 +5.6356354
1 100000 128.91928 -14.637222 391.97278 +85.6402
10 100 2.3103774 -3.2760022 129.23532 +95.475624
10 500 8.379684 -3.5343227 136.63066 +94.042435
10 1000 14.121025 -4.9828243 147.41917 +92.0408
10 5000 70.50238 -6.3713303 215.90045 +37.882004
10 10000 129.53075 -26.582184 355.7182 +23.48935

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 2.0000691 -1.9137878 126.01536 +90.93979
1 5000 6.620839 -2.7931633 133.77654 +88.74582
1 10000 12.603658 -3.379408 143.44571 +87.43385
1 50000 65.57844 -6.5108566 222.70433 +5.4050293
1 100000 117.03743 -26.519073 378.8172 +72.48462
10 100 2.448332 -3.1380475 128.868 +95.1083
10 500 7.4828916 -4.4311156 136.40652 +93.8183
10 1000 14.155463 -4.948386 144.82907 +89.4507
10 5000 69.08501 -7.788704 220.84483 +42.826385
10 10000 130.85938 -25.253555 355.98264 +23.753784

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.8620948 -2.051762 124.57058 +89.49501
1 5000 7.5001144 -1.913888 136.29553 +91.26482
1 10000 12.6726885 -3.3103771 142.86058 +86.848724
1 50000 68.48433 -3.6049652 237.13591 +19.836609
1 100000 117.76902 -25.787483 398.52505 +92.192474
10 100 2.4310906 -3.155289 130.96674 +97.20704
10 500 7.7416034 -4.172404 135.47064 +92.882416
10 1000 14.207195 -4.896654 145.00768 +89.6293
10 5000 67.7575 -9.116211 203.61504 +25.596588
10 10000 130.1785 -25.934433 365.14075 +32.911896

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.8793454 -2.0345116 127.07489 +91.99933
1 5000 6.7415247 -2.6724777 134.4767 +89.445984
1 10000 14.672782 -1.3102837 145.71632 +89.70447
1 50000 65.2255 -6.8637924 225.75148 +8.452179
1 100000 120.84202 -22.714485 386.34872 +80.01614
10 100 2.3276153 -3.2587643 129.65086 +95.89117
10 500 7.569097 -4.34491 140.60048 +98.01225
10 1000 14.345184 -4.758665 144.51468 +89.13631
10 5000 68.8316 -8.042114 214.68239 +36.66394
10 10000 130.31044 -25.80249 364.48895 +32.2601

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.9655533 -1.9483037 127.9305 +92.854935
1 5000 6.6898594 -2.724143 134.44612 +89.415405
1 10000 13.138312 -2.8447533 140.98747 +84.97562
1 50000 65.84674 -6.2425537 223.53287 +6.2335663
1 100000 119.33476 -24.22174 391.03152 +84.698944
10 100 2.5345528 -3.0518267 132.02061 +98.26092
10 500 7.465711 -4.448296 138.18144 +95.593216
10 1000 14.914387 -4.1894627 145.21431 +89.83594
10 5000 70.69228 -6.181427 222.18507 +44.166626
10 10000 130.49014 -25.622787 362.2523 +30.023438

@jmwilliams89 jmwilliams89 marked this pull request as ready for review September 9, 2020 13:26
Copy link
Member

@djaglowski djaglowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great.

@djaglowski djaglowski merged commit 60b8b03 into submod-diskbuff Sep 9, 2020
@djaglowski djaglowski deleted the improved-coverage branch September 9, 2020 13:30
djaglowski added a commit that referenced this pull request Sep 15, 2020
* Renamed package to stanza

* Updated new operator package

* Updated changelog ahead of release

* Submodule exploration

* Finished preliminary reorg of operators into modules

* Fixed google cloud test that was failing on infinite loop

* Fix all tests

* Update changelog

* Fix CI build

* Rename init.go to init_common.go, so that it will not be ignored by any specific OS

* WIP - try fixing tests on windows CI

* WIP - Try another windows testing approach

* Try uploading joined coverage file

* Run all unit tests for windows, and report all coverage

* Fix return paths on windows ci tests

* Try more powershelly command

* Clean up makefile and try codecov upload with glob

* WIP

* WIP

* Make disk buffer work

* Add ReadWait

* Make benchmark for disk buffer

* WIP

* WIP

* WORKING

* Remove debug printlns

* WIP broken

* Rename to stanza

* WIP

* WIP

* WORKING

* Remove unnecessary counter

* Clean up unused

* Start of memory buffer

* Add slow memory buffer implementation

* Reorganize package

* Update comments

* WIP

* Improve performance by only seeking when necessary

* WIP

* WIP

* WIP

* Fix failure to release semaphore

* Some code hygiene

* Add some small tests

* Make NewConfig return a pointer

* Fix tests

* Tidy

* Fix make tidy target

* Fix tests and integrate with Google Cloud

* Fix remaining tests

* Appease linter

* Merged disk buffers again

* Add comments to public functions

* Add test for closing and reopening

* Add comments to flusher

* Remove TODO

* Update diskSizeSemaphore comment

* Update changelog

* Fis issue with creating files in the buffer package

* Add disk buffers

* Tidy dependencies

* Fix all existing tests

* Move pollForNewFiles into its own method

* Deduplicate NewFileReader

* Fix race condition

* Resolved merge conflicts

* Update tests

* Fix some lints

* Minor fixes

* Close file during move on Windows

* Fixed failing test

* Added comments, cleaned up stutter

* Added multi file test

* Improved coverage (#110)

* Completed test coverage for entry and agent packages

* Made a database package and improved tests

* Improved pipeline test coverage

* Fixed open database in offsets

* Moved operator.Duration to helper.Duration

* Increased test coverage for the plugin package

* Fixed graph command

* Improved JSON test coverage

* Improved regex parser test coverage

* Added tests to persister

* Improved operator package test coverage

* Improved drop_output test coverage

* Improved noop operator test coverage

* Added file rotation test

* Add LastSeenTime to readers

* Fix data race

* Synchronize reading to simplify logic

* Update fingerprint on initialize

* Add comments

* Remove unnecessary setOffset function

* Update fingerprint on truncate

* Add a few tests

* Ignore empty lines

* Use nanojack to rotate test logs

* WIP

* WIP

* WIP

* commented

* Working

* Remove commented line

* Merge rotator

* Updated nanojack

* Fix missing iteration

* Rename some things

* Simplify

* Add copy truncate test

* Up the message count

* Fix root test

* CHeck for errors

* Enable profiling

* Added copytruncate test

* Wip - broken copytruncate

* Add reader generation for thinning them out

* Trigger benchmarks

* Turn off profiles

* Allow max backups when using nanojck

* Small cleanup tasks

* File fixes (#113)

Fix some instability in the file input

* Enhanced rotation tests to expose more edge cases

* Fixed test compilation

* Fixed merge issue

* WIP

* Added some csv output

* Added robust file rotation testing

* Refactored rotation test to use more explicit cases

* Fast rotation tests are less fast but still fast

* Fast rotation tests are less fast but still fast

* Add patience to rotation tests

* Readd test file cleanup

* Addressed some pr feedback

Co-authored-by: Camden Cheek <[email protected]>
Co-authored-by: Joshua Williams <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants