Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File fixes #113

Merged
merged 26 commits into from
Sep 11, 2020
Merged

File fixes #113

merged 26 commits into from
Sep 11, 2020

Conversation

camdencheek
Copy link
Contributor

Description of Changes

Please check that the PR fulfills these requirements

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • Add a changelog entry (for non-trivial bug fixes / features)
  • CI passes

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.8276111 -0.1034565 128.52357 +1.3087311
1 5000 6.017381 -1.3793297 136.99501 +0.40611267
1 10000 12.241598 -4.172495 142.57126 -4.5195312
1 50000 59.863518 -19.086494 180.73128 -65.4872
1 100000 114.669106 -34.223244 228.39696 -188.4969
10 100 2.431147 -0.3792479 131.27707 +0.48558044
10 500 6.655271 -2.0346932 137.363 +0.28744507
10 1000 13.34518 -3.378951 145.33513 -1.4606628
10 5000 63.334034 -18.705189 187.8156 -38.287308
10 10000 122.814064 -33.946556 216.90382 -160.68495

@codecov
Copy link

codecov bot commented Sep 10, 2020

Codecov Report

Merging #113 into rc0.10.0 will increase coverage by 0.18%.
The diff coverage is 74.63%.

Impacted file tree graph

@@             Coverage Diff              @@
##           rc0.10.0     #113      +/-   ##
============================================
+ Coverage     70.17%   70.35%   +0.18%     
============================================
  Files            80       80              
  Lines          4945     4965      +20     
============================================
+ Hits           3470     3493      +23     
+ Misses         1163     1161       -2     
+ Partials        312      311       -1     
Impacted Files Coverage Δ
operator/builtin/input/file/config.go 85.71% <ø> (ø)
operator/builtin/input/file/file.go 72.79% <71.43%> (-4.68%) ⬇️
operator/builtin/input/file/reader.go 73.68% <76.92%> (+11.50%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49d23ab...90f8975. Read the comment docs.

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.6896763 -0.2413913 125.592804 -1.6220398
1 5000 6.189773 -1.2069378 133.96632 -2.6225739
1 10000 11.931285 -4.482808 144.18561 -2.905182
1 50000 60.587982 -18.36203 175.69733 -70.52115
1 100000 111.9889 -36.90345 237.43184 -179.46202
10 100 2.6725297 -0.1378653 132.46619 +1.6746979
10 500 6.9656987 -1.7242656 139.33661 +2.2610474
10 1000 13.5005665 -3.2235641 145.16716 -1.6286316
10 5000 63.15198 -18.887241 185.76602 -40.336884
10 10000 126.28434 -30.47628 224.42525 -153.16353

@camdencheek camdencheek marked this pull request as ready for review September 11, 2020 15:01
@camdencheek camdencheek merged commit 9bc0838 into rc0.10.0 Sep 11, 2020
@camdencheek camdencheek deleted the file-fixes branch September 11, 2020 15:01
djaglowski added a commit that referenced this pull request Sep 15, 2020
* Renamed package to stanza

* Updated new operator package

* Updated changelog ahead of release

* Submodule exploration

* Finished preliminary reorg of operators into modules

* Fixed google cloud test that was failing on infinite loop

* Fix all tests

* Update changelog

* Fix CI build

* Rename init.go to init_common.go, so that it will not be ignored by any specific OS

* WIP - try fixing tests on windows CI

* WIP - Try another windows testing approach

* Try uploading joined coverage file

* Run all unit tests for windows, and report all coverage

* Fix return paths on windows ci tests

* Try more powershelly command

* Clean up makefile and try codecov upload with glob

* WIP

* WIP

* Make disk buffer work

* Add ReadWait

* Make benchmark for disk buffer

* WIP

* WIP

* WORKING

* Remove debug printlns

* WIP broken

* Rename to stanza

* WIP

* WIP

* WORKING

* Remove unnecessary counter

* Clean up unused

* Start of memory buffer

* Add slow memory buffer implementation

* Reorganize package

* Update comments

* WIP

* Improve performance by only seeking when necessary

* WIP

* WIP

* WIP

* Fix failure to release semaphore

* Some code hygiene

* Add some small tests

* Make NewConfig return a pointer

* Fix tests

* Tidy

* Fix make tidy target

* Fix tests and integrate with Google Cloud

* Fix remaining tests

* Appease linter

* Merged disk buffers again

* Add comments to public functions

* Add test for closing and reopening

* Add comments to flusher

* Remove TODO

* Update diskSizeSemaphore comment

* Update changelog

* Fis issue with creating files in the buffer package

* Add disk buffers

* Tidy dependencies

* Fix all existing tests

* Move pollForNewFiles into its own method

* Deduplicate NewFileReader

* Fix race condition

* Resolved merge conflicts

* Update tests

* Fix some lints

* Minor fixes

* Close file during move on Windows

* Fixed failing test

* Added comments, cleaned up stutter

* Added multi file test

* Improved coverage (#110)

* Completed test coverage for entry and agent packages

* Made a database package and improved tests

* Improved pipeline test coverage

* Fixed open database in offsets

* Moved operator.Duration to helper.Duration

* Increased test coverage for the plugin package

* Fixed graph command

* Improved JSON test coverage

* Improved regex parser test coverage

* Added tests to persister

* Improved operator package test coverage

* Improved drop_output test coverage

* Improved noop operator test coverage

* Added file rotation test

* Add LastSeenTime to readers

* Fix data race

* Synchronize reading to simplify logic

* Update fingerprint on initialize

* Add comments

* Remove unnecessary setOffset function

* Update fingerprint on truncate

* Add a few tests

* Ignore empty lines

* Use nanojack to rotate test logs

* WIP

* WIP

* WIP

* commented

* Working

* Remove commented line

* Merge rotator

* Updated nanojack

* Fix missing iteration

* Rename some things

* Simplify

* Add copy truncate test

* Up the message count

* Fix root test

* CHeck for errors

* Enable profiling

* Added copytruncate test

* Wip - broken copytruncate

* Add reader generation for thinning them out

* Trigger benchmarks

* Turn off profiles

* Allow max backups when using nanojck

* Small cleanup tasks

* File fixes (#113)

Fix some instability in the file input

* Enhanced rotation tests to expose more edge cases

* Fixed test compilation

* Fixed merge issue

* WIP

* Added some csv output

* Added robust file rotation testing

* Refactored rotation test to use more explicit cases

* Fast rotation tests are less fast but still fast

* Fast rotation tests are less fast but still fast

* Add patience to rotation tests

* Readd test file cleanup

* Addressed some pr feedback

Co-authored-by: Camden Cheek <[email protected]>
Co-authored-by: Joshua Williams <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants