Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[receiver/mongodbatlasreceiver] Add Log collection #12800

Merged
merged 29 commits into from
Aug 15, 2022
Merged

[receiver/mongodbatlasreceiver] Add Log collection #12800

merged 29 commits into from
Aug 15, 2022

Conversation

armstrmi
Copy link
Contributor

@armstrmi armstrmi commented Aug 1, 2022 •
edited
Loading

Description:

  • Added functionality for a logs receiver to collect logs from project clusters specified by the user.
  • Along with cluster logs, audit logs from those clusters are also scraped.
  • New Log and Audit log structures added in to the internal package logs.go
  • Log conversion from new custom log structures into plog.Logs

Link to tracking Issue:
#12347

Testing:

  • Tested changes against a live system and was able to successfully retrieve mongo and audit logs
  • New Unit tests added to config.go_test, receiver_test.go,
  • New Test Files added mongodb_event_to_logdata_test.go, logs_test.go

Documentation:

  • Added changes to the local README to correlate with the updated configuration.

cpheps
cpheps reviewed Aug 1, 2022
View reviewed changes
receiver/mongodbatlasreceiver/README.md Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/README.md Show resolved Hide resolved
receiver/mongodbatlasreceiver/config.go Show resolved Hide resolved
receiver/mongodbatlasreceiver/internal/model/logs.go Show resolved Hide resolved
receiver/mongodbatlasreceiver/internal/model/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/mongodb_event_to_logdata.go Show resolved Hide resolved
receiver/mongodbatlasreceiver/mongodb_event_to_logdata.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/receiver.go Outdated Show resolved Hide resolved
BinaryFissionGames
BinaryFissionGames reviewed Aug 1, 2022
View reviewed changes
receiver/mongodbatlasreceiver/README.md Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/README.md Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/factory.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/internal/mongodb_atlas_client.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/internal/mongodb_atlas_client.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/mongodb_event_to_logdata.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/mongodb_event_to_logdata.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/receiver.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
@armstrmi armstrmi force-pushed the mongo_atlas_logs_receiver branch from ebd6c3d to bbfcf44 Compare August 2, 2022 16:07
@StefanKurek
Copy link
Contributor

@armstrmi once you fix the failures, I think it's looking pretty good for an actual PR

@armstrmi armstrmi force-pushed the mongo_atlas_logs_receiver branch from 3d8ef2b to fe18da6 Compare August 3, 2022 17:30
cpheps
cpheps reviewed Aug 3, 2022
View reviewed changes
receiver/mongodbatlasreceiver/factory.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/internal/mongodb_atlas_client.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
@armstrmi armstrmi force-pushed the mongo_atlas_logs_receiver branch from 0c3259f to 4e78e85 Compare August 4, 2022 14:35
@armstrmi armstrmi changed the title feat: Mongo atlas logs receiver [receiver/mongodbatlasreceiver] Add Log collection Aug 4, 2022
BinaryFissionGames
BinaryFissionGames reviewed Aug 4, 2022
View reviewed changes
receiver/mongodbatlasreceiver/internal/model/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
@armstrmi armstrmi marked this pull request as ready for review August 4, 2022 18:58
@armstrmi armstrmi requested review from a team and Aneurysm9 August 4, 2022 18:58
djaglowski
djaglowski reviewed Aug 4, 2022
View reviewed changes
receiver/mongodbatlasreceiver/README.md Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/README.md Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/README.md Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/config.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/config.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/receiver.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
@armstrmi armstrmi force-pushed the mongo_atlas_logs_receiver branch 3 times, most recently from 4b49b8a to 7d81dc1 Compare August 9, 2022 14:30
@armstrmi armstrmi requested a review from djaglowski August 9, 2022 14:51
armstrmi added 21 commits August 15, 2022 09:43
@armstrmi
edit README to include updated config
09782b7
@armstrmi
implemented changes from feeback
001b1b2
@armstrmi
added necessary feedback for collection interval
629c596
@armstrmi
implemented necessary feedback changes
dc0ab22 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
incorporated start and end times to be included in the resourceInfo s…
25003d1 
…truct to reduce punching holes in more helper functions

Signed-off-by: armstrmi <[email protected]>
@armstrmi
added wait group to ensure client is shutdown properly
c28b738 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
Updated receiver_test to account for unique mongodb config IDs
a0c943f 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
add license and lint fixes
2f78216
@armstrmi
linter fixes
46fa2c0
@armstrmi
feedback changes
97c899f 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
minor fixes before major fixes
e1f8f48 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
removed collection of organizations in KickoffReceiver
2b1f638 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
created separate logsReceiver struct
5a89655 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
make goporto
ca42999 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
updated naming semantics
88f96c5
@armstrmi
updated test to account for no logging enabled
04bf53c 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
enabled logging in component receiver test
2074604
@armstrmi
implemented feedback
8d10f37
@armstrmi
implemented feedback for log conversion
82a6d1f
@armstrmi
updated logrecord to populate only when logrecord is present
748057b 
Signed-off-by: armstrmi <[email protected]>
@armstrmi
removing unnecessary log statement
733d681
@armstrmi armstrmi force-pushed the mongo_atlas_logs_receiver branch from 7303f44 to 733d681 Compare August 15, 2022 13:43
djaglowski
djaglowski reviewed Aug 15, 2022
View reviewed changes
Copy link
Member

@djaglowski djaglowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really good. Just two minor things to resolve yet:

  • Early return in getHostLogs / getHostAuditLogs
  • ProjectConfig variable name in one func

receiver/mongodbatlasreceiver/logs.go Outdated Show resolved Hide resolved
djaglowski
djaglowski approved these changes Aug 15, 2022
View reviewed changes
Copy link
Member

@djaglowski djaglowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for iterating @armstrmi. This looks good.

@djaglowski djaglowski merged commit eb95d08 into open-telemetry:main Aug 15, 2022
@armstrmi armstrmi deleted the mongo_atlas_logs_receiver branch August 15, 2022 15:23
@armstrmi
Copy link
Contributor Author

@djaglowski Thanks for the informative feedback

@cpheps cpheps mentioned this pull request Aug 16, 2022
Merged
2 tasks
@evan-bradley evan-bradley mentioned this pull request Sep 14, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@StefanKurek StefanKurek StefanKurek approved these changes

@BinaryFissionGames BinaryFissionGames BinaryFissionGames approved these changes

@djaglowski djaglowski djaglowski approved these changes

@Aneurysm9 Aneurysm9 Awaiting requested review from Aneurysm9

@cpheps cpheps Awaiting requested review from cpheps

Assignees

@mx-psi mx-psi

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@armstrmi @StefanKurek @djaglowski @BinaryFissionGames @cpheps @mx-psi