Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat: Fix o365 module issues #18948

Merged
merged 11 commits into from
Jun 9, 2020
Merged

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Jun 3, 2020

Assorted fixes to the o365 module:

  • Mark module as beta in docs.
  • get rid of data-loss error on startup: Bad error handling around saved-state loading (unimplemented) caused a data-loss warning on startup instead of a less scary info message:
- [ERROR] Error loading saved state. Will fetch all retained events. Depending on max_retention, this can cause event loss or duplication.
+ [INFO] No saved state found. Will fetch events for the last 168h.
  • Avoid passing API errors to the JS pipeline

Ingestion pipeline errors from o365audit input need not to go through the JS pipeline, it'll add more errors and noise.

  • Prevent dissect error about overriding client.port

  • Fix how API settings are passed to the o365 input

Passing low-level API settings between module and input was broken.

  • Document max_period using the right units.

The sample conf will use 7d which is not valid as hours is the largest supported unit.

@adriansr adriansr added bug in progress Pull request is currently in progress. Team:SIEM labels Jun 3, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 3, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18948 updated]

  • Start Time: 2020-06-08T16:17:45.927+0000

  • Duration: 49 min 32 sec

Test stats 🧪

Test Results
Failed 0
Passed 2226
Skipped 382
Total 2608

@adriansr adriansr changed the title Fix o365 module issues [draft] Filebeat: Fix o365 module issues Jun 8, 2020
@adriansr adriansr added review and removed in progress Pull request is currently in progress. labels Jun 8, 2020
@adriansr adriansr marked this pull request as ready for review June 8, 2020 16:14
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@adriansr adriansr merged commit 83bbd57 into elastic:master Jun 9, 2020
@adriansr adriansr deleted the fix_o365_issues branch June 9, 2020 08:22
adriansr added a commit to adriansr/beats that referenced this pull request Jun 9, 2020
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
@adriansr adriansr added the v7.9.0 label Jun 9, 2020
adriansr added a commit to adriansr/beats that referenced this pull request Jun 9, 2020
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
@adriansr adriansr added the v7.8.0 label Jun 9, 2020
adriansr added a commit to adriansr/beats that referenced this pull request Jun 9, 2020
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
@adriansr adriansr added the v7.7.2 label Jun 9, 2020
adriansr pushed a commit that referenced this pull request Jun 9, 2020
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind #18948
adriansr pushed a commit to adriansr/beats that referenced this pull request Jun 9, 2020
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind elastic#18948

(cherry picked from commit 05935ee)
adriansr pushed a commit to adriansr/beats that referenced this pull request Jun 9, 2020
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind elastic#18948

(cherry picked from commit 05935ee)
adriansr added a commit that referenced this pull request Jun 9, 2020
* Filebeat: Fix o365 module issues (#18948)

- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
adriansr added a commit that referenced this pull request Jun 9, 2020
* Filebeat: Fix o365 module issues (#18948)

- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
adriansr added a commit that referenced this pull request Jun 9, 2020
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind #18948

(cherry picked from commit 05935ee)

Co-authored-by: AndyHunt66 <[email protected]>
adriansr added a commit that referenced this pull request Jun 9, 2020
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit 83bbd57)
adriansr added a commit that referenced this pull request Jun 9, 2020
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind #18948

(cherry picked from commit 05935ee)

Co-authored-by: AndyHunt66 <[email protected]>
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
I think it would be good to get some extra information around the max_retention setting, to make it clear that unless the tenant itself has a longer retention period, then 7 days is going to be all that you can fetch.

Follow-on from investigation behind elastic#18948
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…lastic#19063)

* Filebeat: Fix o365 module issues (elastic#18948)

- Fix scary data-loss warning on startup
- Avoid API errors being processed by the JS pipeline
- Fix dissect error about overiding client.port
- Fix module passing API settings to the input
- Document max_period using the right units

(cherry picked from commit b99a73c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants