[Cisco Secure Endpoint] Parse additional fields to ECS #6258

MakoWish · 2023-05-18T18:08:05Z

Type of change

Enhancement

What does this PR do?

This PR is to parse out the host.ip and host.mac fields from events' cisco.secure_endpoint.computer.network_addresses array of network MAC and IP addresses, as well as map host.id, group.id, error.code, and error.message from the Cisco Secure Endpoint events.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have added an entry to my package's manifest.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Added painless script for parsing IP addresses to host.ip
Added painless script for parsing MAC addresses to host.mac
Rename cisco.secure_endpoint.computer.connector_guid --> host.id
Rename cisco.secure_endpoint.error.description --> error.message
Rename cisco.secure_endpoint.error.error_code --> error.code
Rename cisco.secure_endpoint.group_guids --> group.id
Corrected some discrepancies in ecs.yml fields file
Regenerated expectations

Related issues

Closes [Cisco Secure Endpoint] Parse additional fields to ECS #5352

elasticmachine · 2023-05-18T18:11:20Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-13T07:59:06.909+0000
Duration: 15 min 32 sec

Test stats 🧪

Test	Results
Failed	0
Passed	14
Skipped	0
Total	14

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

…b.com/MakoWish/integrations into cisco_secure_endpoint_mac_address_fix

elasticmachine · 2023-05-30T20:36:18Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

MakoWish · 2023-05-30T20:39:20Z

I for some reason cannot set a value for error.message. If I try either a set or rename processor, I get:

FAILURE DETAILS:
cisco_secure_endpoint/event test-cisco-amp1.log:
[0] unexpected pipeline error: Object name not found
[1] unexpected pipeline error: Object path not found
cisco_secure_endpoint/event test-cisco-amp4.log:
[0] unexpected pipeline error: Delete pending
[1] unexpected pipeline error: Object name not found
cisco_secure_endpoint/event test-cisco-amp5.log:
[0] unexpected pipeline error: Cannot delete
[1] unexpected pipeline error: Delete pending
[2] unexpected pipeline error: Object name not found
cisco_secure_endpoint/event test-cisco-amp6.log:
[0] unexpected pipeline error: Cannot delete
[1] unexpected pipeline error: Delete pending
[2] unexpected pipeline error: Object name not found
cisco_secure_endpoint/event test-cisco-amp7.log:
[0] unexpected pipeline error: Delete pending
[1] unexpected pipeline error: Object name not found

When attempting to test. If I use an append processor, I get:

FAILURE DETAILS:
cisco_secure_endpoint/event test-cisco-amp1.log:
[0] unexpected pipeline error (unexpected error.message type []interface {}): [Object name not found]
[1] unexpected pipeline error (unexpected error.message type []interface {}): [Object path not found]
cisco_secure_endpoint/event test-cisco-amp4.log:
[0] unexpected pipeline error (unexpected error.message type []interface {}): [Delete pending]
[1] unexpected pipeline error (unexpected error.message type []interface {}): [Object name not found]
cisco_secure_endpoint/event test-cisco-amp5.log:
[0] unexpected pipeline error (unexpected error.message type []interface {}): [Cannot delete]
[1] unexpected pipeline error (unexpected error.message type []interface {}): [Delete pending]
[2] unexpected pipeline error (unexpected error.message type []interface {}): [Object name not found]
cisco_secure_endpoint/event test-cisco-amp6.log:
[0] unexpected pipeline error (unexpected error.message type []interface {}): [Cannot delete]
[1] unexpected pipeline error (unexpected error.message type []interface {}): [Delete pending]
[2] unexpected pipeline error (unexpected error.message type []interface {}): [Object name not found]
cisco_secure_endpoint/event test-cisco-amp7.log:
[0] unexpected pipeline error (unexpected error.message type []interface {}): [Delete pending]
[1] unexpected pipeline error (unexpected error.message type []interface {}): [Object name not found]

The error when trying to append would suggest error.message cannot be an array (not 100% sure on that), but the error when trying to set or rename doesn't tell me much of anything at all. Not sure how to get around that.

Eric

efd6 · 2023-05-30T23:12:34Z

@MakoWish Can you paste the code snippets for the approaches that you tried?

Note that error.message is for errors in the pipeline, rather than errors that are valid for the dataset. So the tests are interpreting the text you place in error.message (valid information for the dataset) as a complaint about the dataset. The "unexpected error.message type" warning is a distraction — that I have to personally apologise for; I'll fix it now — due to how we check the error messages and doesn't change the meaning of what I have above.

MakoWish · 2023-05-31T13:26:12Z

@efd6

Note that error.message is for errors in the pipeline, rather than errors that are valid for the dataset.

Should the ECS Field Reference be updated to reflect that? I have been using error.message whenever the event contains an error message, as suggested in the documentation:

Use them for errors that happen while fetching events or in cases where the event itself contains an error.

Given an event with cisco.secure_endpoint.error.description = "Object name not found", any of these:

  - rename:
      field: cisco.secure_endpoint.error.description
      target_field: error.message
      ignore_missing: true

  - set:
      field: error.message
      value: '{{{cisco.secure_endpoint.error.description}}}'
      if: ctx.cisco?.secure_endpoint?.error?.description != null && ctx.cisco.secure_endpoint.error.description != ''

  - set:
      field: error.message
      copy_from: cisco.secure_endpoint.error.description
      if: ctx.cisco?.secure_endpoint?.error?.description != null && ctx.cisco.secure_endpoint.error.description != ''

causes elastic-package test -v to return:

FAILURE DETAILS:
cisco_secure_endpoint/event test-cisco-amp1.log:
[0] unexpected pipeline error: Object name not found

If I try an append processor:

  - append:
      field: error.message
      value: '{{{cisco.secure_endpoint.error.description}}}'
      if: ctx.cisco?.secure_endpoint?.error?.description != null && ctx.cisco.secure_endpoint.error.description != ''

the test results in:

FAILURE DETAILS:
cisco_secure_endpoint/event test-cisco-amp1.log:
[0] unexpected pipeline error (unexpected error.message type []interface {}): [Object name not found]

In all of these cases, the expected results JSON files do correctly get the error.message field populated, but elastic-package test -v gives the errors regardless.

efd6 · 2023-06-01T00:30:00Z

Should the ECS Field Reference be updated to reflect that? I have been using error.message whenever the event contains an error message, as suggested in the documentation:

Use them for errors that happen while fetching events or in cases where the event itself contains an error.

I agree, the documentation is unclear.

The part of the error message (unexpected error.message type []interface {}) can be ignored here, it is there to be helpful, but misses the mark in this situation. This PR addresses that and when it's merged will result in the same kind of error message as you see for the set processor setting of error.message. The reason for the parenthetic is that we want to check that people are not putting non-string values into error.message, but the check I put there failed to take into account how Go deserialises arrays of strings without a known type context. The PR above is intended to fix that.

efd6 · 2023-06-01T00:31:10Z

/test

elasticmachine · 2023-06-01T00:46:44Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	94.118% (`16/17`)	❕
Lines	91.055% (`509/559`)	❕
Conditionals	100.0% (`0/0`)	💚

packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml

packages/cisco_secure_endpoint/data_stream/event/fields/ecs.yml

LaZyDK · 2023-06-01T07:37:47Z

Should the ECS Field Reference be updated to reflect that? I have been using error.message whenever the event contains an error message, as suggested in the documentation:

Use them for errors that happen while fetching events or in cases where the event itself contains an error.

I agree, the documentation is unclear.

The part of the error message (unexpected error.message type []interface {}) can be ignored here, it is there to be helpful, but misses the mark in this situation. This PR addresses that and when it's merged will result in the same kind of error message as you see for the set processor setting of error.message. The reason for the parenthetic is that we want to check that people are not putting non-string values into error.message, but the check I put there failed to take into account how Go deserialises arrays of strings without a known type context. The PR above is intended to fix that.

I agree on this one. I have been using error fields for all kinds of errors, just as ECS is stating:

These fields can represent errors of any kind.
Use them for errors that happen while fetching events or in cases where the event itself contains an error.

efd6 · 2023-06-01T08:44:35Z

It may be that this is elastic-package over-interpreting error.message. Ideally it should check for "event.kind":"pipeline_error", but we don't always set that. I'd like to see what @P1llus thinks.

Co-authored-by: Dan Kortschak <[email protected]>

MakoWish · 2023-06-02T17:09:51Z

Until that change happens, are you okay with utilizing error.message and just ignoring the test failures? Again, the "expected results" documents are properly created, and it would be compliant with the ECS documentation; it is just that the Elastic Package test fails (incorrectly IMO).

efd6 · 2023-06-04T22:03:36Z

Until that change happens, are you okay with utilizing error.message and just ignoring the test failures?

This is not possible; CI will not allow us to merge with failing builds. It may seem reasonable to allow this, but if we do that, now we have to assess whether the failure was a real failure or an unreasonable failure. This is not tenable.

I've raised this internally and the situation is essentially as I describe above; there is a dependency chain on behaviours.

In order to allow relaxation of error.message state checks in testing we need to have an alternative method to detect pipeline error.
This exists in the form of "event.kind":"pipeline_error", but is not uniformly applied, so
We need to have all packages using "event.kind":"pipeline_error" for pipeline errors before we can change EP's behaviour.

This is a reasonable thing to want to do, but it is a non-trivial amount of work given the number of packages and teams that need to engage with it.

efd6 · 2023-06-04T22:19:49Z

/test

MakoWish · 2023-06-06T17:11:16Z

it is a non-trivial amount of work given the number of packages and teams that need to engage with it.

I don't know how your internal team would like to break out that task, but since I would benefit from it, I am willing to help get it done. Let me know, and I could start knocking out some PR's. I am sure you would want some sort of tracking on which have been completed, and which are pending? It would basically just be adding event.kind: pipeline_error like we did in the Arista integration, correct?

efd6 · 2023-06-07T01:08:57Z

I am willing to help get it done. Let me know, and I could start knocking out some PR's.

Adding the event.kind set to "pipeline_error" certainly would be helpful. We should file an issue; I'll raise it in our meeting.

efd6 · 2023-06-07T01:18:32Z

/test

MakoWish · 2023-06-08T21:47:54Z

@efd6,

Doesn't look like the test kicked off. Can you run another test?

efd6 · 2023-06-08T23:21:24Z

/test

LaZyDK · 2023-06-09T09:47:13Z

@MakoWish I think that you will need to run pipeline and system tests again, after I added the new test file.

…b.com/MakoWish/integrations into cisco_secure_endpoint_mac_address_fix

LaZyDK · 2023-06-13T07:53:49Z

@efd6 Can you start the test on this one?

efd6 · 2023-06-13T07:58:52Z

/test

elasticmachine · 2023-06-13T07:59:44Z

💔 Build Failed

Buildkite Build
Commit: d83bff6

Failed CI Steps

:pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

LaZyDK · 2023-06-13T08:44:58Z

LGTM

MakoWish · 2023-06-13T20:59:34Z

I have never seen "buildkite" before. What is that? I get a "Page not found" if I click on "Details".

efd6 · 2023-06-13T21:32:13Z

@MakoWish It's the new build system we are using.

efd6

Thanks

MakoWish · 2023-06-13T21:43:02Z

@MakoWish It's the new build system we are using.

Will non-Elasticians have visibility into failures with that system? I used to be able to see issues in Jenkins when one of you kicked off a test and it failed, but I got a 404 when I tried to see why it failed with buildkite.

elasticmachine · 2023-06-13T21:54:00Z

Package cisco_secure_endpoint - 2.14.0 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_endpoint

* Add parsing for `host.ip` and `host.mac` * Correction to parsing of host.ip

efd6 · 2023-06-19T07:16:46Z

@MakoWish I have asked internally and will get back to you when I have an answer. Also, WRT your query above #6258 (comment), there is this now #6582.

MakoWish · 2023-06-19T20:10:08Z

@MakoWish I have asked internally and will get back to you when I have an answer.

Looks like they already took action on that ask. I can now see the build statuses in buildkite.

Add parsing for host.ip and host.mac

0c2df93

MakoWish requested a review from a team as a code owner May 18, 2023 18:08

Updated manifest and changelog

ccc8f76

MakoWish force-pushed the cisco_secure_endpoint_mac_address_fix branch from f2281ea to ccc8f76 Compare May 18, 2023 18:09

MakoWish changed the title ~~Cisco secure endpoint mac address fix~~ [Cisco Secure Endpoint] Parse host.ip and host.mac from events May 18, 2023

MakoWish marked this pull request as draft May 18, 2023 20:14

MakoWish changed the title ~~[Cisco Secure Endpoint] Parse host.ip and host.mac from events~~ [Cisco Secure Endpoint] Parse additional fields to ECS May 18, 2023

jamiehynds added Team:Security-External Integrations Integration:cisco_secure_endpoint Cisco Secure Endpoint labels May 22, 2023

MakoWish added 4 commits May 30, 2023 11:02

Merge branch 'main' into cisco_secure_endpoint_mac_address_fix

db12ad5

Correction to parsing of host.ip

0b66037

Corrected conflicts and additional parsing

49728eb

Merge branch 'cisco_secure_endpoint_mac_address_fix' of https://githu…

3b60397

…b.com/MakoWish/integrations into cisco_secure_endpoint_mac_address_fix

MakoWish marked this pull request as ready for review May 30, 2023 20:36

MakoWish mentioned this pull request May 30, 2023

[Cisco Secure Endpoint] Parse additional fields to ECS #5352

Closed

efd6 mentioned this pull request May 30, 2023

internal/testrunner/runners/pipeline: clarify warning of error.message type elastic/elastic-package#1286

Merged

MakoWish added 2 commits May 31, 2023 11:50

Merge branch 'main' into cisco_secure_endpoint_mac_address_fix

80924da

Increment version due to conflict with recent merge

a380b05

efd6 reviewed Jun 1, 2023

View reviewed changes

packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

packages/cisco_secure_endpoint/data_stream/event/fields/ecs.yml Outdated Show resolved Hide resolved

Commit suggested change

93998a3

Co-authored-by: Dan Kortschak <[email protected]>

MakoWish mentioned this pull request Jun 6, 2023

[Arista NG Firewall] Initial Release #6347

Merged

5 tasks

MakoWish added 2 commits June 7, 2023 08:38

Remove field host.id from ecs.yml

fb630e9

Remove field host.id from ecs.yml

580b347

MakoWish added 3 commits June 9, 2023 09:07

Re-run system and pipeline tests

be46992

Re-run system and pipeline tests

dcf40b1

Merge branch 'cisco_secure_endpoint_mac_address_fix' of https://githu…

d83bff6

…b.com/MakoWish/integrations into cisco_secure_endpoint_mac_address_fix

efd6 approved these changes Jun 13, 2023

View reviewed changes

efd6 merged commit 13992bf into elastic:main Jun 13, 2023

MakoWish deleted the cisco_secure_endpoint_mac_address_fix branch June 13, 2023 21:44

sodhikirti07 pushed a commit that referenced this pull request Jun 15, 2023

[Cisco Secure Endpoint] Parse additional fields to ECS (#6258)

6b82136

* Add parsing for `host.ip` and `host.mac` * Correction to parsing of host.ip

MakoWish mentioned this pull request Jul 20, 2023

[azure] Ensure event.kind is correctly set for pipeline errors #7040

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cisco Secure Endpoint] Parse additional fields to ECS #6258

[Cisco Secure Endpoint] Parse additional fields to ECS #6258

MakoWish commented May 18, 2023 •

edited

Loading

elasticmachine commented May 18, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented May 30, 2023

MakoWish commented May 30, 2023 •

edited

Loading

efd6 commented May 30, 2023

MakoWish commented May 31, 2023

efd6 commented Jun 1, 2023

efd6 commented Jun 1, 2023

elasticmachine commented Jun 1, 2023 •

edited

Loading

LaZyDK commented Jun 1, 2023

efd6 commented Jun 1, 2023

MakoWish commented Jun 2, 2023 •

edited

Loading

efd6 commented Jun 4, 2023

efd6 commented Jun 4, 2023

MakoWish commented Jun 6, 2023

efd6 commented Jun 7, 2023

efd6 commented Jun 7, 2023

MakoWish commented Jun 8, 2023

efd6 commented Jun 8, 2023

LaZyDK commented Jun 9, 2023

LaZyDK commented Jun 13, 2023

efd6 commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

LaZyDK commented Jun 13, 2023

MakoWish commented Jun 13, 2023

efd6 commented Jun 13, 2023

efd6 left a comment

MakoWish commented Jun 13, 2023 •

edited

Loading

elasticmachine commented Jun 13, 2023

efd6 commented Jun 19, 2023

MakoWish commented Jun 19, 2023

[Cisco Secure Endpoint] Parse additional fields to ECS #6258

[Cisco Secure Endpoint] Parse additional fields to ECS #6258

Conversation

MakoWish commented May 18, 2023 • edited Loading

Type of change

What does this PR do?

Checklist

Author's Checklist

Related issues

elasticmachine commented May 18, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 30, 2023

MakoWish commented May 30, 2023 • edited Loading

efd6 commented May 30, 2023

MakoWish commented May 31, 2023

efd6 commented Jun 1, 2023

efd6 commented Jun 1, 2023

elasticmachine commented Jun 1, 2023 • edited Loading

🌐 Coverage report

LaZyDK commented Jun 1, 2023

efd6 commented Jun 1, 2023

MakoWish commented Jun 2, 2023 • edited Loading

efd6 commented Jun 4, 2023

efd6 commented Jun 4, 2023

MakoWish commented Jun 6, 2023

efd6 commented Jun 7, 2023

efd6 commented Jun 7, 2023

MakoWish commented Jun 8, 2023

efd6 commented Jun 8, 2023

LaZyDK commented Jun 9, 2023

LaZyDK commented Jun 13, 2023

efd6 commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

💔 Build Failed

Failed CI Steps

LaZyDK commented Jun 13, 2023

MakoWish commented Jun 13, 2023

efd6 commented Jun 13, 2023

efd6 left a comment

Choose a reason for hiding this comment

MakoWish commented Jun 13, 2023 • edited Loading

elasticmachine commented Jun 13, 2023

efd6 commented Jun 19, 2023

MakoWish commented Jun 19, 2023

MakoWish commented May 18, 2023 •

edited

Loading

elasticmachine commented May 18, 2023 •

edited

Loading

MakoWish commented May 30, 2023 •

edited

Loading

elasticmachine commented Jun 1, 2023 •

edited

Loading

MakoWish commented Jun 2, 2023 •

edited

Loading

MakoWish commented Jun 13, 2023 •

edited

Loading