[iptables] Fix failing test daily: system test: journald in iptables.log

[aleksmaus]

Proposed commit message

[iptables] Fix failing test daily: system test: journald in iptables.log

The latest version of filebeat sends the new field journald.custom.realtime_timestamp that didn't have mapping defined. Adding the mapping resolves the issue.

Example:

          "journald": {
            "custom": {
              "realtime_timestamp": "1642033008518660"
            },
            "host": {
              "boot_id": "c2f79f985830406a9e08241d015eff05"
            }
          },

The "auto-mapping" that was created for the index

I'm not 100% sure if this field should present in the first place, but adding the mapping fixes the test failure.
@andrewkroh @taylor-swanson

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes [Stack 8.16.0-SNAPSHOT] [iptables] Failing test daily: system test: journald in iptables.log #10757

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[andrewkroh]

How about adding config to drop this field entirely until the input is sorted out?

Like a beat processor is what I was thinking:

- drop_fields:
    ignore_missing: true
    fields:
      - journald.custom.realtime_timestamp

[aleksmaus]

- drop_fields:

We could do that with the processor, but the filebeat would need to be fixed before 8.16 release anyways I think.

As far as I understand this approach would fail if the user upgrades to 8.16 filebeat without updating the package and rolling out the agent policy (that includes this drop_fields processor). So there is a chance that the beat would be still sending that field to Elasticsearch.

[andrewkroh]

True, neither solution addresses the problem of the user upgrading the agent before upgrading the package. Putting a remove processor into ingest pipeline is good mix of the two.

The reason I prefer field deletion over adding a mapping is because when we no longer need this change and can revert it, then will look less like we are making a breaking change.

[aleksmaus]

The reason I prefer field deletion over adding a mapping is because when we no longer need this change and can revert it, then will look less like we are making a breaking change.

Sounds good. Will add remove processor.

[aleksmaus]

Rolled back the mapping change, added drop_fields processor as agreed

[andrewkroh]

Putting a remove processor into ingest pipeline is good mix of the two.

Sounds good. Will add remove processor.

added drop_fields processor as agreed

I think we got our wires crossed 😄 . I meant an ingest node pipeline remove processor (https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html). This marginally better than drop_fields since it would cover cases like standalone agent where the user might not immediately update their edge configuration agent policy similar to what you mentioned.

e.g.

# packages/iptables/data_stream/log/elasticsearch/ingest_pipeline/default.yml
- remove:  
    description: Temporary fix to remove realtime_timestamp until the journald input is fixed.  
    if: ctx.agent?.version != null && ctx.agent.version.startsWith("8.16")  
    field: journald.custom.realtime_timestamp  
    ignore_missing: true

But given that this is meant to be very temporary, what you have is totally fine with me. I expect we'll have the input updated long before 8.16 is released.

[aleksmaus]

I think we got our wires crossed 😄 . I meant an ingest node pipeline remove processor

ohh, though you mentioned drop_field before. Kk can redo with the pipeline processor.

[aleksmaus]

Updated.
So many options.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 2e0f691

History

• 💚 Build #15079 succeeded 9f25299
• 💚 Build #15041 succeeded bee5410
• 💔 Build #15036 failed 11de8cd

cc @aleksmaus

[aleksmaus]

I guess we would not need this change at all if the fix is upstream (in the filebeat), instead would have to add the mapping for the realtime_timestamp

[aleksmaus]

Closing this PR for the fix upstream (in filebeat)
elastic/beats#40658