Cursor input skeleton

[urso]

• Enhancement

What does this PR do?

This PR is part of introducing a new input architecture in filebeat. The current state of the full implementation can be seen here and sample inputs based on the new API.

The full list of changes will include:

• Introduce v2 API interfaces
• Introduce compatibility layer to integrate API with existing functionality
• Introduce helpers for writing stateless inputs.
• Introduce helpers for writing inputs that store a state between restarts.
• Integrate new API with existing inputs and modules in filebeat.

The change introduces the skeleton and documentation with details for
cursor based inputs. Future updates will add the actual implementation
and tests.

Why is it important?

Filebeat input v2 API.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works
  - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates: Filebeat input v2 API #15324

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [urso commented: jenkins run the tests please]

• Start Time: 2020-06-29T18:39:07.862+0000

• Duration: 80 min 36 sec

Test stats 🧪

Test	Results
Failed	0
Passed	543
Skipped	127
Total	670

[kvch]

🎉

[kvch]

If I understand correctly, the finished states are not claimed by any input and there is no pending state update. So this means that as long as Filebeat is running, it must claim the resource to persist its state. Does this mean that Filebeat cannot close an input source temporarily, e.g. a journal to wait until new entries show up otherwise its state is lost? Or is this going to implement functionality similar to those of the options clean_* for log input?

[urso]

finished states are not claimed by any input and there is no pending state update.

correct.

So this means that as long as Filebeat is running, it must claim the resource to persist its state.

The resource is claimed by an active input. If there is no active input (e.g. autodiscovery closed the input after the container has been deleted, or the configuration file has changed between restarts), it is 'free' and might be removed by the cleaning process.

Does this mean that Filebeat cannot close an input source temporarily,

More or less correct. With the implementation of this input manager we add a TTL to each entry in the store. unclaimed resources are removed if latest update + TTL < now.

This input manager requires sources to be configured statically and upfront based on the configuration. Internally resources can't be released and reclaimed without an external signal.
But:

• the input.Run method is allowed to close connections, wait, and reopen. The resource is claimed for as long as the go-routine is alive (Run did not return).
• We might consider to add a 'flag' to a resource saying that it can not be cleaned yet. This flag would be active in memory only and reset on beats restart

Filebeat can have different input manager instances and even different input manager implementations that can all coexist and provide different coordination behavior. For the log input we can not configure []Source upfront, as we need a discovery mechanism (file watcher) adding and removing sources dynamically. Here we have two options, enhance this input manager, or develop a second input manager that can handle dynamic sources. For dynamic sources we should also consider some signaling like 'source removed' or 'source renamed', so we implement close_removed and clean_removed on top of these signals.

Almost all stateful filebeat inputs we have do configure the source statically at init-time. Even the windows event logs and the journald log (configure system_logs, path, or a filename). In the later case the journald libraries do the heavy lifting for use (based on our configured source).
AFAICT the only special input is the log input.

Or is this going to implement functionality similar to those of the options clean_* for log input?

Yes. The TTL and cleaner are similar to clean_inactive. This input manager does not have a clean_removed, as the sources are 'static' and we have no way to tell it that a resource is gone for good.

For an input manager that supports dynamic sources, having a file watcher we could emit a remove immediately, even cancelling pending update operations (or set TTL to 0 and wait for the resource to be 'finished').

For an example usage check out the journald input implementation: https://github.com/urso/beats/blob/fb-input-v2-combined/filebeat/features/input/journald/input.go
The journald input creates it's own InputManager, that coordinates journald input instances only.

[urso]

jenkins run the tests please