[Meta] Remove Elastic Agent V1 control protocol

[jlind23]

Now that the V2 protocol is out from 8.3 on, the V1 will not be updated and thus removed in a few releases. (Around end of 2022)

https://docs.google.com/document/d/1wqBWAhFdao93h2SE__koisuOOMGbwA5DTXDVFSKFYBs/edit

Parallel

• Re-structure the binary layout for V2 to be flat (no tarball, compression, etc.)
  • Change all beats to start without reading a configuration file from the disk (preferably they should start, context to control protocol and wait for initial configuration)
    • Beats run under agent should start without reading a configuration file from disk. beats#31901
• Change beats to understand configuration directly from policy (all rules are removed in the V2 binary specification)
  • Migrate filebeat and metricbeat to the V2 agent specification file format beats#31900
• Implement V2 control protocol in libbeat (having all beats using V2)
  • Migrate libbeat agent management to the agent V2 control protocol beats#32078
• Implement V2 control protocol in Endpoint Security

Sequential

• Implement validation and reading of the V2 binary specification
• Implement V2 control protocol server in Elastic Agent
• Parse Elastic Agent policy with V2 binary specification to spawn Units and be controlled through the V2 protocol

[jlind23]

@pierrehilbert this is something you should read and take into consideration for the next upcoming releases.

[michalpristas]

who;s in charge of Change all beats to start without reading a configuration file from the disk (preferably they should start, context to control protocol and wait for initial configuration) ?

[ph]

That should be on data plane team @cmacknz its closely related to the libbeat changes.

[cmacknz]

Yes, we can help with changing how the beats start up under agent. None of us are particularly familiar with how beats run under agent yet though, so if you can send us a detailed issue about what to change it would be greatly appreciated.

The same for Change beats to understand configuration directly from policy (all rules are removed in the V2 binary specification). I know we are relying on rules for filebeat and metricbeat but we didn't write the spec files originally.

[michalpristas]

i can write something up about how config is loaded now and what needs to be changed tomorrow

[michalpristas]

@cmacknz

Beats behavior change

Current

At the moment we handle process as this:

• elastic agent starts
• elastic agent is notified about or detects config change
• elastic agent creates a specific change for specific beats (let's assume filebeat from now on)
• elastic agent starts filebeat process with -management.enabled flag
• filebeat process starts
• filebeat loads a config from filebeat.yml

	cfg, err := cfgfile.Load("", settings.ConfigOverrides)
	if err != nil {
		return fmt.Errorf("error loading config file: %w", err)
	}

• filebeat initializes manager to get comms with agent going

	// initialize config manager
	b.Manager, err = management.Factory(b.Config.Management)(b.Config.Management, reload.Register, b.Beat.Info.ID)
	if err != nil {
		return err
	}

• filebeat fetches config from fleet

Proposed

• elastic agent starts
• elastic agent is notified about or detects config change
• elastic agent creates a specific change for specific beats (let's assume filebeat from now on)
• elastic agent starts filebeat process with -management.enabled flag
• filebeat process starts
• in case management.enabled flag is not set filebeat loads a config from filebeat.yml otherwise use sane default configuration

	cfg, err := cfgfile.Load("", settings.ConfigOverrides)
	if err != nil {
		return fmt.Errorf("error loading config file: %w", err)
	}

• filebeat initializes manager to get comms with agent going

	// initialize config manager
	b.Manager, err = management.Factory(b.Config.Management)(b.Config.Management, reload.Register, b.Beat.Info.ID)
	if err != nil {
		return err
	}

• filebeat fetches config from fleet

[cmacknz]

Thanks! I've created two issues to track the needed work in beats and updated this issue description to track them:

• Beats run under agent should start without reading a configuration file from disk. beats#31901
• Migrate filebeat and metricbeat to the V2 agent specification file format beats#31900

For migrating to the V2 specification file/removing config transformations I've only considered filebeat and metricbeat since those are the only beats we own. They are also probably the most work to port.

[ph]

@cmacknz AFAIK Auditbeat should behave the same as metricbeat I believe it inherit all the same runtime, we would need to solve it for packetbeat too.

[cmacknz]

Issue to migrate libbeat to the V2 protocol: elastic/beats#32078