Elastic stack: support Fleet Server

[mtojek]

The goal of this issue is to ensure that the Fleet Server is enabled for the Elastic stack setup.

Known differences / to be solved:

• new policy for the fleet server (should be supported by test runners)
• Blocker: [Fleet] Support POLICY_REASSIGN action beats#24447 [Fleet] Handle POLICY_REASSIGN action (#24616) beats#24631 (backport)
  - [ ] Blocker: Elastic-Agent: failed: could not decode the response, raw response: no matching action beats#24467 (most likely not a blocker anymore, but no-go for support of 7.12 and 7.11, environment variables are in conflict between versions)
• Blocker: Elastic Agent: expecting Dict and received *transpiler.Key for '0' beats#24453 (can't observe anymore)
• Blocker: [Fleet] Agents can't pick up the policy? kibana#95179
• Blocker: [Fleet] HandlerUnknown type: UNKNOWN (original type: INTERNAL_POLICY_REASSIGN)' received beats#24725 [Fleet] creating a policy without integration break fleet server fleet-server#176 Fix issue with rolling out policies from the coordinator. (bp #180) fleet-server#183
• Blocker: elastic-agent docker: support healthcheck for the container beats#24503 Add status subcommand to report status of running daemon. beats#24856 Cherry-pick #24856 to 7.x: Add status subcommand to report status of running daemon. beats#24905
• Blocker: [Fleet] setup failed: An internal server error occurred kibana#95896 (related? [Fleet] An internal server error occurred: ResponseError: security_exception kibana#95388)

Are older stacks working?
- [ ] 7.12.0 no: elastic/beats#24467 (comment)
- [ ] 7.11.2 no: elastic/beats#24467 (comment)

[ruflin]

Comment on the older stacks: 7.11 and 7.12 are not expected to work.

[mtojek]

It depends on the set of environment variables. Maybe it's possible to select such set that all stacks work correctly. Even though I agree to give priority to the latest (7.13).

[mtojek]

Status update:

With support of @nchaulet , I managed to start the agent, connect to the fleet server and enroll. I suppose we eliminated this way one blocker. Unfortunately it seems that we can't comprise a complete list of environment vars that will be suitable for 7.13, 7.12, 7.11 as vars are in conflict with each other (different behavior).

[mtojek]

Next status update:

Another issue spotted. Either the agent can't pick up the policy or it doesn't confirm that it was picked - elastic/kibana#95179 .

[mtojek]

Status update:

It looks like we fixed all issues or at least they're not bumping up here.

[ruflin]

@mtojek Thanks a lot for pushing through all these. It caused a lot of work on your end but helped the other teams to make the product better ❤️