[Fleet Server] [Agent] Support "mixed environments" of cloud stack + local (self-managed) Fleet Servers

[EricDavisX]

This ticket is a place-holder, for the technical + testing requirements to be confirmed to support "mixed environments" (and relating Docs) where the stack is in the Elastic Cloud as hosted, but a user wishes to add in additional 'self managed' local Fleet Servers.

• one customer requested using this, to support testing vs production environments, with the same version fleet server. I've lost in slack who cited this, they never said what customer it was and didn't post to the ticket when requested.

• Another customer references this in terms of intranet vs DMZ access by different sets of hosts. I've similarly not been able to capture who that was.

• one customer is internal, the "QA" test group - it is a desired (preferred) test strategy.

I'll bring forward my note from a prior discussing this: I'm not sure we can assume customers won't do this. See issue here, which notes customers sometimes intentionally test on-prem before moving to the cloud: elastic/kibana#101830

We had a ticket that was tracking this request prior, but it was confusing and too minimal and was closed out before the actual need was met. if helpful for reference:
#25940 (comment)

• see a note from PM citing we do not fully support it now, though it may work in some lesser complex scenarios currently.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[mukeshelastic]

@EricDavisX I am struggling to understand what is being asked here. Can you update the description to clearly articulate the ask with customer evidence and the benefits it will bring? The current description of the issue makes it hard to triage and determine whether we should be acting on this or not.

[EricDavisX]

I'm following up on an email thread that was started on this, just fyi for all.

[EricDavisX]

I cleaned up the description.

Here is a brief summary of post from email:
It is possible this works for some use cases. If a user is comfortable shutting down the hosted 'Fleet Server' in cloud, then self-managed Fleet Servers should work fine.

The Round-Robin connection approach prevents some explicit usage, or at least prevents it without some messiness and log errors, but it doesn't mean the desired setups cannot be implemented.

Still, because it is not clean and not in a position to be documented, and requires work-arounds, it isn't truly ready to be cited as 'supported' @ruflin 's citation. So I'll leave this open for now.

---

From a technical side:
...the reason this is not supported today is that the "Fleet URL" that is set in Kibana is global and there is no way to set [protocol, if different, and cert usage] per policy. Adding the ability to set it up per policy would open the door to solving this use case.

Cloud Elastic Agent policy (fleet_server integration) - Pointing at Cloud proxy
Default policy - Pointing at Cloud proxy

On-prem Fleet Server policy (fleet_server integration) - Pointing at self on-prem
Default on-prem policy - Pointing at self on-prem

[nicpenning]

Hey all, I will chime in on this.

Today we use Logstash in the DMZ and on the internal network.

Assets that live out in the world not on our network can still securely connect via Mutual TLS authentication to our DMZ Logstash Node.

Assets the live inside the network can connect directly to the much more robust load balanced Logstash nodes and cannot access the DMZ.

Managing the internal assets are fairly easy as no additional firewall holes must be created. The external assets however need to be managed and maintained assuming they don't VPN into the network so they need to reach a cloud or other Fleet server to receive the latest updates.

However, if there was a single cloud fleet server then we would need to expose our internal network to a cloud instance rather than a DMZ which seems to be doable but at the expense needing to manage a cloud service instead of a local deployment on hardware we control.

Just thinking out loud, but would it cause more latency to have our Kibana (hosted internally) to connect to Fleet in the cloud and then back to internal again for the thousands of agents we will have?

I am just wondering about performance for 10K+ agents on prem (and the <1K agents externally) when forcing things to the cloud instead of leveraging our on prem network yet satisfying the requirement to update and manage agents outside of our network.

Speaking with experience of 10K Winlogbeat agents, 3 Logstash nodes, and 4 Elasticsearch nodes all hosted on premise. No cloud deployment experience.

I am not opposed to a new architecture, but need to figure out the direction Elastic is heading with the scaling fleet servers and handling multiple environments.

[ruflin]

@nicpenning There are various way here on how you could build this for your need. First, in the future you will be able to likely configure multiple outputs in Fleet. You could manage your local Elastic Agents with your local Elastic Stack inside your DMZ but ship the data outside. Alternative you can keep doing what you are doing now just with the standalone Elastic Agent and not use Fleet. For the Kibana part, this should not matter where it runs. fleet-server does not connect to Kibana, Elasticsearch only.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!