Preconfiguration API behavior for missing package policies

[simitt]

Using the preconfiguration API to set up agent policies results in an empty agent policy, if the package registry cannot be reached. If it eventually is reachable, the agent policy is not updated, also not on Kibana startup.

This potentially leaves agent policies in a half-baked state, that is not automatically recoverable.

There's probably a bunch of questions to answer if policies from the preconfig API need to update existing agent policies, but can we discuss comparing preconfigured agent policies with installed ones and add missing package policies (not updating existing ones if they differ)?

cc @jen-huang @joshdover @ruflin

[elasticmachine]

Pinging @elastic/fleet (Feature:Fleet)

[joshdover]

@nchaulet this seems like something we should be able to handle. We shouldn't have any of the fleet-preconfiguration-deletion-record objects around for these, so I think this is something we can achieve.

The challenge is deciding when to execute this retry, which will be something that should be tackled as part of #111859

[nchaulet]

@nchaulet this seems like something we should be able to handle. We shouldn't have any of the fleet-preconfiguration-deletion-record objects around for these, so I think this is something we can achieve.

Yes this something we can achieve, I have been digging a little more in our code and there probably a few things that can lead to error in the way the preconfigure service:

• If a policy is managed, we update the agent policy fields (not the associated package policy we should probably change that)
• We have a bug: we do not use the is_managed flag from the config file for the previous feature only the one from the saved object, this cause the previous point to doesn't work correctly as we first create the policy saved object without the is_managed flag

[joshdover]

Seems like we need to audit the existing behavior of preconfiguration, including how it handles failures and kibana.yml config changes, so we can determine a consistent model that should meet all of these types of requirements. I suspect this might be better than patching one thing at a time?

[simitt]

As discussed with @nchaulet async, this is important for the cloud setup for 8.0 for the apm-server. The cloud setup is in managed mode, so adding the missing packages would only be required for the managed policy. Is this doable for 8.0?

[nchaulet]

@simitt I have a draft PR that does that. I will work on having on solid test here but we should be able to land this for 8.0 #119488

[joshdover]

Test instructions:

1. Start an on-prem 8.1 Kibana and ES cluster, add this to the kibana.yml config before starting Kibana:

xpack.fleet.packages:
  - name: fleet-server
    version: latest
  - name: apm
    version: latest
xpack.fleet.agentPolicies:
  # Cloud Agent policy
  - name: Elastic Cloud agent policy
    id: policy-elastic-cloud
    description: Default agent policy for agents hosted on Elastic Cloud
    is_default: false
    is_managed: true
    is_default_fleet_server: true
    namespace: default
    monitoring_enabled: []
    package_policies:
      - name: Fleet Server
        package:
          name: fleet_server
        inputs:
          - type: fleet-server
            keep_enabled: true
            vars:
              - name: host
                value: 0.0.0.0
              - name: port
                value: 8220

2. Go to Fleet app and wait for UI to load
3. Stop Kibana
4. Change the kibana.yml to use these settings:

xpack.fleet.packages:
  - name: fleet-server
    version: latest
  - name: apm
    version: latest
xpack.fleet.agentPolicies:
  # Cloud Agent policy
  - name: Elastic Cloud agent policy
    id: policy-elastic-cloud
    description: Default agent policy for agents hosted on Elastic Cloud
    is_default: false
    is_managed: true
    is_default_fleet_server: true
    namespace: default
    monitoring_enabled: []
    package_policies:
      - name: apm-cloud-123
        package:
          name: apm
      - name: Fleet Server
        package:
          name: fleet_server
        inputs:
          - type: fleet-server
            keep_enabled: true
            vars:
              - name: host
                value: 0.0.0.0
              - name: port
                value: 8220

5. Start Kibana and go to Fleet app
6. Go to Elastic Cloud policy and verify that an APM integration policy was added

[amolnater-qasource]

Hi @joshdover
We have revalidated this on self-managed 8.1 BC-4 and had below observations:

• We added the required configuration to kibana.yml, however we get below error on logging in to kibana.
  

Could you please confirm if we need to create any policy(manually) for this?

Build details:
VERSION: 8.1.0
BUILD: 50428
COMMIT: 015578b

Please let us know if we are missing anything here.
Thanks

[joshdover]

@amolnater-qasource Apologies, I've updated the test instructions above to include the id

[amolnater-qasource]

Hi @joshdover
Thanks for updating the configuration, we revalidated this with the updated configuration on 8.1 BC-6 self-managed environment.

Build details:
BUILD: 50485
COMMIT: 4aaeda2
Artifact Link: https://staging.elastic.co/8.1.0-6b89df21/summary-8.1.0.html

We observed that we are still getting some errors, due to which we are unable to access Fleet tab.
There should be something that we are still missing in configuration.

Screenshots:

We tried to troubleshoot, however didn't get any success.
Could please look into the same?

Thanks!

[joshdover]

@amolnater-qasource Good catch, these need to be updated for the removal of default packages. I've updated the configs above again to include the fleet_server package.

[amolnater-qasource]

Hi @joshdover
Thank you for updating the configuration. We have revalidated this on 8.1 BC-6 self-managed environment and found it working fine.

Steps followed:

1. Started self-managed kibana with below config in kibana.yml.

xpack.fleet.packages:
  - name: fleet_server
    version: latest
  - name: apm
    version: latest
xpack.fleet.agentPolicies:
  # Cloud Agent policy
  - name: Elastic Cloud agent policy
    id: policy-elastic-cloud
    description: Default agent policy for agents hosted on Elastic Cloud
    is_default: false
    is_managed: true
    is_default_fleet_server: true
    namespace: default
    monitoring_enabled: []
    package_policies:
      - name: Fleet Server
        package:
          name: fleet_server
        inputs:
          - type: fleet-server
            keep_enabled: true
            vars:
              - name: host
                value: 0.0.0.0
              - name: port
                value: 8220

2. On Fleet>Agent Policies> Elastic Cloud agent policy is available with only Fleet Server integration.
3. Stopped kibana and added updated configuration:

xpack.fleet.packages:
  - name: fleet_server
    version: latest
  - name: apm
    version: latest
xpack.fleet.agentPolicies:
  # Cloud Agent policy
  - name: Elastic Cloud agent policy
    id: policy-elastic-cloud
    description: Default agent policy for agents hosted on Elastic Cloud
    is_default: false
    is_managed: true
    is_default_fleet_server: true
    namespace: default
    monitoring_enabled: []
    package_policies:
      - name: apm-cloud-123
        package:
          name: apm
      - name: Fleet Server
        package:
          name: fleet_server
        inputs:
          - type: fleet-server
            keep_enabled: true
            vars:
              - name: host
                value: 0.0.0.0
              - name: port
                value: 8220

4. Rerun kibana.bat and Elastic Cloud agent policy is now available with Fleet Server and Elastic APM integrations.

Build details:

BUILD: 50485
COMMIT: 4aaeda23aea9c3bf29698878c70a0107ea3c1659
Artifact Link: https://staging.elastic.co/8.1.0-6b89df21/summary-8.1.0.html

Screenshots:

Hence marking this as QA: Validated.
Thanks!