[Fleet] Add package information to _meta on Elasticsearch assets

[joshdover]

Ingest assets installed into Elasticsearch by Fleet do not all contain information about the package that was installed. We should add this to be sure that all assets are displayed as managed and are tied to the appropriate package that created them.

• Index templates
  • Current:

    "_meta" : {
      "package" : {
        "name" : "system"
      },
      "managed_by" : "fleet",
      "managed" : true
    },

  • Should add:
    • nothing
• Component templates
  • Current:

    "_meta" : {
      "package" : {
        "name" : "elastic_agent"
      }
    }

  • Should add:
    • managed
    • managed_by
• Ingest pipelines
  • Current status: no meta data at all
  • Should add all fields
  • Elasticsearch blocker: Add support for _meta field to ingest pipelines elasticsearch#75905
• ILM policies
  • Currently we do not add any _meta to policies from packages, only to the default Fleet policy
  • Should add all fields
  • Note: only the Elastic APM integration currently ships an ILM policy
• Transforms
  • Currently we do not add any _meta to policies from packages, only to the default Fleet policy
  • Should add all fields
  • Note: only the Elastic Endpoint integration currently ships a transform
  • Elasticsearch blocker: Add support for _meta field on transforms elasticsearch#77506

Adding package.version

We don't have any current plans to leverage this field for upgrades, but it potentially may be useful for rollbacks or general bookkeeping and debugging. In theory, we should tell Elasticsearch to install all objects but as seen in #110500, there are some performance challenges we need to solve for this. By adding the version information, we may be able to optimize this from the Kibana side in the meantime.

Decision: for now let's not do this

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

I had mistakenly assumed that the fields in _meta on index templates, etc. were being supplied by the package installation code in Kibana, however it seems they’re coming directly from packages since Kibana does not modify the assets currently.

Can anyone anticipate any problems with populating this data from Kibana rather than from the packages? cc @ruflin @mtojek

[ruflin]

The Fleet code builds the index template and component templates out of the fields.yml. So I also assume the meta info is added by Fleet. What do I miss here? In any case, agree that Fleet should add the data.

[amolnater-qasource]

Hi @joshdover
We have validated this on latest 8.0 Snapshot.
Build details:
Build: 48703
Commit: a739f06

Please find our observations below:

• Index templates:
  

• Component templates:
  

• Ingest Pipelines
  

Could you please confirm above results?
Further, could you please share details for ILM policies and transform if any testing is required.

Thanks

[dikshachauhan-qasource]

Hi @joshdover

While performing testing on related PR at 8.0 snapshot build, we have observed following :

1. No '_meta' field is provided at Index and component templates.
   Query: Here in the ticket description data is provided under _meta braces. However at UI a column is provided for metadata under which same data is shared. Is the changes at UI is working as expected.

Screenshot:

2. However, 'managed' and 'managed_by' fields are provided for installed integrations at both Index and component templates.
3. Further, for ILM policies, we have noticed that in summary it is mentioned :

ILM Policies:
Currently we do not add any _meta to policies from packages, only to the default Fleet policy
Should add all fields
Note: only the Elastic APM integration currently ships an ILM policy

However, we have observed that _meta field with managed fields was available for endpoint ILM policies too.

We have validated that, APM policy includes all fields.

However, Fleet default policy did not included all fields like managed and managed by. Could you please confirm if it is expected.

Please let us know if we are missing anything to validate here.

Thanks
QAS

[joshdover]

@dikshachauhan-qasource Apologies for the confusing issue description. The screenshots you share show the functionality that I expect. All objects we install should have these meta properties.

You did catch the .fleet-actions-results-ilm-policy which is missing the managed_by: fleet, but I believe this is actually installed by Elasticsearch. I'm less worried about it since it is more explicitly named as a Fleet object, but we should get it updated too. Could you also check any other Fleet installed assets that aren't part of packages, such as .fleet_component_template-1 and .fleet_final_pipeline (not sure if name is exactly correct). We should file an issue for updating those separately.

[dikshachauhan-qasource]

Hi @joshdover

Thanks for the feedback.

We have validated the above mentioned templates on 8.0 snapshot build and found only one as working fine.

Screenshots:
.fleet_component_template-1:

For the .fleet_final_pipeline-1 its not working fine and we have reported issue at : #121854

Further, we have created below test case for same:

• C164219

Please review and let us know if we are missing anything else to mention in it.

Thanks
QAS

[joshdover]

@dikshachauhan-qasource Looks great, thanks.