GET /package/${pkgName}/${pkgVersion} does not return policy_template.vars for input packages

[hop-dev]

I am using the custom_logs-1.0.0 example input package to develop the Fleet side of input packages work: elastic/kibana#137750

When retrieving the package from the 8.3.3 package registry (also tried main) the GET /package/${pkgName}/${pkgVersion} endpoint, policy_template.vars is not returned, here is a relevant snippet from the manifest.yml:

format_version: 1.0.0
name: custom_logs
title: Custom Logs
description: >-
  Read lines from active log files with Elastic Agent.
type: input
version: 1.2.0-beta
license: basic
categories:
  - custom
policy_templates:
  - name: first_policy_template
    type: logs
    title: Custom log file
    description: Collect your custom log files.
    input: logfile
    template_path: input.yml.hbs
# vvvvvvvvvv THIS IS NOT RETURNED vvvv
    vars:
      - name: paths
        type: text
        title: Paths
        multi: true
        required: true
        show_user: true
      - name: tags
        type: text
        title: Tags
        multi: true
        required: true
        show_user: false
      - name: data_stream.dataset
        type: constant_keyword
        description: Data stream dataset.
        required: true
        show_user: true
      - name: ignore_older
        type: text
        title: Ignore events older than
        required: false
        default: 72h

Here is the policy_templates in the API response:

"policy_templates": [
            {
                "name": "first_policy_template",
                "title": "Custom log file",
                "description": "Collect your custom log files.",
                "multiple": true,
                "type": "logs",
                "input": "logfile",
                "template_path": "input.yml.hbs"
            }
        ],

[jsoriano]

Hey @hop-dev,

Kibana should have access to the manifest of installed packages, could this information be obtained from there?

We wouldn't like to expose additional information on registry endpoints that is not needed for discovery and retrieval of packages.

See elastic/kibana#115032.

[hop-dev]

@kpollich @jsoriano we could obtain this information from the manifest and I think this is the way we want to go (now or eventually) with elastic/kibana#115032, but it is quite a big change and would delay the delivery of input packages.

@kpollich I'll try and put something in tomorrow to have a chat about what direction to take here 📆

[kpollich]

@hop-dev - Would it be possible to maybe add a "one-off" call during the package installation process that fetches just this piece of data from the manifest, maybe even just for input packages? That way we don't have to commit to a substantial refactor in order to support this one piece of data?

[hop-dev]

I have been thinking about this, I agree we should move to getting as much info as possible from the package its-self, however we need the vars to render the package policy creation UI before package install time. It would be a shame to have to unzip the archive to render the package policy UI.

Could we consider having these as part of the registry API @jsoriano, so the principal would be that the API provides all info to search for packages and the variables taht are needed to install them?

[jsoriano]

I have been thinking about this, I agree we should move to getting as much info as possible from the package its-self, however we need the vars to render the package policy creation UI before package install time. It would be a shame to have to unzip the archive to render the package policy UI.

The way I think about this is that when Kibana is going to install a package, it should already have it, and if Kibana has it, it can obtain any information from it.

There are some reasons to follow this model:

• It makes package installation independent of the source of the package. It doesn't matter if the package is bundled in Kibana, uploaded by the user, downloaded from epr.elastic.co or from any other registry, installation works always the same. There is a separation of concerns between the source of the package and its installation.
• Kibana validates the package before doing anything with it. For that, it should be downloaded and its signature validated. From this point, the validated package should be the single source of trust. If not, we could be rendering something while ending up installing a different thing.
• The package registry and all related infra for indexing packages doesn't need to be updated with any new feature supported in the package spec. Their implementations regarding package parsing can be minimal, focused only on its purpose of helping on the discovery and distribution of packages.

It would be a shame to have to unzip the archive to render the package policy UI.

Think it in a different way: what would be to have to make a request to a remote service to obtain something that you already have locally?

Could we consider having these as part of the registry API @jsoriano, so the principal would be that the API provides all info to search for packages and the variables taht are needed to install them?

We could do this as a workaround to unblock this, as we have done in the past. But this has some implications:

• It forces us to extend an API and continue maintaining this long term when we actually want to reduce it.
• It can delay more the implementation of [Fleet] Rely on the content of a package when installing it kibana#115032 and [Fleet] Allow to install a package without the registry. kibana#70582, as there will be another thing depending on the registry API for package installation.

I would prefer not having to continue adding this kind of settings to the API.

[jsoriano]

Btw, Marcin reminded me that with the storage v2, extending the registry API to include more information in packages also implies:

• Modifying the indexing jobs.
• Reindexing all available packages.

This won't scale. The sooner we do elastic/kibana#115032 the better. @kpollich @jen-huang, it'd be great if we can prioritize this.

[kpollich]

Great point about having to modify all the indexing jobs to generate package archives, @jsoriano and @mtojek - I agree that this isn't a solution that will scale.

I'd like to make elastic/kibana#115032 a high priority item in 8.6 as it will unblock multiple cross-team efforts as well as Fleet's support for input packages.

[hop-dev]

We opted to get this info from the package manifest (in the archive) for input packages only, see elastic/kibana#140035.