Do not ignore vars when there are no inputs

[jalvz]

Consider this snippet of an 0.1.0/manifest.yml example:

policy_templates:
  - name: apm
    inputs:
      - type: traces
        vars:
          - name: host
            type: int
            title: APM Server port
            required: true
            show_user: true
            default: 8200

If my data stream(s) don't have a streams key, the generated policy will just have an input with streams: [], and the variable host will not be there even if I had filled it in the UI.

For apm-server we don't have streams, so we need that the vars defined at the policy level get picked up and rendered in the generated policy.

Not sure on which repo this issue belongs, let me know if it should be transferred to Kibana.

Update: The ideal way forward is that the top level config shows right under inputs, and not under a streams entry. For this we need hbs templates at the top level too.

[ruflin]

@jen-huang As most of these changes need to happen in Kibana, would be great to get your input on that.
@jalvz I guess the current spec allows the above? If yes, sounds like the spec is fine, it is just the missing support in Kibana?

Currently all the templates are living inside the agent/stream directory. I guess for the above we need to have support for this also on the top level? It might be better to call it there agent/input as I guess it is a template for the full input?

[jalvz]

I guess the current spec allows the above?

If you mean if it validates, yes it does.

Currently all the templates are living inside the agent/stream directory. I guess for the above we need to have support for this also on the top level?

Good point; but IIUC that wouldn't ever work without major changes... The apm-server spec expects an input with type:apm in https://github.com/elastic/beats/blob/master/x-pack/elastic-agent/spec/apm-server.yml.disabled#L8-L11

I don't know how else we could instruct Elastic Agent to spin up an apm-server process and pass it that config. Not only that, but we also need the fields.yml to generate docs, and currently that is only per input as well.

So it doesn't look like we can just omit the input; rather Kibana should be able to create an input (for apm-server in this case) silently behind the curtains.

Maybe the way forward is something similar to the proposal in #69: define a hidden property on the input, such that if hidden:true, Kibana will not show it, but will still add it to the policy. Thinking something like this:

0.1.0/manifest.yml

policy_templates:
  - name: apm
    title: Elastic APM Integration
    inputs:
      - type: apm
        title: Collect application traces
        force_enabled: true    # don't display toggle switch 
        vars:
        ... 

0.1.0/data_stream/apm/manifest.yml

title: APM data stream
type: traces
streams:
  - input: apm
    enabled: true
    hidden: true              # don't display the input
    title: (It doesn't matter, no one will see)

Will produce the policy:

elastic-agent.yml

inputs:
  - id: <randomly-generated>
    name: <user-defined>
    type: apm
    meta:
      package:
        name: apm
        version: 0.1.0
    data_stream:
      namespace: <user-defined>
    streams:
      - id: <generated>
        data_stream:
          dataset: <generated>
          type: traces
        apm-server:   # this would come from the hbs template
        ...

WDYT?

[ruflin]

I think I don't get why there is streams in 0.1.0/data_stream/apm/manifest.yml. My assumption is also that streams does not exist in the final apm input config? Can you share a full config example on how it should look in the end? Not what you think the packages can create today, but your idea format? BTW endpoint is in a similar situation but they don't have any vars and create part of it in their UI: https://github.com/elastic/package-storage/blob/production/packages/endpoint/0.16.0/manifest.yml#L13

[jalvz]

Yes, your assumption is correct. Without that limitation, a full config for APM I think should look like this:

id: dac0e3c0-19c8-11eb-9ffd-b9afa78c7e76
revision: 1
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
agent:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
inputs:
  - id: bda84570-19ec-11eb-8f6d-192e8cf5c9dd
    name: apm-integration
    revision: 1
    type: apm
    use_output: default
    meta:
      package:
        name: apm
        version: 0.1.0
    data_stream:
      namespace: default
    apm-server:
       rum.enabled: true
       # more specific apm-server settings
fleet:
  kibana:
    protocol: http
    hosts:
      - 'localhost:5601'

I will update the original issue 👌

[ruflin]

Great, lets make sure we can make this possible.

@jen-huang Currently when building a config, we rely on agent/stream files. I was thinking to introduce a global agent/input/template.hbs file which can be used here as no streams are needed but only an input. Thoughts?

[jalvz]

Kibana issue

[jen-huang]

I was thinking to introduce a global agent/input/template.hbs file which can be used here as no streams are needed but only an input.

@ruflin Makes sense to me. Thanks for filing the Kibana issue @jalvz.