Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime APM UI index configuration #39741

Closed
graphaelli opened this issue Jun 26, 2019 · 35 comments
Closed

Runtime APM UI index configuration #39741

graphaelli opened this issue Jun 26, 2019 · 35 comments
Assignees
Labels
design Team:APM All issues that need APM UI Team support

Comments

@graphaelli
Copy link
Member

graphaelli commented Jun 26, 2019

Describe the feature:

The APM UI currently relies on kibana.yml settings for configuration of indices to use for APM data. This is a request to allow configuration after kibana has started, similar to what is available for the logs and infra uis.

We may also consider making this space-specific, so that different spaces can be set up to use different APM indices. That may simplify future work regarding spaces for the APM UI.

@graphaelli graphaelli added :apm Team:APM All issues that need APM UI Team support labels Jun 26, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/apm-ui

@graphaelli graphaelli removed the :apm label Jun 27, 2019
@graphaelli
Copy link
Member Author

fyi @tbragin

@formgeist
Copy link
Contributor

@elastic/apm-ui Could use some help understanding the implications of configuring the default indices in this way. What would need to happen for us to be able to support it? I'm currently considering where these settings might sit within Kibana, and will also be investigating how users might be able to configure these settings per space in Kibana.

@tbragin
Copy link
Contributor

tbragin commented Jul 8, 2019

@graphaelli @formgeist We could start with Kibana Advanced Settings, and not make these space-specific? Similar to the following SIEM setting.

Screen Shot 2019-07-07 at 10 41 20 PM

Alternative would be to put this into the APM app itself, like we did with the Logs and Infra UI source configuration. I believe these settings are space-specific.

Screen Shot 2019-07-07 at 10 46 42 PM

@roncohen @alvarolobato What do we currently default to when selecting whether to make new functionality we build in Observability apps Kibana Space aware or not?

cc @nehaduggal

@sorenlouv
Copy link
Member

sorenlouv commented Jul 8, 2019

The APM UI currently relies on kibana.yml settings for configuration of indices to use for APM data. This is a request to allow configuration after kibana has started

As I understand this proposal the default settings should still come from kibana.yml but be changable at runtime? So if Kibana is restarted it will go back to kibana.yml settings? Or will the runtime changes be persisted somewhere?

Perhaps this overriding behaviour should be generic and built-in to Kibana. So any setting specified in kibana.yml can be changed at runtime, and not only specific APM settings.

@graphaelli
Copy link
Member Author

As I understand this proposal the default settings should still come from kibana.yml but be changable at runtime?

Yes, at least initially for backwards compatibility in 7.x.

So if Kibana is restarted it will go back to kibana.yml settings? Or will the runtime changes be persisted somewhere?

I didn't originally dive into the implementation details but a look at infra-ui shows it stores the configuration as a saved object per space - /api/saved_objects/infrastructure-ui-source/$space. The object is created only when the configuration is updated.

@sorenlouv
Copy link
Member

@katrin-freihofner We have decided to move forward with this so I've moved it to the design board. There are probably other higher priority items so just whenever it makes sense for you to look into.

@katrin-freihofner
Copy link
Contributor

katrin-freihofner commented Aug 22, 2019

If we decide for a per app settings page this could be solved by: #40236

@sorenlouv
Copy link
Member

I'm probably missing something but how is this issue related to #40236?
#40236 is about re-designing the header, whereas this issue is about a new settings page.

@katrin-freihofner
Copy link
Contributor

@sqren sorry for the confusion. This is what I'm working on in the header issue:
Screenshot 2019-08-22 at 15 25 32

If we want a settings page per app, we could navigate to it by one of the links/tabs in this header.

@katrin-freihofner
Copy link
Contributor

I'm not sure what exactly is needed from a design perspective to get this going?

@sorenlouv
Copy link
Member

I'm not sure what exactly is needed from a design perspective to get this going?

Currently the link to "Settings" takes the user to the agent configuration page. This issue is about creating a page where the user can configure which indices to use for apm (eg. apm-*-transaction-*)

Kibana has an "Advanced Settings Page":

Screen Shot 2019-08-26 at 17 20 59

Should it look like that? LMK if you want to zoom about this.

@katrin-freihofner
Copy link
Contributor

katrin-freihofner commented Aug 27, 2019

As discussed via zoom with @sqren we need an APM specific settings page where the following can be configured:

apm_oss.sourcemapIndices
apm_oss.errorIndices
apm_oss.onboardingIndices
apm_oss.spanIndices
apm_oss.transactionIndices
apm_oss.metricsIndices
apm_oss.apmAgentConfigurationIndex

# additional ones we might consider:
xpack.apm.ui.maxTraceItems
xpack.apm.ui.transactionGroupBucketSize

@sorenlouv
Copy link
Member

sorenlouv commented Aug 27, 2019

@katrin-freihofner Sounds good 👍
Btw. I updated your list with a few more settings we may consider.

@sorenlouv
Copy link
Member

sorenlouv commented Sep 2, 2019

Since all the fields in "UI settings" are for indicies, we might call it "Index settings" instead.

@katrin-freihofner
Copy link
Contributor

Should we additionally provide a Reset defaults option?

@sorenlouv
Copy link
Member

Should we additionally provide a Reset defaults option?

This would just clear all fields, right? So I'm not sure how useful that'll be.

@katrin-freihofner
Copy link
Contributor

Yes. I guess it is fine for now - if we get more and more settings it might be helpful.

@bmorelli25
Copy link
Member

Since "settings" is used in the top level nav, I think we should avoid using it in the sub-headings.
Screen Shot 2019-09-03 at 8 06 58 AM

What about something like "APM indices", "Index patterns", or even just "indices" instead?

Settings
- Indices
- Agent configuration

@sorenlouv
Copy link
Member

Makes sense to avoid "settings".
Conceptually I like "index patterns" but since Kibana has something (very different) also called "Index patterns" we should probably stick to "Indices" or "UI indices".

@katrin-freihofner
Copy link
Contributor

Design update:

  • text updates according to discussion
  • navigation update (removed the settings tab to fit the currently implemented approach)
    vertical

@bmorelli25 could you please help me with the intro text on the top and the hints under each input field - I'm pretty sure we can do better than my mockup shows ;)

@zube zube bot removed Team:APM All issues that need APM UI Team support design enhancement New value added to drive a business result v7.5.0 labels Sep 5, 2019
@bmorelli25
Copy link
Member

bmorelli25 commented Sep 5, 2019

Here's a starting point for a discussion on the intro text and input hints. @katrin-freihofner

The APM UI uses index patterns to specify the location of your APM indices.
If you've customized the index names that APM Server writes events to,
you may need to update these patterns for certain features in the APM UI to work.

Index pattern for matching indices that contain **source map** data. Overwrites `apm_oss.sourcemapIndices`
Index pattern for matching indices that contain **error** data. Overwrites `apm_oss.errorIndices`
Index pattern for matching indices that contain **onboarding** data. Overwrites `apm_oss.onboardingIndices`
Index pattern for matching indices that contain **span** data. Overwrites `apm_oss.spanIndices`
Index pattern for matching indices that contain **transaction** data. Overwrites `apm_oss.transactionIndices`
Index pattern for matching indices that contain **metrics** data. Overwrites `apm_oss.metricsIndices`
Index pattern for matching indices that contain **Agent configuration** data. Overwrites `apm_oss.apmAgentConfigurationIndex`

@sorenlouv
Copy link
Member

Thanks @bmorelli25. Building on top of that I'll also have a go:

  • "location of your APM indices": Location makes me think of a file path or similar
  • "for certain features in the APM UI to work": Nothing in APM UI will work if the index names has been changed beyond the matcher apm-*.
  • "Index pattern for matching indices that contain": feels a little long. Maybe trim it?
  • "Overwrites": has the connotation of being irreversible (and scary) even though the user is able to remove their value in the ui and then it falls back to the original value again. This temporary aspect makes me think "override" is a better term

Suggestion

The APM UI uses index patterns to query your APM indices. If you've customized the index names that APM Server writes events to, you may need to update these patterns for the APM UI to work.

**source map indices**. Overrides `apm_oss.sourcemapIndices`
**error indices**. Overrides `apm_oss.errorIndices`
**onboarding indices**. Overrides `apm_oss.onboardingIndices`
**span indices**. Overrides `apm_oss.spanIndices`
**transaction indices**. Overrides `apm_oss.transactionIndices`
**metrics indices**. Overrides `apm_oss.metricsIndices`
**Agent configuration indices**. Overrides `apm_oss.apmAgentConfigurationIndex`

@bmorelli25
Copy link
Member

Nice, thanks @sqren! Can we just remove the bolded "source map indices", "error indices", etc., since they match the text above the input?

We could also mention kibana.yml and link to the documentation page. Something like this might work (where kibana.yml is the link).


The APM UI uses index patterns to query your APM indices. If you've customized the index names that APM Server writes events to, you may need to update these patterns for the APM UI to work. Settings here take precedence over those set in kibana.yml.

Source map indices
┌────────────┐
└────────────┘
Overrides apm_oss.sourcemapIndices.

Error indices
┌────────────┐
└────────────┘
Overrides apm_oss.errorIndices.

etc.


@sorenlouv
Copy link
Member

sorenlouv commented Sep 6, 2019

Good idea with link to kibana.yml docs 👍

It's a wrap from here.

@katrin-freihofner
Copy link
Contributor

This leads us to:
vertical

Let me know what you think!

@sorenlouv
Copy link
Member

@katrin-freihofner LGTM 👍

One thing: when we read the value from kibana.yml it's not possible to see if it's set or not because there is a default value if it's not set (apm-*). So we'll always get a value back.

@katrin-freihofner
Copy link
Contributor

Ok, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design Team:APM All issues that need APM UI Team support
Projects
None yet
Development

No branches or pull requests

9 participants