Add "network.publish_host" style configuration in Kibana

[mikecote]

There is a requirement in Alerting to create links on the server side that point to Kibana. These links would be used inside emails (and different types of actions) that allow users to click and get redirected to a section within Kibana. We could create our own configuration under xpack.alerting but seems better to have as a platform level configuration.

The comparable in Elasticsearch would be the network.publish_host configuration https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings.

As suggested by @joshdover, there could also be a core.http.buildUrl() function to help build URLs that considers network.publish_host, basePath and the path within kibana.

[elasticmachine]

Pinging @elastic/kibana-platform

[pmuellr]

I'd guess this is obvious, but such a core.http.buildUrl() function should be able to consider a space as well.

[kobelb]

I'm not opposed to us adding a centralized kibana.yml setting that can be used to build absolute URLs in Kibana. However, we've historically had issues with users misconfiguring a similar setting for Reporting, xpack.reporting.kibanaServer. Users commonly misconfigure these settings, causing their report generation to fail, and it generally requires multiple iterations to get these settings right. At a minimum, when the user opens Kibana in their browser, we should notify them when the kibana.yml settings don't match the URL which we see them using to access Kibana.

I do question the premise that a kibana.yml setting is required to be able to generate absolute URLs for Alerting. When an Alert/Action is created in the UI, it's possible to use the browser APIs to determine the protocol, host, and port where the end-user is accessing Kibana, which can be used to build these absolute URLs.

[pmuellr]

I understand this sort of setting is problematic, but it's also a fairly standard setting for apps which need to generate links back to themselves.

I like the idea of "self-checking" in the browser, but don't think it should be the mechanism to explicitly set it. To suggest it, to compare against the current setting, yes, nice! I suspect we'll run into problems if we try to set it automagically like this.

It also feels like this is bigger than alerting/actions - some other app/solution/plugin will need it eventually. We could also extend the URL shortening API to include the full URL, if we had this.

I have been wondering what devs should do if this URL is not set, when we want to create a URL. Was thinking of having a doc page specifically targeted for this case: "you don't have your publish URL set yet, here's how you can set it".

[kobelb]

I like the idea of "self-checking" in the browser, but don't think it should be the mechanism to explicitly set it. To suggest it, to compare against the current setting, yes, nice! I suspect we'll run into problems if we try to set it automagically like this.

Agreed.

I spent some more time thinking about how this could work, the lessons we can learn from Reporting, and some of the other options we can consider.

Apologies for the wall-of-text which follows... If you want to skip to the solution which I've gravitated towards, it's Solution 2. Add kibana.yml settings. During prior discussions, I've pushed for Solution 1. Use information from the 'client' to build the URL, but after further consideration, there are some drawbacks which are noted below.

Problem

Generally, most functionality in Kibana works perfectly fine with relative URLs, which exclude the protocol, hostname, and port. However, there are certain situations where an absolute public URL is required.

Reporting spawns a headless browser that navigates to a URL to take a screenshot. In some configurations, Reporting can just navigate to http://localhost:5601/app/foo to generate the report. However, when Kibana is hosted behind a reverse-proxy performing TLS termination or implementing a "base path", Reporting must use the "public absolute URL" and can't use localhost.

Various Actions would like to also use the absolute public URL so that they can include links back to Kibana. These links will appear in mediums like e-mail, so they can't use relative URLs.

There are future features that likely require or at least benefit from absolute public URLs also. For example, Slack integration. Based on the architecture of the Slack integration, there might be a way around needing Kibana itself to build the absolute public URL, but this is purely speculation at this point.

Reporting as a cautionary tale

Before discussing possible solutions, there's a lot we can learn from Reporting. Reporting utilizes xpack.reporting.kibanaServer.* settings specified in the kibana.yml to build the absolute public URL, and it's been a constant source of problems. It's common for users to not have to specify any of the xpack.reporting.kibanaServer.* settings initially to generate their reports successfully. However, as soon as they put Kibana behind a reverse-proxy to implement high-availability, TLS or a base-path, Reports will start failing and it's not immediately obvious what changed to cause these failures.

By default, Reporting uses the protocol, hostname and port which Kibana's HTTP server is listening, but allows these values to be overridden by disparate settings:

• xpack.reporting.kibanaServer.protocol
• xpack.reporting.kibanaServer.hostname
• xpack.reporting.kibanaServer.port

The most common configuration mistake that I've seen is when the user doesn't realize that Kibana uses the non-standard port of 5601, and sets xpack.reporting.kibanaServer.protocol: https which doesn't drop the port when building the absolute URL, and they must also set xpack.reporting.kibanaServer.port: 443. A similar misconfiguration also occurs when Kibana is behind a reverse-proxy which uses the standard http port, and they must also set xpack.reporting.kibanaServer.port: 80

When this misconfiguration does occur, an opaque error is thrown and the user is not provided any instructions on how to remedy the error.

Solutions

1. Use information from the "client" to build the URL

In a majority of situations, it's possible to use information from the "client" to determine the missing pieces to build an absolute public URL. For example, when the user creates an Alert using the UI it's possible to use the browser's window.location to determine the public protocol, host, and port and then later use this when constructing the URL. It's also theoretically possible to do this with the Slack integration. The Slack integration will need to know how to communicate with Kibana's API, which is very likely to be the absolute public URL of Kibana, so it's then possible to translate from relative URLs to absolute public URLs.

There are some caveats though. Kibana can't safely use the information provided by the client in all situations because there's potential that this could be abused to perform a server-side request forgery. In the situation of Reporting, Reporting uses a headless browser to visit the absolute URL. If we used the values from window.location, communicated via a Kibana API, an attacker could use this to make the headless browser to visit an internal network address and take a screenshot.

This approach can also be quite complicated, as the protocol, host and port have to be available to the consumers which want to build the full URL, and we have to pass these values all the way through from the client to the portion of the code that needs to consume them. For Alerting, these are asynchronous background jobs that are building the full URLs, so they're stored in an Elasticsearch document, and it's possible that the protocol, host and port changed since the last time they were stored.

There are also some potential situations where there isn't a "client" initiating the operation which creates the absolute public URL. For example, Alerting would like to have an Alert that comes pre-packaged with Kibana and sends an e-mail using values purely from the kibana.yml. There's potential this could be configured by Cloud making a call to Kibana's API, but we haven't required Cloud to do something like this before.

Despite all these caveats, the major benefit of this approach is that it "just works" in most situations without any user intervention and simplifies the getting started experience.

2. Add `kibana.yml` settings

I hope it's obvious, but we shouldn't copy exactly what Reporting is doing at the moment and generalize it. Instead of allowing the user to configure the protocol, hostname and port separately, I'd recommend allowing the user to specify them all together. For example: server.public_base_url: https://some-domain/

We should also display a warning notification when the user logs into Kibana and we notice a mismatch between window.location and the configured server.public_base_url with explicit instructions on what to set in the kibana.yml or in Cloud's user settings overrides to rectify the misconfiguration.

When a user first starts using Kibana and they access it at http://localhost:5601, they will not be required to set server.public_base_url. However, as soon as they begin accessing Kibana externally or put it behind a reverse-proxy, they will get a warning notification and should set server.public_base_url.

Users generally shouldn't need to configure server.public_base_url in Cloud. However, if the user is using a reverse-proxy with a specific domain name, they will be required to change this setting.

Since this is configured in the kibana.yml, it's safe to use in all situations where we need an absolute public URL, and there are no server-side request forgery concerns.

3. Add Advanced Settings

This is relatively similar to the "Add kibana.yml settings" option, but it can't be used in situations where a server-side request forgery is a concern. It is somewhat easier for the user to configure than the kibana.yml option since they don't have to modify a file on the Kibana server or go to Cloud console. However, I don't think that the benefits overcome the drawbacks and we shouldn't pursue this option, so I haven't elaborated upon it.

4. Hybrid approach with fallback

Theoretically, we could use option 2 in combination with option 1 where the server.public_base_url takes priority to the values provided by the client. However, I think this would create a rather inconsistent and somewhat confusing user experience. It would also require the most engineering effort, so I don't think we should pursue this option.

[mikecote]

After going through the considerations, I'm in favour of Solution 2, it will work nicely for being able to append a View in Kibana link to all emails sent from alerts.

[mikecote]

cc @elastic/kibana-alerting-services

[mikecote]

@elastic/kibana-platform any thoughts around this or timeline if we decide to implement something like this at the core level?

[pgayvallet]

TBH I suggested a very similar API during the globalSearch RFC, for the same need of knowing the 'public' url of the server.

Adding the property to the configuration file and adding a simple 'string getter api' for it should be rather trivial.

However,

I would like to take some time to think a little about the implications on existing APIs here. For example, we got a http.getServerInfo() API that returns the protocol, host and port, as defined in the http/ hapi configuration. If the server.public_base_url was to be configured, isn't it an issue that these two APIs would return different values?

Also,

When a user first starts using Kibana and they access it at http://localhost:5601, they will not be required to set server.public_base_url. However, as soon as they begin accessing Kibana externally or put it behind a reverse-proxy, they will get a warning notification and should set server.public_base_url

We should also display a warning notification when the user logs into Kibana and we notice a mismatch between window.location and the configured server.public_base_url with explicit instructions on what to set in the kibana.yml or in Cloud's user settings overrides to rectify the misconfiguration.

Not sure displaying that kind of warning to the 'end' user really makes sense, as it's very unlikely he's in charge of configuring the Kibana instance?

[kobelb]

TBH I suggested a very similar API during the globalSearch RFC, for the same need of knowing the 'public' url of the server.

Apologies for shooting down the suggestion then. I was wrong :hiding:

I would like to take some time to think a little about the implications on existing APIs here. For example, we got a http.getServerInfo() API that returns the protocol, host and port, as defined in the http/ hapi configuration. If the server.public_base_url was to be configured, isn't it an issue that these two APIs would return different values?

I don't necessarily think it's an issue if an API returns information about the protocol/host/port that the HTTP server is listening on, and another API returns the "public base URL", as long as the difference is clear. This is all based on the assumption that there are valid situations where we want to know the protocol/host/port that the HTTP server is listening on.

Not sure displaying that kind of warning to the 'end' user really makes sense, as it's very unlikely he's in charge of configuring the Kibana instance?

I agree that it could be frustrating to see an alert that you can't take any action on. However, there's no way to know whether or not the currently authenticated end-user is has access to modify the kibana.yml, and they generally should be able to bug their system administrator to do so. I do think that we should allow a way for the user to dismiss this alert and not see it again.

[pgayvallet]

I do think that we should allow a way for the user to dismiss this alert and not see it again.

I guess we gonna need user settings for that 😄

This is all based on the assumption that there are valid situations where we want to know the protocol/host/port that the HTTP server is listening on

We would need to check on consumers of the getServerInfo to be sure. I just know for sure that reporting is using it, probably to be used as a fallback when xpack.reporting.kibanaServer is not populated. @tsullivan could you confirm that?

@mikecote for when would you ideally need that feature?

@kobelb If there is urgency, how acceptable would it be to just add the configuration option / access API in a first step, and then add the check + notification you suggested in a follow-up (the team short term capacity is quite small).

[mikecote]

@mikecote for when would you ideally need that feature?

7.11 would be ideal for the alerting team, we would start work in parallel on our side.

[kobelb]

I guess we gonna need user settings for that 😄

😬 else-where, we store values like this in local storage because we don't have persistent user-specific storage.

@kobelb If there is urgency, how acceptable would it be to just add the configuration option / access API in a first step, and then add the check + notification you suggested in a follow-up (the team short term capacity is quite small).

I think that's a good idea.

[legrego]

I haven't exhausted all other possibilities yet, but this is potentially useful for #69411, which I'm also planning for 7.11 in order to support Drilldowns. I'd like to be able to take an arbitrary URL or URL path, and determine if the destination is external to the Kibana instance. Having a canonical public URL would make this a lot easier

[mikecote]

I started putting some code together for how alerting would use this solution. There's two use cases (look for KIBANA_ROOT usage):

• Root URL: https://github.com/elastic/kibana/pull/84371/files#diff-bbb8ade8ee893fae9c5412d41911528c0cca1e646f04b81be9f306c7080e7cb5R245
• Page specific URL: https://github.com/elastic/kibana/pull/84371/files#diff-bbb8ade8ee893fae9c5412d41911528c0cca1e646f04b81be9f306c7080e7cb5R253

[joshdover]

One issue that has been raised is that some customers have multiple addresses at which they access Kibana, one public (with IP filtering) and one private (via a VPN or similar service such as PrivateLink on AWS). If we were to add a warning toast (like in #85344) when a user access the instance from the private address, it would create noise for these customers.

I wonder if we should try to be a bit more intelligent here to support this use case, for example, we could:

• Allow server.publicBaseUrl to be an array of URL strings
• On the client-side, only show the warning if the current URL does not match any of the configured public URLs
• When generating URLs, try to use the base URL that matches the address the current user is accessing currently. This can only be done on the client-side, or on the server-side within the context of a request.
• When the current address cannot be determined (eg. in a task manager task), use the first configured base URL. We should document that the first URL in the configuration should always be the preferred URL to which most users should have access.

@mikecote Thoughts?

[mikecote]

From alerting side, we are ok with such change. Though, we will always be using the "primary" URL since the executions happen in a task manager task.

[kobelb]

One issue that has been raised is that some customers have multiple addresses at which they access Kibana, one public (with IP filtering) and one private (via a VPN or similar service such as PrivateLink on AWS). If we were to add a warning toast (like in #85344) when a user access the instance from the private address, it would create noise for these customers.

In this situation, are we just concerned with the toast warning being noisy or will features not work if we use the public URL instead of the private URL?

If we're only concerned with the toast warning being noisy, I wonder if we should just add a kibana.yml setting to suppress the warning for all users.