Optimize Remote Clusters UX on Cloud

[cjcenizal]

Related to #61503. We recently received feedback from @bleskes on ways to enhance this UX. Much of this scope was suggested by @zanbel (thanks!). CC @andrew-moldovan @juliocamarero

Make Cloud guidance more prominent

One piece of feedback we got was that Boaz didn't notice the Cloud-specific guidance. We should make this more prominent. I like Roy's suggestion of placing this guidance at the top of the page instead of within the form row. However, I think we should also consider using a more impactful component than the info callout. For example, what if we placed the guidance inside of a panel with the Cloud logo inside? Would this be more noticeable?

In terms of layout, here's roughly what I have in mind, courtesy of Roy:

Tailor inline guidance to the Cloud UX

Roy also suggested making the inline guidance Cloud-specific. I think this is a good suggestion. Localized guidance makes it easier to associate the guidance with the relevant action. By replacing the existing help text when on Cloud, we also remove the existing guidance around TLS, which is always enabled on Cloud.

Sanitize or validate input

One potentially common error is for users to include the protocol, as below.

To solve this, we can either sanitize the input on the server by removing the protocol, or we can show the user a validation error asking them to remove the protocol.

Reiterate guidance for remote clusters that aren't connected

Roy also made the great suggestion that we surface this same feedback as a callout inside of the flyout if the remote cluster is broken. I imagine an error callout at the top of the flyout saying something like, "Remote cluster not connected. This could be caused by an incorrect configuration. Configure the remote cluster with the proxy address and server name from the Security page of your deployment, under Remote cluster parameters."

Fix flyout layout when proxy address is long

As you can see in the above screenshot, the Proxy address is pushing the Skip unavailable column off-screen. We can fix by adding word-break: break-all or similar to the <dd> element.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[zanbel]

This is awesome! Thanks @cjcenizal for capturing our feedback and for the quick response.

An additional potential issue discussed over slack is that misconfigured trust (deployments must both trust each other) can be another common reason users will not be able to successfully configure the remote cluster. I think we should mention it in both the info message and in case the remote cluster is configured but disconnected. We can also add a learn more link to the step-by-step guide in our user guide.

Example for the info message:
To configure an Elastic Cloud deployment as a remote cluster, make sure both deployments trust each other. Additionally, the proxy mode should be enabled and you use the correct proxy address and server name values. Those can be copied from the Security page of your deployment menu and search for Remote cluster parameters, in the Elastic Cloud user console. You can also use this step-by-step guide.

Example for the disconnected remote cluster message:
Remote cluster not connected. This could be caused by an incorrect configuration. Make sure both deployments trust each other, the proxy mode is enabled and you use the correct Proxy address and Server name value. Those values can be copied from the Security page of your deployment, under Remote cluster parameters, in the Elastic Cloud user console. You can also use this step-by-step guide.

I know it's a bit long so feel free to rephrase.

[juliocamarero]

Thanks a lot @cjcenizal!

An additional potential issue discussed over slack is that misconfigured trust (deployments must both trust each other) can be another common reason users will not be able to successfully configure the remote cluster. I think we should mention it in both the info message and in case the remote cluster is configured but disconnected. We can also add a learn more link to the step-by-step guide in our user guide.

The link to learn more about trust would be this one: https://www.elastic.co/guide/en/cloud/current/ec-trust-management.html (it may look empty now, but it will be updated later today)

The link that explains how to configure the remote clusters in Cloud is this one: https://www.elastic.co/guide/en/cloud/current/ec-enable-ccs.html (it will also be updated with the new info later today)

EDIT: The docs are live now

[cristina-eleonora]

Since on the cloud side of things we generally talk about deployments, we should refer to it as such in the callout title: Configure a remote Elastic Cloud deployment

The description could say: "Use proxy mode to configure an Elastic Cloud deployment as a remote cluster. The proxy address and server name can be found on the Security page of your remote deployment under Remote cluster parameters"

I'm sure the @elastic/cloud-writers can do a better job and also help with writing / shortening the rest of the text.

[cjcenizal]

Thanks all for the super-helpful feedback!

[kilfoyle]

I agree with Cristina's suggestions for the title and description.

For under the Proxy address and Server name fields, how about using this for both?
Copy this value from your deployment page under Security -> Remote cluster parameters.

(No need to mention the setting name since it's already clear from the label)

[bleskes]

@cjcenizal thanks so much for writing this up and doing the improvement. I agree with all the things said, but I think we can take a few things to the next level.

Through the Cloud Plugin, can we build a dedicated Cloud UI for this screen where all users need to do is paste in a URL to a deployment - it can be the HTTP port, it can be the transport URL, it can have no protocol, or it can have the HTTPS:// prefix. The UI takes all of that and populates the ES setting.

Is that possible? how much work is it than the work described in this ticket? I hope we all agree that the experience will be much better for users and also make all the above work redundant. If we don't agree - please speak up.. I'm really would love to hear it.

[bleskes]

@cjcenizal asked me via slack to give an example:

Here is one: say I paste an http url: https://aaec15d04a031fe37ec6f024b5de9982.us-east-1.aws.found.io:9243. The plugin should convert that to an address: aaec15d04a031fe37ec6f024b5de9982.us-east-1.aws.found.io:9400 , stripping away the https:// and switching the port to 9400. It should also automatically populate the server_name in the ES config with aaec15d04a031fe37ec6f024b5de9982.us-east-1.aws.found.io

[cristina-eleonora]

I like that approach, @bleskes, it would much improve the user experience.
I assume we wouldn't show those other fields, just process everything in the background from the URL.

[zanbel]

Through the Cloud Plugin, can we build a dedicated Cloud UI for this screen where all users need to do is paste in a URL to a deployment - it can be the HTTP port, it can be the transport URL, it can have no protocol, or it can have the HTTPS:// prefix. The UI takes all of that and populates the ES setting.
Is that possible? how much work is it than the work described in this ticket? I hope we all agree that the experience will be much better for users and also make all the above work redundant. If we don't agree - please speak up.. I'm really would love to hear it.

@juliocamarero it's been a while now but, AFAIR, when we talked about it with Jose way back when preparing for the PR this was not possible. There could be scenarios where these values will not match and we can't manipulate the ES URL to automatically populate these fields. Maybe it was an ECE specific thing in case customers will have a reverse proxy in front of ECE and we wanted to use the same for ESS/ECE for consistency?

It should also automatically populate the server_name in the ES config with aaec15d04a031fe37ec6f024b5de9982.us-east-1.aws.found.io

If it is doable, why not have a single field that is actually called Elasticsearch address and, once filled, automatically populate the two other fields that should be read-only. It will be less confusing for users than calling it Proxy address

[bleskes]

I assume we wouldn't show those other fields, just process everything in the background from the URL.

Exactly.

Maybe it was an ECE specific thing in case customers will have a reverse proxy in front of ECE and we wanted to use the same for ESS/ECE for consistency?

I agree this is a concern, but my thinking was that - we do ask users to put in a URL, so that works also if people have a reverse proxy (just like we do in ESS). It may be that we need to add some configuration options to the Kibana.yml. For example - what is the transport port to use (9400). W.r.t Server name, my hope is that that level customization is not needed, but if so, we can still have an advanced UI option that's hidden by default to override the behavior I described.

[cjcenizal]

For the initial release of these Cloud-optimized enhancements, I think we should reduce the amount of complexity we're introducing in the UX, to reduce the risk of introducing bugs. An entirely new and simplified UI for configuring remote clusters on Cloud introduces two dimensions of increased complexity that I can see:

• Edge cases might still require access to the original "advanced" settings. Per Boaz's comment above, if we uncover edge cases where users need the ability to override this Cloud-optimized behavior, then we'll be adding new fields while preserving the original ones. This will represent a net increase in complexity, not a decrease.
• Changes to the configuration of remote clusters affect their read-only representations. If Cloud users configure remote clusters with a new set of fields, they'll expect to see these fields represented in the table and details flyout. This becomes even more complex if we also need to consider the edge cases above, such as deciding if and how we should surface both the simplified settings and the "advanced" settings in the flyout.

For these reasons I think we should stick with the original proposal for now, and consider further optimizations as a separate scope of work.

[cjcenizal]

I chatted with @jethr0null and realized I was too hasty in my assertion above. I'd like to clarify the requirements of Boaz's proposal first.

There could be scenarios where these values will not match and we can't manipulate the ES URL to automatically populate these fields. Maybe it was an ECE specific thing in case customers will have a reverse proxy in front of ECE and we wanted to use the same for ESS/ECE for consistency?

@zanbel Can you provide a detailed example to help me understand this scenario better? To Boaz's point, how much customization would we need to provide a user? Defining just the port? Or the server name too?

[zanbel]

Sure. Just a small warning, this is based on my recollection of what was discussed with Jose a while back.

Consider a scenario where an ECE user has a reverse proxy/LB in front of our proxy layer (that's the recommended setup). In some cases, they want to expose a pretty name so instead of <UUID>.acme.com they configure their reverse proxy to route requests and expose my-awesome-deployment.acme.com. In that case, AFAIU, the proxy address needs to be my-awesome-deployment.acme.com:9400 and the server name needs to be <UUID>.acme.com. Is that accurate?

As Boaz suggested, I think that we can still improve the UX for most users that should be able to copy-paste the ES URL from the deployment overview page and we can do the rest. We should still allow users to manually edit those values for the reason mentioned above.

Lastly, we also need to make sure this will work with deployment aliases, which are coming shortly.

[cjcenizal]

Thanks for the added info @zanbel! Very helpful level of detail.

[cjcenizal]

Based on the edge case @zanbel outlined and @bleskes's suggestions on streamlining the UX, I'd like to propose the below approach.

Treat the originally proposed changes as the "baseline" UX

It's clear that Cloud users will need to have access to the full remote cluster configuration. This is critical to addressing scenarios like the reverse proxy/LB scenario Roy outlined above. The original proposal satisfies this requirement.

Note: I think we'll need to tweak the inline help text defined in the "Tailor inline guidance to the Cloud UX" section, to accommodate the reverse proxy/LB scenario. The inline help text directs the user to copy/paste the proxy address from the Security page, which sounds incorrect for this scenario. We might need to add a caveat to this guidance, for example "If you're using a reverse proxy, please see the docs."

Enhance the baseline UX with a streamlining feature

When the user first visits this page, we can detect that they're on Cloud and display a modal or flyout that surfaces content like this:

Add Cloud deployment as remote cluster

If you're connecting to a Cloud deployment, you can copy and paste the Elasticsearch endpoint URL into the field below. The remote cluster will be automatically configured to connect to the Cloud deployment.

Note that if the Cloud deployment is behind a load balance or reverse proxy, you'll need to configure the remote cluster manually. Read the docs for more info.

Elasticsearch endpoint URL:
[____________________________]

[ Configure remote cluster ] [ Close ]

The user can retrieve the Elasticsearch endpoint URL from their Cloud deployment page here:

When the user clicks the "Configure remote cluster" button, the modal/flyout will close and the form will be prepopulated with all of the necessary values. If the user closes the modal/flyout and changes their mind, they will be able to click a "Connect to Cloud deployment" button to open it back up again.

Note: This streamlining feature is unidirectional. It results in the same remote cluster configuration as if the user had configured it using the baseline form. This addresses my concerns about complexity, because there will be no effect on the read-only representations of the remote clusters that have been configured with this feature.

Separate scopes of work

We can treat each of these as a separate scope of work and ship them in stages. This will reduce the time to shipping a unit of value and possibly allow us to parallelize this effort.

[cjcenizal]

In terms of prioritizing the above scopes of work, I chatted with @jasontedor and we agreed that it will be more impactful to ship the streamlining feature first and then follow it up with the enhanced baseline experience.

[zanbel]

@juliocamarero and I chatted yesterday about deployment aliases. It seems that the same instructions to copy-paste the ES endpoint from the deployment overview page will continue to work by replacing the port with 9400 and removing the protocol. E.g. using my-rad-deployment.es.us-east-1.aws.elastic-cloud.com:9400 will work since our proxies will route requests to the relevant ES cluster with the right port.

The server name, on the other hand, can't be automatically populated based on the alias since my-rad-deployment.es.us-east-1.aws.elastic-cloud.com can't be used, AFAWCT. This means that if using a deployment alias, the remote cluster won't connect, the user will see an error with a hint instructing them to use the "advance mode" and pasting the proxy address and server name values from the deployment's security page.

We wanted to call that out and see if folks can think of a way to avoid having to send the server name (which can simplify things regardless), or another way support deployment aliases. We can also open another issue for that if you think we should move forward with this suggestion even if it doesn't support deployment aliases.

[kellyemurphy]

We might need to add a caveat to this guidance, for example "If you're using a reverse proxy, please see the docs."

Does this documentation already exist? I did a search of the cloud docs for this term "reverse proxy" and I'm not seeing anything.

[yuliacech]

I'm going to close this issue with a new form for Cloud users added in #94450 and follow up work for flyout is being tracked in #97831