[O11y][AWS ELB] Lens Migration

[milan-elastic]

Manually migrate AWS ELB visualizations to the lens in the current Kibana version 8.7.1 itself.

Preparation of data for testing

• Use sample_event.json / setup live instance using docker / mock data using a mock server to populate dashboards

Migration stats

AWS ELB Dashboards	Before Migration				After Migration
	Maps	Search	Lens	Visualization	Maps	Search	Lens	Visualization
[Metrics AWS] ELB Overview	0	0	0	9	0	0	9	0
[Logs AWS] ELB Access Log Overview	1	0	0	8	1	0	8	0

Dashboard : [Logs AWS] ELB Access Log Overview

• Migrate 8 visualization to the lens.
  • [AWS][ELB] Migrate AWS ELB Access Log dashboard visualizations to lens #6521

Verification and Validation

• Verification of functionality remains the same after migration in Kibana
• Verification of data count of after-migrated visualization with before-migrated visualization in Kibana
• Verification of Dashboards is not distorted in the supported Kibana version 8.7.1

[rajvi-patel-22]

@zmoog, While migrating AWS ELB access logs panels to lens, we are facing one issue in ELB Top User Agents [Logs AWS] panel. The values of user agent are very large and it's not getting truncated in lens. Due to which visualization is not getting populated properly!

Please find below screenshot after lens migration:

Below is the screenshot of the entire dashboard:

Can you please explain the use case of this particular panel or Can you suggest alternative fields which could be used instead of entire user agent field (user_agent.original)?

CC: @SubhrataK

[zmoog]

@rajvi-elastic, using user_agent.original creates this problem, and I'm not sure we are extracting all the value from these data points. We could combine the other fields (device name, name, os name, and version) in one or more visualizations.

For a piece of advice on the visualizing of such data, I believe @drewdaemon can probably help us. Drew, I read your reviews on similar dashboards: do you have ideas? 😇

[drewdaemon]

Thanks for the ping. We have an open issue describing this: elastic/kibana#100231. It's a more complicated problem than it appears, but we have added it to our next team meeting agenda to discuss a good approach.

I'm assuming there isn't a shorter version of the field in the data you could use instead?

If not, a couple of tactical workarounds come to mind:

• Move that particular visualization to its own row in the layout. Pros—quick, easy, probably fixes the problem. Cons—interrupts your nice grid
• Create a runtime field to truncate the field name to a specific number of characters. You could do this in a by-value data view for just that visualization. Pros—guarantees a nice layout. Cons—user won't be able to see full data, possibly a small performance hit

Do either of those sound reasonable?

[rajvi-patel-22]

@drewdaemon, We don't have the shorter version of field that we can use instead of user_agent.original .

Solution-1

However, as @zmoog suggested we could combine the other fields (name, os name, and version) and use that field instead of user_agent.original .

I have combined user_agent.name, user_agent.os.name and user_agent.version and used that in top values.

Pros:

1. We won't need to create runtime field!
2. Dashboard layout will remain intact!

Cons:

1. If either of field is missing then count will be incorrect.

Solution-2

Below is the screenshot of entire dashboard after extending panel.

Let me know what are your thoughts?

[drewdaemon]

Thanks for the screenshots! Sounds like we have three options on the table

• adjusting layout
• use runtime field to truncate terms
• multi-term dimension

I would personally prioritize correct counts over aesthetics (so avoid the multi-term strategy).

But, I'm just here to advise on our current visualization capabilities. This decision belongs to @zmoog .

[drewdaemon]

FWIW, I also just noticed that if you go the runtime field route (not advocating for this, just adding an insight), you don't actually have to perform the truncation serverside. Instead, you can just apply a truncation field formatter.

[zmoog]

I like the runtime field solution, and it seems the best trade-off.

Mostly because:

• It preserves correct count.
• We don't need to see the whole user agent, just the top distinct values.
• If users want to dive more, they still have the option to build visualization tailored to their needs.

[drewdaemon]

My colleague pointed out yet one more option: convert the bar chart to a table. Pro—the user can always see the complete user agent string and the layout is preserved. Con—different aesthetic.

[rajvi-patel-22]

I tried the runtime field solution but it is not showing expected results when the beginning of string is same and field length for truncated string is small (In my case 50):

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/112.0.5615.29 Safari/537.36 Elastic/Synthetics
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36

Here, beginning of both the string is same till some length hence it is grouping it in one bar after truncation!

I think creating table chart would be better option here. What do you think @zmoog?

Below is the table chart for reference:

[zmoog]

I tried some experiments truncating the user agent, but the results were usually weird.

I am starting to think that the user agent is such a particular field (a very long string composed of many sub-elements) that a table chart may better serve it.

If there are outliers, you can spot them immediately, with full details at hand.

[zmoog]

I'm gravitating towards a +1 on the table chart; @drewdaemon, do you think changing the aesthetic is acceptable in this scenario?

[drewdaemon]

I agree that user agent is a really particular field. I don't think it looks bad to have a table in there. I would go for it!

[rajvi-patel-22]

As Y-axis labels were not getting truncated in lens bar chart, the ELB Top User Agents has been represented using the table chart as best alternative.

PR for AWS ELB Access log lens migration: #6521