Add "Popular fields" to Lens as part of the "Unified Field List" initiative

[VladLasitsa]

Summary

Closes: #145687

Added "Popular fields" to Lens.

The field will appear in 'popular fields' section in following scenarios:

• User will set popularity for field in dataview management section
• Use the field in discover (as Lens and Discover use the same popularity counter if user add field in discover, that field also will appear in popular section in Lens)
• User clicks the add field button from the field list
• User drags and drops a field from the field list

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[VladLasitsa]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 9f53fe9

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
lens	1.3MB	1.3MB	+1.9KB

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	61	67	+6
lens	19	20	+1
osquery	109	115	+6
securitySolution	439	445	+6
total			+21

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	70	76	+6
lens	22	23	+1
osquery	110	117	+7
securitySolution	516	522	+6
total			+22

History

• 💚 Build #95604 succeeded 8c73d10
• 💔 Build #95445 failed bb18536
• 💔 Build #95438 failed e5647cc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @VladLasitsa

[dej611]

I would propose a small refactoring of the populaizeField function to do the actual work only in success case.

Found also a bug: when Combining two or more fields (using Top values multi-terms) the popularized function is not called.

I wonder if it makes sense to fire off a request every time one user uses a field. Would it be best to batch them by some longer time slot somehow? Do not think this is a vital information to have.

[dej611]

I would refactor a bit here to do the async work only in case of success for a persisted dataView:

Suggested change

	if (!indexPatternId || !capabilities?.indexPatterns?.save) return;
	const dataView = await dataViewsService.get(indexPatternId);
	const field = dataView.fields.getByName(fieldName);
	if (!field) {
	return;
	}
	field.count++;
	const refreshIndexPatternsListArgs = {
	activeDatasources: Object.keys(datasourceStates).reduce(
	(acc, datasourceId) => ({
	...acc,
	[datasourceId]: datasourceMap[datasourceId],
	}),
	{}
	),
	indexPatternId,
	indexPatternService,
	indexPatternsCache,
	};
	if (!dataView.isPersisted()) {
	refreshIndexPatternsList(refreshIndexPatternsListArgs);
	return;
	}
	// Catch 409 errors caused by user adding columns in a higher frequency that the changes can be persisted to Elasticsearch
	try {
	await dataViewsService.updateSavedObject(dataView, 0, true);
	refreshIndexPatternsList(refreshIndexPatternsListArgs);
	// eslint-disable-next-line no-empty
	} catch {}
	if (!indexPatternId || !capabilities?.indexPatterns?.save) return;
	const lensDataView = indexPatternsCache[indexPatternId];
	if (!lensDataView.getFieldByName(fieldName)) {
	return;
	}
	if (lensDataView.isPersisted) {
	// Catch 409 errors caused by user adding columns in a higher frequency that the changes can be persisted to Elasticsearch
	try {
	const dataView = await dataViewsService.get(indexPatternId);
	const field = dataView.fields.getByName(fieldName);
	if (field) {
	field.count++;
	await dataViewsService.updateSavedObject(dataView, 0, true);
	}
	// eslint-disable-next-line no-empty
	} catch {}
	}
	const refreshIndexPatternsListArgs = {
	activeDatasources: Object.keys(datasourceStates).reduce(
	(acc, datasourceId) => ({
	...acc,
	[datasourceId]: datasourceMap[datasourceId],
	}),
	{}
	),
	indexPatternId,
	indexPatternService,
	indexPatternsCache,
	};
	refreshIndexPatternsList(refreshIndexPatternsListArgs);

It is a little bit more verbose but it saves some work for adHoc dataviews

[VladLasitsa]

thank you @dej611, I will update my function. About batch requests I don't think that is a good idea. I call refreshIndexPatternsList here to show correct state of fields after updating popularity. In you proposal we should call refreshIndexPatternsList after some timeout which lead to showing correct list of popular fields with delay.

[VladLasitsa]

@dej611, also for adHoc dataviews we also should use this field.count++; not only for usual dataviews.

[dej611]

Let's wait for the main decision on the feature first.
But for the adHoc I think that operating on the cached version should be ok, no?

[stratoula]

@dej611 one important note on this feature is that we want the counter to initialize only for actions made by the fields list.
This means:

• add icon
• drag and drop
  Personally I don't agree with this addition, in my point of view the popularity in lens is fields that are used in saved visualizations taking under consideration that I can change fields all the time (and operations).
  But this was the product request :)

[VladLasitsa]

@dej611 one important note on this feature is that we want the counter to initialize only for actions made by the fields list. This means:

• add icon
• drag and drop
  Personally I don't agree with this addition, in my point of view the popularity in lens is fields that are used in saved visualizations taking under consideration that I can change fields all the time (and operations).
  But this was the product request :)

I've already added this info in PR description:)

[dej611]

@dej611 one important note on this feature is that we want the counter to initialize only for actions made by the fields list. This means:

• add icon
• drag and drop
  Personally I don't agree with this addition, in my point of view the popularity in lens is fields that are used in saved visualizations taking under consideration that I can change fields all the time (and operations).
  But this was the product request :)

But combine is a drag and drop action 🤔

[stratoula]

Yes I have tried to convince against this one and take some time to think about it more but it seems that it has been decided

cc @timductive @ninoslavmiskovic

[dej611]

Besides the specific implementation I have few questions regarding this feature: what value is it going to bring to the user?

• After trying it for testing I found the feature very invasive as "enabled by default" as it was using too much UI space to deliver very few useful information. At the second visualization I built I closed the accordion as it was just reducing the available list of fields I wanted to use.
• the most confusing part is that the counter is shared between Lens and Discover now, which makes little sense to me. I think at least a different counter should be used for each use case: one for discover and another for data analysis.
• too much noise is generated now: fields are dancing too much when creating a visualization, and the remove action is not taken into account at all. To me "popular" should be counted only for those fields actually used by the user - so, for instance also those picked from the field picker in the configuration panel, not covered here.
• the amount of traffic generated can be quite high to the datasource. The code here is ignoring saving errors as fallback, but why poll the datasource in the first place?

Just few thoughts for a discussion here. cc @stratoula @ninoslavmiskovic

[VladLasitsa]

But combine is a drag and drop action 🤔

Hm, but as I understand combine can be from field list or between dimensions, right?

[stratoula]

I couldn't agree more @dej611. I will try to raise it again!

[dej611]

But combine is a drag and drop action 🤔

Hm, but as I understand combine can be from field list or between dimensions, right?

[VladLasitsa]

@dej611, Some thinking:

• the most confusing part is that the counter is shared between Lens and Discover now, which makes little sense to me. I think at least a different counter should be used for each use case: one for discover and another for data analysis.

If we separate counter, what should we do with popularity which can be set in dataview management section?

• too much noise is generated now: fields are dancing too much when creating a visualization, and the remove action is not taken into account at all. To me "popular" should be counted only for those fields actually used by the user - so, for instance also those picked from the field picker in the configuration panel, not covered here.
• the amount of traffic generated can be quite high to the datasource. The code here is ignoring saving errors as fallback, but why poll the datasource in the first place?

About these two: I have used the same approach as in Discover (see here src/plugins/discover/public/utils/popularize_field.ts). One exception: In Discover if you remove field it will also increase popularity

[stratoula]

@VladLasitsa @dej611 let's pause it for now. I pinged Tim to revisit because 3 engineers disagree with this feature and I think we should re-evaluate.

[stratoula]

Moving this to draft as we decided to discuss more.

[ninoslavmiskovic]

I have setup a meeting in the new year,so that we can align. I will go through "what we know today" regarding popular fields, what we aim to learn from market feedback and what user problems we are aiming to solve, when added to Lens. The aim is to bring clarity and most importantly provide value to the users, like it does in Discover.

[VladLasitsa]

Since I will finish work on the project this year, removed my assignment, feel free to continue working here or close this PR.

[kertal]

Besides the specific implementation I have few questions regarding this feature: what value is it going to bring to the user?

• After trying it for testing I found the feature very invasive as "enabled by default" as it was using too much UI space to deliver very few useful information. At the second visualization I built I closed the accordion as it was just reducing the available list of fields I wanted to use.

Yes, I do agree. In Lens it takes too much screen space, and when there are just a few popular fields, a new field that was just popularized is eating up vertical space in "realtime"

Users can collapse the field sections, but it's not remembered - we should build in functionality to remember the collapse state in e.g. localStorage. FYI @jughosta because it would be a neat unified field list feature

• the most confusing part is that the counter is shared between Lens and Discover now, which makes little sense to me. I think at least a different counter should be used for each use case: one for Discover and another for data analysis.

Yes, I also agree that different use cases need different field stats. And the pattern of increasing the statistics directly in the data view saved object I would consider has legacy behavior. The statistics isn't even accurate because of:

Catch 409 errors caused by user adding columns in a higher frequency that the changes can be persisted to Elasticsearch
So when you add field in Discover in a higher frequency than Elasticsearch allows, not every counter increase is persisted.

• The quick solution was to hide this error (a long time ago)
• The better solution would be to batch the updates
• The best solution, IMO, would be, not to updated statistics in the data view saved object

I think popular fields make sense to help the user find his frequently used fields in a quicker way. This may vary by application and also by user type. What could be a solution to a) still use the counter in the data views saved object b) add statistics by user / application ?

For example we could still use the counter of the data view saved object, so admins could use those to provide base values. And then we would collect the the increments of the the counter values by application in localStorage. Then Lens/Discover would have different statistics based on a users field selection.

This would be the simplest solution that comes to my mind (and we should aim for simplicity)

FYI @mattkime because we chatted about the field popularization a while ago, you mentioned to could be collected in a separate saved object ... a solution like this would also provide a way to collect different statistic / application ... however it sounds like more effort

• too much noise is generated now: fields are dancing too much when creating a visualization, and the remove action is not taken into account at all. To me "popular" should be counted only for those fields actually used by the user - so, for instance also those picked from the field picker in the configuration panel, not covered here.

I do agree about the "dancing", but I think it's also fine to limit the popularity to those fields the user interacted with in the sidebar field selection.

• the amount of traffic generated can be quite high to the datasource. The code here is ignoring saving errors as fallback, but why poll the datasource in the first place?

You mean the 409 ignorance, when clicking faster than Elasticsearch allows, right?

[jughosta]

Adding functionality to persist collapsed/expanded state of sections in localStorage as @kertal suggested: #148373

[dej611]

So when you add field in Discover in a higher frequency than Elasticsearch allows, not every counter increase is persisted.

• The quick solution was to hide this error (a long time ago)
• The better solution would be to batch the updates
• The best solution, IMO, would be, not to updated statistics in the data view saved object

I do agree about the "dancing", but I think it's also fine to limit the popularity to those fields the user interacted with in the sidebar field selection.

I would be ok, to me, to record user interactions as long as all add/remove operation are used. But this is not the case here, as only "some" adding flows are tracked and no removal.
Perhaps a radical approach here: why not update scores on editor exit? That can be either a save or "return to" action. That would reduce the traffic to ES and provide a meaningful score to the user, in my opinion.

You mean the 409 ignorance, when clicking faster than Elasticsearch allows, right?

Yes.

[kertal]

I would be ok, to me, to record user interactions as long as all add/remove operation are used. But this is not the case here, as only "some" adding flows are tracked and no removal. Perhaps a radical approach here: why not update scores on editor exit? That can be either a save or "return to" action. That would reduce the traffic to ES and provide a meaningful score to the user, in my opinion.

I do agree, when the user is interacting with the interface, popularity increase should not change the interface. And updating the score not immediately, in a patched way on certain events would be a good way to get rid of the 409s and get better statistics. One thing I've always been wondering, why should fields, when they are removed get more "popular"?

You mean the 409 ignorance, when clicking faster than Elasticsearch allows, right?

Yes.

Thx

Great work @jughosta BTW for already implementing state of the sections in localStorage 👍

[stratoula]

@kertal tbh I was wondering the same thing. Following the code in Discover I see that we popularizeField when:

• adding a column
• removing a column
• creating a filter from a field

but not when I am doing other things with the fields such as:

• Visualize field
• Add a field as a breakdown
• etc

While I strongly believe that the popular fields in Lens should be fields that are actually used a lot on saved visualizations I think that we might consider re-thinking what popular fields mean in discover too.