[Discover] Redundant data view fields request when switching data views

[kertal]

Kibana version:
Main, 8.10.x

Describe the bug:
When switching data views there are 3 requests to the _fields_for_wildcard API, instead of 2

Steps to reproduce:

1. Go to Discover
2. Switch data view
3. In the network tab of your browser there are 3 requests

Expected behavior:
There should be just 2 requests
Screenshots (if relevant):

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[lukasolson]

Possibly related: #162403

[kertal]

I think this is happening because the data_view_picker fetches the fields of the currently selected data view, when the data view is switched

kibana/src/plugins/unified_search/public/dataview_picker/change_dataview.tsx

Lines 305 to 327 in 4fc0369

	onChangeDataView={async (newId) => {
	try {
	// refreshing the field list
	await dataViews.get(newId, false, true);
	} catch (e) {
	//
	}
	setSelectedDataViewId(newId);
	setPopoverIsOpen(false);
	if (isTextBasedLangSelected && !isTextLangTransitionModalDismissed) {
	setIsTextLangTransitionModalVisible(true);
	} else if (isTextBasedLangSelected && isTextLangTransitionModalDismissed) {
	setIsTextBasedLangSelected(false);
	// clean up the Text based language query
	onTextLangQuerySubmit?.({
	language: 'kuery',
	query: '',
	});
	onChangeDataView(newId);
	setTriggerLabel(trigger.label);
	} else {
	onChangeDataView(newId);
	}

and the callback function of the component is given the actual id of dataView (onChangeDataView(newId))

Then the consuming application, in this case Discover, uses the id to fetch the DataView and it's fields again.

kibana/src/plugins/discover/public/application/main/hooks/utils/change_data_view.ts

Line 40 in 6440075

nextDataView = typeof id === 'string' ? await dataViews.get(id, false) : id;

this is redundant, and could be resolved by passing in the data view object on the callback function

onChangeDataView(dataViewObj)

then the consumer can use the the dataViewObj, and doesn't need to trigger another operation fetching the fields again, which causes the redundancy described in this issue

[jughosta]

I think the second request is happening because of this parameter #160195

[kertal]

I think the second request is happening because of this parameter #160195

but I think is just should be triggered when entering Discover, not when switching data views? (which does not mean it shouldn't also be consolidated to reduce requests)

[mattkime]

A while back...unfortunately I'm not sure how far back...data views seemed to be refreshed whenever the user switched kibana apps. Perhaps this relied on an idiosyncrasy of the kibana app lifecycle that no longer exists. Perhaps it never really worked that way and I was simply mistaken. Whatever the case, they clearly don't work that way now.

Initially we saw new fields as a rare case. You had to push a button to pick up new fields. The button was removed, but not without debate as to whether it should be left, simply devoid of actual function. Well, on occasion new fields are exactly what people are working on and want to see and they complain when they don't - #157239

Then we decided its best to get fresh fields whenever we load a data view - #160195 - generally speaking, this is very effective. Unfortunately when field lists are slow, this makes them much slower. We've made the field list update more frequently at the expense of performance under specific conditions.

This is when discover started refreshing the field list although its unclear why - #136256

...anyways, I need to put this down for a bit. I'll come back to this and see if I arrive at either a conclusion or a well defined paradox.

[kertal]

Then we decided its best to get fresh fields whenever we load a data view - #160195 - generally speaking, this is very effective. Unfortunately when field lists are slow, this makes them much slower. We've made the field list update more frequently at the expense of performance under specific conditions.

Thx for the research 👍 We might think of discussing why the field list needs to be refreshed and if it wouldn't be better to allow the user to do this, if needed. This would prevent a lot of redundant fetching, improving the performance.

I was having another look at #160195, you're both right, this is the cause for the redundant request when switching data views in the UI. ChangeDataView is refreshing the fields, and then it's done again when resolving the selected data view in Discover. We might consider to revert this change, to find a better solution, e.g. we could pass down configuration to ChangeDataView not to refresh the fields (and enable this in Discover), or something similar, reducing redundant requests.

[kertal]

@jughosta we are currently requesting the filtered fields in the UnifiedFieldList to distinguish between available fields and "empty" fields, which is done here, right?

kibana/packages/kbn-unified-field-list/src/services/field_existing/field_existing_utils.ts

Lines 115 to 121 in 7f0d57d

	export function existingFields(filteredFields: FieldSpec[], allFields: Field[]): string[] {
	const filteredFieldsSet = new Set(filteredFields.map((f) => f.name));
	return allFields
	.filter((field) => field.isScript || field.runtimeField || filteredFieldsSet.has(field.name))
	.map((f) => f.name);
	}

Couldn't we use those filtered fields, in case there are new fields, to add those to the displayed fields?

[jughosta]

@kertal Yes, we might be able to add new fields or replace properties of the existing ones with this approach. But not to hide former fields.

[kertal]

@jughosta thx, so former fields that we couldn't hide, these would be e.g. if there were indices deleted and no other indices available with this fields, right? Those would then become empty, which IMO would be an acceptable trade off for reducing _field_caps requests. Also in the UnifiedFieldList, we could offer UI to refresh the fieldlist by the user, when needed

[kertal]

Fixed by #168315