Skip to content

Commit

Permalink
[Discover] One Discover context awareness (#183797)
Browse files Browse the repository at this point in the history
## Summary
This PR includes the initial implementation of the Discover contextual
awareness framework and composable profiles:

![logs_table](https://github.com/elastic/kibana/assets/25592674/0815687a-c4d8-4a80-8f67-5e1de0c65adf)

### Context
We currently support three levels of context in Discover:
- Root context:
  - Based on the current solution type, navigational parameters, etc.
  - Resolved at application initialization and on parameter changes.
  - Runs synchronously or asynchronously.
- Data source context:
  - Based on the current ES|QL query or data view.
- Resolved on ES|QL query or data view change, before data fetching
occurs.
  - Runs synchronously or asynchronously.
- Document context:
  - Based on individual ES|QL records or ES documents.
- Resolved individually for each ES|QL record or ES document after data
fetching runs.
  - Runs synchronously only.

### Composable profiles
To support application extensibility based on context, we've introduced
the concept of "composable profiles". Composable profiles are
implementations of a core `Profile` interface (or a subset of it)
containing all of the available extension points Discover supports. A
composable profile can be implemented at any context level through a
"profile provider", responsible for defining the composable profile and
its associated context resolution method. The context resolution method,
named `resolve`, determines if its composable profile is a match for the
current Discover context, and returns related metadata in a `context`
object.

### Merged accessors
Composable profiles operate similarly to middleware in that each of
their extension point implementations are passed a `prev` argument,
which can be called to access the results from profiles at previous
context levels, and allows overwriting or composing a final result from
the previous results. The method Discover calls to trigger the extension
point merging process and obtain a final result from the combined
profiles is referred to as a "merged accessor".

The following diagram illustrates the extension point merging process:

![image](https://github.com/davismcphee/kibana/assets/25592674/59f7cd23-c1e0-4d8e-99ed-02460211ed96)

### Supporting services
The contextual awareness framework is driven by two main supporting
services called `ProfileService` and `ProfilesManager`.

Each context level has a dedicated profile service, e.g.
`RootProfileService`, which is responsible for accepting profile
provider registrations and running through each provider in order during
context resolution to identify a matching profile.

A single `ProfilesManager` is instantiated on Discover load, or one per
saved search panel in a dashboard. The profiles manager is responsible
for the following:
- Managing state associated with the current Discover context.
- Coordinating profile services and exposing resolution methods for each
context level.
- Providing access to the combined set of resolved profiles.
- Deduplicating profile resolution attempts with identical parameters.
- Error handling and fallback behaviour on profile resolution failure.

### Bringing it all together
The following diagram models the overall Discover contextual awareness
framework and how each of the above concepts come together:

![image](https://github.com/elastic/kibana/assets/25592674/49193141-c50a-473f-9d38-eb09fbaaffbe)

### Followup work
- We'll want to add developer documentation as a followup, which I've
created an issue for here: #184698. The summary for this PR can be used
as the basis for the documentation.
- Since we currently have no profile or extension point implementations,
this PR does not include any functional tests. We should create example
implementations for functional testing and ensure they're only enabled
when running the test suite or when developers want them enabled. I've
created a followup issue for this work here: #184699.

### Testing notes
Testing the framework is tricky since we have no actual profile or
extension point implementations yet. However, I previously added example
implementations that I was using for testing while building the
framework. I've removed the example implementations so they don't get
merged, but they can be temporarily restored for testing by reverting
the commit where I removed them: `git revert
5752651`.

You'll also need to uncomment the following lines in
`src/plugins/discover/public/plugin.tsx`:
https://github.com/elastic/kibana/blob/ce85a6a35fa3623bfdfac7dae41df2d840394154/src/plugins/discover/public/plugin.tsx#L458-L463

To test the root profile resolution based on solution type, I'd
recommend enabling the solution nav locally by adding the following to
`kibana.dev.yml`:
```yml
xpack.cloud_integrations.experiments.enabled: true
xpack.cloud_integrations.experiments.flag_overrides:
  "solutionNavEnabled": true

xpack.cloud.id: "ftr_fake_cloud_id:aGVsbG8uY29tOjQ0MyRFUzEyM2FiYyRrYm4xMjNhYmM="
xpack.cloud.base_url: "https://cloud.elastic.co"
xpack.cloud.deployment_url: "/deployments/deploymentId"
```

In order to change the active solution type, modify the `mockSpaceState`
in `src/plugins/navigation/public/plugin.tsx`:
https://github.com/elastic/kibana/blob/79e51d64f83da6af56107a633a5a3b49947f1ebe/src/plugins/navigation/public/plugin.tsx#L159-L162

For test data, I'd recommend running the following commands to generate
sample ECS compliant logs and metrics data:
```
node scripts/synthtrace.js --target "http://elastic:changeme@localhost:9200/" --kibana "http://elastic:changeme@localhost:5601/" --live --logLevel debug simple_logs.ts
node scripts/synthtrace.js --target "http://elastic:changeme@localhost:9200/" --kibana "http://elastic:changeme@localhost:5601/" --live --logLevel debug simple_trace.ts
```

And lastly a couple of the ES|QL queries I used for testing:
```
// resolves to the example logs data source context
from logs-synth-default

// mixed dataset that falls back to vanilla Discover
// helpful for testing document context in the doc viewer flyout
from logs-synth-default,metrics-*
```

Resolves #181962.

### Checklist

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
davismcphee and kibanamachine authored Jun 10, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
1 parent ad83c67 commit 59912e8
Showing 51 changed files with 2,008 additions and 239 deletions.
12 changes: 12 additions & 0 deletions packages/kbn-discover-utils/src/utils/build_data_record.test.ts
Original file line number Diff line number Diff line change
@@ -30,5 +30,17 @@ describe('Data table record utils', () => {
expect(doc).toHaveProperty('isAnchor');
});
});

test('should support processing each record', () => {
const result = buildDataTableRecordList(esHitsMock, dataViewMock, {
processRecord: (record) => ({ ...record, id: 'custom-id' }),
});
result.forEach((doc) => {
expect(doc).toHaveProperty('id', 'custom-id');
expect(doc).toHaveProperty('raw');
expect(doc).toHaveProperty('flattened');
expect(doc).toHaveProperty('isAnchor');
});
});
});
});
10 changes: 7 additions & 3 deletions packages/kbn-discover-utils/src/utils/build_data_record.ts
Original file line number Diff line number Diff line change
@@ -35,9 +35,13 @@ export function buildDataTableRecord(
* @param docs Array of documents returned from Elasticsearch
* @param dataView this current data view
*/
export function buildDataTableRecordList(
export function buildDataTableRecordList<T extends DataTableRecord = DataTableRecord>(
docs: EsHitRecord[],
dataView?: DataView
dataView?: DataView,
{ processRecord }: { processRecord?: (record: DataTableRecord) => T } = {}
): DataTableRecord[] {
return docs.map((doc) => buildDataTableRecord(doc, dataView));
return docs.map((doc) => {
const record = buildDataTableRecord(doc, dataView);
return processRecord ? processRecord(record) : record;
});
}
3 changes: 3 additions & 0 deletions src/plugins/discover/public/__mocks__/services.ts
Original file line number Diff line number Diff line change
@@ -41,6 +41,7 @@ import { SearchSourceDependencies } from '@kbn/data-plugin/common';
import { SearchResponse } from '@elastic/elasticsearch/lib/api/types';
import { urlTrackerMock } from './url_tracker.mock';
import { createElement } from 'react';
import { createContextAwarenessMocks } from '../context_awareness/__mocks__';

export function createDiscoverServicesMock(): DiscoverServices {
const dataPlugin = dataPluginMock.createStartContract();
@@ -137,6 +138,7 @@ export function createDiscoverServicesMock(): DiscoverServices {
...uiSettingsMock,
};

const { profilesManagerMock } = createContextAwarenessMocks();
const theme = themeServiceMock.createSetupContract({ darkMode: false });

corePluginMock.theme = theme;
@@ -236,6 +238,7 @@ export function createDiscoverServicesMock(): DiscoverServices {
contextLocator: { getRedirectUrl: jest.fn(() => '') },
singleDocLocator: { getRedirectUrl: jest.fn(() => '') },
urlTracker: urlTrackerMock,
profilesManager: profilesManagerMock,
setHeaderActionMenu: jest.fn(),
} as unknown as DiscoverServices;
}
Original file line number Diff line number Diff line change
@@ -31,6 +31,7 @@ const customisationService = createCustomizationService();

async function mountComponent(fetchStatus: FetchStatus, hits: EsHitRecord[]) {
const services = discoverServiceMock;

services.data.query.timefilter.timefilter.getTime = () => {
return { from: '2020-05-14T11:05:13.590', to: '2020-05-14T11:20:13.590' };
};
@@ -69,6 +70,10 @@ async function mountComponent(fetchStatus: FetchStatus, hits: EsHitRecord[]) {
}

describe('Discover documents layout', () => {
beforeEach(() => {
jest.clearAllMocks();
});

test('render loading when loading and no documents', async () => {
const component = await mountComponent(FetchStatus.LOADING, []);
expect(component.find('.dscDocuments__loading').exists()).toBeTruthy();
@@ -131,4 +136,22 @@ describe('Discover documents layout', () => {
expect(discoverGridComponent.prop('externalCustomRenderers')).toBeDefined();
expect(discoverGridComponent.prop('customGridColumnsConfiguration')).toBeDefined();
});

describe('context awareness', () => {
it('should pass cell renderers from profile', async () => {
customisationService.set({
id: 'data_table',
logsEnabled: true,
});
await discoverServiceMock.profilesManager.resolveRootProfile({ solutionNavId: 'test' });
const component = await mountComponent(FetchStatus.COMPLETE, esHitsMock);
const discoverGridComponent = component.find(DiscoverGrid);
expect(discoverGridComponent.exists()).toBeTruthy();
expect(Object.keys(discoverGridComponent.prop('externalCustomRenderers')!)).toEqual([
'content',
'resource',
'rootProfile',
]);
});
});
});
Original file line number Diff line number Diff line change
@@ -68,6 +68,7 @@ import { onResizeGridColumn } from '../../../../utils/on_resize_grid_column';
import { useContextualGridCustomisations } from '../../hooks/grid_customisations';
import { useIsEsqlMode } from '../../hooks/use_is_esql_mode';
import { useAdditionalFieldGroups } from '../../hooks/sidebar/use_additional_field_groups';
import { useProfileAccessor } from '../../../../context_awareness';

const containerStyles = css`
position: relative;
@@ -263,6 +264,12 @@ function DiscoverDocumentsComponent({
useContextualGridCustomisations() || {};
const additionalFieldGroups = useAdditionalFieldGroups();

const getCellRenderersAccessor = useProfileAccessor('getCellRenderers');
const cellRenderers = useMemo(() => {
const getCellRenderers = getCellRenderersAccessor(() => customCellRenderer ?? {});
return getCellRenderers();
}, [customCellRenderer, getCellRenderersAccessor]);

const documents = useObservable(stateContainer.dataState.data$.documents$);

const callouts = useMemo(
@@ -373,66 +380,64 @@ function DiscoverDocumentsComponent({
</>
)}
{!isLegacy && (
<>
<div className="unifiedDataTable">
<CellActionsProvider
getTriggerCompatibleActions={uiActions.getTriggerCompatibleActions}
>
<DiscoverGridMemoized
ariaLabelledBy="documentsAriaLabel"
columns={currentColumns}
columnsMeta={columnsMeta}
expandedDoc={expandedDoc}
dataView={dataView}
loadingState={
isDataLoading
? DataLoadingState.loading
: isMoreDataLoading
? DataLoadingState.loadingMore
: DataLoadingState.loaded
}
rows={rows}
sort={(sort as SortOrder[]) || []}
searchDescription={savedSearch.description}
searchTitle={savedSearch.title}
setExpandedDoc={setExpandedDoc}
showTimeCol={showTimeCol}
settings={grid}
onFilter={onAddFilter as DocViewFilterFn}
onSetColumns={onSetColumns}
onSort={onSort}
onResize={onResizeDataGrid}
useNewFieldsApi={useNewFieldsApi}
configHeaderRowHeight={3}
headerRowHeightState={headerRowHeight}
onUpdateHeaderRowHeight={onUpdateHeaderRowHeight}
rowHeightState={rowHeight}
onUpdateRowHeight={onUpdateRowHeight}
isSortEnabled={true}
isPlainRecord={isEsqlMode}
rowsPerPageState={rowsPerPage ?? getDefaultRowsPerPage(services.uiSettings)}
onUpdateRowsPerPage={onUpdateRowsPerPage}
maxAllowedSampleSize={getMaxAllowedSampleSize(services.uiSettings)}
sampleSizeState={getAllowedSampleSize(sampleSizeState, services.uiSettings)}
onUpdateSampleSize={!isEsqlMode ? onUpdateSampleSize : undefined}
onFieldEdited={onFieldEdited}
configRowHeight={uiSettings.get(ROW_HEIGHT_OPTION)}
showMultiFields={uiSettings.get(SHOW_MULTIFIELDS)}
maxDocFieldsDisplayed={uiSettings.get(MAX_DOC_FIELDS_DISPLAYED)}
renderDocumentView={renderDocumentView}
renderCustomToolbar={renderCustomToolbarWithElements}
services={services}
totalHits={totalHits}
onFetchMoreRecords={onFetchMoreRecords}
componentsTourSteps={TOUR_STEPS}
externalCustomRenderers={customCellRenderer}
customGridColumnsConfiguration={customGridColumnsConfiguration}
customControlColumnsConfiguration={customControlColumnsConfiguration}
additionalFieldGroups={additionalFieldGroups}
/>
</CellActionsProvider>
</div>
</>
<div className="unifiedDataTable">
<CellActionsProvider
getTriggerCompatibleActions={uiActions.getTriggerCompatibleActions}
>
<DiscoverGridMemoized
ariaLabelledBy="documentsAriaLabel"
columns={currentColumns}
columnsMeta={columnsMeta}
expandedDoc={expandedDoc}
dataView={dataView}
loadingState={
isDataLoading
? DataLoadingState.loading
: isMoreDataLoading
? DataLoadingState.loadingMore
: DataLoadingState.loaded
}
rows={rows}
sort={(sort as SortOrder[]) || []}
searchDescription={savedSearch.description}
searchTitle={savedSearch.title}
setExpandedDoc={setExpandedDoc}
showTimeCol={showTimeCol}
settings={grid}
onFilter={onAddFilter as DocViewFilterFn}
onSetColumns={onSetColumns}
onSort={onSort}
onResize={onResizeDataGrid}
useNewFieldsApi={useNewFieldsApi}
configHeaderRowHeight={3}
headerRowHeightState={headerRowHeight}
onUpdateHeaderRowHeight={onUpdateHeaderRowHeight}
rowHeightState={rowHeight}
onUpdateRowHeight={onUpdateRowHeight}
isSortEnabled={true}
isPlainRecord={isEsqlMode}
rowsPerPageState={rowsPerPage ?? getDefaultRowsPerPage(services.uiSettings)}
onUpdateRowsPerPage={onUpdateRowsPerPage}
maxAllowedSampleSize={getMaxAllowedSampleSize(services.uiSettings)}
sampleSizeState={getAllowedSampleSize(sampleSizeState, services.uiSettings)}
onUpdateSampleSize={!isEsqlMode ? onUpdateSampleSize : undefined}
onFieldEdited={onFieldEdited}
configRowHeight={uiSettings.get(ROW_HEIGHT_OPTION)}
showMultiFields={uiSettings.get(SHOW_MULTIFIELDS)}
maxDocFieldsDisplayed={uiSettings.get(MAX_DOC_FIELDS_DISPLAYED)}
renderDocumentView={renderDocumentView}
renderCustomToolbar={renderCustomToolbarWithElements}
services={services}
totalHits={totalHits}
onFetchMoreRecords={onFetchMoreRecords}
componentsTourSteps={TOUR_STEPS}
externalCustomRenderers={cellRenderers}
customGridColumnsConfiguration={customGridColumnsConfiguration}
customControlColumnsConfiguration={customControlColumnsConfiguration}
additionalFieldGroups={additionalFieldGroups}
/>
</CellActionsProvider>
</div>
)}
</EuiFlexItem>
</>
Original file line number Diff line number Diff line change
@@ -64,7 +64,7 @@ export function fetchAll(
savedSearch,
abortController,
} = fetchDeps;
const { data } = services;
const { data, expressions, profilesManager } = services;
const searchSource = savedSearch.searchSource.createChild();

try {
@@ -100,14 +100,15 @@ export function fetchAll(

// Start fetching all required requests
const response = isEsqlQuery
? fetchEsql(
? fetchEsql({
query,
dataView,
data,
services.expressions,
abortSignal: abortController.signal,
inspectorAdapters,
abortController.signal
)
data,
expressions,
profilesManager,
})
: fetchDocuments(searchSource, fetchDeps);
const fetchType = isEsqlQuery ? 'fetchTextBased' : 'fetchDocuments';
const startTime = window.performance.now();
Original file line number Diff line number Diff line change
@@ -30,6 +30,10 @@ const getDeps = () =>
} as unknown as FetchDeps);

describe('test fetchDocuments', () => {
beforeEach(() => {
jest.clearAllMocks();
});

test('resolves with returned documents', async () => {
const hits = [
{ _id: '1', foo: 'bar' },
@@ -38,10 +42,17 @@ describe('test fetchDocuments', () => {
const documents = hits.map((hit) => buildDataTableRecord(hit, dataViewMock));
savedSearchMock.searchSource.fetch$ = <T>() =>
of({ rawResponse: { hits: { hits } } } as IKibanaSearchResponse<SearchResponse<T>>);
const resolveDocumentProfileSpy = jest.spyOn(
discoverServiceMock.profilesManager,
'resolveDocumentProfile'
);
expect(await fetchDocuments(savedSearchMock.searchSource, getDeps())).toEqual({
interceptedWarnings: [],
records: documents,
});
expect(resolveDocumentProfileSpy).toHaveBeenCalledTimes(2);
expect(resolveDocumentProfileSpy).toHaveBeenCalledWith({ record: documents[0] });
expect(resolveDocumentProfileSpy).toHaveBeenCalledWith({ record: documents[1] });
});

test('rejects on query failure', async () => {
Original file line number Diff line number Diff line change
@@ -67,7 +67,9 @@ export const fetchDocuments = (
.pipe(
filter((res) => !isRunningResponse(res)),
map((res) => {
return buildDataTableRecordList(res.rawResponse.hits.hits as EsHitRecord[], dataView);
return buildDataTableRecordList(res.rawResponse.hits.hits as EsHitRecord[], dataView, {
processRecord: (record) => services.profilesManager.resolveDocumentProfile({ record }),
});
})
);

Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import type { EsHitRecord } from '@kbn/discover-utils';
import type { ExecutionContract } from '@kbn/expressions-plugin/common';
import { RequestAdapter } from '@kbn/inspector-plugin/common';
import { of } from 'rxjs';
import { dataViewWithTimefieldMock } from '../../../__mocks__/data_view_with_timefield';
import { discoverServiceMock } from '../../../__mocks__/services';
import { fetchEsql } from './fetch_esql';

describe('fetchEsql', () => {
beforeEach(() => {
jest.clearAllMocks();
});

it('resolves with returned records', async () => {
const hits = [
{ _id: '1', foo: 'bar' },
{ _id: '2', foo: 'baz' },
] as unknown as EsHitRecord[];
const records = hits.map((hit, i) => ({
id: String(i),
raw: hit,
flattened: hit,
}));
const expressionsExecuteSpy = jest.spyOn(discoverServiceMock.expressions, 'execute');
expressionsExecuteSpy.mockReturnValueOnce({
cancel: jest.fn(),
getData: jest.fn(() =>
of({
result: {
columns: ['_id', 'foo'],
rows: hits,
},
})
),
} as unknown as ExecutionContract);
const resolveDocumentProfileSpy = jest.spyOn(
discoverServiceMock.profilesManager,
'resolveDocumentProfile'
);
expect(
await fetchEsql({
query: { esql: 'from *' },
dataView: dataViewWithTimefieldMock,
inspectorAdapters: { requests: new RequestAdapter() },
data: discoverServiceMock.data,
expressions: discoverServiceMock.expressions,
profilesManager: discoverServiceMock.profilesManager,
})
).toEqual({
records,
esqlQueryColumns: ['_id', 'foo'],
esqlHeaderWarning: undefined,
});
expect(resolveDocumentProfileSpy).toHaveBeenCalledTimes(2);
expect(resolveDocumentProfileSpy).toHaveBeenCalledWith({ record: records[0] });
expect(resolveDocumentProfileSpy).toHaveBeenCalledWith({ record: records[1] });
});
});
Loading

0 comments on commit 59912e8

Please sign in to comment.