Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Serverless Search] Improve awareness on using pipelines #182177

Merged
merged 17 commits into from
May 3, 2024
Merged

[Serverless Search] Improve awareness on using pipelines #182177

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
3baa1af
improve pipelines awareness
saarikabhasi Apr 30, 2024
323136f
add optional badge
saarikabhasi Apr 30, 2024
f54bf1f
update tests and FTR
saarikabhasi Apr 30, 2024
d9281e6
add links
saarikabhasi May 1, 2024
d80d267
add suggested copy changes
saarikabhasi May 2, 2024
19bc39c
fix ci
saarikabhasi May 2, 2024
f5cdf64
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine May 2, 2024
5a6a534
replace with svg images
saarikabhasi May 2, 2024
3759857
Merge branch 'data-preprocessing' of github.com:saarikabhasi/kibana i…
saarikabhasi May 2, 2024
6d1c985
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine May 2, 2024
1433dd0
fix FTR tests error
saarikabhasi May 2, 2024
e6afa5d
Merge branch 'data-preprocessing' of github.com:saarikabhasi/kibana i…
saarikabhasi May 2, 2024
68171dc
Merge branch 'main' into data-preprocessing
kibanamachine May 2, 2024
55b88b6
clean up
saarikabhasi May 2, 2024
7e82642
Merge branch 'data-preprocessing' of github.com:saarikabhasi/kibana i…
saarikabhasi May 2, 2024
59e464a
move default ingestion pipeline to constant
saarikabhasi May 2, 2024
61a8579
Merge branch 'main' into data-preprocessing
saarikabhasi May 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
228 changes: 228 additions & 0 deletions packages/kbn-search-api-panels/components/preprocess_data.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,228 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/
import {
EuiFlexGroup,
EuiFlexItem,
EuiSpacer,
EuiText,
EuiThemeProvider,
EuiTitle,
EuiToken,
EuiPanel,
EuiLink,
EuiFlexGrid,
} from '@elastic/eui';
import { i18n } from '@kbn/i18n';
import React from 'react';

export const PreprocessDataPanel: React.FC = () => {
return (
<EuiThemeProvider colorMode="dark">
<EuiPanel paddingSize="xl">
<EuiFlexGrid direction="column" columns={2} gutterSize="xl">
<EuiFlexItem>
<EuiFlexGroup alignItems="flexStart" justifyContent="flexStart">
<EuiFlexItem grow={false}>
<EuiToken iconType="sparkles" size="l" color="#466B8D" />
saarikabhasi marked this conversation as resolved.
Show resolved Hide resolved
</EuiFlexItem>
<EuiFlexItem>
<EuiTitle size="s">
<h3>
{i18n.translate(
'searchApiPanels.preprocessData.overview.dataEnrichment.title',
{
defaultMessage: 'Data enrichment',
}
)}
</h3>
</EuiTitle>
<EuiSpacer size="s" />
<EuiText size="s">
<p>
{i18n.translate(
'searchApiPanels.preprocessData.overview.dataEnrichment.description',
{
defaultMessage:
'Add information from external sources or apply transformations to your documents for more contextual, insightful search.',
}
)}
</p>
</EuiText>
<EuiSpacer size="s" />
<EuiText>
<p>
<EuiLink href={'#'} target="_blank">
Copy link
Contributor

@TattdCodeMonkey TattdCodeMonkey Apr 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this is for serverless we will need to populate all links before we can merge. We have to assume any commit can go to production.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I agree

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated all links - d9281e6

{i18n.translate(
'searchApiPanels.preprocessData.overview.dataEnrichment.learnMore',
{
defaultMessage: 'Learn More',
}
)}
</EuiLink>
</p>
</EuiText>
</EuiFlexItem>
</EuiFlexGroup>
</EuiFlexItem>
<EuiFlexItem>
<EuiFlexGroup alignItems="flexStart" justifyContent="flexStart">
<EuiFlexItem grow={false}>
<EuiToken iconType="logstashFilter" size="l" color="#466B8D" />
</EuiFlexItem>
<EuiFlexItem>
<EuiTitle size="s">
<h3>
{i18n.translate('searchApiPanels.pipeline.overview.dataFiltering.title', {
defaultMessage: 'Data filtering',
})}
</h3>
</EuiTitle>
<EuiSpacer size="s" />
<EuiText size="s">
{i18n.translate('searchApiPanels.pipeline.overview.dataFiltering.description', {
defaultMessage:
'Remove fields from documents like sensitive information from documents before indexing.',
saarikabhasi marked this conversation as resolved.
Show resolved Hide resolved
})}
</EuiText>
<EuiSpacer size="s" />
<EuiText>
<p>
<EuiLink href={'#'} target="_blank">
{i18n.translate(
'searchApiPanels.preprocessData.overview.dataFiltering.learnMore',
{
defaultMessage: 'Learn More',
}
)}
</EuiLink>
</p>
</EuiText>
</EuiFlexItem>
</EuiFlexGroup>
</EuiFlexItem>
<EuiFlexItem>
<EuiFlexGroup alignItems="flexStart" justifyContent="flexStart">
<EuiFlexItem grow={false}>
<EuiToken iconType="tokenArray" size="l" fill="none" color="#466B8D" />
</EuiFlexItem>
<EuiFlexItem>
<EuiTitle size="s">
<h3>
{i18n.translate('searchApiPanels.pipeline.overview.arrayJsonHandling.title', {
defaultMessage: 'Array/JSON handling',
})}
</h3>
</EuiTitle>
<EuiSpacer size="s" />
<EuiText size="s">
{i18n.translate(
'searchApiPanels.pipeline.overview.arrayJsonHandling.description',
{
defaultMessage: 'Run batch processors, parse JSON data and sort elements.',
saarikabhasi marked this conversation as resolved.
Show resolved Hide resolved
}
)}
</EuiText>
<EuiSpacer size="s" />
<EuiText>
<p>
<EuiLink href={'#'} target="_blank">
{i18n.translate(
'searchApiPanels.preprocessData.overview.arrayJsonHandling.learnMore',
{
defaultMessage: 'Learn More',
saarikabhasi marked this conversation as resolved.
Show resolved Hide resolved
}
)}
</EuiLink>
</p>
</EuiText>
</EuiFlexItem>
</EuiFlexGroup>
</EuiFlexItem>
<EuiFlexItem>
<EuiFlexGroup alignItems="flexStart" justifyContent="flexStart">
<EuiFlexItem grow={false}>
<EuiToken iconType="cut" size="l" color="#466B8D" />
</EuiFlexItem>
<EuiFlexItem>
<EuiTitle size="s">
<h3>
{i18n.translate('searchApiPanels.pipeline.overview.dataTransformation.title', {
defaultMessage: 'Data transformation',
})}
</h3>
</EuiTitle>
<EuiSpacer size="s" />
<EuiText size="s">
{i18n.translate(
'searchApiPanels.pipeline.overview.dataTransformation.description',
{
defaultMessage:
'Parse information from your documents to ensure they conform to a standardized format.',
}
)}
</EuiText>
<EuiSpacer size="s" />
<EuiText size="s">
<p>
<EuiLink href={'#'} target="_blank">
{i18n.translate(
'searchApiPanels.preprocessData.overview.dataTransformation.learnMore',
{
defaultMessage: 'Learn More',
}
)}
</EuiLink>
</p>
</EuiText>
</EuiFlexItem>
</EuiFlexGroup>
</EuiFlexItem>
<EuiFlexItem>
<EuiFlexGroup alignItems="flexStart" justifyContent="flexStart">
<EuiFlexItem grow={false}>
<EuiToken iconType="controlsVertical" size="l" color="#466B8D" />
</EuiFlexItem>
<EuiFlexItem>
<EuiTitle size="s">
<h3>
{i18n.translate('searchApiPanels.pipeline.overview.pipelineHandling.title', {
defaultMessage: 'Pipeline handling',
})}
</h3>
</EuiTitle>

<EuiText size="s">
{i18n.translate(
'searchApiPanels.pipeline.overview.pipelineHandling.description',
{
defaultMessage:
'Handle error exceptions, execute another pipeline or reroute document to another index',
saarikabhasi marked this conversation as resolved.
Show resolved Hide resolved
}
)}
</EuiText>
<EuiSpacer size="s" />
<EuiText>
<p>
<EuiLink href={'#'} target="_blank">
{i18n.translate(
'searchApiPanels.preprocessData.overview.pipelineHandling.learnMore',
{
defaultMessage: 'Learn More',
}
)}
</EuiLink>
</p>
</EuiText>
</EuiFlexItem>
</EuiFlexGroup>
</EuiFlexItem>
</EuiFlexGrid>
</EuiPanel>
</EuiThemeProvider>
);
};
1 change: 1 addition & 0 deletions packages/kbn-search-api-panels/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ export * from './components/pipeline_panel';
export * from './components/select_client';
export * from './components/try_in_console_button';
export * from './components/install_client';
export * from './components/preprocess_data';

export * from './types';
export * from './utils';
Expand Down
67 changes: 66 additions & 1 deletion ...search/public/application/components/connectors/connector_config/connector_index_name.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,21 @@
* 2.0.
*/

import { EuiButton, EuiFlexGroup, EuiFlexItem, EuiSpacer, EuiText, EuiTitle } from '@elastic/eui';
import {
EuiButton,
EuiFlexGroup,
EuiFlexItem,
EuiLink,
EuiPanel,
EuiSpacer,
EuiText,
EuiTitle,
} from '@elastic/eui';
import { i18n } from '@kbn/i18n';
import { Connector, ConnectorStatus } from '@kbn/search-connectors';
import React, { useState } from 'react';
import { useQueryClient, useMutation } from '@tanstack/react-query';
import { FormattedMessage } from '@kbn/i18n-react';
import { isValidIndexName } from '../../../../utils/validate_index_name';
import { SAVE_LABEL } from '../../../../../common/i18n_string';
import { useConnector } from '../../../hooks/api/use_connector';
Expand Down Expand Up @@ -80,6 +90,61 @@ export const ConnectorIndexName: React.FC<ConnectorIndexNameProps> = ({ connecto
onChange={(name) => setNewIndexname(name)}
/>
<EuiSpacer />
<EuiPanel hasBorder>
<EuiFlexGroup direction="column" justifyContent="spaceBetween">
<EuiFlexItem grow={false}>
<EuiTitle size="s">
<h3>
{i18n.translate('xpack.serverlessSearch.connectors.config.preprocessData.title', {
defaultMessage: 'Preprocess your data',
})}
</h3>
</EuiTitle>
</EuiFlexItem>
<EuiFlexItem grow={false}>
<EuiText>
<p>
<FormattedMessage
id="xpack.serverlessSearch.connectors.config.preprocessData.description"
defaultMessage="You can use ingest pipelines to preprocess data before indexing into Elasticseearch. Connectors clients use {clientIngestionPipeline} pipeline before indexing data into Elasticsearch"
saarikabhasi marked this conversation as resolved.
Show resolved Hide resolved
values={{
clientIngestionPipeline: (
<strong>
{i18n.translate(
'xpack.serverlessSearch.connectors.config.preprocessData.clientIngestionPipeline',
{
defaultMessage: 'search-default-ingestion',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since this is the name of the default ingest pipeline I think we should

  1. use a constant for it
  2. we should not translate it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe use code format too?

search-default-ingestion

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 59e464a

}
)}
,
</strong>
),
}}
/>
</p>
</EuiText>
</EuiFlexItem>
<EuiFlexItem>
<EuiText>
<p>
<EuiLink
data-test-subj="serverlessSearchConnectorIndexNameLearnMoreLink"
href={'#'}
target="_blank"
>
{i18n.translate(
'xpack.serverlessSearch.connectors.config.preprocessDataTitle.learnMore',
{
defaultMessage: 'Learn More',
}
)}
</EuiLink>
</p>
</EuiText>
</EuiFlexItem>
</EuiFlexGroup>
</EuiPanel>
<EuiSpacer />
<ApiKeyPanel connector={connector} />
<EuiSpacer />
<EuiFlexGroup>
Expand Down
8 changes: 6 additions & 2 deletions x-pack/plugins/serverless_search/public/application/components/overview.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,13 @@ describe('<Overview />', () => {
const { getByRole } = render(<Overview />);
expect(getByRole('heading', { name: 'Build your first search query' })).toBeDefined();
});
test('transform data', () => {
test('preprocess data', () => {
const { getByRole } = render(<Overview />);
expect(getByRole('heading', { name: 'Transform and enrich your data' })).toBeDefined();
expect(
getByRole('heading', {
name: 'Preprocess your data by enriching, transforming or filtering with pipelines',
})
).toBeDefined();
});
test("what's next?", () => {
const { getByRole } = render(<Overview />);
Expand Down
Loading