[Ingest Node Pipelines] Pipeline Processors Editor (#66021)

* initial plugin setup * add smoke test * fix license check * refactor plugin setup * Server-side create/update ingest pipelines (#62744) * List pipelines (#62785) * First iteration of ingest table * Add action placeholders * Refactor list and address PR feedback Refactored the list into smaller pieces and assemble in main.tsx Also addressed feedback on copy, removed unused notifications dep * WiP on flyout Showing name in title * Add reload button * Finish first version of flyout * Slight update to copy * `delete` -> `edit` * Address PR feedback Copy and a11y updates * Add on failure JSON to flyout if it is available * Add details json block file and remove ununsed import Co-authored-by: Elastic Machine <[email protected]> * [Ingest pipelines] Create pipeline UI (#63017) * First vertical slice of pipeline editor component * Made a space for common parameters * [Ingest pipelines] Edit pipeline page (#63522) * First iteration of CRUD functionality working * WiP on moving the pipeline editor to pipeline processor editor * Finish refactor to work with passing state out * Refactor and fix tests * [Ingest pipelines] Polish details panel and empty list (#63926) * Address some early feedback and use FormDataProvider FormDataProvider gives a more declarative way to listen to form state updates. Also refactored the state reader mechanism. * [Ingest pipelines] Delete pipeline (#63635) * [Ingest Node Pipelines] Clone Pipeline (#64049) * First iteration of clone functionality Wired up for both the list table and the details flyout in the list section. * satisfy eslint * Turn on sorting for the list table * Clean up const declarations * Address PR feedback Sentence-casify and update some other copy. * Mark edit and delete as primary actions in list table * Handle URI encoded chars in pipeline name when cloning * Update to using the more flexible controlled component pattern To make this component and mappings editor more consistent, both will expose a more traditional controlled component interface to consumers. In the current implementation this requires that consumers use `useCallback` for `onUpdate` handlers to avoid an infinite rendering cycle and to avoid staleness bugs in their `onUpdate` handlers they need to think about what might make the handler stale. This approach comes with the benefits and flexibility of controlled components at the cost of slightly more complex consumption of the components. In future, we can explore adding the uncontrolled component interface too so that consumers have the option to more simply render and pull data out only when needed * Handle sub-form validity The pipelines processor editor not emits overall validity to consumers * Fix Jest test * Refactor some names prepareDataOut -> serialize prepareDataIn -> deserialize EditorProcessor -> ProcessorInternal * Mark as private * Major WiP Started working on the drag-and-drop-tree and updated some typings stuff * [Ingest node pipelines] Privileges (#63850) * Create privileges check for ingest pipelines app Also moved the public side logic for checking and rendering privilege related messages to es_ui_shared/public following the new __packages_do_not_import__ convention. * Add , * Fix import paths * Address PR feedback Fix i18n strings (remove reference to snapshot and restore) and fix copy referencing snapshot and restore - all copy-pasta errors. Also remove unused field from missing privileges object. * Fix issue from resolving merge conflicts * Add missing app privilege * Use non-deprecated privilige name * First iteration of drag and drop tree on feature parity * First steps toward add on failure handler Updated reducer logic to create the next copy value using immer. * First iteration of nested tree with add on failure working * Refactor and some UI layout updates - Remove the "id" field on processors for now - Implement the nested remove and update functionality again * Remove immer (not call stack safe) Refac to remove immer and reimplemented the immutable set and get functions. * [Ingest Node Pipelines] More lenient treatment of on-failure value (#64411) * Move file to components folder * [Ingest pipelines] Simulate pipeline (#64223) * Updated tree rendering - turn off dropzones for children - fixed up some padding and margins - fixed integration with pipeline_form.tsx The current implementation still has a lot of jank stemming from the UX with DnD. Unfortunately the nesting has opened a number of issues here. * [Ingest Node Pipelines] Show flyout after editing or creating a pipeline (#64409) * Show flyout after editing or creating a pipeline * JSX comment * Show not found flyout Copied from CCR * update not found flyout and fix behavior when viewing details from table * Reset pipeline name in URI when closing flyout * Remove encodeURI Already using encodingURIComponent for unsafe string. Co-authored-by: Alison Goryachev <[email protected]> * Clarification of terms - addProcessor -> addTopLevelProcessor (same in editor modes) - Expanded comment on ProcessorSelector * Implement move between lists functionality Added tests to the reducer for moving in and out of a recusrsive structure. * fix TS * Prevent nesting a parent inside of its own child * Add comment * [Ingest pipelines] Cleanup (#64794) * address review feedback * remove unused import * Big refactor to tree rendering structure DnD tree now converts the nested tree to a flat one and only consists of _1_ droppable area with a flat array of draggables that can be combined. Using the existing logic in the reducer combined with translating the flat structure changes to a format the nested reducer can understand looks like a really promising avenue. There still seems to be a bug with a longer list where items do not interact properly. * Remove unused component * A number of NB changes - Fixed a subtle serialisation bug with on_failure in options - Fixed a performance bug by using useRef in pipeline form - Memoized the drag and drop tree - Removed use of "isValid" from form lib... I think this should be removed entirely. * fix bad conflict resolution * Implemented a slightly better destination resolution algo. Also added tests. * Fix subtle staleness bug, whitelist keys for setValue * NB styling fix!! Due to a parent's setting of overflow: hidden the drag and drop tree had a dead zone that would equal the page overflow limit, unsetting on the component itself (overflow: auto) relaxes this limitation. * Fix stale delete bug too * Update naming of editor modes and update comments * Use field types combo box * Add delete confirmation modal * Refactor delete modal component file name * Better visual integration with existing form * Update layout and styling of form - added some padding around the new processors editor field - Updated use of flex box in the pipeline form fields * Move pipeline processor copy into pipeline processor component The test button is also now inside of the pipeline processor editor. Eventually all of this functionality should be moved into the pipeline processor editor. * First step of refactor to moving between trees * First iteration of x-tree drag and drop * Remove unused import * Fix jest test types * Fix up minor i18n issues and fix up layout of on failure * Remove unnecessary prop * Update spacing above add processor button to make it more center * Fix destination resolution algo * Update dragging resolver unit tests and add a lot more comments * Use one sorting algo (removed use of euiReorder for now) * Add placeholder tests and update comments * Quite a big refactor - Remove DraggableLocation entirely, only use ProcessorSelector this simplifies mapping back to reducer instructions quite a lot - Add tests for special case dragging behaviour * Fix off by one bug in tests and implementation 🤦🏼‍♂ - Also move processor reducer to it's own folder * Update behaviour for dragging up across trees and add tests * Fix combine instruction * Fix test and i18n issues * Remove background color * Fix selector after selector refactor * A major performance - Do not re-render the entire tree when editing settings * Fix component smoke test * Fix reading value from processor state using processor selector * [Ingest pipelines] Custom processor form (#66022) * Re add background color and refactor name of processor item * Fix file naming and refactor 🚜 dnd tree rendering - deserialze and serialize were backwards, fixed - the dnd tree was rendering a flat tree which needed to be mapped back to the nested tree for updates. This is still the case but we do use recursive rendering. This enables tree nodes to better hold local state like whether they are collapsed or not. * Fix getting of initial on failure processors value * Update padding styles for containers * A lot of styling updates to get closer to look of mockup * A WiP version of th click-tree an alternative to dnd As a response to really long pipelines we may pivot away from dnd. * Remove dnd tree * clean up reamining dnd tree references * Clean up and refactor of tree component To simplify the logic of the tree component processor id's were added. Also had to update the jest spec to support this * Added the ability to duplicate a processor * Fix types in test * Added duplicate functionality to ui * Memoize tree components * address es lint issues * remove unused import * Fix editing of custom json * Address form performance issues * Add all known missing processors * Add ability to cancel move * Fix staleness in test and view request flyouts * fix type issue * Remove unused translations and skip funcitonal test for now * add todo comment * Fix type issues * remove poc styles * disable other move buttons if we have a selected processor * Refactor drop zone pin to button and add some styling * Refactor processor editor item * Update styling and use icon for cancel move action too * fix nasty integration bug * some minor optimizations * prevent parent from being placed in own on failure handler * Re-add cancel button * Re-introduce failure processors toggle * Fix typo * Add Handler types for processor editor item * Fix staleness bug for type, refactor classname and fix duplicate bug not immutably copying values * Experimenting with padels (revert this to undo if no further changes have been made) * Add description and unique ids * Share links via component-wide context rather than props * Virtualized list and back to outline dropzones * Refactor id getter to a service and make it an incrementing number * Temporary fix for double rendering issue. This means the pipeline processors component is not controllable but fixes the double rendering even when processors have not changed. This is a problem coming from the ingest pipelines plugin form system outside of this component * add todo comment * remove euicode element * properly handle duplicate flow * attempt to fix i18n * split private_tree into it's own component and add comments * refactor 🚜. rename Tree to ProcessorsTree and move things around * do not delete the top level arrays for processors and onfailures * fix typescript error * Move duplicate, addOnFailure and delete actions into ctx menu * remove unused import * add support for pressing esc key to cancel move * Add outside click listener * always prompt before deleting a processor * refactor remove distinction between adding top level and on fail * add processor button to tree * Hide the add on failure context menu item for processors with failure handlers * Reinstated x-tree moving and highlight and disable for buttons on move and on edit * removing ids step 1: remove idGenerator * remove ids step 2: added inline text input Also refactored a lot of the tree actions away. Now using context and the processors editor tree item component for dispatching actions. The tree item has access to the dispatch and to the selector which makes it a well positioned component. Also saves on some props drilling. * Slight improvement to styling of text input (border) * Re-implement missing failure toggle test * address type todo * Address many type issues and fix yarn.lock * re-enable create pipeline functional test * prevent multiple flyouts from opening * change flyout title when editing an on-failure processor * absolutely position the failure handlers label when we render a dropzone on the label, then we move the label up without affecting overall component height * fix description behaviour not removing tag if empty * some minor clean up * add onflyoutopen cb to tests * refactor processors editor item to multiple files also refactored i18n into it's own file. would be good to come up with a pttaern for doing this more broadly. * fix add on-failure handler in context menu after refactor * tag -> new description field Co-authored-by: Alison Goryachev <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Alison Goryachev <[email protected]>
elastic · Jun 16, 2020 · 7511a6e · 7511a6e
1 parent dd1f64d
commit 7511a6e
Show file tree

Hide file tree

Showing 59 changed files with 3,438 additions and 197 deletions.
diff --git a/x-pack/package.json b/x-pack/package.json
@@ -331,6 +331,7 @@
     "react-syntax-highlighter": "^5.7.0",
     "react-tiny-virtual-list": "^2.2.0",
     "react-use": "^13.27.0",
+    "react-virtualized": "^9.21.2",
     "react-vis": "^1.8.1",
     "react-visibility-sensor": "^5.1.1",
     "recompose": "^0.26.0",

diff --git a/x-pack/plugins/ingest_pipelines/__jest__/client_integration/ingest_pipelines_create.test.tsx b/x-pack/plugins/ingest_pipelines/__jest__/client_integration/ingest_pipelines_create.test.tsx
@@ -68,21 +68,6 @@ describe('<PipelinesCreate />', () => {
       expect(exists('versionField')).toBe(true);
     });
 
-    test('should toggle the on-failure processors editor', async () => {
-      const { actions, component, exists } = testBed;
-
-      // On-failure editor should be hidden by default
-      expect(exists('onFailureEditor')).toBe(false);
-
-      await act(async () => {
-        actions.toggleOnFailureSwitch();
-        await nextTick();
-        component.update();
-      });
-
-      expect(exists('onFailureEditor')).toBe(true);
-    });
-
     test('should show the request flyout', async () => {
       const { actions, component, find, exists } = testBed;
 

diff --git a/x-pack/plugins/ingest_pipelines/common/types.ts b/x-pack/plugins/ingest_pipelines/common/types.ts
@@ -4,10 +4,16 @@
  * you may not use this file except in compliance with the Elastic License.
  */
 
-interface Processor {
-  [key: string]: {
-    [key: string]: unknown;
-  };
+export interface ESProcessorConfig {
+  on_failure?: Processor[];
+  ignore_failure?: boolean;
+  if?: string;
+  tag?: string;
+  [key: string]: any;
+}
+
+export interface Processor {
+  [typeName: string]: ESProcessorConfig;
 }
 
 export interface Pipeline {

diff --git a/...ck/plugins/ingest_pipelines/public/application/components/pipeline_form/pipeline_form.tsx b/...ck/plugins/ingest_pipelines/public/application/components/pipeline_form/pipeline_form.tsx
@@ -4,7 +4,7 @@
  * you may not use this file except in compliance with the Elastic License.
  */
 
-import React, { useState } from 'react';
+import React, { useState, useCallback, useRef } from 'react';
 import { FormattedMessage } from '@kbn/i18n/react';
 import { EuiButton, EuiButtonEmpty, EuiFlexGroup, EuiFlexItem, EuiSpacer } from '@elastic/eui';
 
@@ -16,6 +16,11 @@ import { PipelineTestFlyout } from './pipeline_test_flyout';
 import { PipelineFormFields } from './pipeline_form_fields';
 import { PipelineFormError } from './pipeline_form_error';
 import { pipelineFormSchema } from './schema';
+import {
+  OnUpdateHandlerArg,
+  OnUpdateHandler,
+  SerializeResult,
+} from '../pipeline_processors_editor';
 
 export interface PipelineFormProps {
   onSave: (pipeline: Pipeline) => void;
@@ -30,8 +35,8 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
   defaultValue = {
     name: '',
     description: '',
-    processors: '',
-    on_failure: '',
+    processors: [],
+    on_failure: [],
     version: '',
   },
   onSave,
@@ -44,10 +49,25 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
 
   const [isTestingPipeline, setIsTestingPipeline] = useState<boolean>(false);
 
+  const processorStateRef = useRef<OnUpdateHandlerArg>();
+
   const handleSave: FormConfig['onSubmit'] = async (formData, isValid) => {
-    if (isValid) {
-      onSave(formData as Pipeline);
+    let override: SerializeResult | undefined;
+
+    if (!isValid) {
+      return;
+    }
+
+    if (processorStateRef.current) {
+      const processorsState = processorStateRef.current;
+      if (await processorsState.validate()) {
+        override = processorsState.getData();
+      } else {
+        return;
+      }
     }
+
+    onSave({ ...formData, ...(override || {}) } as Pipeline);
   };
 
   const handleTestPipelineClick = () => {
@@ -60,6 +80,10 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
     onSubmit: handleSave,
   });
 
+  const onEditorFlyoutOpen = useCallback(() => {
+    setIsRequestVisible(false);
+  }, [setIsRequestVisible]);
+
   const saveButtonLabel = isSaving ? (
     <FormattedMessage
       id="xpack.ingestPipelines.form.savingButtonLabel"
@@ -77,6 +101,11 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
     />
   );
 
+  const onProcessorsChangeHandler = useCallback<OnUpdateHandler>(
+    (arg) => (processorStateRef.current = arg),
+    []
+  );
+
   return (
     <>
       <Form
@@ -90,10 +119,13 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
 
         {/* All form fields */}
         <PipelineFormFields
+          onEditorFlyoutOpen={onEditorFlyoutOpen}
+          initialProcessors={defaultValue.processors}
+          initialOnFailureProcessors={defaultValue.on_failure}
+          onProcessorsUpdate={onProcessorsChangeHandler}
           hasVersion={Boolean(defaultValue.version)}
           isTestButtonDisabled={isTestingPipeline || form.isValid === false}
           onTestPipelineClick={handleTestPipelineClick}
-          hasOnFailure={Boolean(defaultValue.on_failure)}
           isEditing={isEditing}
         />
 
@@ -147,13 +179,19 @@ export const PipelineForm: React.FunctionComponent<PipelineFormProps> = ({
         {/* ES request flyout */}
         {isRequestVisible ? (
           <PipelineRequestFlyout
+            readProcessors={() =>
+              processorStateRef.current?.getData() || { processors: [], on_failure: [] }
+            }
             closeFlyout={() => setIsRequestVisible((prevIsRequestVisible) => !prevIsRequestVisible)}
           />
         ) : null}
 
         {/* Test pipeline flyout */}
         {isTestingPipeline ? (
           <PipelineTestFlyout
+            readProcessors={() =>
+              processorStateRef.current?.getData() || { processors: [], on_failure: [] }
+            }
             closeFlyout={() => {
               setIsTestingPipeline((prevIsTestingPipeline) => !prevIsTestingPipeline);
             }}

diff --git a/...ins/ingest_pipelines/public/application/components/pipeline_form/pipeline_form_fields.tsx b/...ins/ingest_pipelines/public/application/components/pipeline_form/pipeline_form_fields.tsx
@@ -6,39 +6,41 @@
 
 import React, { useState } from 'react';
 import { FormattedMessage } from '@kbn/i18n/react';
-import { i18n } from '@kbn/i18n';
-import { EuiButton, EuiSpacer, EuiSwitch, EuiLink } from '@elastic/eui';
+import { EuiSpacer, EuiSwitch } from '@elastic/eui';
 
-import {
-  getUseField,
-  getFormRow,
-  Field,
-  JsonEditorField,
-  useKibana,
-} from '../../../shared_imports';
+import { Processor } from '../../../../common/types';
+import { FormDataProvider } from '../../../shared_imports';
+import { PipelineProcessorsEditor, OnUpdateHandler } from '../pipeline_processors_editor';
+
+import { getUseField, getFormRow, Field, useKibana } from '../../../shared_imports';
 
 interface Props {
+  initialProcessors: Processor[];
+  initialOnFailureProcessors?: Processor[];
+  onProcessorsUpdate: OnUpdateHandler;
   hasVersion: boolean;
-  hasOnFailure: boolean;
   isTestButtonDisabled: boolean;
   onTestPipelineClick: () => void;
+  onEditorFlyoutOpen: () => void;
   isEditing?: boolean;
 }
 
 const UseField = getUseField({ component: Field });
 const FormRow = getFormRow({ titleTag: 'h3' });
 
 export const PipelineFormFields: React.FunctionComponent<Props> = ({
+  initialProcessors,
+  initialOnFailureProcessors,
+  onProcessorsUpdate,
   isEditing,
   hasVersion,
-  hasOnFailure,
   isTestButtonDisabled,
   onTestPipelineClick,
+  onEditorFlyoutOpen,
 }) => {
   const { services } = useKibana();
 
   const [isVersionVisible, setIsVersionVisible] = useState<boolean>(hasVersion);
-  const [isOnFailureEditorVisible, setIsOnFailureEditorVisible] = useState<boolean>(hasOnFailure);
 
   return (
     <>
@@ -110,127 +112,32 @@ export const PipelineFormFields: React.FunctionComponent<Props> = ({
         />
       </FormRow>
 
-      {/* Processors field */}
-      <FormRow
-        title={
-          <FormattedMessage
-            id="xpack.ingestPipelines.form.processorsFieldTitle"
-            defaultMessage="Processors"
-          />
-        }
-        description={
-          <>
-            <FormattedMessage
-              id="xpack.ingestPipelines.form.processorsFieldDescription"
-              defaultMessage="The processors to use to transform the documents before indexing. {learnMoreLink}"
-              values={{
-                learnMoreLink: (
-                  <EuiLink
-                    href={services.documentation.getProcessorsUrl()}
-                    target="_blank"
-                    external
-                  >
-                    {i18n.translate('xpack.ingestPipelines.form.processorsDocumentionLink', {
-                      defaultMessage: 'Learn more',
-                    })}
-                  </EuiLink>
-                ),
-              }}
-            />
-
-            <EuiSpacer />
+      {/* Pipeline Processors Editor */}
+      <FormDataProvider pathsToWatch={['processors', 'on_failure']}>
+        {({ processors, on_failure: onFailure }) => {
+          const processorProp =
+            typeof processors === 'string' && processors
+              ? JSON.parse(processors)
+              : initialProcessors ?? [];
 
-            <EuiButton
-              size="s"
-              onClick={onTestPipelineClick}
-              disabled={isTestButtonDisabled}
-              data-test-subj="testPipelineButton"
-            >
-              <FormattedMessage
-                id="xpack.ingestPipelines.form.testPipelineButtonLabel"
-                defaultMessage="Test pipeline"
-              />
-            </EuiButton>
-          </>
-        }
-      >
-        <UseField
-          path="processors"
-          component={JsonEditorField}
-          componentProps={{
-            euiCodeEditorProps: {
-              'data-test-subj': 'processorsEditor',
-              height: '300px',
-              'aria-label': i18n.translate('xpack.ingestPipelines.form.processorsFieldAriaLabel', {
-                defaultMessage: 'Processors JSON editor',
-              }),
-            },
-          }}
-        />
-      </FormRow>
+          const onFailureProp =
+            typeof onFailure === 'string' && onFailure
+              ? JSON.parse(onFailure)
+              : initialOnFailureProcessors ?? [];
 
-      {/* On-failure field */}
-      <FormRow
-        title={
-          <FormattedMessage
-            id="xpack.ingestPipelines.form.onFailureTitle"
-            defaultMessage="Failure processors"
-          />
-        }
-        description={
-          <>
-            <FormattedMessage
-              id="xpack.ingestPipelines.form.onFailureDescription"
-              defaultMessage="The alternate processors to execute after a processor fails. {learnMoreLink}"
-              values={{
-                learnMoreLink: (
-                  <EuiLink
-                    href={services.documentation.getHandlingFailureUrl()}
-                    target="_blank"
-                    external
-                  >
-                    {i18n.translate('xpack.ingestPipelines.form.onFailureDocumentionLink', {
-                      defaultMessage: 'Learn more',
-                    })}
-                  </EuiLink>
-                ),
-              }}
-            />
-            <EuiSpacer size="m" />
-            <EuiSwitch
-              label={
-                <FormattedMessage
-                  id="xpack.ingestPipelines.form.onFailureToggleDescription"
-                  defaultMessage="Add failure processors"
-                />
-              }
-              checked={isOnFailureEditorVisible}
-              onChange={(e) => setIsOnFailureEditorVisible(e.target.checked)}
-              data-test-subj="onFailureToggle"
+          return (
+            <PipelineProcessorsEditor
+              onFlyoutOpen={onEditorFlyoutOpen}
+              learnMoreAboutProcessorsUrl={services.documentation.getProcessorsUrl()}
+              learnMoreAboutOnFailureProcessorsUrl={services.documentation.getHandlingFailureUrl()}
+              isTestButtonDisabled={isTestButtonDisabled}
+              onTestPipelineClick={onTestPipelineClick}
+              onUpdate={onProcessorsUpdate}
+              value={{ processors: processorProp, onFailure: onFailureProp }}
             />
-          </>
-        }
-      >
-        {isOnFailureEditorVisible ? (
-          <UseField
-            path="on_failure"
-            component={JsonEditorField}
-            componentProps={{
-              euiCodeEditorProps: {
-                'data-test-subj': 'onFailureEditor',
-                height: '300px',
-                'aria-label': i18n.translate('xpack.ingestPipelines.form.onFailureFieldAriaLabel', {
-                  defaultMessage: 'Failure processors JSON editor',
-                }),
-              },
-            }}
-          />
-        ) : (
-          // <FormRow/> requires children or a field
-          // For now, we return an empty <div> if the editor is not visible
-          <div />
-        )}
-      </FormRow>
+          );
+        }}
+      </FormDataProvider>
     </>
   );
 };