[Ingest] Editable Pipelines Prototype

[BigFunger]

pipeline menu

• Get filter working
• Redesign it to whatever spec Jurgen/CJ want
• Some sort of hand-holding documentation for new users. See ycombinator's comment below.

Pipeline Header

• On-failure pipeline header needs some design work.

Store Samples with pipeline

• define mapping via kibana during index creation

foreach processor

• Watch issue about exposing output from on-error processors in 'black box' processors
• modifiy pipeline header to display info/navigation for foreach processor

processor plugins

• build bano processor plugin
• build user-agent processor plugin ES #19074ES #19402
• Build repository/merge process for the geoip processor

joi validation

• Add processor validation into the joi validation by adding a pre function to the routes
• Add function to schema of processor_array to recursively fire off validation

Existing Bugs

• If you add a processor, and then save the pipeline before dirtying the new processor, the processor is not saved with the pipeline
• Allow a pipeline with no processors to be executed/saved. It's still a valid pipeline
• Select boxes other than field select do not show that they have focus (css)
• Weird behavior with breadcrumbs
• Typing values with handlebar syntax too quickly causes elasticsearch to throw an error.

Cleanup

• DRY up the dependencies/less of the processor-container/global on fail sections
• figure out why bindToController doesn't work for processor directives
• convert all of the pipeline directives to use bindToController
• remove failure_action from all tests that don't pertain to failure_action (since it's optional anyway)

Tests

• Once structure/functionality is ok'ed, need tests, lots of tests!

Long term/large goals

• Exporting/Importing pipelines
• Undo/Redo | Back/Forward state management
• Fully featured GROK utility
• More automated Date pattern discovery processor
• Visual representation of processor hierarchy
• Rethink the real-time simulation process

[ycombinator]

Jim has welcomed reviews from the Kibana team for the ingest UI. Here's my feedback:

• When there are no ingest pipelines created, consider hiding the filter bar so only the Add button is shown. That way it is obvious to the user what they can do next in the UI.

• My guess is that most users who visit the pipeline UI know what they are doing. However, there might be a class of users who are there out of curiosity. For them especially it might help to link "Elasticsearch Ingest pipelines" in the copy on top of the page to https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html or some other introductory documentation/video. This is similar to how we link to the Lucene expressions documentation in the Add Scripted Field UI.

• I like how you focus the "Select a Processor..." select list when the New button is clicked. There are other UI elements on the page but I am instantly drawn to the select list, which is the immediate next step I need to take. 👍

• Nit: How difficult would it be to not show the trailing comma here:

  

• Consider labeling or otherwise clearly separating the pipeline output metadata section from the pipeline output data section:

  

• There seem to be two different UI treatments for displaying help text; consider using the same UI treatment for a consistent UX:

  

• Nit: Currently, when a pipeline has one or more processors in it, the "Select a Processor..." at the very bottom isn't immediately obvious (to me) as the thing to click on to add a new processor. Consider changing it to "Add a Processor...". If you make this change, you'll probably want to change the initial "Your pipeline is currently empty. Add a processor to get started!" message to just "Your pipeline is currently empty."

• Consider moving the input selection arrows closer to the inputs section:
  

• I imagine a typical pipeline has at least a couple of processors. If so, the page could get vertically long when defining a pipeline. Consider providing the Save button at the bottom of the page as well so the user doesn't need to either scroll up or collapse all processors to Save the pipeline.

• Along similar lines as the previous point, consider auto-collapsing any processors the user isn't currently focussed on.

• Nice to have: Consider allowing the user to change the order of processors using drag-n-drop, similar to what we recently implemented in the Visualize editor:

  

• Is it intentional that the processor input and pipeline output show JSON objects but the processor output shows JS objects?

  

[epixa]

@BigFunger What do you want to do with this issue?