[Ingest Pipelines Editor] First round of UX improvements

[jloleysens]

Summary

Pretty faithful implementation of this #66021 (comment) by @cjcenizal to improve the click tree experience.

Also included some minor style tweaks for the descriptions field and reordered the buttons to be before the description.

How to test

At this time, to test adding and persisting descriptions you would need to build ES master from source. See the details of the yarn es source command. TL;DR fork ES repo in a sibling directory to Kibana and run yarn es source inside of kibana to build from source.

1. Go the Ingest Pipelines app in the Kibana management section
2. Open an existing pipeline or start creating a new one; create some processors too
3. Attempt to update the description of a specific processor
4. Attempt to reorder the processors

Screenshots

Description with text overflow

New move UX

Gifs

[mdefazio]

@jloleysens can you provide an example test document so I can try the debug flow. And perhaps also an example of a custom processor?

[mdefazio]

Some initial thoughts:

Errors:

• Can we list out each one so they are a bit more user-friendly?
• With that said, perhaps we show a max amount of 5 or so and then have a link for '+ 30 more' that expands to show the whole list.
• I think we went through a similar exercise with the Index Mapping editor in regards to the state of the button and validation errors.

reordered the buttons to be before the description

What was the reason for this?

Add processor:

• As mentioned previously, I think we need to show the list of processors initially instead of forcing the user to open the dropdown or start typing.

Reorder UX:

• I like the idea of clicking the move button and getting into a reorder state.

• Perhaps we disable the other rows similar to how we've disabled the 'Add processor' button
  For example:
  

• Then I think we could also add in the lines for where the user can click to drop, but simply have a different color/state for default and hover (gray vs blue)

This screenshot is also showing that by removing the borders around the processors, the drop areas become the main focus. Probably a better way to do this, but all the borders make the UI more confusing.

Apologies for poor screenshots/design hacks. Let me know your thoughts and I can explore some of these a bit more.

[jloleysens]

can you provide an example test document so I can try the debug flow. And perhaps also an example of a custom processor?

I usually have just added something like:

[ { "_source": {}} ] 😅. @alisonelizabeth do you have any cool test docs that you used in the past?

[alisonelizabeth]

@mdefazio It's very basic, but you can test using the example in the docs.

Processor:

{
        "set" : {
          "field" : "field2",
          "value" : "_value"
        }
      }

Documents:

[
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "rab"
      }
    }
  ]

[P1llus]

Here is a test document I use when testing it right now, have to be reformatted a bit from running it in the dev console though, its all testdata generated by me:

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "remove": {
          "field": [
            "message",
            "json.evidence",
            "json.comments",
            "host"
          ],
          "ignore_missing": true
        }
      },
      {
        "set": {
          "field": "cloud.provider",
          "value": "azure"
        }
      },
      {
        "set": {
          "field": "event.kind",
          "value": "alert"
        }
      },
      {
        "set": {
          "field": "event.provider",
          "value": "defender_atp"
        }
      },
      {
        "set": {
          "field": "event.category",
          "value": "malware",
          "if": "ctx.json?.category == 'Malware'"
        }
      },
      {
        "set": {
          "field": "threat.framework",
          "value": "MITRE ATT&CK",
          "if": "ctx.json?.category != null"
        }
      },
      {
        "set": {
          "field": "@timestamp",
          "value": "{{json.alertCreationTime}}",
          "if": "ctx.json?.alertCreationTime != null"
        }
      },
      {
        "append": {
          "field": "event.type",
          "value": ["creation", "start"],
          "if": "ctx.json?.status == 'New'"
        }
      },
      {
        "append": {
          "field": "event.type",
          "value": ["end"],
          "if": "ctx.json?.status == 'Resolved'"
        }
      },
      {
        "remove": {
          "field": "json.alertCreationTime",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.id",
          "target_field": "event.id",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.category",
          "target_field": "threat.technique.name",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.aadTenantId",
          "target_field": "cloud.account.id",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.machineId",
          "target_field": "cloud.instance.id",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.title",
          "target_field": "message",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.description",
          "target_field": "rule.description",
          "ignore_missing": true,
          "if": "(ctx.json?.description).length() < 1020"
        }
      },
      {
        "rename": {
          "field": "json.firstEventTime",
          "target_field": "event.start",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.lastEventTime",
          "target_field": "event.end",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.computerDnsName",
          "target_field": "host.hostname",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.detectionSource",
          "target_field": "observer.name",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.severity",
          "target_field": "event.severity",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.relatedUser.userName",
          "target_field": "host.user.name",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json.relatedUser.domainName",
          "target_field": "host.user.domain",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "json",
          "target_field": "atp",
          "ignore_missing": true
        }
      },
      {
        "remove": {
          "field": "atp.relatedUser",
          "ignore_missing": true
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "@timestamp": "2020-06-13T11:10:14.193Z",
        "@metadata": {
          "beat": "filebeat",
          "type": "_doc",
          "version": "8.0.0",
          "pipeline": "filebeat-8.0.0-microsoft-atp-pipeline"
        },
        "input": {
          "type": "httpjson"
        },
        "agent": {
          "type": "filebeat",
          "version": "8.0.0",
          "ephemeral_id": "e5ecd9ac-7536-496a-a0e5-724ce4c59219",
          "id": "d9234419d-a2c5-cbc71b5c0c08",
          "name": "Hades"
        },
        "fileset": {
          "name": "atp"
        },
        "service": {
          "type": "microsoft"
        },
        "json": {
          "assignedTo": null,
          "computerDnsName": "testmachine1",
          "detectionSource": "WindowsDefenderAv",
          "determination": null,
          "incidentId": 2,
          "investigationId": 1,
          "relatedUser": null,
          "aadTenantId": "291254e-9e30-40034eb7c6f3",
          "category": "Malware",
          "evidence": [],
          "firstEventTime": "2020-05-12T17:50:12.0207431Z",
          "investigationState": "TerminatedBySystem",
          "lastUpdateTime": "2020-05-19T17:57:24.7533333Z",
          "machineId": "4d036540e54dc6ea75196b4b39db297f110",
          "severity": "Medium",
          "threatFamilyName": "Wintapp",
          "title": "An active 'Wintapp' backdoor was detected",
          "comments": [],
          "alertCreationTime": "2020-05-12T17:51:41.2535496Z",
          "classification": null,
          "id": "da12312712346_-193583864",
          "lastEventTime": "2020-05-12T17:50:12.0207431Z",
          "resolvedTime": null,
          "status": "New"
        },
        "ecs": {
          "version": "1.5.0"
        },
        "host": {
          "architecture": "x86_64",
          "os": {
            "kernel": "5.3.0-53-generic",
            "codename": "tricia",
            "platform": "linuxmint",
            "version": "19.3 (Tricia)",
            "family": "",
            "name": "Linux Mint"
          },
          "id": "aaa642a1efb049fba8d6ca0c8d22ff0a",
          "containerized": false,
          "name": "Hades",
          "ip": [
            "192.168.2.2",
            "172.18.0.1",
            "172.17.0.1"
          ],
          "mac": [
            "1c:1b:0d:99:65:3d",
            "02:42:e5:3d:ad:9c",
            "02:42:8a:d1:cc:07"
          ],
          "hostname": "Hades"
        },
        "event": {
          "created": "2020-06-13T11:10:14.193Z",
          "dataset": "microsoft.atp",
          "module": "microsoft"
        }
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "@timestamp": "2020-06-13T11:10:14.194Z",
        "@metadata": {
          "beat": "filebeat",
          "type": "_doc",
          "version": "8.0.0",
          "pipeline": "filebeat-8.0.0-microsoft-atp-pipeline"
        },
        "service": {
          "type": "microsoft"
        },
        "json": {
          "assignedTo": null,
          "classification": null,
          "comments": [],
          "firstEventTime": "2020-05-12T17:49:44.3584856Z",
          "investigationState": "TerminatedBySystem",
          "resolvedTime": null,
          "status": "New",
          "investigationId": 1,
          "threatFamilyName": null,
          "title": "Suspicious PowerShell command line",
          "aadTenantId": "6578-4c7e-9e30-40034eb7c6f3",
          "computerDnsName": "testmachine1",
          "detectionSource": "WindowsDefenderAtp",
          "determination": null,
          "severity": "Medium",
          "machineId": "3454dc6ea75196b4b39db297f110",
          "alertCreationTime": "2020-05-12T17:50:48.5549141Z",
          "category": "Execution",
          "evidence": [],
          "id": "da637249026486486380_684702768",
          "incidentId": 2,
          "lastEventTime": "2020-05-12T17:50:12.0207431Z",
          "lastUpdateTime": "2020-05-19T17:57:24.7533333Z",
          "relatedUser": {
            "domainName": "TestMachine1",
            "userName": "administrator1"
          }
        },
        "event": {
          "created": "2020-06-13T11:10:14.194Z",
          "module": "microsoft",
          "dataset": "microsoft.atp"
        },
        "fileset": {
          "name": "atp"
        },
        "input": {
          "type": "httpjson"
        },
        "ecs": {
          "version": "1.5.0"
        },
        "host": {
          "hostname": "Hades",
          "architecture": "x86_64",
          "os": {
            "name": "Linux Mint",
            "kernel": "5.3.0-53-generic",
            "codename": "tricia",
            "platform": "linuxmint",
            "version": "19.3 (Tricia)",
            "family": ""
          },
          "name": "Hades",
          "id": "aaa642a1efb049fba8d6ca0c8d22ff0a",
          "containerized": false,
          "ip": [
            "192.168.2.2",
            "172.18.0.1",
            "172.17.0.1"
          ],
          "mac": [
            "1c:1b:0d:99:65:3d",
            "02:42:e5:3d:ad:9c",
            "02:42:8a:d1:cc:07"
          ]
        },
        "agent": {
          "id": "d946a2a5-04b3-419d-a2c5-cbc71b5c0c08",
          "name": "Hades",
          "type": "filebeat",
          "version": "8.0.0",
          "ephemeral_id": "e5ecd9ac-7536-496a-a0e5-724ce4c59219"
        }
      }
    }
  ]
}

[jloleysens]

Can we list out each one so they are a bit more user-friendly?

Yeah I agree that we can improve this! I am happy to iterate a bit on your suggestions.

What was the reason for this? (re reordering buttons and description)

IIRC one of the designs had the description next to the processor name which visually communicated well that the description was of the processor, but after adding some content into the descriptions the UI started to look a bit messy to me in certain cases when a number of descriptions of different lengths were being added. Imagine the move and edit button are at the very end of the max potential length of the processor description (like first line in the PR description screenshot).

There may be a different way to solve this, happy to hear alternate suggestions.

This screenshot is also showing that by removing the borders around the processors, the drop areas become the main focus. Probably a better way to do this, but all the borders make the UI more confusing.

This is a cool idea! Changing the color more drastically of the processors for a stronger visual distinction is a really great idea. Also would like to add this when you enter edit mode, it is currently way too subtle.

That being said, I'm not too sold on removing borders on processors. I explored a similar idea with the grey borders and I totally agree that it quickly looks a bit too messy. Are you concerned about the discoverability of the drop zones?

[mdefazio]

That being said, I'm not too sold on removing borders on processors. I explored a similar idea with the grey borders and I totally agree that it quickly looks a bit too messy. Are you concerned about the discoverability of the drop zones?

I'm not sure if my screenshot made enough sense, but my thought was only removing the borders/styling when in 'reorder' mode. But yes, my goal is to try and get the drop zones the focus instead of the rows themselves. (again, only when you click the reorder icon)

I'll do some more thinking around the placement of the icons. I see your point and I also know we tried them being on the right side of the row in the index mapping editor and it came with its own problems.

[jloleysens]

@elasticmachine merge upstream

[jloleysens]

@mdefazio I have updated some of the UX according to your suggestions. I think the dimmed effect makes it much clearer which processor you are currently performing an action on!

Let me know what you think and if you have any other feedback about the positioning of buttons.

[jloleysens]

@elasticmachine merge upstream

[jloleysens]

@elasticmachine merge upstream

[elasticmachine]

merge conflict between base and head

[jloleysens]

@elasticmachine merge upstream

[mdefazio]

Nice work collecting all the various directions/feedback on this.

1. I'm concerned with removing the move icon when in reorder mode—it's not clear that clicking outside will exit me from reordering. Keeping it place and showing it as active will also reinforce which processor they are moving, and that they are in the reorder mode. I know this might lead to some other issues and I'm curious to hear other issues with this approach.
2. Can we remove the box-shadow on the nested panels? It breaks some EUI guidelines with nested panels otherwise.

[P1llus]

Might be a silly question, but any reason we cannot do the order and moving with drag-n-drop instead? There is plenty of libraries out there doing this already, though I don't know how often we want to include anything third party.

Just treating the whole processor box as something that can be pressed and hold to move around, moving it between 2 should just instantly move the other boxes aside, similar to how we do it in other places like dashboards.

[jloleysens]

Hey @P1llus ! That's an excellent question, and one we had asked ourselves when starting this project :). The first iteration started out with drag and drop but there were a few reasons we pivoted away from it (you can check out this commit 74b69c7, although I cannot speak to its overall stability 😅). The reasons are:

1. Drag and drop is a great solution for UI where moving and interactions with the UI need to be pixel-level granular. There are of course implementations where it is used for re-ordering or moving items between boxes (or drop zones), but these are also almost always complemented with a solution that provides a more controlled approach (or at least they should be!). I think the folks over at ES UI put it well in the "Consider your users" section in their docs https://elastic.github.io/eui/#/display/drag-and-drop
2. Drag and drop generally has a slew of accessibility complications. Consider the processors UI where we want to move processors into and out of other processors potentially (as failure handlers), how would we achieve this on a keyboard if I am on a processor that I want to move into a processor below it, but there is another processor above it, it is not obvious that a good a11y solution exists here.
3. Drag and drop does not scale very well in a few important ways:
   1. If the tree is really tall and I want to duplicate and move a processor some 100 processors down, the UX kind of breaks
   2. In our particular implementation, the library was not built for handling DnD at a big scale (I think other libs could work better, but again at some cost to point 2 I mentioned).
4. We did not have time to build a drag and drop solution that would work well for our use case in the time frame we had, other libs were investigated (react-beautiful-dnd used by EUI and react-dnd, more bare bones and even dragula). Given that DnD was not a requirement for the core mechanism to be provided we pivoted away from it.

These points are not to say that we are not open to considering DnD in the future for the tree, I just think for the first iteration it did not quite fit into what we were trying to achieve.

There is some thinking going into how we could build that kind of experience on top of what we have, but I hope this response answers your question.

[P1llus]

Thank you very much for the detailed response, it makes it really clear now why it went the way it did and makes perfect sense.

Especially around the accessibility and performance on large pipelines, wouldn't really be easy with a keyboard unless you was able to tab through the processors and use the up and down key to move it one up or down, and even then we would still have performance issues.

Looking forward to test out the later iterations of the UX! :)

[cjcenizal]

🍻 🍻 🍻

This is looking and feeling solid to me! I think there are refinements we can make and clear avenues of exploration we can pursue in separate PRs, but I think this fundamental UX is sound and merge-worthy. Aside from my comments I also had some other thoughts I want to share.

Making "move mode" more obvious

@mdefazio made some good points that the mechanism for exiting move mode (clicking outside of the tree) was non-obvious and that the transition to move mode could also be communicated more obviously.

I suggest adding a "Cancel move" button to the processor that's being moved, which will somewhat address both issues. I'm hoping that by showing this text to the user, it will be clearer that they're in move mode. My hope is that the button provides a clear point of exit.

Clicking outside of the tree to exit move mode would become a nice enhancement rather than the primary mechanism. The current behavior hitting ESC is also a nice enhancement as well as a boon to accessibility. Nice touch there!

In the future we could explore adding a bottom bar when you're in move mode, implementing my EuiMouseTip suggestion, and/or other ideas.

Processors in move mode

I think this is a big step forward in terms of removing extraneous information when you're in move mode. I suggest two more tweaks to the way a processor is rendered when in this mode:

• I think we should reduce the gap between the processor type and description (above screenshot demonstrates a reduced spacing that I think would work).
• I think we should also hide the "No description" text if there's no description.

This placeholder text is useful to enable the user to do inline editing and add a description. But in move mode the user can't do this, so I think the placeholder text becomes noise.

Processor editor

I suggest changing the label to "Configuration". "Configuration options" seems verbose and it doesn't seem like "options" imparts any additional information.

When there's an error can we disable the "Submit" button? Otherwise the user can click it repeatedly without seeing any response. You have to infer that it's not working because there's an error, but the immediate instinct might be to think something is broken in the UI.

Can we also change the label from "Submit" to "Update"? I think this is a bit more specific to the action being performed. We can also add an "Update processor" tooltip to the edit icon in the processor tree item, for some continuity.

[cjcenizal]

Out of curiosity, what are the scenarios in which there wouldn't be a model?

[jloleysens]

This is quite specific to how monaco works, and tbh I am not entirely certain 😅 . They do provide types that support this claim and I have seen that, on occasion, at runtime when editing the model is unavailable.

[cjcenizal]

Thanks for explaining! Might be worth leaving a comment in the code with this info.

[cjcenizal]

I wonder whether things would feel simpler to the user to just be allowed to move the processor anywhere, even if it's just to "drop" it back in place. If you click move accidentally and want to get out of that mode, would this be convenient?

Put another way: are we helping the user by telling them they can't move the processor above or below where it currently is? Technically, they could move the processor there -- it just wouldn't change position. If a user did this, would that experience be so disorienting that we'd want to prevent it?

I'm just asking these questions aloud, I don't have any answers. :)

[jloleysens]

Hm, that is an interesting point! And kind of philosophical 😄 . If something is moved, but its position did not change did it move?

I gave this some thought; the way I think about it is "wouldn't change position" is equivalent to not moving and this to me is the noop described by cannot move.

The primary goal I think this serves at the moment is giving us a way to make drop zones more discoverable (and the entire drop mode).

[mdefazio]

Per our conversation, approving to move forward with gathering feedback.

[jloleysens]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 8ffad8c

Build metrics

✅ unchanged

History

• 💚 Build #56580 succeeded c38f0dd
• 💚 Build #56248 succeeded b4ea683
• 💚 Build #55682 succeeded 6ac957f
• 💚 Build #55450 succeeded 7f25e6a
• 💚 Build #55408 succeeded fe68f96

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[sebelga]

@jloleysens After watching the recording of @cjcenizal feedback on the design, I had the following comments in case they can be useful:

• Put the "Move" icon on the left of the processor name. This is where I use to find it when I reorganize lists. This way, when entering "Move mode", and all the icons are hidden, it does not suddenly hang in the middle.
• Give a default name to each processor at creation time (if no name was provided) with the following pattern: <processorType> (<total>). So for example Bytes (1). And the counter increments for each new "Bytes" processor added. This name would have a dash underline to make it clear to the user the he can inline edit it. And then I would put the actual processor type as a badge, the same way we do with field "types" in the mappings editor.
• Why not use 2 tabs for the configuration form and the processors as we do in the mappings editor? (fields and advanced configuration)

I also had a question: I am not sure I follow why we don't want ES engineer to give us a unique "name" field instead of the "description". Name can be multiple words but would be a text input. Description means multiple paragraphs and would require a textarea. I wonder how is the search experience going to look like with paragraphs to search into instead of keywords?

[jloleysens]

@sebelga Thanks for this feedback, I really like your idea regarding putting the move button to the left and using badges for the names.

I think in the context of a processor the expectation is not necessarily that users write that much text, because generally I don't think there will be much to describe 😅. The intention of the description field is more in line with what you have described in giving users a way to uniquely name things. At the time we have chosen the term description.