Switch to embeddable factory interface with optional override

[stacey-gammon]

The main goal of this PR is to support the ability for another plugin to modify the creation of turning EmbeddableFactoryDefinitions to EmbeddableFactory instances.

The downside is that only one plugin can do it. This is the same limitation the same pattern that we use for savedObjectClient has.

Broken down, this PR:

• EmbeddableFactory is an interface now, not an abstract class.
• Introduction of new Embeddable API setEmbeddableFactoryProvider. This is tested in a jest test but not used. There is no sample plugin explaining how to use this. Since we can only have one plugin extend it, and we know AppArch team will be registering the one plugin that we will do this (EmbeddableEnhanced), I don't think we should advertise this extension point.
• If we can figure out how to support multiple plugins intercepting this creation process, which would be beneficial, then we should add tests/example usage.
• Many tests needed to be updated to take into account the difference between EmbeddableFactoryDefinition and EmbeddableFactory and now use embeddablePluginMock. This is better for testing anyway.
• VisualizeEmbeddable and MapEmbeddable exposed new functionality on the EmbeddableFactory that is no longer carried over to the instance of EmbeddableFactory, so those usages needed to be removed/changed.
• One downside to this new approach is to preserve this context in the definition functions, I have to use const create = async () => semantics instead of create() {....
  • I am now binding all the functions so implementors aren't required to do this.

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[nreese]

maps changes LGTM
code review

[spong]

SIEM code changes look good, however when testing locally I'm seeing the provided layerList duplicated in both error and loading states. All map requests to /internal/search/es also fail with a 400.

Could this be an error in our configuration, or perhaps the provided layerList is getting initialized twice?

Here's the failed request payload

{
  "params": {
    "ignoreThrottled": true,
    "preference": 1585690218105,
    "index": "filebeat-*",
    "body": {
      "size": 0,
      "aggs": {
        "destSplit": {
          "terms": {
            "script": {
              "source": "doc['destination.geo.location'].value.toString()",
              "lang": "painless"
            },
            "order": {
              "_count": "desc"
            },
            "size": 100
          },
          "aggs": {
            "sourceGrid": {
              "geotile_grid": {
                "field": "source.geo.location",
                "precision": 2,
                "size": 500
              },
              "aggs": {
                "sourceCentroid": {
                  "geo_centroid": {
                    "field": "source.geo.location"
                  }
                },
                "sum_of_source.bytes": {
                  "sum": {
                    "field": "source.bytes"
                  }
                },
                "sum_of_destination.bytes": {
                  "sum": {
                    "field": "destination.bytes"
                  }
                }
              }
            }
          }
        }
      },
      "stored_fields": [
        "*"
      ],
      "script_fields": {},
      "docvalue_fields": [
        {
          "field": "@timestamp",
          "format": "date_time"
        },
        {
          "field": "azure.auditlogs.properties.activity_datetime",
          "format": "date_time"
        },
        {
          "field": "azure.enqueued_time",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.agentReceiptTime",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.deviceCustomDate1",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.deviceCustomDate2",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.deviceReceiptTime",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.endTime",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.fileCreateTime",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.fileModificationTime",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.flexDate1",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.managerReceiptTime",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.oldFileCreateTime",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.oldFileModificationTime",
          "format": "date_time"
        },
        {
          "field": "cef.extensions.startTime",
          "format": "date_time"
        },
        {
          "field": "event.created",
          "format": "date_time"
        },
        {
          "field": "event.end",
          "format": "date_time"
        },
        {
          "field": "event.ingested",
          "format": "date_time"
        },
        {
          "field": "event.start",
          "format": "date_time"
        },
        {
          "field": "file.accessed",
          "format": "date_time"
        },
        {
          "field": "file.created",
          "format": "date_time"
        },
        {
          "field": "file.ctime",
          "format": "date_time"
        },
        {
          "field": "file.mtime",
          "format": "date_time"
        },
        {
          "field": "kafka.block_timestamp",
          "format": "date_time"
        },
        {
          "field": "misp.campaign.first_seen",
          "format": "date_time"
        },
        {
          "field": "misp.campaign.last_seen",
          "format": "date_time"
        },
        {
          "field": "misp.intrusion_set.first_seen",
          "format": "date_time"
        },
        {
          "field": "misp.intrusion_set.last_seen",
          "format": "date_time"
        },
        {
          "field": "misp.observed_data.first_observed",
          "format": "date_time"
        },
        {
          "field": "misp.observed_data.last_observed",
          "format": "date_time"
        },
        {
          "field": "misp.report.published",
          "format": "date_time"
        },
        {
          "field": "misp.threat_indicator.valid_from",
          "format": "date_time"
        },
        {
          "field": "misp.threat_indicator.valid_until",
          "format": "date_time"
        },
        {
          "field": "netflow.collection_time_milliseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.exporter.timestamp",
          "format": "date_time"
        },
        {
          "field": "netflow.flow_end_microseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.flow_end_milliseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.flow_end_nanoseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.flow_end_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.flow_start_microseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.flow_start_milliseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.flow_start_nanoseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.flow_start_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.max_export_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.max_flow_end_microseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.max_flow_end_milliseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.max_flow_end_nanoseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.max_flow_end_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.min_export_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.min_flow_start_microseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.min_flow_start_milliseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.min_flow_start_nanoseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.min_flow_start_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.monitoring_interval_end_milli_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.monitoring_interval_start_milli_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.observation_time_microseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.observation_time_milliseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.observation_time_nanoseconds",
          "format": "date_time"
        },
        {
          "field": "netflow.observation_time_seconds",
          "format": "date_time"
        },
        {
          "field": "netflow.system_init_time_milliseconds",
          "format": "date_time"
        },
        {
          "field": "package.installed",
          "format": "date_time"
        },
        {
          "field": "process.parent.start",
          "format": "date_time"
        },
        {
          "field": "process.start",
          "format": "date_time"
        },
        {
          "field": "suricata.eve.flow.end",
          "format": "date_time"
        },
        {
          "field": "suricata.eve.flow.start",
          "format": "date_time"
        },
        {
          "field": "suricata.eve.timestamp",
          "format": "date_time"
        },
        {
          "field": "suricata.eve.tls.notafter",
          "format": "date_time"
        },
        {
          "field": "suricata.eve.tls.notbefore",
          "format": "date_time"
        },
        {
          "field": "tls.client.not_after",
          "format": "date_time"
        },
        {
          "field": "tls.client.not_before",
          "format": "date_time"
        },
        {
          "field": "tls.server.not_after",
          "format": "date_time"
        },
        {
          "field": "tls.server.not_before",
          "format": "date_time"
        },
        {
          "field": "zeek.kerberos.valid.from",
          "format": "date_time"
        },
        {
          "field": "zeek.kerberos.valid.until",
          "format": "date_time"
        },
        {
          "field": "zeek.ocsp.revoke.time",
          "format": "date_time"
        },
        {
          "field": "zeek.ocsp.update.next",
          "format": "date_time"
        },
        {
          "field": "zeek.ocsp.update.this",
          "format": "date_time"
        },
        {
          "field": "zeek.pe.compile_time",
          "format": "date_time"
        },
        {
          "field": "zeek.smb_files.times.accessed",
          "format": "date_time"
        },
        {
          "field": "zeek.smb_files.times.changed",
          "format": "date_time"
        },
        {
          "field": "zeek.smb_files.times.created",
          "format": "date_time"
        },
        {
          "field": "zeek.smb_files.times.modified",
          "format": "date_time"
        },
        {
          "field": "zeek.smtp.date",
          "format": "date_time"
        },
        {
          "field": "zeek.snmp.up_since",
          "format": "date_time"
        },
        {
          "field": "zeek.x509.certificate.valid.from",
          "format": "date_time"
        },
        {
          "field": "zeek.x509.certificate.valid.until",
          "format": "date_time"
        }
      ],
      "_source": {
        "excludes": []
      },
      "query": {
        "bool": {
          "must": [],
          "filter": [
            {
              "match_all": {}
            },
            {
              "match_all": {}
            },
            {
              "geo_bounding_box": {
                "destination.geo.location": {
                  "top_left": [
                    -180,
                    89.78601
                  ],
                  "bottom_right": [
                    180,
                    -89.78601
                  ]
                }
              }
            },
            {
              "range": {
                "@timestamp": {
                  "gte": "2020-03-30T21:36:06.271Z",
                  "lte": "2020-03-31T21:36:06.271Z",
                  "format": "strict_date_optional_time"
                }
              }
            }
          ],
          "should": [],
          "must_not": []
        }
      }
    },
    "rest_total_hits_as_int": true,
    "ignore_unavailable": true,
    "ignore_throttled": true,
    "timeout": "30000ms"
  },
  "serverStrategy": "es"
}

[stacey-gammon]

ugghhh, thanks for looking @spong. I don't have an easy way to get SIEM data , the old creds I had for the hosted es instance are causing errors ( FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that can obfuscate privilege-related issues. You should use the "kibana" user instead.
). Will try to sync up with you tomorrow so I can debug.

What do you mean by

the provided layerList duplicated in both error and loading states.

btw, probably a good spot to add a functional test for so the error would be caught on ci.

[stacey-gammon]

@nreese - would you mind helping me out with this error? I stepped through and setLayerList is being called as I'd expect. This function already existed on the MapEmbeddable, but I don't think it was called anywhere prior to this PR, so maybe there is a bug when setting the layer list outside of the constructor? I'm a bit out of my element digging into the maps code to debug.

[nreese]

I have located the problem that @spong is running into. The problem is that replaceLayerList is getting called before the map has loaded/rendered. This means that layers are getting added to waitingForMapReadyLayerList state instead of layerList. replaceLayerList has a bug and does not clear out waitingForMapReadyLayerList so thats why the layers are showing up multiple times. Replace https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/maps/public/actions/map_actions.js#L175 with

export function replaceLayerList(newLayerList) {
  return (dispatch, getState) => {
    dispatch({
      type: CLEAR_WAITING_FOR_MAP_READY_LAYER_LIST,
    });
    getLayerListRaw(getState()).forEach(({ id }) => {
      dispatch(removeLayerFromLayerList(id));
    });

    newLayerList.forEach(layerDescriptor => {
      dispatch(addLayer(layerDescriptor));
    });
  };
}

[nreese]

created #62202 to address the problem in replaceLayerList action creator that is resulting in duplicated layers.

[majagrubic]

Tested dashboard and visualize embeddable flow, works as expected. Had a look at other embeddables, all seems fine to me. Haven't tested other scenarios.

[majagrubic]

LGTM

[nreese]

#62202 is merged so the problem should be resolved now

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 971faaa

History

• 💚 Build #37930 succeeded cb40ba2
• 💚 Build #37554 succeeded 06bdd5a
• 💔 Build #37530 failed 218066e
• 💚 Build #37318 succeeded 245075d
• 💔 Build #37273 failed 61bf750

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[spong]

Checked out and verified layerList fix is in place. The SIEM Network Map now no longer has duplicate layers in both loading/error states. In testing I found another issue with map requests failing (and thus no data is shown), but have opened #62356 to track.

Approving as base SIEM changes LGTM -- thanks @stacey-gammon and @nreese!