[Meta] Refactor Kibana querying infrastructure ("courier")

[lukasolson]

Related: #29215, #20364

Summary

Currently, our querying infrastructure (a.k.a. "courier") has been around since the early days of 4.x. Over time, lots of functionality has been stuffed/mashed in, not always considering best practices. As a result, courier has lots of complicated functionality and is a cause of a lot of technical debt. Also, it doesn't have a simple or stable API, so usage by core and 3rd party plugins is limited, resulting in a disjointed system for data access across plugins. (Things like Timelion and TSVB don't use courier.)

There are several steps we will take in refactoring courier:

Create a server API (7.4)

• (In progress: [skip ci] Data access service #38737) We'd like to eventually get rid of our Elasticsearch proxy inside Kibana. By creating a server API (similar to how the rollups search strategy works), we can remove courier's dependence on the proxy.

Add support for request cancellation (7.4, pending elastic/elasticsearch#43332)

(In progress: #39963) Kibana users & admins would like queries to be cancellable in many situations:

• On long queries (perhaps after prompting user)
• On window unload/route change
• On resubmit/refresh/update/time range change/agg config change/panel config change/panel resize

Elasticsearch has this ability, yet courier has not supported it. We'd like to add these capabilities to courier and allow apps to take advantage of query cancellation.

Refactor search source to use the new server APIs and remove Elasticsearch proxy (7.5)

• We will refactor the existing search source to use the new server-side APIs introduced above, which will remove the need for our Elasticsearch proxy, at which point we will remove it. In this step we will still be using our existing filter/agg config models.
  • Switch management of index patterns over to the new data service.
  • Switch vega over to using the new data service.
  • Remove usage of es service to keep session alive in security plugin.
  • Remove 'es' angular service.
  • Remove elasticsearch/{indexpattern}/search endpoint
  • Switch usage of elasticsearch/_msearch endpoint in tutorials to data.search service
  • Switch usage of elasticsearch/_msearch endpoint in field_editor to data.search service
  • Remove elasticsearch/_msearch endpoint

Associate search requests with a user/saved object (7.5)

• (POC: [skip-ci] [POC] Use x-opaque-id to associate search request with user/saved object #41530) We'd like to use the X-Opaque-Id header in Elasticsearch to associate search requests with a given user/saved object.

Add support for query progress (7.6, pending Elasticsearch ability to do so)

• As soon as Elasticsearch supports it, we would like to expose the ability to provide consumers with the current progress (as a percentage of complete) of the query execution.

Create a high-level API with abstractions (7.7)

• Refactor & replace the agg config model with something better (TBD)
• Create Typescript interfaces/models for the following:
  • Specifying which fields to query (scripted fields, stored fields, _source fields, _score, aggregation results)
  • Filtering (on fields, scripted fields, aggregation results)
  • Sorting (on fields, scripted fields, aggregation results)
  • Paging (on fields, scripted fields, aggregation results)
  • Highlighting

Related issues:

Refactoring

• Dashboard avoid reliance on broadcast emits
• Remove angular promises from Courier

Performance/batching

• Courier should fail early if it receives a response it can't process
• Improve dashboard load performance
• HTTP response streaming
• Surface more information for diagnosing shard failures
• Keep same order of requests in _msearch
• Don't send scripted fields if not needed

Query cancellation

• Cancel outstanding requests when new time period is selected
• Cancel query / filter automatically if new query entered or running query is removed
• Cancel underlying query when switching context in Kibana or old query is replaced
• Cancel underlying aggregation query if aggregation is modified
• Ignore responses for previous query when filter/query has changed
• Inflight _msearch requests are retried if visualizations are resized on dashboard

Elasticsearch query options

• Allow sorting on multiple fields
• All Kibana requests with a client timeout should propagate these to elasticsearch
• Provide support for request_cache in msearch queries
• Users selection to allow queries to run beyond configurable timeout
• Add search IDs to requests

[lukasolson]

Closing in favor of https://github.com/elastic/kibana/projects/43.