[DISCUSS] Saved-objects find with aggregations

[kobelb]

All of Kibana's saved-objects are stored as Elasticsearch documents, and are used to represent Dashboards, Visualizations, etc. For example, the following Elasticsearch document represents a Visualization:

{
    "visualization" : {
      "title" : "A Visualization",
      "visState" : """{"type":"metric","aggs":[{"id":"1","enabled":true,"type":"count","schema":"metric","params":{}}],"params":{"addTooltip":true,"addLegend":false,"type":"metric","metric":{"percentageMode":false,"useRanges":false,"colorSchema":"Green to Red","metricColorMode":"None","colorsRange":[{"from":0,"to":10000}],"labels":{"show":true},"invertColors":false,"style":{"bgFill":"#000","bgColor":false,"labelColor":false,"subText":"","fontSize":60}}},"title":"A Visualization"}""",
      "uiStateJSON" : "{}",
      "description" : "",
      "version" : 1,
      "kibanaSavedObjectMeta" : {
        "searchSourceJSON" : """{"query":{"query":"","language":"kuery"},"indexRefName":"kibanaSavedObjectMeta.searchSourceJSON.index","filter":[]}"""
      }
    },
    "type" : "visualization",
    "references" : [
      {
        "name" : "kibanaSavedObjectMeta.searchSourceJSON.index",
        "type" : "index-pattern",
        "id" : "800135f0-8429-11ea-98cc-13021164c023"
      }
    ],
    "namespace" : "marketing",
    "migrationVersion" : {
      "visualization" : "7.8.0"
    },
    "updated_at" : "2020-04-21T23:44:13.660Z"
  }

End-users of Kibana don't have access to all saved-objects, and they're commonly restricted to only access a subset of saved-object types in specific namespaces.

Kibana enforces this by requiring that end-users access saved-objects using a service which first performs an authorization check to ensure the user is authorized to access a subset of saved-objects before performing the query against Elasticsearch. There's a REST API which is a facade over the service itself, and our docs enumerate all of the functions that are currently provided: https://www.elastic.co/guide/en/kibana/current/saved-objects-api.html

For example, the following HTTP request to Kibana's saved-objects API:

GET http://localhost:5601/s/marketing/api/saved_objects/_find?type=visualization&fields=title&per_page=0

will be translated into the following ES query:

POST http://localhost:5601/.kibana/_search?size=10000&from=0&_source=visualization.title%2Cnamespace%2Cnamespaces%2Ctype%2Creferences%2CmigrationVersion%2Cupdated_at%2Ctitle&rest_total_hits_as_int=true

{
  "seq_no_primary_term": true,
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "type": "visualization"
                      }
                    },
                    {
                      "term": {
                        "namespace": "marketing"
                      }
                    }
                  ],
                  "must_not": [
                    {
                      "exists": {
                        "field": "namespaces"
                      }
                    }
                  ]
                }
              }
            ],
            "minimum_should_match": 1
          }
        }
      ]
    }
  }
}

There's a PR being reviewed, which expands upon the current functionality of the saved-objects find to allow consumers to now request aggregations.

For example, we want to allow the user to request a max aggregation using the following HTTP request to Kibana's saved-objects API:

GET http://localhost:5601/s/marketing/api/saved_objects/_find?type=visualization&fields=title&per_page=0&aggs=encodeAggs({
 max_version: { 
   max: { 
     field: 'visualization.attributes.version'
   } 
 }
}) 

which will be translated to the following ES query:

POST /.kibana/_search?size=0&from=0&_source=visualization.title%2Cnamespace%2Cnamespaces%2Ctype%2Creferences%2CmigrationVersion%2Cupdated_at%2Ctitle&rest_total_hits_as_int=true

{
  "seq_no_primary_term": true,
  "aggs": {
    "max_version": {
      "max": {
        "field": "visualization.version"
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "type": "visualization"
                      }
                    },
                    {
                      "term": {
                        "namespace": "marketing"
                      }
                    }
                  ],
                  "must_not": [
                    {
                      "exists": {
                        "field": "namespaces"
                      }
                    }
                  ]
                }
              }
            ],
            "minimum_should_match": 1
          }
        }
      ]
    }
  }
}

The query in the request body is necessary to restrict the user to only view the documents which they have access to. As long as the query is limiting the user to the proper subset of documents, are we safe to allow any arbitrary aggregations to be specified by the end-user?

[kobelb]

/cc @elastic/kibana-platform

[nik9000]

Aggregations get filtered by the query so it is probably fairly safe. Plugins could almost certainly break the rules but that's always going to be the case.

[rudolf]

To add to what @kobelb said, we're also doing whitelist validation on the passed in aggregation to prevent any scripts from executing.

[polyfractal]

In addition to what Nik said, you should disallow global aggs, since the purpose of those is to "escape" the filter context and run aggregations against the whole dataset.

Related, parent/child and nested aggs may be a bit "dangerous" too. E.g. the query restricts the scope to matching documents, but a parent/children/nested aggregation can pull in documents that don't necessarily match the query but will show up because their parent/child/nested relationship matches. I suspect this a moot point since I don't think Kibana uses nested/parent-child.

[rudolf]

I don't think Kibana uses nested/parent-child.

Correct, but good to be aware of.

[kobelb]

Thank you everyone for your feedback. I believe that we've come to the consensus that the approach we're intending to implement in Kibana will work, but there are certain aggregations which can't be supported, so care should be taken when adding a new aggregation type.