[Search Sessions] Kibana fails to update or delete sessions in non-default space

[Dosant]

Kibana version: 7.12.0

Elasticsearch version: 7.12.0

Server OS version:

Original install method (e.g. download page, yum, from source, etc.): From source or cloud

Describe the bug:

Kibana fails to update and delete search sessions created in non-default space.

This leads to cluster performance degradation caused by a growing number of periodic _async_search/status/<id> requests from Kibana to ES.

Steps to reproduce:

1. Create non-default space. Run from non-default space some searches from Discover or Dashboard
2. Using dev tools observe a growing number of search-sessions objects

GET .kibana*/_search
{
  "query": {
    "match": {
      "type": "search-session"
    }
  }
}

3. Search session objects created in non-default space not transition from IN-PROGRESS -> COMPLETED state and not deleted.

Expected behavior:

When searches complete, the session object should be transitioned to COMPLETED state and then in ~1m completely deleted.
The behavior in the default space is expected.

Screenshots (if relevant):

Symptoms:

Any additional context:

Workaround

To stabilize cluster manually delete search-sessions objects. This has to be done periodically:

POST .kibana_*/_delete_by_query
{
  "query": {
    "bool": {
      "must": [{
        "match": {
          "type": "search-session"
        }
      }, {
        "match": {
          "search-session.persisted": false
        }
      }]
    }
  }
}

And/or disable search-sessions feature in kibana.yml:

xpack.data_enhanced.search.sessions.enabled: false

With search-sessions disabled, session monitoring which calls _async_search/status/<id> will not be scheduled: the number of search-session objects will still grow in a cluster, but tasks that processing them will not run hence not affect cluster performance.

After restarting a 7.12 cluster with xpack.data_enhanced.search.sessions.enabled: false also make sure that the task doesn't exist from the previous run.

GET .kibana_task_manager*/_search
{
  "query": {
    "match": {
      "_id": "task:data_enhanced_search_sessions_monitor"
    }
  }
}

If it exists, then delete it:

POST /.kibana_task_manager*/_delete_by_query
{
  "query": {
    "match": {
      "_id": "task:data_enhanced_search_sessions_monitor"
    }
  }
}

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[geekpete]

Wondering if this could be related in any way (might not be, but wasn't originally reproduced/solved) #53255

[Dosant]

@geekpete,

Wondering if this could be related in any way (might not be, but wasn't originally reproduced/solved) #53255

Looks like different bugs since the current issue applied only to 7.12.0 and search-sesison features.
The underlying reason could be related - misuse of namespaces/permissions APIs, but this should be investigated/fixed separately

[Dosant]

Fixed by #96123