[Search Sessions] don't stay stable during upgrade

[flash1293]

Kibana version: 7.14

Describe the bug: Search session can't be restored after an upgrade form 7.13 to 7.14

Steps to reproduce:

1. Save sessions for discover/dashboard in 7.13
2. Upgrade stack to 7.14
3. Try to restore session
4. Additional requests are made

Expected behavior:
Session should be restored

Any additional context:
I'm not sure whether it's possible to really solve this - apps change their behavior in different versions and emit different queries, so it might not even be possible to retain a session. Maybe we should just delete all search sessions on upgrade and start over?

E.g. this is the Discover difference which seems legit to me:

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[Dosant]

FYI to someone who will work on this, there is a hacky way to skip documents during migration: #96938 currently we skip non-persisted sessions, but we could skip all the sessions.

cc @ppisljar, this even doesn't look like a persistable state use-case 😮

I think the best would be to indeed NOT move those sessions to new kibana index. But then what can we do with hanging async searches which might execute in a background for a week

[flash1293]

I think the best would be to indeed NOT move those sessions to new kibana index. But then what can we do with hanging async searches which might execute in a background for a week

Can we delete the async searches and drop the search session saved objects?

Wrt implementation it could be done as part of the data plugin on startup - check whether there are sessions from older versions, if yes delete them and any associated async searches

[ppisljar]

I agree, it sounds we should delete those.

[Dosant]

Just to sum up:

I think we can’t do anything about it, I think we can only delete search sessions from previous kibana versions.
There are two ways to do it:

1. The simplest way is to drop search sessions when migrating kibana index (we already drop non-persisted sessions) . I think the caveat is that any existing async searches will stay running in es.
2. Or do manual clean-up as part of our background task. This will include cleaning up async searches and could free up some resources

We could also leave those search sessions, but add a warning on search sessions from previous versions saying “Might not fully be restorable” 😞