[Reporting] CSV reporting does not work on frozen indices even with Search enabled in Advanced settings

[radoondas]

Kibana version: 7.5.+ (maybe sooner)

Elasticsearch version: 7.5.+

Server OS version: any

Browser version: any

Original install method (e.g. download page, yum, from source, etc.): ECE, ESS, other

Describe the bug:
CSV reporting is not providing any data from frozen indices.
Even search on frozen indices is enabled in Advanced settings.

When the index is back in unfrozen state, CSV reporting on that index is working as expected.

Steps to reproduce:
1 Add demo data
2. Freeze the demo data index
3. Enable search on frozen indices
4. Go on discovery & check the data is there
6. Save the saved search
6. Export from CSV -> Fails with error
Error: Expected _scroll_id in the following Elasticsearch response: {"took":0,"timed_out":false,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]}}
7. Create a Dashboard, add the saved search, save the dashboard, export csv from visualization > Fails with the same error

Same error in both cases.

Checking the Kibana logs we get the query:
POST /kibana_sample_data_logs/_search?scroll=30s&size=500

Kibana doesn’t pass the ignore_throttled=false

Expected behavior:
Reporting will search and get documents from frozen indices

Thanks!

[aidofitz]

+1 - am also being impacted by this issue

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

@elastic/kibana-app @elastic/kibana-app-arch I'd like to get some inputs here. There is a server-side data plugin in Kibana, which didn't exist when CSV export was first created.

Since we are not using any data access services for CSV export, the reporting team ends up adding all features which seem like they should be automatic. It is super hard for the Reporting team to invest effort to re-engineer CSV export. Is this something we can collaborate on? Is there any possible timeline for doing so?

[slesru]

Hit this yesterday, is there any workaround? May be it is at least unfreeze indexes from kibana?
Thank you!

[lukeelmers]

@tsullivan The search service in the data plugin automatically handles setting ignore_throttled based on the "search frozen indices" advanced setting. We are in the process of exposing the rest of the search service on the server, in which case reporting should be able to leverage it for cases like this.

@lizozom or @lukasolson may have more thoughts on this

[tsullivan]

I'm currently working on some code cleanup for Export CSV (Discover) and Download CSV (Saved Search Dashboard Panel) and one of the goals is to streamline the code a bit so the basic functionality can be replaced with the data plugin on the server. I need to collaborate with someone on this 😄

Thank you @lukeelmers for following up.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[tsullivan]

This issue will be closed out when CSV Export moves to using SearchSource on the server. That work is prioritized since it is needed for CSV Export to support Runtime Fields: #81079