[7.17] Some CSV reporting tests fail when run against ES 8.0

[lukeelmers]

When testing 7.17 Kibana against 8.0 Elasticsearch, we get some CSV reporting-related failures from CI:

[job] [logs] Default CI Group #25 / discover Discover CSV Export Generate CSV: new search generates a report from a new search with data: default
[job] [logs] Default CI Group #25 / discover Discover CSV Export Generate CSV: new search generates a report from a new search with data: default
[job] [logs] Default CI Group #2 / Reporting APIs CSV Generation from SearchSource Exports CSV with almost all fields when using fieldsFromSource
[job] [logs] Default CI Group #2 / Reporting APIs CSV Generation from SearchSource Exports CSV with almost all fields when using fieldsFromSource

https://buildkite.com/elastic/kibana-7-dot-latest-es-8-dot-0-forward-compatibility/builds/15

In 7.17, basic Kibana functions are still expected to work when run against ES 8.0, because during an upgrade process users will upgrade ES first.

So the ask here is to:

• investigate the failures from the build
• determine if they are simply bugs in our tests, or legit issues
• if they are legit issues, we should determine if they need to be raised as 7.17 blockers

To test locally (be sure to update the config paths & string to grep for):

git checkout 7.17
TEST_ES_BRANCH=8.0.0 node scripts/functional_tests_server.js --config=path/to/suite/config.ts
# in another terminal session
node scripts/functional_test_runner --config=path/to/suite/config.ts --grep="whatever"

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

@lukeelmers thank you for including the commands to run the test locally!

I am guessing these Reporting CSV tests are failing due to a problem with the test themselves. They error out as soon as I try to load the es archive: https://github.com/elastic/kibana/blob/no-include_type_name/x-pack/test/reporting_api_integration/services/scenarios.ts#L54

That triggers this rejection:

ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: request [/ecommerce] contains unrecognized parameter: [include_type_name]

Is there an alternative way to load the archive data?

[rudolf]

@tsullivan Was this fixed by #123093 or is further work required?

[tsullivan]

Hi @rudolf,
Mikhail reverted the PR as it causes the tests to fail when run against 7.x ES.

These tests are specific to 7.x. The tests blindly query for all of the fields that can be returned from ES. In 7.x, the results include _type and in 8.0, they do not.

This doesn't seem like a blocker to me. The test coverage for CSV export with ES 8.0 is adequate in the main and 8.0 branches. We also have working tests when 7.17 is run against 8.0, where the request to generate CSV includes a set list of fields, and there is no _type interplay.

Could we flag these 7.17 tests in a way that will skip them when the test server is using ES 8.0?

[tsullivan]

Looks like these tests will be skipped in this PR: #123312

It will add this.onlyEsVersion('<=7') to our tests in 7.17 branch only. This will be the best outcome for the tests in 7.17: the work to adapt them for cross-version is going to be more work than it's worth. And, there is already a lot of test coverage on other csv exports.

[tsullivan]

I'm not sure the tests were skipped as selectively as they should have been: https://github.com/elastic/dev/issues/1855#issuecomment-1021444827

[tsullivan]

@jloleysens @dokmic could one of you take a look into this? In the 7.17 branch, we have this.onlyEsVersion('<=7') statements in the code for CSV tests:

> x-pack/test/functional/apps/discover/reporting.ts:    this.onlyEsVersion('<=7');
  x-pack/test/reporting_api_integration/reporting_and_security/ilm_migration_apis.ts:    this.onlyEsVersion('<=7');
  x-pack/test/reporting_api_integration/reporting_and_security/generate_csv_discover.ts:    this.onlyEsVersion('<=7');
  x-pack/test/reporting_api_integration/reporting_and_security/search_frozen_indices.ts:    this.onlyEsVersion('<=7');
  x-pack/test/reporting_api_integration/reporting_and_security/download_csv_dashboard.ts:    this.onlyEsVersion('<=7');

We should make sure these statements are not skipping more tests than is necessary. The only tests that make sense to skip when Kibana 7.17 is run against ES 8.0 are the ones that export "all" the fields from the index. In those tests, ES in 7.x will return a _type field, but in 8.0 it will not.

[tsullivan]

Resolved by #123981