Async Discover search test #64388

LeeDr · 2020-04-24T00:50:35Z

Summary

This PR is for new functional UI tests in x-pack for Discover with async search.

There are currently 2 tests;
query should show failed shards pop up
query return results with valid scripted field (fails on 7.7.0 BC8)

The trick to these tests are in the scripted fields. The test originally started with "empty_kibana" .kibana index and created the index pattern and scripted fields through the UI. But the last commit has that already set in the kibana_scripted_fields_on_logstash esArchive to make the test much faster.
With a couple of scripted fields that cause Elasticsearch to have to do some work, the initial response from Elasticsearch in that case wasn't being handled properly and would cause the "no results" message to show. With the fixes which are already merged the query returns the expected results.

The first test causes the failed shards popup. A recent fix in Elasticsearch fixed this.

NOTE: One thing unusually about the test file in it's current version is that it has all the steps commented-out which were used to create the scripted fields which are now in the esArchive. I know we don't usually like to leave a bunch of commented-out code in our files but this might be a practice to consider. If someone needed to change what was in the esArchive it would be pretty painful without those steps.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios
This was checked for cross-browser compatibility, including a check against IE11

For maintainers

This was checked for breaking API changes and was labeled appropriately

kertal · 2020-04-29T16:42:54Z

ACK & THX @LeeDr, will review it tomorrow

LeeDr · 2020-04-30T15:45:11Z

Results of the 2 new tests;

x-pack/test/functional/apps/discover/async_scripted_fields·js	1 min 12 sec	0		0		2	+2	2	+2

LeeDr · 2020-04-30T15:45:29Z

@elasticmachine merge upstream

kertal · 2020-05-05T13:26:24Z

started a flaky test runner for it, looks stable: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/424/

elasticmachine · 2020-05-07T18:07:20Z

Pinging @elastic/kibana-app (Team:KibanaApp)

…earchTest

LeeDr · 2020-05-08T21:13:51Z

I ran the new test 42 times on the flaky test runner and all passed.

LeeDr · 2020-05-08T21:14:09Z

@elasticmachine merge upstream

LeeDr · 2020-05-13T13:00:41Z

@elasticmachine merge upstream

LeeDr · 2020-05-15T15:53:01Z

Last Jenkins job test failure indicates some problem loading data;

[00:09:11]             └-> "before all" hook
[00:09:11]               │ info [kibana_scripted_fields_on_logstash] Loading "mappings.json"
[00:09:11]               │ info [kibana_scripted_fields_on_logstash] Loading "data.json.gz"
[00:09:11]               │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1589374883702750076] [.kibana_2/JGNjvdJ6RlmhSgo3Rld8GA] deleting index
[00:09:11]               │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1589374883702750076] [.kibana_1/EBvRVAWZQ_idG5p26KCaOw] deleting index
[00:09:11]               │ info [kibana_scripted_fields_on_logstash] Deleted existing index [".kibana_2",".kibana_1"]
[00:09:11]               │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/discover async search with scripted fields _before all_ hook.png"
[00:09:11]               │ info Current URL is: http://localhost:6191/s/another-space/app/discover#/c760fbb0-9522-11ea-963a-eba904777d7f?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(_source),filters:!(),index:%27logstash-*%27,interval:auto,query:(language:kuery,query:%27%27),sort:!())
[00:09:12]               │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/discover async search with scripted fields _before all_ hook.html
[00:09:12]               └- ✖ fail: "discover async search with scripted fields "before all" hook for "query should show failed shards pop up""

Stacktrace
{ Error: [illegal_argument_exception] unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
    at respond (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/dev/shm/workspace/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 400,
  displayName: 'BadRequest',
  message:
   '[illegal_argument_exception] unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings',
  path: '/.kibana_1',

LeeDr · 2020-05-15T15:53:42Z

@elasticmachine merge upstream

…earchTest

LeeDr · 2020-06-24T16:02:13Z

@elasticmachine merge upstream

LeeDr · 2020-06-24T21:40:43Z

@kertal remember this PR? :-) It fell out of date but I've got it up and passing again if you have a chance to review it.

kertal · 2020-06-25T07:05:34Z

@LeeDr thx, sure, how could I forget :), will review it

kertal · 2020-06-26T06:35:27Z

x-pack/test/functional/apps/discover/async_scripted_fields.js

+      // changing the timepicker default here saves us from having to set it in Discover (~8s)
+      await kibanaServer.uiSettings.update({
+        'timepicker:timeDefaults':
+          '{  "from": "Sep 18, 2015 @ 19:37:13.000",  "to": "Sep 23, 2015 @ 02:30:09.000"}',
+      });


Yes, this is nice, imagine how much time would be saved if we regularly do it that way. which reminds me, I recently created a helper for that, in the TimePickerProvider, but it's using a different start, end time

kibana/test/functional/page_objects/time_picker.ts

Line 58 in 3ee0bf2

async setDefaultAbsoluteRangeViaUiSettings() {

kertal · 2020-06-26T06:37:28Z

x-pack/test/functional/apps/discover/async_scripted_fields.js

+      await kibanaServer.uiSettings.replace({});
+      await kibanaServer.uiSettings.update({});


this is just a question, there is no way to replace a single uiSetting ? just asking because of

kibana/test/functional/page_objects/time_picker.ts

Line 64 in 3ee0bf2

async resetDefaultAbsoluteRangeViaUiSettings() {

kertal · 2020-06-26T06:38:44Z

x-pack/test/functional/apps/discover/async_scripted_fields.js

+      /* If you had to modify the scripted fields, you could un-comment all this, run it, use es_archiver to update 'kibana_scripted_fields_on_logstash'
+       */
+
+      // await PageObjects.settings.navigateTo();
+      // await PageObjects.settings.clickKibanaIndexPatterns();
+      // await PageObjects.settings.createIndexPattern('logsta');
+      // await PageObjects.settings.clickScriptedFieldsTab();
+      // await log.debug('add scripted field');
+      // await PageObjects.settings.addScriptedField(
+      //   'sharedFail',
+      //   'painless',
+      //   'string',
+      //   null,
+      //   '1',
+      //   // Scripted field below with multiple string checking actually should cause share failure message
+      //   // bcause it's not checking if all the fields it uses exist in each doc (and they don't)
+      //   "if (doc['response.raw'].value == '200') { return 'good ' + doc['url.raw'].value } else { return 'bad ' + doc['machine.os.raw'].value } "
+      // );


yes, makes sense to keeps this here

kertal

Code LGTM, minor remarks and questions added, maybe a final flaky test suite linked in the description of this PR. Crazy how much time it took the reproduce this problem once, and it's now done in a few lines of code 👍

kertal · 2020-06-26T06:40:22Z

x-pack/test/functional/apps/discover/async_scripted_fields.js

+      // the commented-out steps below were used to create the scripted fields in the logstash-* index pattern
+      // which are now saved in the esArchive.
+
+      // await PageObjects.settings.navigateTo();
+      // await PageObjects.settings.clickKibanaIndexPatterns();
+      // await PageObjects.settings.clickIndexPatternLogstash();
+      // const startingCount = parseInt(await PageObjects.settings.getScriptedFieldsTabCount());
+      // await PageObjects.settings.clickScriptedFieldsTab();
+      // await log.debug('add scripted field');
+      // await PageObjects.settings.addScriptedField(
+      //   'goodScript',
+      //   'painless',
+      //   'string',
+      //   null,
+      //   '1',
+      //   // Scripted field below with should work
+      //   "if (doc['response.raw'].value == '200') { if (doc['url.raw'].size() > 0) { return 'good ' + doc['url.raw'].value } else { return 'good' } } else { if (doc['machine.os.raw'].size() > 0) { return 'bad ' + doc['machine.os.raw'].value } else { return 'bad' } }"
+      // );
+      // await retry.try(async function() {
+      //   expect(parseInt(await PageObjects.settings.getScriptedFieldsTabCount())).to.be(
+      //     startingCount + 1
+      //   );
+      // });
+
+      // await PageObjects.settings.addScriptedField(
+      //   'goodScript2',
+      //   'painless',
+      //   'string',
+      //   null,
+      //   '1',
+      //   // Scripted field below which should work
+      //   "if (doc['url.raw'].size() > 0) { String tempString = \"\"; for ( int i = (doc['url.raw'].value.length() - 1); i >= 0 ; i--) { tempString = tempString + (doc['url.raw'].value).charAt(i); } return tempString; } else { return \"emptyUrl\"; }"
+      // );
+      // await retry.try(async function() {
+      //   expect(parseInt(await PageObjects.settings.getScriptedFieldsTabCount())).to.be(
+      //     startingCount + 2
+      //   );
+      // });


just wonder if it would make sense, extracting these code into helper functions, uncommenting the usage of those?

LeeDr · 2020-06-29T15:11:10Z

@elasticmachine merge upstream

…earchTest

kibanamachine · 2020-06-29T20:23:51Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 615a85f

Build metrics

✅ unchanged

History

💚 Build #57186 succeeded 2a0d480
💚 Build #55969 succeeded c238e60
💚 Build #55916 succeeded 169c36e
💔 Build #55895 failed 0348d9b
💔 Build #55751 failed 72b15a9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Co-authored-by: Elastic Machine <[email protected]>

…ata-streams * 'master' of github.com:elastic/kibana: (50 commits) [Logs UI] [Alerting] "Group by" functionality (elastic#68250) [Discover] Deangularize Skip to bottom button (elastic#69811) Implement recursive plugin discovery (elastic#68811) Use ts-expect-error in platform code (elastic#69883) [SIEM][Detection Engine][Lists] Moves getQueryFilter to common folder for use by both front and backend [Ingest Manager][SECURITY SOLUTION] adjust config reassign link and add roundtrip to Reassignment flow (elastic#70208) [Security][Lists] Add API functions and react hooks for value list APIs (elastic#69603) [ILM] Fix bug when clearing priority field (elastic#70154) [Platform][Security] Updates cluster_manager ignorePaths to include security scripts (elastic#70139) [IngestManager] Allow to filter agent by packages (elastic#69731) [code coverage] exclude folders: test_helpers, tests_bundle (elastic#70199) [Metrics UI] UX improvements for saved views (elastic#69910) [APM] docs: unique transaction troubleshooting (elastic#69831) Cross cluster search functional test with minimun privileges assigned to the test_user (elastic#70007) [Maps] choropleth layer wizard (elastic#69699) Make custom errors by extending Error (elastic#69966) [Ingest Manager] Support updated package output structure (elastic#69864) Resolver test coverage (elastic#70246) Async Discover search test (elastic#64388) [ui-shared-deps] include styled-components (elastic#69322) ... # Conflicts: # x-pack/plugins/snapshot_restore/server/types.ts

…bana into alerting/consumer-based-rbac * 'alerting/consumer-based-rbac' of github.com:gmmorris/kibana: (49 commits) [Discover] Deangularize Skip to bottom button (elastic#69811) Implement recursive plugin discovery (elastic#68811) Use ts-expect-error in platform code (elastic#69883) [SIEM][Detection Engine][Lists] Moves getQueryFilter to common folder for use by both front and backend [Ingest Manager][SECURITY SOLUTION] adjust config reassign link and add roundtrip to Reassignment flow (elastic#70208) [Security][Lists] Add API functions and react hooks for value list APIs (elastic#69603) [ILM] Fix bug when clearing priority field (elastic#70154) [Platform][Security] Updates cluster_manager ignorePaths to include security scripts (elastic#70139) [IngestManager] Allow to filter agent by packages (elastic#69731) [code coverage] exclude folders: test_helpers, tests_bundle (elastic#70199) [Metrics UI] UX improvements for saved views (elastic#69910) [APM] docs: unique transaction troubleshooting (elastic#69831) Cross cluster search functional test with minimun privileges assigned to the test_user (elastic#70007) [Maps] choropleth layer wizard (elastic#69699) Make custom errors by extending Error (elastic#69966) [Ingest Manager] Support updated package output structure (elastic#69864) Resolver test coverage (elastic#70246) Async Discover search test (elastic#64388) [ui-shared-deps] include styled-components (elastic#69322) SECURITY-ENDPOINT: add host properties (elastic#70238) ...

Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

Co-authored-by: Elastic Machine <[email protected]>

LeeDr added 4 commits April 23, 2020 19:45

add x-pack test for discover async search with scripted field

11628db

one test for failed shards and one that *should* work but fails on BC8

674130c

speed up setScriptedFieldScript by 20 sec for most cases

421d7db

use index pattern with scripted fields from esArchive

64c3150

LeeDr requested a review from kertal April 29, 2020 15:52

LeeDr added 2 commits April 29, 2020 18:09

merge

ad9e362

enable test for failed shards

ad80789

Merge branch 'master' into asyncSearchTest

8530255

LeeDr added release_note:skip Skip the PR/issue when compiling release notes Team:Visualizations Visualization editors, elastic-charts and infrastructure test_xpack_functional v7.8.0 v8.0.0 labels May 7, 2020

LeeDr added 3 commits May 8, 2020 11:54

added more to esArchive files to eliminate steps from test

e3fa31c

Merge branch 'asyncSearchTest' of github.com:LeeDr/kibana into asyncS…

13b4462

…earchTest

Merge branch 'master' of github.com:elastic/kibana into asyncSearchTest

6d2178a

Merge branch 'master' into asyncSearchTest

8413697

Merge branch 'master' into asyncSearchTest

46538a8

LeeDr changed the title ~~Async search test~~ Async Discover search test May 13, 2020

Merge branch 'master' into asyncSearchTest

fdccf5c

LeeDr added 6 commits June 23, 2020 16:26

Merge branch 'master' of github.com:elastic/kibana into asyncSearchTest

4672fd4

remove prefer_v2_templates: true

208ba76

Merge branch 'asyncSearchTest' of github.com:LeeDr/kibana into asyncS…

72b15a9

…earchTest

delete bad yarn.lock file

0348d9b

Merge branch 'master' of github.com:elastic/kibana into asyncSearchTest

aa58dc9

restore yarn.lock file

169c36e

Merge branch 'master' into asyncSearchTest

c238e60

LeeDr added v7.9.0 and removed v7.8.1 labels Jun 24, 2020

LeeDr marked this pull request as ready for review June 24, 2020 18:13

kertal reviewed Jun 26, 2020

View reviewed changes

kertal approved these changes Jun 26, 2020

View reviewed changes

elasticmachine and others added 3 commits June 29, 2020 10:11

Merge branch 'master' into asyncSearchTest

2a0d480

un-commented the setup code but put it in unreachable condition

4afcc20

Merge branch 'asyncSearchTest' of github.com:LeeDr/kibana into asyncS…

615a85f

…earchTest

LeeDr merged commit 752fa6e into elastic:master Jun 29, 2020

LeeDr deleted the asyncSearchTest branch June 29, 2020 20:40

LeeDr mentioned this pull request Jun 29, 2020

[7.x] Async Discover search test (#64388) #70261

Merged

LeeDr pushed a commit to LeeDr/kibana that referenced this pull request Jun 29, 2020

Async Discover search test (elastic#64388)

637c377

Co-authored-by: Elastic Machine <[email protected]>

LeeDr pushed a commit that referenced this pull request Jun 30, 2020

[7.x] Async Discover search test (#64388) (#70261)

9b00a2a

Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

Bamieh pushed a commit to Bamieh/kibana that referenced this pull request Jul 1, 2020

Async Discover search test (elastic#64388)

57fc0d1

Co-authored-by: Elastic Machine <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Async Discover search test #64388

Async Discover search test #64388

LeeDr commented Apr 24, 2020 •

edited

Loading

kertal commented Apr 29, 2020

LeeDr commented Apr 30, 2020

LeeDr commented Apr 30, 2020

kertal commented May 5, 2020

elasticmachine commented May 7, 2020

LeeDr commented May 8, 2020

LeeDr commented May 8, 2020

LeeDr commented May 13, 2020

LeeDr commented May 15, 2020

LeeDr commented May 15, 2020

LeeDr commented Jun 24, 2020

LeeDr commented Jun 24, 2020

kertal commented Jun 25, 2020

kertal Jun 26, 2020

kertal Jun 26, 2020

kertal Jun 26, 2020

kertal left a comment

kertal Jun 26, 2020

LeeDr commented Jun 29, 2020

kibanamachine commented Jun 29, 2020

		await kibanaServer.uiSettings.replace({});
		await kibanaServer.uiSettings.update({});

Async Discover search test #64388

Async Discover search test #64388

Conversation

LeeDr commented Apr 24, 2020 • edited Loading

Summary

Checklist

For maintainers

kertal commented Apr 29, 2020

LeeDr commented Apr 30, 2020

LeeDr commented Apr 30, 2020

kertal commented May 5, 2020

elasticmachine commented May 7, 2020

LeeDr commented May 8, 2020

LeeDr commented May 8, 2020

LeeDr commented May 13, 2020

LeeDr commented May 15, 2020

LeeDr commented May 15, 2020

LeeDr commented Jun 24, 2020

LeeDr commented Jun 24, 2020

kertal commented Jun 25, 2020

kertal Jun 26, 2020

Choose a reason for hiding this comment

kertal Jun 26, 2020

Choose a reason for hiding this comment

kertal Jun 26, 2020

Choose a reason for hiding this comment

kertal left a comment

Choose a reason for hiding this comment

kertal Jun 26, 2020

Choose a reason for hiding this comment

LeeDr commented Jun 29, 2020

kibanamachine commented Jun 29, 2020

💚 Build Succeeded

Build metrics

History

LeeDr commented Apr 24, 2020 •

edited

Loading