[Lens] Support index pattern runtime fields in existence and field stats API #90600

flash1293 · 2021-02-08T13:12:05Z

Fixes #90175 This PR reads the index pattern and includes index pattern level runtime fields for the requests of field existence and field stats API

To test:

Download and import index pattern including runtime field definition index-pattern-so.ndjson.txt
Go to Lens and select imported index pattern
Runtime field what_a_fielding should work correctly

To simplify a little, this PR also changes the field stats API to include the index pattern id and field name instead of the index pattern title and field definition (as these things can be looked up on the server side). I don't have a strong opinion on this, but it looks like a simpler API to me (which probably wasn't possible in earlier versions because index patterns were not available on the server)

…tence

flash1293 · 2021-02-09T17:42:01Z

@elasticmachine merge upstream

flash1293 · 2021-02-10T08:10:54Z

@elasticmachine merge upstream

elasticmachine · 2021-02-10T10:28:08Z

Pinging @elastic/kibana-app (Team:KibanaApp)

flash1293 · 2021-02-11T11:08:19Z

@elasticmachine merge upstream

flash1293 · 2021-02-15T09:59:23Z

@elasticmachine merge upstream

timroes · 2021-02-15T11:21:25Z

x-pack/plugins/lens/server/routes/field_stats.ts

            track_total_hits: true,
            body: {
              query,
              aggs,
+              runtime_mappings:
+                !field.isMapped && field.runtimeField ? { [fieldName]: field.runtimeField } : {},


I am not 100% sure if this is the "correct" behavior for runtime fields shadowing mapped fields. @mattkime Will runtime fields shadowing mapped fields report as isMapped: true and runtimeField: true (that was my understanding so far). If it's the case we should not check on isMapped here and always write the script if runtimeField === true.

Good point Tim, fixed this.

Tim's understanding is correct.

dej611

Tested with Chrome and Safari, but I am a bit lost on what the expected result of this PR.

The only difference I noticed with master is this + button feature enabled:

The error notification is still showing up and the data popover is still empty (or in general it does not show any extra information).

Edit: also with this PR everytime I open the data popover I see this error on the server (before there was none):

server   error  [12:49:52.958]  Error: Internal Server Error
    at HapiResponseAdapter.toError (~/kibana/src/core/server/http/router/response_adapter.ts:122:19)
    at HapiResponseAdapter.toHapiResponse (~/kibana/src/core/server/http/router/response_adapter.ts:72:19)
    at HapiResponseAdapter.handle (~/kibana/src/core/server/http/router/response_adapter.ts:67:17)
    at Router.handle (~/kibana/src/core/server/http/router/router.ts:273:34)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at handler (~/kibana/src/core/server/http/router/router.ts:227:11)
    at exports.Manager.execute (~/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (~/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (~/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (~/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (~/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)

mbondyra

Tested in Chrome, the described test case works as expected. Code looks good to me.

dej611 · 2021-02-15T13:32:02Z

The error notification is still showing up and the data popover is still empty (or in general it does not show any extra information).

Edit: also with this PR everytime I open the data popover I see this error on the server (before there was none):

server   error  [12:49:52.958]  Error: Internal Server Error
    at HapiResponseAdapter.toError (~/kibana/src/core/server/http/router/response_adapter.ts:122:19)
    at HapiResponseAdapter.toHapiResponse (~/kibana/src/core/server/http/router/response_adapter.ts:72:19)
    at HapiResponseAdapter.handle (~/kibana/src/core/server/http/router/response_adapter.ts:67:17)
    at Router.handle (~/kibana/src/core/server/http/router/router.ts:273:34)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at handler (~/kibana/src/core/server/http/router/router.ts:227:11)
    at exports.Manager.execute (~/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (~/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (~/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (~/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (~/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)

Installed the ecommerce dataset and all is fixed now.

wylieconlon

These changes make sense and I tested that they work as expected. It seems like there are some missing pieces that are pretty important:

No functional test
There is still no UI for managing these, I assume this is coming soon?

wylieconlon · 2021-02-16T18:58:40Z

x-pack/test/api_integration/apis/lens/field_stats.ts

@@ -32,17 +32,14 @@ export default ({ getService }: FtrProviderContext) => {
    describe('field distribution', () => {


I would appreciate a functional test for an index pattern runtime field, seems like it's missing

…tence

flash1293 · 2021-02-17T16:45:20Z

@elasticmachine merge upstream

flash1293 · 2021-02-17T16:46:21Z

I'm going to postpone merging this PR until after feature freeze to reduce some risk as index pattern runtime fields won't land in 7.12 anyway.

kibanamachine · 2021-02-17T18:58:25Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lens`	907.9KB	907.9KB	-35.0B

History

💔 Build #107196 failed 86954d7
💚 Build #106570 succeeded 70d62f7
💚 Build #106523 succeeded e309896
💚 Build #105968 succeeded 3cb360c
💚 Build #105482 succeeded 5f169c3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…ats API (elastic#90600)

kibanamachine · 2021-02-22T16:17:07Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

kibanamachine · 2021-02-23T16:21:20Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

kibanamachine · 2021-02-24T16:31:20Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

flash1293 · 2021-02-24T16:33:08Z

I'm getting there, kibana machine, I'm trying really hard

…eld stats API (#90600) (#91872) * [Lens] Support index pattern runtime fields in existence and field stats API (#90600) * fix fixture Co-authored-by: Kibana Machine <[email protected]>

fix runtime existence

888d6e6

flash1293 added Team:Visualizations Visualization editors, elastic-charts and infrastructure v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:Lens v7.12.0 labels Feb 8, 2021

flash1293 added 2 commits February 9, 2021 16:10

Merge remote-tracking branch 'upstream/master' into lens/runtime-exis…

61dc731

…tence

fix tests

e984392

Merge branch 'master' into lens/runtime-existence

c11f854

Merge branch 'master' into lens/runtime-existence

5f169c3

flash1293 marked this pull request as ready for review February 10, 2021 10:28

flash1293 requested a review from a team February 10, 2021 10:28

Merge branch 'master' into lens/runtime-existence

3cb360c

Merge branch 'master' into lens/runtime-existence

e309896

timroes reviewed Feb 15, 2021

View reviewed changes

dej611 reviewed Feb 15, 2021

View reviewed changes

shadow ES level runtime fields

70d62f7

mbondyra approved these changes Feb 15, 2021

View reviewed changes

wylieconlon approved these changes Feb 16, 2021

View reviewed changes

flash1293 added 3 commits February 17, 2021 09:33

add functional test

4537e2f

Merge remote-tracking branch 'upstream/master' into lens/runtime-exis…

86954d7

…tence

fix functional tests

2370e5e

Merge branch 'master' into lens/runtime-existence

fdd910f

flash1293 added v7.13.0 and removed v7.12.0 labels Feb 17, 2021

flash1293 merged commit 759f343 into elastic:master Feb 18, 2021

flash1293 mentioned this pull request Feb 18, 2021

[7.x] [Lens] Support index pattern runtime fields in existence and field stats API (#90600) #91872

Merged

flash1293 added a commit to flash1293/kibana that referenced this pull request Feb 18, 2021

[Lens] Support index pattern runtime fields in existence and field st…

5a2418a

…ats API (elastic#90600)

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 22, 2021

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lens] Support index pattern runtime fields in existence and field stats API #90600

[Lens] Support index pattern runtime fields in existence and field stats API #90600

flash1293 commented Feb 8, 2021 •

edited

Loading

flash1293 commented Feb 9, 2021

flash1293 commented Feb 10, 2021

elasticmachine commented Feb 10, 2021

flash1293 commented Feb 11, 2021

flash1293 commented Feb 15, 2021

timroes Feb 15, 2021

flash1293 Feb 15, 2021

mattkime Feb 15, 2021

dej611 left a comment •

edited

Loading

mbondyra left a comment

dej611 commented Feb 15, 2021

wylieconlon left a comment

wylieconlon Feb 16, 2021

flash1293 commented Feb 17, 2021

flash1293 commented Feb 17, 2021

kibanamachine commented Feb 17, 2021

kibanamachine commented Feb 22, 2021

kibanamachine commented Feb 23, 2021

kibanamachine commented Feb 24, 2021

flash1293 commented Feb 24, 2021

		@@ -32,17 +32,14 @@ export default ({ getService }: FtrProviderContext) => {
		describe('field distribution', () => {

[Lens] Support index pattern runtime fields in existence and field stats API #90600

[Lens] Support index pattern runtime fields in existence and field stats API #90600

Conversation

flash1293 commented Feb 8, 2021 • edited Loading

flash1293 commented Feb 9, 2021

flash1293 commented Feb 10, 2021

elasticmachine commented Feb 10, 2021

flash1293 commented Feb 11, 2021

flash1293 commented Feb 15, 2021

timroes Feb 15, 2021

Choose a reason for hiding this comment

flash1293 Feb 15, 2021

Choose a reason for hiding this comment

mattkime Feb 15, 2021

Choose a reason for hiding this comment

dej611 left a comment • edited Loading

Choose a reason for hiding this comment

mbondyra left a comment

Choose a reason for hiding this comment

dej611 commented Feb 15, 2021

wylieconlon left a comment

Choose a reason for hiding this comment

wylieconlon Feb 16, 2021

Choose a reason for hiding this comment

flash1293 commented Feb 17, 2021

flash1293 commented Feb 17, 2021

kibanamachine commented Feb 17, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

History

kibanamachine commented Feb 22, 2021

kibanamachine commented Feb 23, 2021

kibanamachine commented Feb 24, 2021

flash1293 commented Feb 24, 2021

flash1293 commented Feb 8, 2021 •

edited

Loading

dej611 left a comment •

edited

Loading