[index pattern management] load index pattern list without loading field lists #108823

mattkime · 2021-08-17T02:36:30Z

Summary

Previously we were loading the full index pattern for each item in the index pattern management index pattern list. Obviously this isn't efficient but urgency was added by reports that the index pattern list may fail to load with problematic index patterns (hitting thousands of indices with RBAC enabled).

Release notes

In some circumstances involving Role Based Access Control and index patterns matching thousands of indices, the index pattern may fail to load. This affects the index pattern management index pattern list as well. This fix no longer longer loads the fields lists for the index patterns which sidesteps the loading error.

elasticmachine · 2021-08-17T02:45:39Z

Pinging @elastic/kibana-app-services (Team:AppServices)

Dosant

Code LGTM, tested index pattern page

Dosant · 2021-08-17T07:44:35Z

src/plugins/data/common/index_patterns/index_patterns/index_patterns.ts

-  getIdsWithTitle = async (
-    refresh: boolean = false
-  ): Promise<Array<{ id: string; title: string }>> => {
+  getIdsWithTitle = async (refresh: boolean = false): Promise<IndexPatternListItem[]> => {


Would be good to rename getIdsWithTitle -> getIndexPatternList
maybe later

kibanamachine · 2021-08-17T14:47:35Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: 6b216e3
Storybooks Preview
Flaky suites:
- xpack-kibana-ciGroup6

Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/security_solution/network_dns·ts.apis SecuritySolution Endpoints Network DNS With packetbeat Make sure that we get Dns data and sorting by uniqueDomains descending

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://github.com/elastic/kibana/issues/87793

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook in "apis"
[00:07:29]           └-: SecuritySolution Endpoints
[00:07:29]             └-> "before all" hook in "SecuritySolution Endpoints"
[00:08:01]             └-: Network DNS
[00:08:01]               └-> "before all" hook in "Network DNS"
[00:08:01]               └-: With packetbeat
[00:08:01]                 └-> "before all" hook for "Make sure that we get Dns data and sorting by uniqueDomains ascending"
[00:08:01]                 └-> "before all" hook for "Make sure that we get Dns data and sorting by uniqueDomains ascending"
[00:08:01]                   │ info [x-pack/test/functional/es_archives/packetbeat/dns] Loading "mappings.json"
[00:08:01]                   │ info [x-pack/test/functional/es_archives/packetbeat/dns] Loading "data.json.gz"
[00:08:01]                   │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [packetbeat-7.0.0-iot-2019.11] creating index, cause [api], templates [], shards [3]/[1]
[00:08:01]                   │ info [x-pack/test/functional/es_archives/packetbeat/dns] Created index "packetbeat-7.0.0-iot-2019.11"
[00:08:01]                   │ debg [x-pack/test/functional/es_archives/packetbeat/dns] "packetbeat-7.0.0-iot-2019.11" settings {"index":{"lifecycle":{"name":"packetbeat-7.0.0","rollover_alias":"packetbeat-7.0.0"},"mapping":{"total_fields":{"limit":"10000"}},"number_of_replicas":"1","number_of_shards":"3","query":{"default_field":["tags","message","agent.version","agent.name","agent.type","agent.id","agent.ephemeral_id","client.address","client.mac","client.domain","client.geo.continent_name","client.geo.country_name","client.geo.region_name","client.geo.city_name","client.geo.country_iso_code","client.geo.region_iso_code","client.geo.name","cloud.provider","cloud.availability_zone","cloud.region","cloud.instance.id","cloud.instance.name","cloud.machine.type","cloud.account.id","container.runtime","container.id","container.image.name","container.image.tag","container.name","destination.address","destination.mac","destination.domain","destination.geo.continent_name","destination.geo.country_name","destination.geo.region_name","destination.geo.city_name","destination.geo.country_iso_code","destination.geo.region_iso_code","destination.geo.name","ecs.version","error.id","error.message","error.code","event.id","event.kind","event.category","event.action","event.outcome","event.type","event.module","event.dataset","event.hash","event.timezone","file.path","file.target_path","file.extension","file.type","file.device","file.inode","file.uid","file.owner","file.gid","file.group","file.mode","group.id","group.name","host.hostname","host.name","host.id","host.mac","host.type","host.architecture","host.os.platform","host.os.name","host.os.full","host.os.family","host.os.version","host.os.kernel","host.geo.continent_name","host.geo.country_name","host.geo.region_name","host.geo.city_name","host.geo.country_iso_code","host.geo.region_iso_code","host.geo.name","http.request.method","http.request.body.content","http.request.referrer","http.response.body.content","http.version","log.level","network.name","network.type","network.iana_number","network.transport","network.application","network.protocol","network.direction","network.community_id","observer.mac","observer.hostname","observer.vendor","observer.version","observer.serial_number","observer.type","observer.os.platform","observer.os.name","observer.os.full","observer.os.family","observer.os.version","observer.os.kernel","observer.geo.continent_name","observer.geo.country_name","observer.geo.region_name","observer.geo.city_name","observer.geo.country_iso_code","observer.geo.region_iso_code","observer.geo.name","organization.name","organization.id","os.platform","os.name","os.full","os.family","os.version","os.kernel","process.name","process.args","process.executable","process.title","process.working_directory","server.address","server.mac","server.domain","server.geo.continent_name","server.geo.country_name","server.geo.region_name","server.geo.city_name","server.geo.country_iso_code","server.geo.region_iso_code","server.geo.name","service.id","service.name","service.type","service.state","service.version","service.ephemeral_id","source.address","source.mac","source.domain","source.geo.continent_name","source.geo.country_name","source.geo.region_name","source.geo.city_name","source.geo.country_iso_code","source.geo.region_iso_code","source.geo.name","url.original","url.full","url.scheme","url.domain","url.path","url.query","url.fragment","url.username","url.password","user.id","user.name","user.full_name","user.email","user.hash","user.group.id","user.group.name","user_agent.original","user_agent.name","user_agent.version","user_agent.device.name","user_agent.os.platform","user_agent.os.name","user_agent.os.full","user_agent.os.family","user_agent.os.version","user_agent.os.kernel","agent.hostname","error.type","cloud.project.id","kubernetes.pod.name","kubernetes.pod.uid","kubernetes.namespace","kubernetes.node.name","kubernetes.container.name","kubernetes.container.image","type","server.process.name","server.process.args","server.process.executable","server.process.working_directory","server.process.start","client.process.name","client.process.args","client.process.executable","client.process.working_directory","client.process.start","flow.id","status","method","resource","path","query","params","request","response","amqp.reply-text","amqp.exchange","amqp.exchange-type","amqp.consumer-tag","amqp.routing-key","amqp.queue","amqp.content-type","amqp.content-encoding","amqp.delivery-mode","amqp.correlation-id","amqp.reply-to","amqp.expiration","amqp.message-id","amqp.timestamp","amqp.type","amqp.user-id","amqp.app-id","cassandra.request.headers.flags","cassandra.request.headers.stream","cassandra.request.headers.op","cassandra.request.query","cassandra.response.headers.flags","cassandra.response.headers.stream","cassandra.response.headers.op","cassandra.response.result.type","cassandra.response.result.rows.meta.keyspace","cassandra.response.result.rows.meta.table","cassandra.response.result.rows.meta.flags","cassandra.response.result.rows.meta.paging_state","cassandra.response.result.keyspace","cassandra.response.result.schema_change.change","cassandra.response.result.schema_change.keyspace","cassandra.response.result.schema_change.table","cassandra.response.result.schema_change.object","cassandra.response.result.schema_change.target","cassandra.response.result.schema_change.name","cassandra.response.result.schema_change.args","cassandra.response.result.prepared.prepared_id","cassandra.response.result.prepared.req_meta.keyspace","cassandra.response.result.prepared.req_meta.table","cassandra.response.result.prepared.req_meta.flags","cassandra.response.result.prepared.req_meta.paging_state","cassandra.response.result.prepared.resp_meta.keyspace","cassandra.response.result.prepared.resp_meta.table","cassandra.response.result.prepared.resp_meta.flags","cassandra.response.result.prepared.resp_meta.paging_state","cassandra.response.authentication.class","cassandra.response.warnings","cassandra.response.event.type","cassandra.response.event.change","cassandra.response.event.host","cassandra.response.event.schema_change.change","cassandra.response.event.schema_change.keyspace","cassandra.response.event.schema_change.table","cassandra.response.event.schema_change.object","cassandra.response.event.schema_change.target","cassandra.response.event.schema_change.name","cassandra.response.event.schema_change.args","cassandra.response.error.msg","cassandra.response.error.type","cassandra.response.error.details.read_consistency","cassandra.response.error.details.write_type","cassandra.response.error.details.keyspace","cassandra.response.error.details.table","cassandra.response.error.details.stmt_id","cassandra.response.error.details.num_failures","cassandra.response.error.details.function","cassandra.response.error.details.arg_types","dhcpv4.transaction_id","dhcpv4.flags","dhcpv4.client_mac","dhcpv4.server_name","dhcpv4.op_code","dhcpv4.hardware_type","dhcpv4.option.message_type","dhcpv4.option.parameter_request_list","dhcpv4.option.class_identifier","dhcpv4.option.domain_name","dhcpv4.option.hostname","dhcpv4.option.message","dhcpv4.option.boot_file_name","dns.op_code","dns.response_code","dns.question.name","dns.question.type","dns.question.class","dns.question.registered_domain","dns.answers.name","dns.answers.type","dns.answers.class","dns.answers.data","dns.authorities.name","dns.authorities.type","dns.authorities.class","dns.additionals.name","dns.additionals.type","dns.additionals.class","dns.additionals.data","dns.opt.version","dns.opt.ext_rcode","http.response.status_phrase","icmp.version","icmp.request.message","icmp.response.message","memcache.protocol_type","memcache.request.line","memcache.request.command","memcache.response.command","memcache.request.type","memcache.response.type","memcache.response.error_msg","memcache.request.opcode","memcache.response.opcode","memcache.response.status","memcache.request.raw_args","memcache.request.automove","memcache.response.version","mongodb.error","mongodb.fullCollectionName","mongodb.startingFrom","mongodb.query","mongodb.returnFieldsSelector","mongodb.selector","mongodb.update","mongodb.cursorId","mysql.insert_id","mysql.num_fields","mysql.num_rows","mysql.query","mysql.error_message","nfs.tag","nfs.opcode","nfs.status","rpc.xid","rpc.status","rpc.auth_flavor","rpc.cred.gids","rpc.cred.machinename","pgsql.error_message","pgsql.error_severity","pgsql.num_fields","pgsql.num_rows","redis.return_value","redis.error","thrift.params","thrift.service","thrift.return_value","thrift.exceptions","tls.version","tls.resumption_method","tls.client_hello.version","tls.client_hello.extensions.server_name_indication","tls.client_hello.extensions.application_layer_protocol_negotiation","tls.client_hello.extensions.session_ticket","tls.client_hello.extensions.supported_versions","tls.client_hello.extensions.supported_groups","tls.client_hello.extensions.signature_algorithms","tls.client_hello.extensions.ec_points_formats","tls.client_hello.extensions._unparsed_","tls.server_hello.version","tls.server_hello.selected_cipher","tls.server_hello.selected_compression_method","tls.server_hello.session_id","tls.server_hello.extensions.session_ticket","tls.server_hello.extensions.supported_versions","tls.server_hello.extensions.ec_points_formats","tls.server_hello.extensions._unparsed_","tls.client_certificate.serial_number","tls.client_certificate.public_key_algorithm","tls.client_certificate.signature_algorithm","tls.client_certificate.raw","tls.client_certificate.subject.country","tls.client_certificate.subject.organization","tls.client_certificate.subject.organizational_unit","tls.client_certificate.subject.province","tls.client_certificate.subject.common_name","tls.client_certificate.issuer.country","tls.client_certificate.issuer.organization","tls.client_certificate.issuer.organizational_unit","tls.client_certificate.issuer.province","tls.client_certificate.issuer.common_name","tls.client_certificate.fingerprint.md5","tls.client_certificate.fingerprint.sha1","tls.client_certificate.fingerprint.sha256","tls.server_certificate.serial_number","tls.server_certificate.public_key_algorithm","tls.server_certificate.signature_algorithm","tls.server_certificate.raw","tls.server_certificate.subject.country","tls.server_certificate.subject.organization","tls.server_certificate.subject.organizational_unit","tls.server_certificate.subject.province","tls.server_certificate.subject.common_name","tls.server_certificate.issuer.country","tls.server_certificate.issuer.organization","tls.server_certificate.issuer.organizational_unit","tls.server_certificate.issuer.province","tls.server_certificate.issuer.common_name","tls.server_certificate.fingerprint.md5","tls.server_certificate.fingerprint.sha1","tls.server_certificate.fingerprint.sha256","tls.alert_types","tls.fingerprints.ja3.hash","tls.fingerprints.ja3.str","fields.*"]},"refresh_interval":"5s"}}
[00:08:01]                   │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-7.0.0-iot-2019.11/ycQHe-5pSkqCxgsdLYarrQ] update_mapping [_doc]
[00:08:03]                   │ info [x-pack/test/functional/es_archives/packetbeat/dns] Indexed 5700 docs into "packetbeat-7.0.0-iot-2019.11"
[00:08:03]                 └-> Make sure that we get Dns data and sorting by uniqueDomains ascending
[00:08:03]                   └-> "before each" hook: global before each for "Make sure that we get Dns data and sorting by uniqueDomains ascending"
[00:08:03]                   └- ✓ pass  (41ms) "apis SecuritySolution Endpoints Network DNS With packetbeat Make sure that we get Dns data and sorting by uniqueDomains ascending"
[00:08:03]                 └-> Make sure that we get Dns data and sorting by uniqueDomains descending
[00:08:03]                   └-> "before each" hook: global before each for "Make sure that we get Dns data and sorting by uniqueDomains descending"
[00:08:03]                   └- ✖ fail: apis SecuritySolution Endpoints Network DNS With packetbeat Make sure that we get Dns data and sorting by uniqueDomains descending
[00:08:03]                   │      Error: expected 0 to equal 10
[00:08:03]                   │       at Assertion.assert (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:08:03]                   │       at Assertion.be.Assertion.equal (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:227:8)
[00:08:03]                   │       at Assertion.be (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:69:22)
[00:08:03]                   │       at Context.<anonymous> (test/api_integration/apis/security_solution/network_dns.ts:87:44)
[00:08:03]                   │       at runMicrotasks (<anonymous>)
[00:08:03]                   │       at processTicksAndRejections (internal/process/task_queues.js:95:5)
[00:08:03]                   │       at Object.apply (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:08:03]                   │ 
[00:08:03]                   │

Stack Trace

Error: expected 0 to equal 10
    at Assertion.assert (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.be.Assertion.equal (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:227:8)
    at Assertion.be (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/expect/expect.js:69:22)
    at Context.<anonymous> (test/api_integration/apis/security_solution/network_dns.ts:87:44)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at Object.apply (/dev/shm/workspace/parallel/20/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`indexPatternManagement`	384.6KB	384.6KB	+23.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	791.3KB	791.9KB	+651.0B

Unknown metric groups

API count

id	before	after	diff
`data`	3482	3495	+13

API count missing comments

id	before	after	diff
`data`	2970	2983	+13

Non-exported public API item count

id	before	after	diff
`data`	51	50	-1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…eld lists (elastic#108823) * don't load field list for index pattern list

kibanamachine · 2021-08-17T15:06:33Z

💔 Backport failed

Status	Branch	Result
✅	7.x
❌	7.14	Commit could not be cherrypicked due to conflicts

Successful backport PRs will be merged automatically after passing CI.

To backport manually run:
node scripts/backport --pr 108823

…eld lists (elastic#108823) * don't load field list for index pattern list

…eld lists (#108823) (#108929) * don't load field list for index pattern list Co-authored-by: Matthew Kime <[email protected]>

…ding field lists (#108823) (#108936) * [index pattern management] load index pattern list without loading field lists (#108823) * don't load field list for index pattern list * update snapshot

mattkime added 2 commits August 16, 2021 21:31

don't load field list for index pattern list

fb8cb3c

update docs

329a8de

remove commented code

d7c45e2

mattkime marked this pull request as ready for review August 17, 2021 02:45

mattkime requested a review from a team as a code owner August 17, 2021 02:45

mattkime requested a review from Dosant August 17, 2021 02:45

mattkime added 2 commits August 16, 2021 22:25

typescript fixes

52f9751

typescript fixes

29a0385

lukasolson added v8.0.0 and removed 8.0.0 labels Aug 17, 2021

type fixes

e2d364f

Dosant approved these changes Aug 17, 2021

View reviewed changes

mattkime added 2 commits August 17, 2021 06:54

update jest test

3e36272

Merge branch 'master' into index_pattern_mgmt_more_efficient_list

6b216e3

mattkime merged commit d62ff55 into elastic:master Aug 17, 2021

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Aug 17, 2021

[index pattern management] load index pattern list without loading fi…

027253f

…eld lists (elastic#108823) * don't load field list for index pattern list

kibanamachine mentioned this pull request Aug 17, 2021

[7.x] [index pattern management] load index pattern list without loading field lists (#108823) #108929

Merged

mattkime mentioned this pull request Aug 17, 2021

[7.14] [index pattern management] load index pattern list without loading field lists (#108823) #108936

Merged

mattkime added a commit to mattkime/kibana that referenced this pull request Aug 17, 2021

[index pattern management] load index pattern list without loading fi…

985a098

…eld lists (elastic#108823) * don't load field list for index pattern list

kibanamachine added a commit that referenced this pull request Aug 17, 2021

[index pattern management] load index pattern list without loading fi…

08202ff

…eld lists (#108823) (#108929) * don't load field list for index pattern list Co-authored-by: Matthew Kime <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[index pattern management] load index pattern list without loading field lists #108823

[index pattern management] load index pattern list without loading field lists #108823

mattkime commented Aug 17, 2021 •

edited

Loading

elasticmachine commented Aug 17, 2021

Dosant left a comment

Dosant Aug 17, 2021

kibanamachine commented Aug 17, 2021

Standard Out

Stack Trace

API count

API count missing comments

Non-exported public API item count

kibanamachine commented Aug 17, 2021

[index pattern management] load index pattern list without loading field lists #108823

[index pattern management] load index pattern list without loading field lists #108823

Conversation

mattkime commented Aug 17, 2021 • edited Loading

Summary

Release notes

elasticmachine commented Aug 17, 2021

Dosant left a comment

Choose a reason for hiding this comment

Dosant Aug 17, 2021

Choose a reason for hiding this comment

kibanamachine commented Aug 17, 2021

💛 Build succeeded, but was flaky

Test Failures

Standard Out

Stack Trace

Metrics [docs]

Async chunks

Page load bundle

API count

API count missing comments

Non-exported public API item count

History

kibanamachine commented Aug 17, 2021

💔 Backport failed

mattkime commented Aug 17, 2021 •

edited

Loading