Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Logstash] Disables mapping update #86045

Closed
wants to merge 2 commits into from
Closed

[Logstash] Disables mapping update #86045

wants to merge 2 commits into from

Conversation

tylersmalley
Copy link
Contributor

@tylersmalley tylersmalley commented Dec 15, 2020 •
edited
Loading

The mapping is updated when a user visits the edit pipeline route in the UI. This can not happen in 8.0 due to it being a system index. I am not sure why this mapping exists, or if it's even still needed. I am disabling now to unblock the promotion of the Elasticsearch snapshot and the Logstash/Ingest team can follow-up.

Closes #86038

Tyler Smalley added 2 commits December 15, 2020 14:31
[Logstash] Disables mapping update
fd15b58 
The mapping is updated when a user visits the edit pipeline route in the
UI. This can not happen in 8.0 due to it being a system index. I am not
sure why this mapping exists, or if it's even still needed. I am
disabling now to unblock the promotion of the Elasticsearch snapshot and
the Logstash/Ingest team can follow-up.

Signed-off-by: Tyler Smalley <[email protected]>
TEMP: Test against unverified snapshot
e894a0c 
Signed-off-by: Tyler Smalley <[email protected]>
@tylersmalley tylersmalley mentioned this pull request Dec 16, 2020
Closed
@kibanamachine
Copy link
Contributor

kibanamachine commented Dec 16, 2020 •
edited
Loading

💔 Build Failed

Failed CI Steps

Test Failures

X-Pack EPM API Integration Tests.x-pack/test/fleet_api_integration/apis/epm/data_stream·ts.Fleet Endpoints EPM Endpoints datastreams should list the logs and metrics datastream

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://dryrun

[00:00:00]       │
[00:00:00]         └-: Fleet Endpoints
[00:00:00]           └-> "before all" hook
[00:03:51]           └-: EPM Endpoints
[00:03:51]             └-> "before all" hook
[00:08:58]             └-: datastreams
[00:08:58]               └-> "before all" hook
[00:08:58]               └-> should list the logs and metrics datastream
[00:08:58]                 └-> "before each" hook: global before each
[00:08:58]                 └-> "before each" hook: beforeSetupWithDockerRegistry
[00:08:58]                 └-> "before each" hook
[00:08:58]                   │ proc [kibana]   log   [00:54:24.000] [info][fleet][plugins] Custom registry url is an experimental feature and is unsupported.
[00:08:58]                   │ info [docker:registry] 2020/12/16 00:54:24 source.ip: 172.17.0.1:39708, url.original: /search?package=datastreams&internal=true&experimental=true
[00:08:58]                   │ proc [kibana]   log   [00:54:24.003] [info][fleet][plugins] Custom registry url is an experimental feature and is unsupported.
[00:08:58]                   │ info [docker:registry] 2020/12/16 00:54:24 source.ip: 172.17.0.1:39712, url.original: /package/datastreams/0.1.0
[00:08:58]                   │ info [docker:registry] 2020/12/16 00:54:24 source.ip: 172.17.0.1:39716, url.original: /package/datastreams/0.1.0/
[00:08:58]                   │ proc [kibana]   log   [00:54:24.009] [info][fleet][plugins] Custom registry url is an experimental feature and is unsupported.
[00:08:58]                   │ info [docker:registry] 2020/12/16 00:54:24 source.ip: 172.17.0.1:39720, url.original: /epr/datastreams/datastreams-0.1.0.zip
[00:09:00]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] updating index lifecycle policy [all_assets]
[00:09:01]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding index lifecycle policy [logs-datastreams.test_logs-all_assets]
[00:09:05]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding component template [logs-datastreams.test_logs-settings]
[00:09:05]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding component template [logs-datastreams.test_logs-mappings]
[00:09:05]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding index template [metrics-datastreams.test_metrics] for index patterns [metrics-datastreams.test_metrics-*]
[00:09:05]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding index template [logs-datastreams.test_logs] for index patterns [logs-datastreams.test_logs-*]
[00:09:05]                   │ proc [kibana]   log   [00:54:31.141] [info][fleet][plugins] Found previous transform references:
[00:09:05]                   │ proc [kibana]  []
[00:09:05]                   │ proc [kibana]   log   [00:54:31.141] [info][fleet][plugins] Deleting currently installed transform ids 
[00:09:06]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] creating index, cause [initialize_data_stream], templates [logs-datastreams.test_logs], shards [1]/[1]
[00:09:06]                   │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding data stream [logs-datastreams.test_logs-default] with write index [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] and backing indices []
[00:09:06]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [logs]
[00:09:06]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [logs]
[00:09:06]                   │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] [.ds-logs-datastreams.test_logs-default-2020.12.16-000001/fNuPtzJETZmk9JSB0DN03A] update_mapping [_doc]
[00:09:06]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [logs]
[00:09:06]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] creating index, cause [initialize_data_stream], templates [metrics-datastreams.test_metrics], shards [1]/[1]
[00:09:06]                   │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding data stream [metrics-datastreams.test_metrics-default] with write index [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] and backing indices []
[00:09:07]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [metrics]
[00:09:07]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [metrics]
[00:09:07]                   │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001/IU-WEA6DR5-KzmEBMFOQ3Q] update_mapping [_doc]
[00:09:07]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [metrics]
[00:09:07]                 └- ✖ fail: Fleet Endpoints EPM Endpoints datastreams should list the logs and metrics datastream
[00:09:07]                 │      Error: expected '.ds-logs-datastreams.test_logs-default-2020.12.16-000001' to equal '.ds-logs-datastreams.test_logs-default-000001'
[00:09:07]                 │       at Assertion.assert (/dev/shm/workspace/parallel/24/kibana/packages/kbn-expect/expect.js:100:11)
[00:09:07]                 │       at Assertion.equal (/dev/shm/workspace/parallel/24/kibana/packages/kbn-expect/expect.js:227:8)
[00:09:07]                 │       at Context.<anonymous> (test/fleet_api_integration/apis/epm/data_stream.ts:91:76)
[00:09:07]                 │       at runMicrotasks (<anonymous>)
[00:09:07]                 │       at processTicksAndRejections (internal/process/task_queues.js:93:5)
[00:09:07]                 │       at Object.apply (/dev/shm/workspace/parallel/24/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:09:07]                 │ 
[00:09:07]                 │

Stack Trace

Error: expected '.ds-logs-datastreams.test_logs-default-2020.12.16-000001' to equal '.ds-logs-datastreams.test_logs-default-000001'
    at Assertion.assert (/dev/shm/workspace/parallel/24/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.equal (/dev/shm/workspace/parallel/24/kibana/packages/kbn-expect/expect.js:227:8)
    at Context.<anonymous> (test/fleet_api_integration/apis/epm/data_stream.ts:91:76)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at Object.apply (/dev/shm/workspace/parallel/24/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
X-Pack EPM API Integration Tests.x-pack/test/fleet_api_integration/apis/epm/data_stream·ts.Fleet Endpoints EPM Endpoints datastreams should list the logs and metrics datastream

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: Fleet Endpoints
[00:00:00]           └-> "before all" hook
[00:03:55]           └-: EPM Endpoints
[00:03:55]             └-> "before all" hook
[00:09:02]             └-: datastreams
[00:09:02]               └-> "before all" hook
[00:09:02]               └-> should list the logs and metrics datastream
[00:09:02]                 └-> "before each" hook: global before each
[00:09:02]                 └-> "before each" hook: beforeSetupWithDockerRegistry
[00:09:02]                 └-> "before each" hook
[00:09:02]                   │ proc [kibana]   log   [00:02:08.841] [info][fleet][plugins] Custom registry url is an experimental feature and is unsupported.
[00:09:02]                   │ info [docker:registry] 2020/12/16 00:02:08 source.ip: 172.17.0.1:51942, url.original: /search?package=datastreams&internal=true&experimental=true
[00:09:02]                   │ proc [kibana]   log   [00:02:08.844] [info][fleet][plugins] Custom registry url is an experimental feature and is unsupported.
[00:09:02]                   │ info [docker:registry] 2020/12/16 00:02:08 source.ip: 172.17.0.1:51946, url.original: /package/datastreams/0.1.0
[00:09:02]                   │ info [docker:registry] 2020/12/16 00:02:08 source.ip: 172.17.0.1:51950, url.original: /package/datastreams/0.1.0/
[00:09:02]                   │ proc [kibana]   log   [00:02:08.851] [info][fleet][plugins] Custom registry url is an experimental feature and is unsupported.
[00:09:02]                   │ info [docker:registry] 2020/12/16 00:02:08 source.ip: 172.17.0.1:51954, url.original: /epr/datastreams/datastreams-0.1.0.zip
[00:09:04]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] updating index lifecycle policy [all_assets]
[00:09:05]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding index lifecycle policy [logs-datastreams.test_logs-all_assets]
[00:09:09]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding component template [logs-datastreams.test_logs-mappings]
[00:09:09]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding component template [logs-datastreams.test_logs-settings]
[00:09:09]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding index template [metrics-datastreams.test_metrics] for index patterns [metrics-datastreams.test_metrics-*]
[00:09:09]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding index template [logs-datastreams.test_logs] for index patterns [logs-datastreams.test_logs-*]
[00:09:09]                   │ proc [kibana]   log   [00:02:16.001] [info][fleet][plugins] Found previous transform references:
[00:09:09]                   │ proc [kibana]  []
[00:09:09]                   │ proc [kibana]   log   [00:02:16.002] [info][fleet][plugins] Deleting currently installed transform ids 
[00:09:10]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] creating index, cause [initialize_data_stream], templates [logs-datastreams.test_logs], shards [1]/[1]
[00:09:10]                   │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding data stream [logs-datastreams.test_logs-default] with write index [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] and backing indices []
[00:09:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [logs]
[00:09:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [logs]
[00:09:10]                   │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] [.ds-logs-datastreams.test_logs-default-2020.12.16-000001/GU7MGKMUTnWCjW96-RATBg] update_mapping [_doc]
[00:09:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-logs-datastreams.test_logs-default-2020.12.16-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [logs]
[00:09:10]                   │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] creating index, cause [initialize_data_stream], templates [metrics-datastreams.test_metrics], shards [1]/[1]
[00:09:10]                   │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] adding data stream [metrics-datastreams.test_metrics-default] with write index [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] and backing indices []
[00:09:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [metrics]
[00:09:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [metrics]
[00:09:10]                   │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001/wdrJSpOWSxGYWFrvsEO3Jg] update_mapping [_doc]
[00:09:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1608071708458203138] moving index [.ds-metrics-datastreams.test_metrics-default-2020.12.16-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [metrics]
[00:09:10]                 └- ✖ fail: Fleet Endpoints EPM Endpoints datastreams should list the logs and metrics datastream
[00:09:10]                 │      Error: expected '.ds-logs-datastreams.test_logs-default-2020.12.16-000001' to equal '.ds-logs-datastreams.test_logs-default-000001'
[00:09:10]                 │       at Assertion.assert (/dev/shm/workspace/parallel/24/kibana/packages/kbn-expect/expect.js:100:11)
[00:09:10]                 │       at Assertion.equal (/dev/shm/workspace/parallel/24/kibana/packages/kbn-expect/expect.js:227:8)
[00:09:10]                 │       at Context.<anonymous> (test/fleet_api_integration/apis/epm/data_stream.ts:91:76)
[00:09:10]                 │       at runMicrotasks (<anonymous>)
[00:09:10]                 │       at processTicksAndRejections (internal/process/task_queues.js:93:5)
[00:09:10]                 │       at Object.apply (/dev/shm/workspace/parallel/24/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:09:10]                 │ 
[00:09:10]                 │

Stack Trace

Error: expected '.ds-logs-datastreams.test_logs-default-2020.12.16-000001' to equal '.ds-logs-datastreams.test_logs-default-000001'
    at Assertion.assert (/dev/shm/workspace/parallel/24/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.equal (/dev/shm/workspace/parallel/24/kibana/packages/kbn-expect/expect.js:227:8)
    at Context.<anonymous> (test/fleet_api_integration/apis/epm/data_stream.ts:91:76)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at Object.apply (/dev/shm/workspace/parallel/24/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)

Metrics [docs]

Distributable file count

id before after diff
default 47249 48009 +760

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Failing ES Promotion: logstash pipeline list route add button links to the empty pipeline editor
2 participants
@tylersmalley @kibanamachine