Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Reporting] Track down cause of 408 status response #41594

Closed
tsullivan opened this issue Jul 19, 2019 · 3 comments
Closed

[Reporting] Track down cause of 408 status response #41594

tsullivan opened this issue Jul 19, 2019 · 3 comments
Labels
(Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead

Comments

@tsullivan
Copy link
Member

Kibana version:
7.2.x

Elasticsearch version:
7.2.x

There have been cases where a Reporting job fails and a 408 status code comes back from Kibana (?).

As seen in a failing functional test with Watcher integration, Watcher logs that Kibana sent a 408 status response:

00:15:05 .watcher-history*={"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":0.6931472,"hits":[{"_index":".watcher-history-10-2019.07.19","_type":"_doc","_id":"watcher_report1563494492409_c5df9980-d432-45c6-8b2a-d70cd736ad2c-2019-07-19T00:13:47.957914Z","_score":0.6931472,"_source":{"watch_id":"watcher_report1563494492409","node":"FYTJoHuyQXyKHy78p9F5lQ","state":"executed","status":{"state":{"active":true,"timestamp":"2019-07-19T00:13:47.957Z"},"last_checked":"2019-07-19T00:13:47.957Z","last_met_condition":"2019-07-19T00:13:47.957Z","actions":{"email_admin":{"ack":{"timestamp":"2019-07-19T00:13:47.957Z","state":"awaits_successful_execution"},"last_execution":{"timestamp":"2019-07-19T00:13:47.957Z","successful":false,"reason":""}}},"execution_state":"executed","version":-1},"trigger_event":{"type":"schedule","triggered_time":"2019-07-19T00:13:47.957Z","schedule":{"scheduled_time":"2019-07-19T00:13:47.706Z"}},"input":{"none":{}},"condition":{"always":{}},"result":{"execution_time":"2019-07-19T00:13:47.957Z","execution_duration":8,"input":{"type":"none","status":"success","payload":{}},"condition":{"type":"always","status":"success","met":true},"actions":[{"id":"email_admin","type":"email","status":"failure","error":{"root_cause":[{"type":"exception","reason":"Watch[watcher_report1563494492409] reporting[test_report.png] Error response when trying to trigger reporting generation host[localhost], port[5601] method[POST], path[/api/reporting/generate/png], status[408]"}],"type":"exception","reason":"Watch[watcher_report1563494492409] reporting[test_report.png] Error response when trying to trigger reporting generation host[localhost], port[5601] method[POST], path[/api/reporting/generate/png], status[408]"}}]},"messages":[]}}]}}
00:15:05 
00:15:05 02:15:05.527: 
00:15:05 response={"_index":".watches","_type":"_doc","_id":"watcher_report1563494492409","_version":2,"_seq_no":4,"_primary_term":1,"found":true,"_source":{"trigger":{"schedule":{"interval":"300s"}},"actions":{"email_admin":{"email":{"to":["[email protected]","[email protected]","[email protected]","[email protected]"],"subject":"PNG 7.3.0 watcher_report1563494492409, VM=centos7_rpm a79f418e4bba1f76a0bf30d6a0981dffc751642c by:worker-950891","attachments":{"test_report.png":{"reporting":{"url":"https://localhost:5601/api/reporting/generate/png?jobParams=(browserTimezone:Europe%2FBucharest,layout:(dimensions:(height:1400,width:1864)),objectType:dashboard,relativeUrl:%27%2Fapp%2Fkibana%23%2Fdashboard%2FMetricbeat-system-overview-ecs%3F_g%3D()%26_a%3D(description:!%27Overview%2Bof%2Bsystem%2Bmetrics!%27,filters:!!(),fullScreenMode:!!f,options:(darkTheme:!!f),panels:!!((gridData:(h:5,i:!%279!%27,w:48,x:0,y:0),id:System-Navigation-ecs,panelIndex:!%279!%27,type:visualization,version:!%277.0.0-beta1!%27),(embeddableConfig:(vis:(defaultColors:(!%270%2B-%2B100!%27:!%27rgb(0,104,55)!%27))),gridData:(h:10,i:!%2711!%27,w:8,x:0,y:5),id:c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs,panelIndex:!%2711!%27,type:visualization,version:!%277.0.0-beta1!%27),(embeddableConfig:(vis:(defaultColors:(!%270%2B-%2B100!%27:!%27rgb(0,104,55)!%27))),gridData:(h:25,i:!%2712!%27,w:24,x:24,y:15),id:fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs,panelIndex:!%2712!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:25,i:!%2713!%27,w:24,x:0,y:15),id:!%27855899e0-1b1c-11e7-b09e-037021c4f8df-ecs!%27,panelIndex:!%2713!%27,type:visualization,version:!%277.0.0-beta1!%27),(embeddableConfig:(vis:(defaultColors:(!%270%2525%2B-%2B15%2525!%27:!%27rgb(247,252,245)!%27,!%2715%2525%2B-%2B30%2525!%27:!%27rgb(199,233,192)!%27,!%2730%2525%2B-%2B45%2525!%27:!%27rgb(116,196,118)!%27,!%2745%2525%2B-%2B60%2525!%27:!%27rgb(35,139,69)!%27))),gridData:(h:30,i:!%2714!%27,w:48,x:0,y:40),id:!%277cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs!%27,panelIndex:!%2714!%27,type:visualization,version:!%277.0.0-beta1!%27),(embeddableConfig:(vis:(defaultColors:(!%270%2B-%2B100!%27:!%27rgb(0,104,55)!%27))),gridData:(h:10,i:!%2716!%27,w:8,x:32,y:5),id:!%27522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs!%27,panelIndex:!%2716!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:10,i:!%2717!%27,w:8,x:40,y:5),id:!%271aae9140-1b93-11e7-8ada-3df93aab833e-ecs!%27,panelIndex:!%2717!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:10,i:!%2718!%27,w:8,x:24,y:5),id:!%27825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs!%27,panelIndex:!%2718!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:10,i:!%2719!%27,w:8,x:16,y:5),id:d3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs,panelIndex:!%2719!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:10,i:!%2720!%27,w:8,x:8,y:5),id:!%2783e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs!%27,panelIndex:!%2720!%27,type:visualization,version:!%277.0.0-beta1!%27)),query:(language:lucene,query:(query_string:(analyze_wildcard:!!t,query:!%27*!%27))),timeRestore:!!f,title:!%27%255BMetricbeat%2BSystem%255D%2BOverview%2BECS!%27,viewMode:view)%27,title:%27%5BMetricbeat%20System%5D%20Overview%20ECS%27)","auth":{"basic":{"username":"elastic","password":"changeit"}}}}}}}},"status":{"last_checked":"2019-07-19T00:13:47.957Z","last_met_condition":"2019-07-19T00:13:47.957Z","execution_state":"executed","actions":{"email_admin":{"last_execution":{"reason":"","timestamp":"2019-07-19T00:13:47.957Z","successful":false},"ack":{"state":"awaits_successful_execution","timestamp":"2019-07-19T00:13:47.957Z"}}},"version":-1}}}
00:15:05 
00:15:05 02:15:05.530: tryForTime failure: expected false to sort of equal true
00:15:06 02:15:06.033: ... sleep(15000) start
00:15:21 02:15:21.043: ... sleep(15000) end
00:15:21 search {watch_id:watcher_report1563494492409}
00:15:21 02:15:21.049: 
00:15:21 .watcher-history*={"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":0.6931472,"hits":[{"_index":".watcher-history-10-2019.07.19","_type":"_doc","_id":"watcher_report1563494492409_c5df9980-d432-45c6-8b2a-d70cd736ad2c-2019-07-19T00:13:47.957914Z","_score":0.6931472,"_source":{"watch_id":"watcher_report1563494492409","node":"FYTJoHuyQXyKHy78p9F5lQ","state":"executed","status":{"state":{"active":true,"timestamp":"2019-07-19T00:13:47.957Z"},"last_checked":"2019-07-19T00:13:47.957Z","last_met_condition":"2019-07-19T00:13:47.957Z","actions":{"email_admin":{"ack":{"timestamp":"2019-07-19T00:13:47.957Z","state":"awaits_successful_execution"},"last_execution":{"timestamp":"2019-07-19T00:13:47.957Z","successful":false,"reason":""}}},"execution_state":"executed","version":-1},"trigger_event":{"type":"schedule","triggered_time":"2019-07-19T00:13:47.957Z","schedule":{"scheduled_time":"2019-07-19T00:13:47.706Z"}},"input":{"none":{}},"condition":{"always":{}},"result":{"execution_time":"2019-07-19T00:13:47.957Z","execution_duration":8,"input":{"type":"none","status":"success","payload":{}},"condition":{"type":"always","status":"success","met":true},"actions":[{"id":"email_admin","type":"email","status":"failure","error":{"root_cause":[{"type":"exception","reason":"Watch[watcher_report1563494492409] reporting[test_report.png] Error response when trying to trigger reporting generation host[localhost], port[5601] method[POST], path[/api/reporting/generate/png], status[408]"}],"type":"exception","reason":"Watch[watcher_report1563494492409] reporting[test_report.png] Error response when trying to trigger reporting generation host[localhost], port[5601] method[POST], path[/api/reporting/generate/png], status[408]"}}]},"messages":[]}}]}}
00:15:21 
00:15:21 02:15:21.052: 
00:15:21 response={"_index":".watches","_type":"_doc","_id":"watcher_report1563494492409","_version":2,"_seq_no":4,"_primary_term":1,"found":true,"_source":{"trigger":{"schedule":{"interval":"300s"}},"actions":{"email_admin":{"email":{"to":["[email protected]","[email protected]","[email protected]","[email protected]"],"subject":"PNG 7.3.0 watcher_report1563494492409, VM=centos7_rpm a79f418e4bba1f76a0bf30d6a0981dffc751642c by:worker-950891","attachments":{"test_report.png":{"reporting":{"url":"https://localhost:5601/api/reporting/generate/png?jobParams=(browserTimezone:Europe%2FBucharest,layout:(dimensions:(height:1400,width:1864)),objectType:dashboard,relativeUrl:%27%2Fapp%2Fkibana%23%2Fdashboard%2FMetricbeat-system-overview-ecs%3F_g%3D()%26_a%3D(description:!%27Overview%2Bof%2Bsystem%2Bmetrics!%27,filters:!!(),fullScreenMode:!!f,options:(darkTheme:!!f),panels:!!((gridData:(h:5,i:!%279!%27,w:48,x:0,y:0),id:System-Navigation-ecs,panelIndex:!%279!%27,type:visualization,version:!%277.0.0-beta1!%27),(embeddableConfig:(vis:(defaultColors:(!%270%2B-%2B100!%27:!%27rgb(0,104,55)!%27))),gridData:(h:10,i:!%2711!%27,w:8,x:0,y:5),id:c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs,panelIndex:!%2711!%27,type:visualization,version:!%277.0.0-beta1!%27),(embeddableConfig:(vis:(defaultColors:(!%270%2B-%2B100!%27:!%27rgb(0,104,55)!%27))),gridData:(h:25,i:!%2712!%27,w:24,x:24,y:15),id:fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs,panelIndex:!%2712!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:25,i:!%2713!%27,w:24,x:0,y:15),id:!%27855899e0-1b1c-11e7-b09e-037021c4f8df-ecs!%27,panelIndex:!%2713!%27,type:visualization,version:!%277.0.0-beta1!%27),(embeddableConfig:(vis:(defaultColors:(!%270%2525%2B-%2B15%2525!%27:!%27rgb(247,252,245)!%27,!%2715%2525%2B-%2B30%2525!%27:!%27rgb(199,233,192)!%27,!%2730%2525%2B-%2B45%2525!%27:!%27rgb(116,196,118)!%27,!%2745%2525%2B-%2B60%2525!%27:!%27rgb(35,139,69)!%27))),gridData:(h:30,i:!%2714!%27,w:48,x:0,y:40),id:!%277cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs!%27,panelIndex:!%2714!%27,type:visualization,version:!%277.0.0-beta1!%27),(embeddableConfig:(vis:(defaultColors:(!%270%2B-%2B100!%27:!%27rgb(0,104,55)!%27))),gridData:(h:10,i:!%2716!%27,w:8,x:32,y:5),id:!%27522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs!%27,panelIndex:!%2716!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:10,i:!%2717!%27,w:8,x:40,y:5),id:!%271aae9140-1b93-11e7-8ada-3df93aab833e-ecs!%27,panelIndex:!%2717!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:10,i:!%2718!%27,w:8,x:24,y:5),id:!%27825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs!%27,panelIndex:!%2718!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:10,i:!%2719!%27,w:8,x:16,y:5),id:d3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs,panelIndex:!%2719!%27,type:visualization,version:!%277.0.0-beta1!%27),(gridData:(h:10,i:!%2720!%27,w:8,x:8,y:5),id:!%2783e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs!%27,panelIndex:!%2720!%27,type:visualization,version:!%277.0.0-beta1!%27)),query:(language:lucene,query:(query_string:(analyze_wildcard:!!t,query:!%27*!%27))),timeRestore:!!f,title:!%27%255BMetricbeat%2BSystem%255D%2BOverview%2BECS!%27,viewMode:view)%27,title:%27%5BMetricbeat%20System%5D%20Overview%20ECS%27)","auth":{"basic":{"username":"elastic","password":"changeit"}}}}}}}},"status":{"last_checked":"2019-07-19T00:13:47.957Z","last_met_condition":"2019-07-19T00:13:47.957Z","execution_state":"executed","actions":{"email_admin":{"last_execution":{"reason":"","timestamp":"2019-07-19T00:13:47.957Z","successful":false},"ack":{"state":"awaits_successful_execution","timestamp":"2019-07-19T00:13:47.957Z"}}},"version":-1}}}
00:15:21 
00:15:21 02:15:21.053: tryForTime failure: expected false to sort of equal true
00:15:21 02:15:21.557: Taking screenshot "/var/lib/jenkins/workspace/elastic+integration-test+7.3/test/screenshots/failure/failure_1563495321556_PNG Reporting watch_should be successful and update revision.png"
00:15:21 >> FAIL: chrome 11 on any platform - reporting app - watcher app - PNG Reporting watch - should be successful and update revision (393920ms)
00:15:21 Error: timeout expected false to sort of equal true
00:15:21   at attempt  <test/support/utils/try.js:23:15>
00:15:21   at tryCatcher  <node_modules/bluebird/js/main/util.js:26:23>
00:15:21   at Promise._settlePromiseFromHandler  <node_modules/bluebird/js/main/promise.js:503:31>
00:15:21   at Promise._settlePromiseAt  <node_modules/bluebird/js/main/promise.js:577:18>
00:15:21   at Promise._settlePromises  <node_modules/bluebird/js/main/promise.js:693:14>
00:15:21   at Async._drainQueue  <node_modules/bluebird/js/main/async.js:123:16>
00:15:21   at Async._drainQueues  <node_modules/bluebird/js/main/async.js:133:10>
00:15:21   at Immediate.Async.drainQueues  <node_modules/bluebird/js/main/async.js:15:14>
00:15:21   at runCallback  <timers.js:637:20>
00:15:21   at tryOnImmediate  <timers.js:610:5>
00:15:21   at processImmediate [as _immediateCallback]  <timers.js:582:5>
00:15:21 02:15:21.645: 
00:15:21 response={"_index":".watches","_type":"_doc","_id":"watcher_report1563494492409","_version":3,"result":"deleted","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":5,"_primary_term":1}
@tsullivan tsullivan added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead Team:Stack Services labels Jul 19, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-stack-services

@tsullivan tsullivan mentioned this issue Jul 19, 2019
Closed
@bmcconaghy bmcconaghy added Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) and removed Team:Stack Services labels Dec 12, 2019
@bmcconaghy bmcconaghy added Team:Reporting Services and removed Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Dec 20, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

@tsullivan
Copy link
Member Author

I don't think a 408 status can be sent from Kibana. Most likely there is a proxy in front of Kibana that controls timeouts of requests that it sees.

If 408 status codes come up, it's probably best to check the proxy logs to see what is happening.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
(Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tsullivan @bmcconaghy @elasticmachine