Fleet: error: could not decode the response

[mtojek]

The goal of the issue to root cause problems with Elastic Agent/Fleet/Kibana with latest snapshots. The stack boots up correctly, then a new policy should be reassigned to the agent, but it seems that it never happens.

Artifacts/logs: https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Felastic-package/detail/PR-319/1/artifacts

�[36mfleet-server_1               |�[0m 2021-04-12T08:52:08.659Z	ERROR	fleet/fleet_gateway.go:203	Could not communicate with Checking API will retry, error: could not decode the response, raw response: 
�[36mfleet-server_1               |�[0m 
�[36mfleet-server_1               |�[0m 2021-04-12T08:53:42.756Z	ERROR	fleet/fleet_gateway.go:203	Could not communicate with Checking API will retry, error: could not decode the response, raw response: 
�[36mfleet-server_1               |�[0m 
�[36mfleet-server_1               |�[0m 2021-04-12T08:56:45.556Z	ERROR	fleet/fleet_gateway.go:203	Could not communicate with Checking API will retry, error: could not decode the response, raw response: 
�[36mfleet-server_1               |�[0m 

We deployed an emergency fix to use last known stable revisions. Here is the PR reverting this change: elastic/elastic-package#319 . This one is expected to be back to green.

It impacts elastic/integrations for ~5 days now.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[mtojek]

@blakerouse @nchaulet Would you mind sharing the status? Do you need more reference data/logs?

For clarity: due to this issue we are unable to test integrations with latest snapshots (current using ones published a week ago).

[blakerouse]

I see that fleet-server is having issues with the following error:

{"log.level":"info","error":"elastic fail 404:index_not_found_exception:no such index [.fleet-actions]","id":"9db653bd-0679-4db1-9419-397be8e22933","code":400,"@timestamp":"2021-04-13T20:36:27.251Z","message":"fail checkin"}

Did some more digging and this was not an error on 8.0. After some more looking I see that Fleet Server missed a backport to 7.x.

elastic/fleet-server#205

[blakerouse]

Backport PR - elastic/fleet-server#231

Once landed and new 7.x snapshot, I expect this to be fixed.

[simitt]

Testing with latest 7.13 snapshot (2021/04/16) and still seeing following error:

{"log.level":"error","@timestamp":"2021-04-16T08:54:34.425Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":203},"message":"Could not communicate with Checking API will retry, error: could not decode the response, raw response: \n","ecs.version":"1.6.0"}

[jen-huang]

Are we still seeing this behavior with recent snapshots? If so, @blakerouse @nchaulet any guess on which side this is on?

[mtojek]

I haven't seen it, but this error is so generic that when it bumps up it usually means something really different. Is it possible to improve logging in this case? For example - flush the raw byte content or something (if debug is enabled).

[ruflin]

Also did not see this recently. ++ on improving logging. I filed #25230, please feel free to edit it and add entries with specific things that should be improved.

[simitt]

It's back - seen with the latest snapshots today.

[mtojek]

I can give you some hints. I think the CI managed to reproduce it if only the Fleet-Server health status fluctuates (see: #25341).

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[jen-huang]

I went ahead and moved this and put it in the agent Iteration board :)

[ph]

This is indeed an agent @elastic/agent Anyone could take a look at this one?

[faec]

Confirmed that this is no longer happening in local snapshots or nightly builds. Closing, though we can reopen if this is spotted again independent of #25341.