Elastic-Agent: failed: could not decode the response, raw response: no matching action #24467

mtojek · 2021-03-10T10:31:03Z

I'm trying to enable the Fleet Server in elastic/elastic-package#279 (7.13.0-SNAPSHOT):

I wanted to assign a new policy clicked out with Kibana:

fleet_server
system

The agent can't pick it up:

2021-03-10T10:27:41.360Z	ERROR	application/fleet_gateway.go:185	failed to dispatch actions, error: acknowledge 20 actions '[action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 210cb410-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE action_id: 32ef4170-818b-11eb-b053-ad3176422d75, type: POLICY_CHANGE]' for elastic-agent '98c3677f-5380-4ea1-8e10-d2541c86289a' failed: could not decode the response, raw response: no matching action

2021-03-10T10:27:41.360Z	WARN	status/reporter.go:233	Elastic Agent status changed to: 'degraded'
2021-03-10T10:27:41.360Z	INFO	status/reporter.go:233	Elastic Agent status changed to: 'online'

... although the UI reports that it's healthy.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-03-10T10:31:05Z

Pinging @elastic/agent (Team:Agent)

ruflin · 2021-03-10T12:43:51Z

What you mean by assign a new policy? Do you have 2 policies and you tried to assign a different policy to the Elastic Agent with the fleet-server?

mtojek · 2021-03-10T13:47:28Z

I think there are multiple ways to see this effect, e.g. I tried to add Apache integration to the "Default Fleet Server policy".

blakerouse · 2021-03-10T13:55:12Z

Changing an Agent to a different policy is something still being worked on #24447. As for adding a new integration to the Fleet Server policy, that part should work.

I assume the Agent that is hitting this issue is also the one running the Fleet Server?

mtojek · 2021-03-10T14:10:08Z

Yes, the standard setup with embedded Fleet Server: https://github.com/elastic/elastic-package/pull/279/files

ruflin · 2021-03-16T12:37:27Z

I tried this out. I used elastic/elastic-package#279 to setup the stack, then selected the fleet-server policy and added system integration with only metrics enabled. The I get system metrics reported for the docker-fleet-agent. I think the status reporting on if a policy was fully deployed to fleet-server is not fully working yet (@nchaulet ) but the rollout happens.

Can you check if you see data coming in?

nchaulet · 2021-03-16T13:42:00Z

@mtojek I tested this locally and you need to remove the FLEET_URL and KIBANA_HOST from your agent docker config also you need to add the fleet server url in the kibana config xpack.fleet.agents.kibana.host: "http://127.0.0.1:8220" otherwise the agent is still talking to Kibana and it cause some issues

Let me know if it's work.

@blakerouse @hbharding During the onboarding and the setup of fleet server as fleet server is an elastic agent. I think we need to have the fleet server url configured before enrolling fleet server

mtojek · 2021-03-16T15:14:37Z

Thanks, that's something we didn't change before (port 8220). Out of curiosity - why the agent supporting fleet server doesn't ignore legacy environment variables if we use FLEET_SERVER_ENABLE=1? It should be sufficient to mark which mode the user wants to use.

I understand that due to above inconvenience, we won't be able to use same compose definitions for 7.11, 7.12 as environment variables exclude each other.

What is the minimum environment set in our case:

Agent's env:

    environment:
    - "FLEET_ENROLL=1"
    - "FLEET_ENROLL_INSECURE=1"
    - "FLEET_SERVER_INSECURE_HTTP=1"
    - "FLEET_INSECURE=1"
    - "FLEET_SERVER_ENABLE=1"
    - "FLEET_SETUP=1"

Kibana's config:

server.name: kibana
server.host: "0.0.0.0"

elasticsearch.hosts: [ "http://elasticsearch:9200" ]
elasticsearch.username: elastic
elasticsearch.password: changeme
xpack.monitoring.ui.container.elasticsearch.enabled: true

xpack.fleet.enabled: true
xpack.fleet.registryUrl: "http://package-registry:8080"
xpack.fleet.agents.enabled: true
xpack.fleet.agents.elasticsearch.host: "http://elasticsearch:9200"
xpack.fleet.agents.fleetServerEnabled: true
xpack.fleet.agents.kibana.host: "http://localhost:8220"
xpack.fleet.agents.tlsCheckDisabled: true
xpack.encryptedSavedObjects.encryptionKey: "12345678901234567890123456789012"

mtojek · 2021-03-16T15:40:17Z

Ok, with the above config I managed to connect to the Fleet Server (port 8220), although xpack.fleet.agents.kibana.host is really misleading (it's not Kibana but fleet server). Thank you for support, @nchaulet .

I think the question I posted above is still valid: what is the minimum set of vars for 7.13.0?

I'm hitting now that issue: #24453 , so I assume it's the next level?

I will leave this issue open to discuss the user experience of fleet configuration.

nchaulet · 2021-03-16T16:01:04Z

Yes the kibana.host is misleading we are in the process to have a new config variables for Kibana and the policy this will need to be updated when this land elastic/kibana#94364

simitt · 2021-04-14T06:21:06Z

FWIW I am seeing a similar error on latest 7.13.0-SNAPSHOTs. Started a fresh ES + Kibana container and started Elastic Agent with

docker run --name elastic-agent-local -it \
  --env KIBANA_FLEET_SETUP=1  \
  --env FLEET_SERVER_ENABLE=1 \
  --env ELASTICSEARCH_HOST="http://elasticsearch:9200" \
  --env ELASTICSEARCH_USERNAME=admin \
  --env ELASTICSEARCH_PASSWORD="password" \
  --env FLEET_ENROLL=1  \
  --env FLEET_INSECURE=1 \
  --env KIBANA_FLEET_HOST="http://kibana:5601" \
  --env KIBANA_FLEET_PASSWORD="changeme" \
  --env KIBANA_FLEET_USERNAME="password"  \
  --network apm-integration-testing \
  -p 8200:8200 --network-alias elastic-agent  --rm docker.elastic.co/beats/elastic-agent:7.13.0-SNAPSHOT

seeing this error:

2021-04-14T06:20:10.295Z	ERROR	fleet/fleet_gateway.go:203	Could not communicate with Checking API will retry, error: could not decode the response, raw response:

simitt · 2021-04-14T06:22:18Z

Just realized, the above mentioned error is covered in https://github.com/elastic/kibana/issues/96779.

mtojek · 2021-04-14T07:43:36Z

@simitt I suppose we've a different problem here. I'm pretty sure I reported this one for a different root cause, but all error message look either similar or even the same.

At the moment I don't reckon if this GH issue exists anymore (too many consecutive ones).

faec · 2022-04-04T18:42:56Z

This looks quite stale, closing as it is probably fixed or at least unrecognizable in current versions of agent.

mtojek added the Team:Elastic-Agent Label for the Agent team label Mar 10, 2021

mtojek mentioned this issue Mar 11, 2021

Elastic stack: support Fleet Server elastic/elastic-package#278

Closed

7 tasks

nchaulet assigned nchaulet and unassigned nchaulet Mar 16, 2021

faec closed this as completed Apr 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastic-Agent: failed: could not decode the response, raw response: no matching action #24467

Elastic-Agent: failed: could not decode the response, raw response: no matching action #24467

mtojek commented Mar 10, 2021 •

edited

Loading

elasticmachine commented Mar 10, 2021

ruflin commented Mar 10, 2021

mtojek commented Mar 10, 2021

blakerouse commented Mar 10, 2021

mtojek commented Mar 10, 2021

ruflin commented Mar 16, 2021

nchaulet commented Mar 16, 2021 •

edited

Loading

mtojek commented Mar 16, 2021 •

edited

Loading

mtojek commented Mar 16, 2021

nchaulet commented Mar 16, 2021

simitt commented Apr 14, 2021

simitt commented Apr 14, 2021

mtojek commented Apr 14, 2021

faec commented Apr 4, 2022

Elastic-Agent: failed: could not decode the response, raw response: no matching action #24467

Elastic-Agent: failed: could not decode the response, raw response: no matching action #24467

Comments

mtojek commented Mar 10, 2021 • edited Loading

elasticmachine commented Mar 10, 2021

ruflin commented Mar 10, 2021

mtojek commented Mar 10, 2021

blakerouse commented Mar 10, 2021

mtojek commented Mar 10, 2021

ruflin commented Mar 16, 2021

nchaulet commented Mar 16, 2021 • edited Loading

mtojek commented Mar 16, 2021 • edited Loading

mtojek commented Mar 16, 2021

nchaulet commented Mar 16, 2021

simitt commented Apr 14, 2021

simitt commented Apr 14, 2021

mtojek commented Apr 14, 2021

faec commented Apr 4, 2022

mtojek commented Mar 10, 2021 •

edited

Loading

nchaulet commented Mar 16, 2021 •

edited

Loading

mtojek commented Mar 16, 2021 •

edited

Loading