Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The CM plugin fails to send orders back to the Management API #154

Closed
2 tasks done
Tracked by #496 ...
AlexRuiz7 opened this issue Nov 29, 2024 · 6 comments · Fixed by #187
Closed
2 tasks done
Tracked by #496 ...

The CM plugin fails to send orders back to the Management API #154

AlexRuiz7 opened this issue Nov 29, 2024 · 6 comments · Fixed by #187
Assignees
Labels
level/task Task issue mvp Minimum Viable Product type/bug Bug issue

Comments

@AlexRuiz7
Copy link
Member

AlexRuiz7 commented Nov 29, 2024

Description

The Command Manager fails to send orders back to the Management API.

[2024-11-29T15:11:01,640][INFO ][c.w.c.r.RestPostCommandAction] [node-1] Received POST /_plugins/_command_manager/commands request id [277] from host [xxxxxxx:9200]
[2024-11-29T15:11:01,641][INFO ][c.w.c.u.h.HttpRestClientDemo] [node-1] Executing POST request
[2024-11-29T15:11:01,647][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Attempting authentication at [https://xxxxxxx:55000/security/user/authenticate]
[2024-11-29T15:11:01,647][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [null] to [https://xxxxxxx:55000/security/user/authenticate]
[2024-11-29T15:11:01,690][ERROR][c.w.c.u.h.HttpRestClient ] [node-1] Execution failed javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[2024-11-29T15:11:01,690][ERROR][c.w.c.u.h.HttpRestClientDemo] [node-1] Error sending data: java.lang.NullPointerException: Cannot invoke "org.apache.hc.client5.http.async.methods.SimpleHttpResponse.getCode()" because "loginResponse" is null
[2024-11-29T15:11:01,691][ERROR][c.w.c.r.RestPostCommandAction] [node-1] Error reading response: Cannot invoke "org.apache.hc.client5.http.async.methods.SimpleHttpResponse.getCode()" because "response" is null

Full Log

Plan

  • Investigate the cause of the problem.
  • Propose a fix.
@AlexRuiz7 AlexRuiz7 added level/task Task issue type/bug Bug issue mvp Minimum Viable Product labels Nov 29, 2024
@f-galland
Copy link
Member

Set up the manager with the following configuration:

root@mvp-api-test:~# cat /etc/wazuh-server/wazuh-server.yml 
server:
  nodes:
    - master
  node:
    name: manager_01
    type: master
    ssl:
      key:  "/etc/wazuh-server/server.key"
      cert: "/etc/wazuh-server/server.crt"
      ca:  "/etc/wazuh-server/server.ca"
management_api:
  host:
    - "0.0.0.0"
    - "::"
  communications_api:
    host: "0.0.0.0"
indexer:
  hosts:
    - host: localhost
      port: 9200
  username: "admin"
  password: "admin"
  ssl:
    use_ssl: true
    key:  "/etc/wazuh-indexer/certs/indexer-key.pem"
    certificate: "/etc/wazuh-indexer/certs/indexer.pem"
    certificate_authorities:
      - "/etc/wazuh-indexer/certs/root-ca.pem"

The indexer's keystore was set up with the following commands:

$ echo 'https://127.0.0.1:55000' | /usr/share/wazuh-indexer/bin/opensearch-keystore add m_api.uri
$ echo 'admin' | /usr/share/wazuh-indexer/bin/opensearch-keystore add m_api.auth.username
$ echo 'admin' | /usr/share/wazuh-indexer/bin/opensearch-keystore add m_api.auth.password

A command was issued to the Indexer's command manager API:

$ curl --cacert /etc/wazuh-indexer/certs/root-ca.pem -u admin:admin https://localhost:9200/_plugins/_command_manager/commands -H "Content-Type: application/json" -d '{
  "source": "Engine",
  "user": "user53",
  "target": {
    "id": "target4",
    "type": "agent"
  },
  "action": {
    "name": "restart",
    "args": [
      "/path/to/executable/arg6",
      "/path/to/executable/arg6",
      "/path/to/executable/arg6",
      "/path/to/executable/arg6",
      "/path/to/executable/arg6"
    ],
    "version": "v4"
  },
  "timeout": 30
}'
{"_index":".commands","_id":"S5HneJMBv1NFW5MWrXxx","result":"CREATED"}

After this, the following logs were found in /var/log/wazuh-indexer/wazuh-cluster.log:

$ grep -E 'HttpRestClient|ERROR' /var/log/wazuh-indexer/wazuh-cluster.log
[2024-11-29T17:00:54,937][INFO ][c.w.c.u.h.HttpRestClientDemo] [node-1] Executing POST request
[2024-11-29T17:00:54,938][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Attempting authentication at [https://127.0.0.1:55000/security/user/authenticate]
[2024-11-29T17:00:54,938][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [null] to [https://127.0.0.1:55000/security/user/authenticate]
[2024-11-29T17:00:54,943][ERROR][c.w.c.u.h.HttpRestClient ] [node-1] Execution failed javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[2024-11-29T17:00:54,943][ERROR][c.w.c.u.h.HttpResponseCallback] [node-1] POST https://127.0.0.1:55000/security/user/authenticate->javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[2024-11-29T17:00:54,943][ERROR][c.w.c.u.h.HttpRestClientDemo] [node-1] Error sending data: java.lang.NullPointerException: Cannot invoke "org.apache.hc.client5.http.async.methods.SimpleHttpResponse.getCode()" because "loginResponse" is null
[2024-11-29T17:00:54,943][ERROR][c.w.c.r.RestPostCommandAction] [node-1] Error reading response: Cannot invoke "org.apache.hc.client5.http.async.methods.SimpleHttpResponse.getCode()" because "response" is null
[2024-11-29T17:01:13,463][ERROR][o.o.h.n.s.SecureNetty4HttpServerTransport] [node-1] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6361742f6865616c746820485454502f312e310d0a486f73743a203132372e302e302e313a393230300d0a417574686f72697a6174696f6e3a2042617369632059575274615734365632463664576846626d6470626d55314b773d3d0d0a4163636570742d436861727365743a207574662d380d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a0d0a

The HttpClient is not handling ssl based connections properly. Our tests so far have been using insecure http. We need to update the plugin appropriately.

@f-galland
Copy link
Member

It seems that a custom SSL context needs to be created whenever we are dealing with self signed certificates:

@f-galland
Copy link
Member

f-galland commented Dec 3, 2024

In order to further test this issue, I'm setting up an imposter instance with self signed certs in the following way:

  1. Download the imposter jar file
  2. Generate certificates using the tools in our documentation
  3. Go into the wazuh-certificates folder and generate a pkcs12 file with the following command:
openssl pkcs12 -export -inkey wazuh-1-key.pem -in wazuh-1.pem -certfile root-ca.pem -out keystore.p12 -name wazuh-1 -passout pass:password
  1. Create a keystore with the above trust chain:
keytool -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -srcstorepass password -destkeystore imposter.jks -deststoretype JKS     -deststorepass password -alias wazuh-1
  1. Run imposter from the jar file with the following options:
IMPOSTER_OPENAPI_REMOTE_FILE_CACHE=true IMPOSTER_JS_PLUGIN=js-graal-compat java -jar ~/.imposter/engines/imposter-4.2.4.jar --plugin openapi --tlsEnabled --configDir ~/IdeaProjects/wazuh-indexer-plugins/imposter --listenPort 55000 --keystorePath ~/IdeaProjects/wazuh-indexer-plugins/imposter/imposter.jks --keystorePassword password

@f-galland
Copy link
Member

At the moment and in order to get the Wazuh Server up exposing a tls encrypted management API, we need to set up a configuration block in /etc/wazuh-server/wazuh-server.yml as follows:

management_api:
  ssl:
    key: /etc/wazuh-server/wazuh-1-key.pem
    cert: /etc/wazuh-server/wazuh-1.pem
    use_ca: false
    ca: /etc/wazuh-server/root-ca.pem

Note that use_ca is set to false. I couldn't get a curl call working with it set to true.

Issuing a request for a token works with the configured root-ca.pem:

root@mvp-api-test:~# curl --cacert /etc/wazuh-indexer/certs/root-ca.pem -u wazuh:wazuh -XPOST https://127.0.0.1:55000/security/user/authenticate
{"data": {"token": "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3YXp1aCIsImF1ZCI6IldhenVoIEFQSSBSRVNUIiwibmJmIjoxNzMzMjIyMTIxLCJleHAiOjE3MzMyMjMwMjEsInN1YiI6IndhenVoIiwicnVuX2FzIjpmYWxzZSwicmJhY19yb2xlcyI6WzFdLCJyYmFjX21vZGUiOiJ3aGl0ZSJ9.8VW9Em5w8n59r_ZUUYEgqZkSV4oLE7p0_9DzuNoq7zUTDlMP_f92w8iWmWfAQly9YvnqMm-XTAYoE8SRVlE-oQ"}, "error": 0}

In order for the Wazuh Indexer to be able to connect to the Management API using self-signed certificates, the root-ca.pem file signing wazuh-1.pem in the server's configuration file needs to be added to the Indexer's JVM keystore.

This can be done with the following command:

/usr/share/wazuh-indexer/jdk/bin/keytool -trustcacerts -keystore /usr/share/wazuh-indexer/jdk/lib/security/cacerts -importcert -alias wazuh-root-ca -file /etc/wazuh-indexer/certs/root-ca.pem

After this is done, issuing a call to the Indexer's command manager seems to work properly:

root@mvp-api-test:~# curl --cacert /etc/wazuh-indexer/certs/root-ca.pem -u admin:admin https://127.0.0.1:9200/_plugins/_command_manager/commands -H "Content-Type: application/json" -d '{
  "source": "Engine",
  "user": "user53",
  "target": {
    "id": "target4",
    "type": "agent"
  },
  "action": {
    "name": "restart",
    "args": [
      "/path/to/executable/arg6",
      "/path/to/executable/arg6",
      "/path/to/executable/arg6",
      "/path/to/executable/arg6",
      "/path/to/executable/arg6"
    ],
    "version": "v4"
  },
  "timeout": 30
}'
{"_index":".commands","_id":"5LsbjJMB1LH4hxV_LnvY","result":"CREATED"}

The Manager seems to receive data (as can be gathered from its stdout logs) but a 401 error code is returned:

2024/12/03 10:41:04 INFO: unknown_user 127.0.0.1 "POST /security/user/authenticate" with parameters {} and body {} done in 0.001s: 401
2024/12/03 10:41:04 INFO: unknown_user 127.0.0.1 "POST /orders" with parameters {} and body {"agent": {"groups": ["groups000"]}, "command": {"source": "Engine", "user": "user53", "target": {"type": "agent", "id": "target4"}, "action": {"name": "restart", "args": ["/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6"], "version": "v4"}, "timeout": 30, "status": "PENDING", "order_id": "5rsbjJMB1LH4hxV_-3vz", "request_id": "5bsbjJMB1LH4hxV_-3vz"}} done in 0.001s: 401
2024/12/03 10:41:04 INFO: unknown_user 127.0.0.1 "POST /security/user/authenticate" with parameters {} and body {} done in 0.001s: 401
2024/12/03 10:41:04 INFO: unknown_user 127.0.0.1 "POST /orders" with parameters {} and body {"agent": {"groups": ["groups000"]}, "command": {"source": "Engine", "user": "user53", "target": {"type": "agent", "id": "target4"}, "action": {"name": "restart", "args": ["/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6"], "version": "v4"}, "timeout": 30, "status": "PENDING", "order_id": "5rsbjJMB1LH4hxV_-3vz", "request_id": "5bsbjJMB1LH4hxV_-3vz"}} done in 0.001s: 401
2024/12/03 10:41:04 INFO: unknown_user 127.0.0.1 "POST /security/user/authenticate" with parameters {} and body {} done in 0.001s: 401
2024/12/03 10:41:04 INFO: unknown_user 127.0.0.1 "POST /orders" with parameters {} and body {"agent": {"groups": ["groups000"]}, "command": {"source": "Engine", "user": "user53", "target": {"type": "agent", "id": "target4"}, "action": {"name": "restart", "args": ["/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6", "/path/to/executable/arg6"], "version": "v4"}, "timeout": 30, "status": "PENDING", "order_id": "5rsbjJMB1LH4hxV_-3vz", "request_id": "5bsbjJMB1LH4hxV_-3vz"}} done in 0.001s: 401

The 401 response codes seem to be related to the Wazuh Server not handling authentication properly:

[2024-12-03T10:41:04,499][INFO ][c.w.c.r.RestPostCommandAction] [node-1] Received POST /_plugins/_command_manager/commands request id [6] from host [127.0.0.1:9200]
[2024-12-03T10:41:04,500][INFO ][c.w.c.u.h.HttpRestClientDemo] [node-1] Executing POST request
[2024-12-03T10:41:04,504][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Attempting authentication at [https://127.0.0.1:55000/security/user/authenticate]
[2024-12-03T10:41:04,504][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [null] to [https://127.0.0.1:55000/security/user/authenticate]
[2024-12-03T10:41:04,512][ERROR][c.w.c.u.h.AuthHttpRestClient] [node-1] Authentication failed due to: SimpleBody{content length=58, content type=application/problem+json; charset=utf-8}
[2024-12-03T10:41:04,512][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [57sbjJMB1LH4hxV_-3vz] to [https://127.0.0.1:55000/orders]
[2024-12-03T10:41:04,515][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Token invalidated
[2024-12-03T10:41:04,515][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Attempting authentication at [https://127.0.0.1:55000/security/user/authenticate]
[2024-12-03T10:41:04,515][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [null] to [https://127.0.0.1:55000/security/user/authenticate]
[2024-12-03T10:41:04,517][ERROR][c.w.c.u.h.AuthHttpRestClient] [node-1] Authentication failed due to: SimpleBody{content length=58, content type=application/problem+json; charset=utf-8}
[2024-12-03T10:41:04,517][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [57sbjJMB1LH4hxV_-3vz] to [https://127.0.0.1:55000/orders]
[2024-12-03T10:41:04,518][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Token invalidated
[2024-12-03T10:41:04,519][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Attempting authentication at [https://127.0.0.1:55000/security/user/authenticate]
[2024-12-03T10:41:04,519][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [null] to [https://127.0.0.1:55000/security/user/authenticate]
[2024-12-03T10:41:04,520][ERROR][c.w.c.u.h.AuthHttpRestClient] [node-1] Authentication failed due to: SimpleBody{content length=58, content type=application/problem+json; charset=utf-8}
[2024-12-03T10:41:04,520][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [57sbjJMB1LH4hxV_-3vz] to [https://127.0.0.1:55000/orders]
[2024-12-03T10:41:04,522][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Token invalidated
[2024-12-03T10:41:04,522][ERROR][c.w.c.u.h.HttpRestClientDemo] [node-1] Error sending data: java.lang.Exception: Max retries [3/3] reached for POST request with id [57sbjJMB1LH4hxV_-3vz]
[2024-12-03T10:41:04,522][ERROR][c.w.c.r.RestPostCommandAction] [node-1] Error reading response: Cannot invoke "org.apache.hc.client5.http.async.methods.SimpleHttpResponse.getCode()" because "response" is null
[2024-12-03T10:41:04,522][INFO ][c.w.c.i.CommandIndex     ] [node-1] Index template index-template-commands already exists. Skipping creation.
[2024-12-03T10:41:04,522][INFO ][c.w.c.i.CommandIndex     ] [node-1] Indexing command with id [57sbjJMB1LH4hxV_-3vz]

The above cannot be reproduced when running authentication against an imposter instance mocking the API as specified in wazuh/wazuh's spec.yaml:

[2024-12-03T10:48:00,833][INFO ][c.w.c.r.RestPostCommandAction] [node-1] Received POST /_plugins/_command_manager/commands request id [7] from host [127.0.0.1:9200]
[2024-12-03T10:48:00,834][INFO ][c.w.c.u.h.HttpRestClientDemo] [node-1] Executing POST request
[2024-12-03T10:48:00,838][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Attempting authentication at [https://127.0.0.1:55000/security/user/authenticate]
[2024-12-03T10:48:00,839][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [null] to [https://127.0.0.1:55000/security/user/authenticate]
[2024-12-03T10:48:02,608][INFO ][c.w.c.u.h.AuthHttpRestClient] [node-1] Authentication successful
[2024-12-03T10:48:02,608][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [6rsijJMB1LH4hxV_VntC] to [https://127.0.0.1:55000/orders]
[2024-12-03T10:48:02,620][INFO ][c.w.c.r.RestPostCommandAction] [node-1] Received response to POST request with code [200]
[2024-12-03T10:48:02,620][INFO ][c.w.c.r.RestPostCommandAction] [node-1] Raw response:
{
  "data" : {
    "affected_items" : [ "l8nhoJIBerRY2Wz26Je8" ],
    "total_affected_items" : 1,
    "total_failed_items" : 0,
    "failed_items" : [ ]
  },
  "message" : "All orders were sent to the server",
  "error" : 0
}
[2024-12-03T10:48:02,620][INFO ][c.w.c.i.CommandIndex     ] [node-1] Index template index-template-commands already exists. Skipping creation.
[2024-12-03T10:48:02,620][INFO ][c.w.c.i.CommandIndex     ] [node-1] Indexing command with id [6rsijJMB1LH4hxV_VntC]

@wazuhci wazuhci moved this from In progress to On hold in XDR+SIEM/Release 5.0.0 Dec 3, 2024
@AlexRuiz7
Copy link
Member Author

AlexRuiz7 commented Dec 4, 2024

The problem is solved if we configure the Management API as depicted above and the root-ca.pem certificate is added to the embedded JDK's trust store.

We'll review this problem with the most recent package.

@wazuhci wazuhci moved this from On hold to Done in XDR+SIEM/Release 5.0.0 Dec 10, 2024
@AlexRuiz7 AlexRuiz7 reopened this Dec 17, 2024
@AlexRuiz7 AlexRuiz7 assigned AlexRuiz7 and unassigned f-galland Dec 17, 2024
@wazuhci wazuhci moved this from Done to In progress in XDR+SIEM/Release 5.0.0 Dec 17, 2024
@wazuhci wazuhci moved this from In progress to Pending review in XDR+SIEM/Release 5.0.0 Dec 17, 2024
@wazuhci wazuhci moved this from Pending review to Pending final review in XDR+SIEM/Release 5.0.0 Dec 17, 2024
@AlexRuiz7
Copy link
Member Author

Check the PR #187 for details.

@wazuhci wazuhci moved this from Pending final review to Done in XDR+SIEM/Release 5.0.0 Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level/task Task issue mvp Minimum Viable Product type/bug Bug issue
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants