[logging] Downgrade hapi connection errors when connecting with the w… #11209

jbudz · 2017-04-12T21:46:20Z

When serving Kibana over http and a request comes in over https, kibana logs a "Parse Error" to the standard application log.

When serving Kibana over https and a request comes in over http, kibana logs a "routines:SSL23_GET_CLIENT_HELLO:http request" to the standard application log. This won't be testable until the http -> https redirect with httpolyglot is removed. See #10930.

This downgrades both log types to debug output.

spalger

If the answer to my question is "yes, that's our only option" then this LGTM

spalger · 2017-04-14T17:37:54Z

src/server/logging/log_interceptor.js

+  }
+
+  downgradeIfHTTPSWhenHTTP(event) {
+    return downgradeIfErrorMessage('Parse Error', event);


Is error message matching our only option here? Parse Error is pretty vague isn't it? I guess we're still limiting it by the tag list, but I'd really like to use a value that's intended for machines (like errno) rather than one that's intended for humans (and much more likely to change/conflict)

Good point, I went straight for text because there was no errno. But there is a code for Parse Error sot hat looks like it's the way to go:

{ event: 'log', timestamp: 1492194353021, tags: ['connection', 'client', 'error'], data: { Error: Parse Error at Error(native) bytesParsed: 0, code: 'HPE_INVALID_METHOD' }, pid: 73405 }

For the http when https case there isn't more:

{ event: 'log', timestamp: 1492194569892, tags: ['connection', 'client', 'error'], data: Error: 140736014037952: error: 1407609 C: SSL routines: SSL23_GET_CLIENT_HELLO: http request: ../deps/openssl/openssl/ssl/s23_srvr.c:394:at Error(native), pid: 74537 }

- httpolyglot is removed, we no longer automatically redirect from http to https - server.ssl.redirectHttpFromPort option added to allow for http -> https redirect from one port to another - We no longer start the dev server with tls by default, it can be turned on with the --ssl flag, npm start -- --ssl, or ./bin/kibana --dev --ssl - There will currently be error log messages if you connect over the wrong protocol, we have #11209 for downgrading these

…rong protocol

…s when serving http errors

kimjoar · 2017-07-10T15:31:46Z

Rebased master, works as expected in my testing.

Below are examples when debug logging is enabled.

ssl.enabled: true + http request:

log   [15:27:52.252] [debug][connection] 140735775310784:error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request:../deps/openssl/openssl/ssl/s23_srvr.c:394:

ssl.enabled: false + https request:

log   [15:30:06.249] [debug][connection][hpe_invalid_method] HPE_INVALID_METHOD: Socket was closed by the client (probably the browser) before it could be read completely

epixa

LGTM on green

## Summary Changes in this PR: * Added `update` route to handle bulk rule migrations docs updates * Exposed `id` field in `RuleMigration` object needed for ES bulk update operation * Updated SIEM migrations schemas to use `NonEmptyString` when it is needed ## Testing locally Enable the flag ``` xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled'] ``` Create and start a rule migration. Then use `update` API to updated corresponding docs. cURL request examples: <details> <summary>Rules migration `create` POST request</summary> ``` curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '[ { "id": "f8c325ea-506e-4105-8ccf-da1492e90115", "vendor": "splunk", "title": "Linux Auditd Add User Account Type", "description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.", "query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *", "query_language":"spl", "mitre_attack_ids": [ "T1136" ] }, { "id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90", "vendor": "splunk", "title": "Linux Auditd Change File Owner To Root", "description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.", "query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`", "query_language": "spl", "mitre_attack_ids": [ "T1222" ] } ]' ``` </details> <details> <summary>Rules migration `start` task request</summary> - Assuming the connector `azureOpenAiGPT4o` is already created in the local environment. - Using the {{`migration_id`}} from the first POST request response ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '{ "connectorId": "azureOpenAiGPT4o" }' ``` </details> <details> <summary>Rules migration rules documents request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration `update` PUT request</summary> - Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from previous GET request response ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' --data '[ { "comments": [ "## Migration Summary\n- The `FROM` command is used to select the `logs-*` index pattern.\n- The `RENAME` command is used to rename the `host` field to `dest`.\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.", "Additional comment 2.0" ], "translation_result": "full", "id": "{{rule_migration_id_1}}" }, { "created_by": "elastic2.0", "elastic_rule": { "severity": "high", "title": "Linux Auditd Change File Owner To Root (UPDATED)" }, "id": "{{rule_migration_id_2}}" } ]' ``` </details> --------- Co-authored-by: kibanamachine <[email protected]>

…lastic#200815) ## Summary Changes in this PR: * Added `update` route to handle bulk rule migrations docs updates * Exposed `id` field in `RuleMigration` object needed for ES bulk update operation * Updated SIEM migrations schemas to use `NonEmptyString` when it is needed ## Testing locally Enable the flag ``` xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled'] ``` Create and start a rule migration. Then use `update` API to updated corresponding docs. cURL request examples: <details> <summary>Rules migration `create` POST request</summary> ``` curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '[ { "id": "f8c325ea-506e-4105-8ccf-da1492e90115", "vendor": "splunk", "title": "Linux Auditd Add User Account Type", "description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.", "query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *", "query_language":"spl", "mitre_attack_ids": [ "T1136" ] }, { "id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90", "vendor": "splunk", "title": "Linux Auditd Change File Owner To Root", "description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.", "query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`", "query_language": "spl", "mitre_attack_ids": [ "T1222" ] } ]' ``` </details> <details> <summary>Rules migration `start` task request</summary> - Assuming the connector `azureOpenAiGPT4o` is already created in the local environment. - Using the {{`migration_id`}} from the first POST request response ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '{ "connectorId": "azureOpenAiGPT4o" }' ``` </details> <details> <summary>Rules migration rules documents request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration `update` PUT request</summary> - Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from previous GET request response ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' --data '[ { "comments": [ "## Migration Summary\n- The `FROM` command is used to select the `logs-*` index pattern.\n- The `RENAME` command is used to rename the `host` field to `dest`.\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.", "Additional comment 2.0" ], "translation_result": "full", "id": "{{rule_migration_id_1}}" }, { "created_by": "elastic2.0", "elastic_rule": { "severity": "high", "title": "Linux Auditd Change File Owner To Root (UPDATED)" }, "id": "{{rule_migration_id_2}}" } ]' ``` </details> --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit f6ac2cf)

…200815) (#200910) # Backport This will backport the following commits from `main` to `8.x`: - [[Rules migration] Add rules migrations update route (#11209) (#200815)](#200815)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

…lastic#200815) ## Summary Changes in this PR: * Added `update` route to handle bulk rule migrations docs updates * Exposed `id` field in `RuleMigration` object needed for ES bulk update operation * Updated SIEM migrations schemas to use `NonEmptyString` when it is needed ## Testing locally Enable the flag ``` xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled'] ``` Create and start a rule migration. Then use `update` API to updated corresponding docs. cURL request examples: <details> <summary>Rules migration `create` POST request</summary> ``` curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '[ { "id": "f8c325ea-506e-4105-8ccf-da1492e90115", "vendor": "splunk", "title": "Linux Auditd Add User Account Type", "description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.", "query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *", "query_language":"spl", "mitre_attack_ids": [ "T1136" ] }, { "id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90", "vendor": "splunk", "title": "Linux Auditd Change File Owner To Root", "description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.", "query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`", "query_language": "spl", "mitre_attack_ids": [ "T1222" ] } ]' ``` </details> <details> <summary>Rules migration `start` task request</summary> - Assuming the connector `azureOpenAiGPT4o` is already created in the local environment. - Using the {{`migration_id`}} from the first POST request response ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' \ --header 'Content-Type: application/json' \ --data '{ "connectorId": "azureOpenAiGPT4o" }' ``` </details> <details> <summary>Rules migration rules documents request</summary> - Using the {{`migration_id`}} from the first POST request response. ``` curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' ``` </details> <details> <summary>Rules migration `update` PUT request</summary> - Using the {{`rule_migration_id_1`}} and {{`rule_migration_id_2`}} from previous GET request response ``` curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \ --header 'kbn-xsrf;' \ --header 'x-elastic-internal-origin: security-solution' \ --header 'elastic-api-version: 1' --data '[ { "comments": [ "## Migration Summary\n- The `FROM` command is used to select the `logs-*` index pattern.\n- The `RENAME` command is used to rename the `host` field to `dest`.\n- The `WHERE` command filters the rows where `process_exec` contains the pattern `*chown *root*`.\n- The `STATS` command is used to aggregate the data, counting the number of occurrences and finding the minimum and maximum timestamps, grouped by `process_exec`, `proctitle`, `normalized_proctitle_delimiter`, and `dest`.\n- The macros `security_content_ctime` and `linux_auditd_change_file_owner_to_root_filter` are placeholders for the corresponding Splunk macros.", "Additional comment 2.0" ], "translation_result": "full", "id": "{{rule_migration_id_1}}" }, { "created_by": "elastic2.0", "elastic_rule": { "severity": "high", "title": "Linux Auditd Change File Owner To Root (UPDATED)" }, "id": "{{rule_migration_id_2}}" } ]' ``` </details> --------- Co-authored-by: kibanamachine <[email protected]>

jbudz added Team:Operations Team label for Operations Team review v6.0.0 labels Apr 12, 2017

jbudz force-pushed the log-ssl-connection branch from ccf058a to 04b0d21 Compare April 12, 2017 21:47

jbudz mentioned this pull request Apr 12, 2017

Revert same port http -> https redirect #10930

Merged

spalger reviewed Apr 14, 2017

View reviewed changes

jbudz added updates_needed and removed review labels Apr 14, 2017

jbudz force-pushed the log-ssl-connection branch from 04b0d21 to f9d0f6e Compare April 18, 2017 20:50

jbudz added review and removed updates_needed labels Apr 18, 2017

epixa self-requested a review May 17, 2017 18:12

epixa added the v6.0.0-alpha2 label May 17, 2017

epixa added v6.0.0-alpha3 and removed v6.0.0-alpha2 labels Jun 2, 2017

jimgoodwin added the v6.0.0-beta1 label Jul 7, 2017

jbudz added 2 commits July 10, 2017 17:11

[logging] Downgrade hapi connection errors when connecting with the w…

56d708c

…rong protocol

[logging] Check for error code instead of message for requesting http…

3d8b0c3

…s when serving http errors

kimjoar force-pushed the log-ssl-connection branch from f9d0f6e to 3d8b0c3 Compare July 10, 2017 15:29

epixa approved these changes Jul 10, 2017

View reviewed changes

kimjoar merged commit 00cd9ea into elastic:master Jul 10, 2017

epixa added v6.0.0-beta1 and removed v6.0.0-beta1 v6.0.0-alpha3 labels Jul 10, 2017

jimgoodwin added release_note:fix release_note:enhancement and removed release_note:fix labels Jul 25, 2017

jportner mentioned this pull request Sep 14, 2020

Error logs when clients connect over HTTP instead of HTTPS #77391

Closed

e40pud added a commit to e40pud/kibana that referenced this pull request Nov 19, 2024

[Rules migration] Add rules migrations update route (elastic#11209)

77db9cf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[logging] Downgrade hapi connection errors when connecting with the w… #11209

[logging] Downgrade hapi connection errors when connecting with the w… #11209

jbudz commented Apr 12, 2017

spalger left a comment

spalger Apr 14, 2017

jbudz Apr 14, 2017

kimjoar commented Jul 10, 2017

epixa left a comment

[logging] Downgrade hapi connection errors when connecting with the w… #11209

[logging] Downgrade hapi connection errors when connecting with the w… #11209

Conversation

jbudz commented Apr 12, 2017

spalger left a comment

Choose a reason for hiding this comment

spalger Apr 14, 2017

Choose a reason for hiding this comment

jbudz Apr 14, 2017

Choose a reason for hiding this comment

kimjoar commented Jul 10, 2017

epixa left a comment

Choose a reason for hiding this comment