Audit Logging for Extensions #5

MaciejMierzwa · 2023-06-22T13:09:52Z

Description

Turn on audit logging in SecurityRestFilter that logs rest requests on endpoints registered by extensions.

Category (Enhancement)
Why these changes are required?
What is the old behavior before changes and new behavior after changes?
The behavior before changes was that audit logging wasn't performed. After changes rest requests are logged.

Issues Resolved

#2700

Testing

Manual testing was performed based on examples provided in pr: #2753, for task: #2752

Audit logging config:

{
    "_readonly": [],
    "config":
        "enabled": true,
        "audit": {
            "ignore_users": [],
            "ignore_requests": [],
            "disabled_rest_categories": [],
            "disabled_transport_categories": [],
            "enable_transport": true,
            "enable_rest": true,
            "log_request_body": true,
            "resolve_indices": true,
            "resolve_bulk_requests": true,
            "exclude_sensitive_headers": true
        }
    }
}

Request/Response:

curl -XGET https://new-user:admin@localhost:9200/_extensions/_hw/hello --insecure
Hello, World

AuditLog:
[{ "_index": "security-auditlog-2023.06.22", "_id": "sOu-4ogByQU68dEm3NGb", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "maciej-Legion-5-15ACH6", "audit_rest_request_method": "GET", "audit_category": "GRANTED_PRIVILEGES", "audit_request_origin": "REST", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "2023-06-22T10:55:39.163+00:00", "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687431339163 ] }, { "_index": "security-auditlog-2023.06.22", "_id": "r-u-4ogByQU68dEm3NGa", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "maciej-Legion-5-15ACH6", "audit_request_initiating_user": "new-user", "audit_rest_request_method": "GET", "audit_category": "AUTHENTICATED", "audit_request_origin": "REST", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "2023-06-22T10:55:39.161+00:00", "audit_request_effective_user_is_admin": false, "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687431339161 ] }]

Request/Response:

curl -XPUT https://new-user:admin@localhost:9200/_extensions/_hw/hello/HW --insecure -H 'Content-Type: application/json' -d '{}'
Updated the world's name to HW

AuditLog:
[ { "_index": "security-auditlog-2023.06.22", "_id": "s-vF4ogByQU68dEmDdEg", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_rest_request_params": { "name": "HW" }, "audit_node_name": "node_name", "audit_rest_request_method": "PUT", "audit_category": "GRANTED_PRIVILEGES", "audit_request_origin": "REST", "audit_request_body": "{}", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello/HW", "@timestamp": "timestamp", "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "content-length": [ "2" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "content-type": [ "application/json" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687431744800 ] }, { "_index": "security-auditlog-2023.06.22", "_id": "tOvF4ogByQU68dEmDdEg", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_rest_request_params": { "name": "HW" }, "audit_node_name": "node_name", "audit_request_initiating_user": "new-user", "audit_rest_request_method": "PUT", "audit_category": "AUTHENTICATED", "audit_request_origin": "REST", "audit_request_body": "{}", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello/HW", "@timestamp": "timestamp", "audit_request_effective_user_is_admin": false, "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "content-length": [ "2" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "content-type": [ "application/json" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687431744799 ]]
Request/Response

curl -XGET https://new-user:admin@localhost:9200/_extensions/_hw/hello --insecurextensions/_hw/hello --insecure
Hello, HW!

AuditLog:
[ { "_index": "security-auditlog-2023.06.22", "_id": "uOvH4ogByQU68dEmNdES", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node_name", "audit_rest_request_method": "GET", "audit_category": "GRANTED_PRIVILEGES", "audit_request_origin": "REST", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "timestamp", "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687431886098 ] }, { "_index": "security-auditlog-2023.06.22", "_id": "t-vH4ogByQU68dEmNdER", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node_name", "audit_request_initiating_user": "new-user", "audit_rest_request_method": "GET", "audit_category": "AUTHENTICATED", "audit_request_origin": "REST", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "timestamp", "audit_request_effective_user_is_admin": false, "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687431886097 ] }]

Request/Response:

curl -XPOST https://new-user:admin@localhost:9200/_extensions/_hw/hello --insecure -H 'Content-Type: application/json' -d '{ }'
no permissions for [hw:greet_with_adjective] and User [name=new-user, backend_roles=[], requestedTenant=null]%

AuditLog:
[ { "_index": "security-auditlog-2023.06.22", "_id": "vOvI4ogByQU68dEmQ9Ha", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node_name", "audit_rest_request_method": "POST", "audit_category": "MISSING_PRIVILEGES", "audit_request_origin": "REST", "audit_request_body": "{ }", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "timestamp", "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "content-length": [ "3" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "content-type": [ "application/json" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687431955417 ] }, { "_index": "security-auditlog-2023.06.22", "_id": "u-vI4ogByQU68dEmQ9HY", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node_name", "audit_request_initiating_user": "new-user", "audit_rest_request_method": "POST", "audit_category": "AUTHENTICATED", "audit_request_origin": "REST", "audit_request_body": "{ }", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "timestamp", "audit_request_effective_user_is_admin": false, "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "content-length": [ "3" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "content-type": [ "application/json" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687431955416 ] }]
Request/Response:

//before performing this request create role-mapping extension_hw_full -> new-user
curl -XPOST https://new-user:admin@localhost:9200/_extensions/_hw/hello --insecure -H 'Content-Type: application/json' -d '{ }'
{"worldAdjectives":["{ }"]}

AuditLog:
[{ "_index": "security-auditlog-2023.06.22", "_id": "fB714ogBq1SgHk27qmi9", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node_name", "audit_rest_request_method": "POST", "audit_category": "GRANTED_PRIVILEGES", "audit_request_origin": "REST", "audit_request_body": "{ }", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "timestamp", "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "content-length": [ "3" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "content-type": [ "application/json" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687434930875 ] }, { "_index": "security-auditlog-2023.06.22", "_id": "ex714ogBq1SgHk27PGgf", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node_name", "audit_request_initiating_user": "new-user", "audit_rest_request_method": "POST", "audit_category": "AUTHENTICATED", "audit_request_origin": "REST", "audit_request_body": "{ }", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "timestamp", "audit_request_effective_user_is_admin": false, "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "content-length": [ "3" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "content-type": [ "application/json" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687434893142 ] }]

Request/Response:

curl -XDELETE https://new-user:admin@localhost:9200/_extensions/_hw/goodbye --insecure
Goodbye, cruel world! Restored default values

AuditLog:
[{ "_index": "security-auditlog-2023.06.22", "_id": "gB7_4ogBq1SgHk27Lmi_", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node_name", "audit_rest_request_method": "DELETE", "audit_category": "GRANTED_PRIVILEGES", "audit_request_origin": "REST", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/goodbye", "@timestamp": "timestamp", "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687435554493 ] }, { "_index": "security-auditlog-2023.06.22", "_id": "fx7_4ogBq1SgHk27IGgh", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node_name", "audit_request_initiating_user": "new-user", "audit_rest_request_method": "DELETE", "audit_category": "AUTHENTICATED", "audit_request_origin": "REST", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/goodbye", "@timestamp": "timestamp", "audit_request_effective_user_is_admin": false, "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687435545536 ] }]

Check List

New functionality includes testing
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Maciej Mierzwa <[email protected]>

DarshitChanpura · 2023-06-22T15:31:11Z

src/main/java/org/opensearch/security/filter/SecurityRestFilter.java

            } else {
-                // auditLog.logMissingPrivileges(action, request, task);
+                auditLog.logMissingPrivileges(route.name(), user.getName(), request);


is there a difference in adding this route.name() here? Maybe i missed it in the logs you posted

you're right, it didn't do anything. It turns about this parameter wasn't used anywhere in one of the methods down the line. Added it in AbstractAuditLog and here's how logs look like now:
audit_request_privilege is logged correctly

{ "_index": "security-auditlog-2023.06.23", "_id": "KJVs54gBDjJ9Ce5u7dZD", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node", "audit_rest_request_method": "POST", "audit_category": "MISSING_PRIVILEGES", "audit_request_origin": "REST", "audit_request_body": "{ }", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "2023-06-23T08:44:15.545+00:00", "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_request_privilege": "hw:greet_with_adjective", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "content-length": [ "3" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "content-type": [ "application/json" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687509855545 ] }, { "_index": "security-auditlog-2023.06.23", "_id": "J5Vs54gBDjJ9Ce5u7dZD", "_score": null, "_source": { "audit_cluster_name": "opensearch", "audit_node_name": "node", "audit_request_initiating_user": "new-user", "audit_rest_request_method": "POST", "audit_category": "AUTHENTICATED", "audit_request_origin": "REST", "audit_request_body": "{ }", "audit_node_id": "dwo7slt-SYiWN0z_aajNYQ", "audit_request_layer": "REST", "audit_rest_request_path": "/_extensions/_hw/hello", "@timestamp": "2023-06-23T08:44:12.445+00:00", "audit_request_effective_user_is_admin": false, "audit_format_version": 4, "audit_request_remote_address": "127.0.0.1", "audit_node_host_address": "127.0.0.1", "audit_rest_request_headers": { "x-http2-stream-id": [ "1" ], "content-length": [ "3" ], "host": [ "localhost:9200" ], "x-http2-scheme": [ "https" ], "content-type": [ "application/json" ], "user-agent": [ "curl/7.81.0" ], "accept": [ "*/*" ] }, "audit_request_effective_user": "new-user", "audit_node_host_name": "127.0.0.1" }, "sort": [ 1687509852445 ] }

Signed-off-by: Maciej Mierzwa <[email protected]>

### Description Adding tests for #2700, relevant PR: DarshitChanpura#5. The goal is to test audit logs hit in authorizeRequest() method from SecurityRestFilterExtension class. Issues Resolved: #2700 ### Testing Integration testing added ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff Signed-off-by: Maciej Mierzwa <[email protected]>

turn on audit logging

8503358

Signed-off-by: Maciej Mierzwa <[email protected]>

MaciejMierzwa requested a review from DarshitChanpura as a code owner June 22, 2023 13:09

MaciejMierzwa changed the title ~~turn on audit logging~~ Audit Logging for Extensions Jun 22, 2023

DarshitChanpura reviewed Jun 22, 2023

View reviewed changes

add privileges info

3c397e2

Signed-off-by: Maciej Mierzwa <[email protected]>

DarshitChanpura merged commit e7d10c7 into DarshitChanpura:authorize-rest-requests Jun 27, 2023

This was referenced Jul 31, 2023

audit logging tests, named routes #7

Closed

SecurityRestFilter, authorizeRequest() audit log tests opensearch-project/security#3084

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Audit Logging for Extensions #5

Audit Logging for Extensions #5

MaciejMierzwa commented Jun 22, 2023 •

edited

Loading

DarshitChanpura Jun 22, 2023

MaciejMierzwa Jun 23, 2023

Audit Logging for Extensions #5

Audit Logging for Extensions #5

Conversation

MaciejMierzwa commented Jun 22, 2023 • edited Loading

Description

Issues Resolved

Testing

Check List

DarshitChanpura Jun 22, 2023

Choose a reason for hiding this comment

MaciejMierzwa Jun 23, 2023

Choose a reason for hiding this comment

MaciejMierzwa commented Jun 22, 2023 •

edited

Loading