[Security Solution] Endpoint lacks permissions on cloud to write to `.logs-endpoint.action.responses-*` on cloud instances #116715

kevinlog · 2021-10-28T21:16:24Z

Kibana version:
7.16, BC2

Elasticsearch version:
7.16, BC2

Describe the bug:
When the Endpoint tries to write to the new .logs-endpoint.action.responses-*, it doesn't have proper index permissions to create the data stream. This is seen in the Endpoint logs (see below).

The Endpoint tries to write to this datastream when it completes a response action such as Host Isolation.

I have reproduced this in Cloud instances only. Locally, everything seems to work correctly and the Endpoint has permissions to create and write to the appropriate indices.

Steps to reproduce:

Deploy an Agent running Endpoint security
Navigate to the Endpoint list
Try to isolate the Endpoint
See that it remains in an "Isolating" state

Expected behavior:
The Endpoint should exit the "Isolating" state and be able to write and create the correct indices. This means that the Endpoint should be able to create and write to the .logs-endpoint.action.responses-* index.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Relavant logs from Endpoint:

{"@timestamp":"2021-10-28T18:14:10.2533692Z","agent":{"id":"08e6c080-c1ac-4a58-8370-5b42e852aae7","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":537,"name":"Client.cpp"}}},"message":"Client.cpp:537 Failed to create in index (.logs-endpoint.action.responses-default) json ({\"@timestamp\":\"2021-10-28T18:14:10.0Z\",\"EndpointActions\":{\"action_id\":\"\",\"completed_at\":\"2021-10-28T18:14:10.0Z\",\"data\":{\"command\":\"isolate\",\"comment\":\"Action completed successfully\"},\"input_type\":\"endpoint\",\"started_at\":\"2021-10-28T18:14:05.0Z\"},\"agent\":{\"id\":\"08e6c080-c1ac-4a58-8370-5b42e852aae7\"},\"data_stream\":{\"dataset\":\"endpoint.action.responses\",\"namespace\":\"default\",\"type\":\".logs\"}}) reason (action [indices:admin/auto_create] is unauthorized for API key id [1o8ayHwBtLbMqpHcSTyf] of user [elastic/fleet-server] on indices [.logs-endpoint.action.responses-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]) status (403)","process":{"pid":5652,"thread":{"id":3340}}}

Any additional context:

I noticed that the data streams are mentioned in the Agent policy that contains the Endpoint package policy.

Here is a snippet from the Agent Policy:

...
    Protect:
      indices:
        - names:
            - .logs-endpoint.action.responses-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - .logs-endpoint.actions-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.alerts-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - .logs-endpoint.diagnostic.collection-default
          privileges:
            - auto_configure
            - create_doc
...

I also noticed that these privileges differ from what's in the Endpoint log message seen above.

A snippet of the permissions mentioned in the log message.
auto_configure,create_index,manage,all

The permissions look like they're set in this section of the code: https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/package_policies_to_agent_permissions.ts#L62

Do we need to set specific permissions for this index in the Endpoint package or otherwise?

Screenshot:

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-10-28T21:16:26Z

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

ashokaditya · 2021-10-29T15:30:13Z

I can reproduce this as well. I get a similar notice when looking into logs.

Formatted log message:

Client.cpp: 537 Failed to create in index (.logs-endpoint.action.responses-default) json ({
        "@timestamp": "2021-10-29T12:53:01.0Z",
        "EndpointActions": {
            "action_id": "acd97fa0-d06c-4920-84e2-e604b45a1891",
            "completed_at": "2021-10-29T12:53:01.0Z",
            "data": {
                "command": "isolate",
                "comment": "Action completed with errors"
            },
            "input_type": "endpoint",
            "started_at": "2021-10-29T12:52:59.0Z"
        },
        "agent": {
            "id": "1f3be6ea-c680-4231-ba15-c861248c9184"
        },
        "data_stream": {
            "dataset": "endpoint.action.responses",
            "namespace": "default",
            "type": ".logs"
        },
        "error": {
            "message": "Failed to apply action; check Endpoint logs for details"
        }
    }) reason (
        action [indices:admin/auto_create] is unauthorized for API key id [segGzHwBhUBC71UIhf2y] of user [elastic/fleet-server] on indices [.logs-endpoint.action.responses-default], 
        this action is granted by the index privileges [auto_configure,create_index,manage,all]) 
    status (403)

ruflin · 2021-11-01T14:25:59Z

Let me start with some definitions to make sure we talk about the same thing:

Cloud setup: Managed Elastic Agent on your machine -> Hosted Elastic Agent -> Elasticsearch <- Fleet
Standalone setup = Manachged on your machine: Managed Elastic Agent on your Machine -> Elastic Agent with fleet-server on a machine -> Elasticsearch <- Fleet

In general, there should be no difference between the two. The permissions are policy specific. Is it correct that the data stream you get an error writing is listed in the policy under the permissions but you still get an error? This makes me wonder if the API permissions were refreshed? Were both installation fresh setup of the cluster and Elastic Agent or was one of these migrated from previous versions?

In the above the conversation is about an index? I assume we are talking about data streams with backing indices?

ashokaditya · 2021-11-01T16:29:58Z

Thanks, @ruflin for listing out the setup accurately. Indeed the indices are listed under policy permissions and have identical privileges (auto_configure and create_doc) in standalone and cloud setups.
Both setups were fresh installs of 7.16.

cloud agent policy:

id: 31b5f290-38b0-11ec-b319-8989c71efd64
revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://b36be9ee308d44a988d80412efb26dce.us-central1.gcp.foundit.no:443'
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    7.16-test-integration:
      indices:
        - names:
            - .logs-endpoint.action.responses-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - .logs-endpoint.actions-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.alerts-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - .logs-endpoint.diagnostic.collection-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.file-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.library-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-endpoint.metadata-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-endpoint.metrics-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.network-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-endpoint.policy-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.process-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.registry-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.security-default
          privileges:
            - auto_configure
            - create_doc
agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: 277c5dc0-1569-4c4a-afdb-cbfa4723cd7e
    name: 7.16-test-integration
    revision: 1
    type: endpoint
    use_output: default
    meta:
      package:
        name: endpoint
        version: 1.2.1
    data_stream:
      namespace: default
    artifact_manifest:
      manifest_version: 1.0.0
      schema_version: v1
      artifacts:
        endpoint-exceptionlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-exceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-exceptionlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-exceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-exceptionlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-exceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-trustlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-trustlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-trustlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-eventfilterlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-eventfilterlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-eventfilterlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-eventfilterlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-eventfilterlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-eventfilterlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-hostisolationexceptionlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-hostisolationexceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-hostisolationexceptionlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-hostisolationexceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-hostisolationexceptionlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-hostisolationexceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
    policy:
      windows:
        events:
          dll_and_driver_load: true
          dns: true
          file: true
          network: true
          process: true
          registry: true
          security: true
        malware:
          mode: prevent
        ransomware:
          mode: prevent
          supported: true
        memory_protection:
          mode: prevent
          supported: true
        behavior_protection:
          mode: prevent
          supported: true
        popup:
          malware:
            enabled: true
            message: ''
          ransomware:
            enabled: true
            message: ''
          memory_protection:
            enabled: true
            message: ''
          behavior_protection:
            enabled: true
            message: ''
        logging:
          file: info
        antivirus_registration:
          enabled: false
      mac:
        events:
          process: true
          file: true
          network: true
        malware:
          mode: prevent
        behavior_protection:
          mode: prevent
          supported: true
        memory_protection:
          mode: prevent
          supported: true
        popup:
          malware:
            enabled: true
            message: ''
          behavior_protection:
            enabled: true
            message: ''
          memory_protection:
            enabled: true
            message: ''
        logging:
          file: info
      linux:
        events:
          process: true
          file: true
          network: true
        malware:
          mode: prevent
        behavior_protection:
          mode: prevent
          supported: true
        memory_protection:
          mode: prevent
          supported: true
        popup:
          malware:
            enabled: true
            message: ''
          behavior_protection:
            enabled: true
            message: ''
          memory_protection:
            enabled: true
            message: ''
        logging:
          file: info
fleet:
  hosts:
    - >-
      https://22090f8741f7411680822760938113f8.fleet.us-central1.gcp.foundit.no:443

standalone agent policy

id: 1b521a70-3b2f-11ec-9cca-1f2753288199
revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    7.16-test-integration:
      indices:
        - names:
            - .logs-endpoint.action.responses-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - .logs-endpoint.actions-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.alerts-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - .logs-endpoint.diagnostic.collection-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.file-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.library-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-endpoint.metadata-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-endpoint.metrics-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.network-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-endpoint.policy-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.process-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.registry-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-endpoint.events.security-default
          privileges:
            - auto_configure
            - create_doc
agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: 1da68b16-f9e4-443e-8569-2bb352d48fad
    name: 7.16-test-integration
    revision: 1
    type: endpoint
    use_output: default
    meta:
      package:
        name: endpoint
        version: 1.2.1
    data_stream:
      namespace: default
    artifact_manifest:
      manifest_version: 1.0.0
      schema_version: v1
      artifacts:
        endpoint-exceptionlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-exceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-exceptionlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-exceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-exceptionlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-exceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-trustlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-trustlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-trustlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-eventfilterlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-eventfilterlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-eventfilterlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-eventfilterlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-eventfilterlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-eventfilterlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-hostisolationexceptionlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-hostisolationexceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-hostisolationexceptionlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-hostisolationexceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-hostisolationexceptionlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/fleet/artifacts/endpoint-hostisolationexceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
    policy:
      windows:
        events:
          dll_and_driver_load: true
          dns: true
          file: true
          network: true
          process: true
          registry: true
          security: true
        malware:
          mode: prevent
        ransomware:
          mode: prevent
          supported: true
        memory_protection:
          mode: prevent
          supported: true
        behavior_protection:
          mode: prevent
          supported: true
        popup:
          malware:
            enabled: true
            message: ''
          ransomware:
            enabled: true
            message: ''
          memory_protection:
            enabled: true
            message: ''
          behavior_protection:
            enabled: true
            message: ''
        logging:
          file: info
        antivirus_registration:
          enabled: false
      mac:
        events:
          process: true
          file: true
          network: true
        malware:
          mode: prevent
        behavior_protection:
          mode: prevent
          supported: true
        memory_protection:
          mode: prevent
          supported: true
        popup:
          malware:
            enabled: true
            message: ''
          behavior_protection:
            enabled: true
            message: ''
          memory_protection:
            enabled: true
            message: ''
        logging:
          file: info
      linux:
        events:
          process: true
          file: true
          network: true
        malware:
          mode: prevent
        behavior_protection:
          mode: prevent
          supported: true
        memory_protection:
          mode: prevent
          supported: true
        popup:
          malware:
            enabled: true
            message: ''
          behavior_protection:
            enabled: true
            message: ''
          memory_protection:
            enabled: true
            message: ''
        logging:
          file: info

You assume right about data streams with backing indices.
.logs-endpoint.actions DS allows for .logs-endpoint.actions-* index patterns and thus we write to .logs-endpoint.actions-default. Similarly with .logs-endpoint.action.responses DS and allowed index pattern.

How do we verify that API permissions were not refreshed?

Also here's another ticket that involves the same indices #116396 but about upgrading from ES v1.1.1 to ES v1.2.0

kevinlog · 2021-11-02T17:01:42Z

@ruflin @ashokaditya thanks for the context.

Some observations:

Remove permission for `.logs-endpoint.action.responses-*` on my local setup

In my local setup, I changed the code slightly so that .logs-endpoint.action.responses-* did not show up in the Policy permissions list. With this, I was able to have the permission check fail on my local instance, which I never was before. I did this to verify that the permissions were being respected locally and that there wasn't anything bypassing it. Again, locally, this is working as I would expect. If the index is in the policy, then the permissions should be granted. If I remove it, the permissions check should fail.

Compare Fleet Server policies in a local vs cloud instance

I took a look at the permissions granted in my local instance vs the cloud instance.

I noticed that the Fleet Server on cloud has a policy that looks different than the one that is used locally. I do not know if this makes a difference, but I'm looking for any difference I can find.

Cloud Fleet Server output permissions:

id: policy-elastic-agent-on-cloud
revision: 2
outputs:
  es-containerhost:
    type: elasticsearch
    hosts:
      - 'http://1b628684a38f48968313f0cf00b4781f.containerhost:9244'
output_permissions:
  es-containerhost:
    _elastic_agent_monitoring:
      indices: []
    _elastic_agent_checks:
      cluster:
        - monitor
 ...

Local Fleet Server output permissions:

id: 8f0bae60-3bf0-11ec-a547-979b41dc625b
revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://10.0.2.2:9200'
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
...

Look at another hidden index that we shipped some time back, `.logs-endpoint.diagnostic.collection-*`

I also looked at .logs-endpoint.diagnostic.collection-* which the Endpoint also streams to. We shipped this in December of last year.

First I verified that the permissions for this index are still working on cloud, and they are working. I investigated what the differences could be and I opened this PR: #117162. I do not know if this fixes the problem, but it seemed odd that we included our older index and not the new one.

I do find it interesting that this index works without issue. This index was initially added when we still used the special fleet user role for all of our permission checks. Was there anything special done when permissions were moved to be driven from the Policy? Are we missing anything additional that we needed to do to ensure we have permission to access this new index?

.logs-endpoint.diagnostic.collection-* and .logs-endpoint.action.responses-* are very similar in how they're set up and how they're used. They're both hidden indices and the Endpoint streams data directly to them.

Is there anyone else I should pull in on this?

nchaulet · 2021-11-02T17:51:18Z

@kevinlog I took a look at your PR on Kibana but I do not think it will solve the problem the default permission. Actually I think Fleet server is not able to give the permission to the agent as it does not have any permissions related to logs-endpoint.action.responses see https://github.com/elastic/elasticsearch/blob/26c86871fc091900952e88e252c36fbfedf8d5fa/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/service/ElasticServiceAccounts.java#L32

kevinlog · 2021-11-02T17:59:53Z

@nchaulet thanks for taking a look! I wasn't aware that there were additional permissions required in ES. Any idea why running locally using yarn es snapshot would work vs using a cloud instance?

nchaulet · 2021-11-02T18:02:20Z

@kevinlog Are you running your fleet server locally with elastic user or any superuser? in this case locally the fleet server user will have all the permissions needed

…nt response index (#80238) This PR adds the proper permissions for fleet server to create and write documents to the .logs-endpoint.action.responses-* index. The Security Endpoint, run by the Agent, streams action responses to this index which is used by the Security app to determine if actions are complete, etc. This was initially missed during testing because of using locally running fleet servers that were given superuser permissions, hence bypassing the fleet server user. This PR adds the index to fleet server so that the Endpoint gets the key that it needs to write to the index properly. For more information, see this ticket: elastic/kibana#116715

…nt response index (elastic#80238) This PR adds the proper permissions for fleet server to create and write documents to the .logs-endpoint.action.responses-* index. The Security Endpoint, run by the Agent, streams action responses to this index which is used by the Security app to determine if actions are complete, etc. This was initially missed during testing because of using locally running fleet servers that were given superuser permissions, hence bypassing the fleet server user. This PR adds the index to fleet server so that the Endpoint gets the key that it needs to write to the index properly. For more information, see this ticket: elastic/kibana#116715

…nt response index (#80238) (#80241) This PR adds the proper permissions for fleet server to create and write documents to the .logs-endpoint.action.responses-* index. The Security Endpoint, run by the Agent, streams action responses to this index which is used by the Security app to determine if actions are complete, etc. This was initially missed during testing because of using locally running fleet servers that were given superuser permissions, hence bypassing the fleet server user. This PR adds the index to fleet server so that the Endpoint gets the key that it needs to write to the index properly. For more information, see this ticket: elastic/kibana#116715 Co-authored-by: Kevin Logan <[email protected]>

…nt response index (#80238) (#80242) This PR adds the proper permissions for fleet server to create and write documents to the .logs-endpoint.action.responses-* index. The Security Endpoint, run by the Agent, streams action responses to this index which is used by the Security app to determine if actions are complete, etc. This was initially missed during testing because of using locally running fleet servers that were given superuser permissions, hence bypassing the fleet server user. This PR adds the index to fleet server so that the Endpoint gets the key that it needs to write to the index properly. For more information, see this ticket: elastic/kibana#116715 Co-authored-by: Kevin Logan <[email protected]>

ruflin · 2021-11-03T10:27:46Z

As @nchaulet pointed out, it is relevant if the Elastic Agent with fleet-server is run with superuser permissions or the fleet-server service token. That could explain the difference. I'm wondering if we should expand the permissions for the fleet-server user account to also include .logs-* indices, same for metrics etc. to make hidden indices support without having to add each one.

kevinlog · 2021-11-03T11:53:53Z

@ruflin @nchaulet - that is what I believe the difference is. As you can see from the PRs above, I've added the the specific index to start. We could discuss making the permissions include .logs-* as you mentioned and it would avoid having to add each one by one. Maybe make a change for 8.0 ?

ruflin · 2021-11-03T12:00:05Z

+1 on the change from my side. Could you open an issue with Elasticsearch about this to discuss it further? @joshdover This might also impact kibana_system permissions, but not sure.

@jlind23 For Elastic Agent, we should deprecate / discourage the usage of the direct Elasticsearch permissions. That is the second issue we have seen related to it in the recent days. In theory it could also be removed but that might break some older deployments.

joshdover · 2021-11-04T10:09:08Z

I'm wondering if we should expand the permissions for the fleet-server user account to also include .logs-* indices, same for metrics etc. to make hidden indices support without having to add each one.
+1 on the change from my side. Could you open an issue with Elasticsearch about this to discuss it further?

Sure I can open the issue. Generally, the security team has been more comfortable granting broad access to dot-prefixed indices since these are well communicated as being managed by the system. For instance, kibana_system has access to .kibana* and .fleet*.

This might also impact kibana_system permissions, but not sure.

It does, we had to do a similar thing here for kibana_system: elastic/elasticsearch#80140. I'll mention both problems in the issue.

jlind23 · 2021-11-04T10:10:52Z

@ruflin shall we create an issue to add a warning message accordingly?

ruflin · 2021-11-04T10:18:31Z

@jlind23 Lets open an issue to discuss the details further. I see more then one option.

…ng actions logic (#117219) * use a feature flag to use the new pending actions logic refs /issues/116715 * switch off pending actions for endpoints when feature flag is disabled review suggestions * update/add tests to use FF * correctly override the FF `parseExperimentalConfigValue` method sets feature flag key values to `true` if passed as arg Co-authored-by: Kibana Machine <[email protected]>

…ng actions logic (elastic#117219) * use a feature flag to use the new pending actions logic refs elastic/issues/116715 * switch off pending actions for endpoints when feature flag is disabled review suggestions * update/add tests to use FF * correctly override the FF `parseExperimentalConfigValue` method sets feature flag key values to `true` if passed as arg Co-authored-by: Kibana Machine <[email protected]>

…ng actions logic (#117219) (#117497) * use a feature flag to use the new pending actions logic refs /issues/116715 * switch off pending actions for endpoints when feature flag is disabled review suggestions * update/add tests to use FF * correctly override the FF `parseExperimentalConfigValue` method sets feature flag key values to `true` if passed as arg Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Ashokaditya <[email protected]>

…ng actions logic (#117219) (#117498) * use a feature flag to use the new pending actions logic refs /issues/116715 * switch off pending actions for endpoints when feature flag is disabled review suggestions * update/add tests to use FF * correctly override the FF `parseExperimentalConfigValue` method sets feature flag key values to `true` if passed as arg Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Ashokaditya <[email protected]>

kevinlog · 2021-11-05T21:10:13Z

Confirmed fixed in the latest BC for 7.16

joshdover · 2021-11-15T12:50:22Z

@ruflin @jlind23 did an issue get created?

ruflin · 2021-11-15T13:11:01Z

@joshdover Not that I'm aware of. Can you take care of it?

jlind23 · 2021-11-15T16:26:57Z

@ruflin @joshdover I'll do it now - elastic/beats#28973

kevinlog · 2021-11-15T16:44:32Z

@jlind23 @joshdover @ruflin - thanks for the followup, I'm also tracking the above ticket on our board.

kevinlog added bug Fixes for quality problems that affect the customer experience impact:critical This issue should be addressed immediately due to a critical level of impact on the product. Team:Defend Workflows “EDR Workflows” sub-team of Security Solution OLM Sprint labels Oct 28, 2021

kevinlog assigned ashokaditya Oct 28, 2021

kevinlog mentioned this issue Nov 2, 2021

[Security Solution] Add endpoint action response permission to defaults #117162

Closed

1 task

This was referenced Nov 2, 2021

[Security Solution] Add proper permissions to fleet server for Endpoint response index elastic/elasticsearch#80231

Merged

[Security Solution] Add proper permissions to fleet server for Endpoint response index (#80231) elastic/elasticsearch#80238

Merged

ywangd mentioned this issue Nov 3, 2021

[Security Solution] Add proper permissions to fleet server for Endpoint response index (#80238) elastic/elasticsearch#80241

Merged

ywangd mentioned this issue Nov 3, 2021

[Security Solution] Add proper permissions to fleet server for Endpoint response index (#80238) elastic/elasticsearch#80242

Merged

joshdover mentioned this issue Nov 4, 2021

[Discuss] Add wildcard privileges for dot-prefixed Fleet indices to kibana_system and elastic/fleet-server elastic/elasticsearch#80332

Open

kevinlog closed this as completed Nov 5, 2021

jlind23 mentioned this issue Nov 15, 2021

Usage of Elasticsearch permissions elastic/beats#28973

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Endpoint lacks permissions on cloud to write to `.logs-endpoint.action.responses-*` on cloud instances #116715

[Security Solution] Endpoint lacks permissions on cloud to write to `.logs-endpoint.action.responses-*` on cloud instances #116715

kevinlog commented Oct 28, 2021 •

edited

Loading

elasticmachine commented Oct 28, 2021

ashokaditya commented Oct 29, 2021

ruflin commented Nov 1, 2021

ashokaditya commented Nov 1, 2021 •

edited

Loading

kevinlog commented Nov 2, 2021

nchaulet commented Nov 2, 2021

kevinlog commented Nov 2, 2021

nchaulet commented Nov 2, 2021

ruflin commented Nov 3, 2021

kevinlog commented Nov 3, 2021

ruflin commented Nov 3, 2021

joshdover commented Nov 4, 2021

jlind23 commented Nov 4, 2021

ruflin commented Nov 4, 2021

kevinlog commented Nov 5, 2021

joshdover commented Nov 15, 2021

ruflin commented Nov 15, 2021

jlind23 commented Nov 15, 2021

kevinlog commented Nov 15, 2021

[Security Solution] Endpoint lacks permissions on cloud to write to .logs-endpoint.action.responses-* on cloud instances #116715

[Security Solution] Endpoint lacks permissions on cloud to write to .logs-endpoint.action.responses-* on cloud instances #116715

Comments

kevinlog commented Oct 28, 2021 • edited Loading

elasticmachine commented Oct 28, 2021

ashokaditya commented Oct 29, 2021

ruflin commented Nov 1, 2021

ashokaditya commented Nov 1, 2021 • edited Loading

kevinlog commented Nov 2, 2021

Remove permission for .logs-endpoint.action.responses-* on my local setup

Compare Fleet Server policies in a local vs cloud instance

Look at another hidden index that we shipped some time back, .logs-endpoint.diagnostic.collection-*

nchaulet commented Nov 2, 2021

kevinlog commented Nov 2, 2021

nchaulet commented Nov 2, 2021

ruflin commented Nov 3, 2021

kevinlog commented Nov 3, 2021

ruflin commented Nov 3, 2021

joshdover commented Nov 4, 2021

jlind23 commented Nov 4, 2021

ruflin commented Nov 4, 2021

kevinlog commented Nov 5, 2021

joshdover commented Nov 15, 2021

ruflin commented Nov 15, 2021

jlind23 commented Nov 15, 2021

kevinlog commented Nov 15, 2021

[Security Solution] Endpoint lacks permissions on cloud to write to `.logs-endpoint.action.responses-*` on cloud instances #116715

[Security Solution] Endpoint lacks permissions on cloud to write to `.logs-endpoint.action.responses-*` on cloud instances #116715

kevinlog commented Oct 28, 2021 •

edited

Loading

ashokaditya commented Nov 1, 2021 •

edited

Loading

Remove permission for `.logs-endpoint.action.responses-*` on my local setup

Look at another hidden index that we shipped some time back, `.logs-endpoint.diagnostic.collection-*`