[Fleet] Secrets storage phase 1 - Remove Kibana's ability to read secrets #154715
Assignees
Labels
QA:Validated
Issue has been validated by QA
Team:Fleet
Team label for Observability Data Collection Fleet team
Comments
hop-dev
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
This was referenced Apr 11, 2023
|
PRs in order they need to be merged:
The kibana PRs re built on top of each other |
|
Remaining work:
|
This was referenced May 12, 2023
Closed
|
Created #157503 for making sure test code is removed |
hop-dev
added a commit
that referenced
this issue
May 12, 2023
#156036) ## Summary The first part of secrets phase 1. This is not a fully working implementation of secrets just yet, hence why it is behind a feature flag. This just implements creating secrets: - on package policy creation, if a package has fields with `secrets: true` set, then their values are stored in the .secrets system index, a reference to the secret is stored on the package policy e.g { id : 1234 isSecretReference : true } - The compiled policy (returned from the get full agent policy API, or stored in the .fleet-policies index) shows secret values in the format `$co.elastic.secret{12345}` and includes a top level secret_references array with all secret IDs in it, allowing fleet server to look them up in one swoop. - This works for pakacge level vars, input level vars and stream level vars Part of #154715 How to test: ``` # clone the elasticsearch repo gh pr checkout 95625 ./gradlew run # now get a service token curl -XPOST -u elastic:password http://localhost:9200/_security/service/elastic/kibana/credential/token/token1 # paste the service token into your kibana config under # elasticsearch.serviceAccountToken: "<your_token>" # once kibana has started, we now need to run our own package registry to get a package with secrets in # replace /Users/markhopkin/dev with the path to your kibana docker run -p 8080:8080 -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main # once kibana has started successfully once and installed the fleet_server package, add this to your kibana config xpack.fleet.registryUrl: http://localhost:8080 # you can now see the 'secrets' package and create a package policy # after creating the package policy, check the .fleet-secrets index, the .fleet-policies index or the get package policy API to see how the secrets have been stored ``` ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
|
Update May 17: Mark has started his leave, remaining work in Phase 1: Finish open prs (Jill):
Not assigned yet:
|
9 tasks
|
There is one task from #154731 that not yet implemented:
|
jasonrhodes
pushed a commit
that referenced
this issue
May 17, 2023
#156036) ## Summary The first part of secrets phase 1. This is not a fully working implementation of secrets just yet, hence why it is behind a feature flag. This just implements creating secrets: - on package policy creation, if a package has fields with `secrets: true` set, then their values are stored in the .secrets system index, a reference to the secret is stored on the package policy e.g { id : 1234 isSecretReference : true } - The compiled policy (returned from the get full agent policy API, or stored in the .fleet-policies index) shows secret values in the format `$co.elastic.secret{12345}` and includes a top level secret_references array with all secret IDs in it, allowing fleet server to look them up in one swoop. - This works for pakacge level vars, input level vars and stream level vars Part of #154715 How to test: ``` # clone the elasticsearch repo gh pr checkout 95625 ./gradlew run # now get a service token curl -XPOST -u elastic:password http://localhost:9200/_security/service/elastic/kibana/credential/token/token1 # paste the service token into your kibana config under # elasticsearch.serviceAccountToken: "<your_token>" # once kibana has started, we now need to run our own package registry to get a package with secrets in # replace /Users/markhopkin/dev with the path to your kibana docker run -p 8080:8080 -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main # once kibana has started successfully once and installed the fleet_server package, add this to your kibana config xpack.fleet.registryUrl: http://localhost:8080 # you can now see the 'secrets' package and create a package policy # after creating the package policy, check the .fleet-secrets index, the .fleet-policies index or the get package policy API to see how the secrets have been stored ``` ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
jillguyonnet
added a commit
that referenced
this issue
Jun 26, 2023
## Summary Closes #154731 Related to #154715 Adds a new component for viewing and editing secret values in the policy editor. Secrets can be written on policy create, when editing a policy, if a secret is filled then we give the user the option to replace the value only. Also when viewing the agent policy yaml, hide secret references and replace them with incremental env var names, and add a callout which tells the user they need to replace the vars. I have kept the UI un-opinionated about which type of field can be a secret. **Secret variable in package policy editor:** https://github.com/elastic/kibana/assets/3315046/b452668f-196e-4ace-a170-be45df135834 Replaced secrets and new callout in policy yaml viewer: https://github.com/elastic/kibana/assets/3315046/75d684ba-d76e-454b-96bc-9093ee2afbfc Testing - start a docker package registry ``` docker run -p 8080:8080 -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main ``` - point your kibana to the registry and use the `secrets-1.0.0` package to test - create a package policy with secrets (should be successful, should show 'normal' input components) - edit a package policy with secrets (should be successful, should show "replace value" panel for filled values) - view agent policy yaml with secrets (should not show secret references in yaml, should show ${SECRET_0} etc) ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Jill Guyonnet <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: jillguyonnet <[email protected]>
|
Created a bugfix issue with repro steps for when editing a secret var after enabling the feature flag for the first time: #160537 |
|
Our team tried integrating Outputs from testing with latest Agent policy snippet: Computed agent policy snippet: streams:
- config_version: "2"
cursor:
timestamp:
value: '[[ formatDate (now (parseDuration "-1h")) "RFC3339" ]]'
data_stream:
dataset: ti_otx.threat
type: logs
id: httpjson-ti_otx.threat-a520c8d4-1c8e-48c6-8525-d046ab1b8bda
interval: 5m
publisher_pipeline:
disable_host: true
request:
method: GET
timeout: 30s
tracer:
filename: ../../logs/httpjson/http-request-trace-*.ndjson
transforms:
- set:
target: header.Content-Type
value: application/json
- set:
target: header.X-OTX-API-KEY
value: $co.elastic.secret{h00vlosB2EJk2Ik-Hsof} ### <= Unexpected valueHTTP request/response tracing also shows the Incorrect API value in the request header: Full trace: |
|
@ebeahan Which version of Fleet Server are you using? Could you share the Fleet Server logs? |
|
|
@ebeahan could you also share the ouput of the followign dev tools request? Thanks |
|
Output of Expand{
"item": {
"id": "elastic-agent-managed-ep",
"outputs": {
"default": {
"type": "elasticsearch",
"hosts": [
"https://elasticsearch:9200"
],
"ssl.ca_trusted_fingerprint": "7299A8DEA66099EDDC427971EFC932F3007F73B93DCE70F3D3EDB8E6CD041C4E"
}
},
"inputs": [
{
"id": "logfile-system-default-system",
"revision": 1,
"name": "system-1",
"type": "logfile",
"data_stream": {
"namespace": "default"
},
"use_output": "default",
"package_policy_id": "default-system",
"streams": [
{
"id": "logfile-system.auth-default-system",
"data_stream": {
"type": "logs",
"dataset": "system.auth"
},
"ignore_older": "72h",
"paths": [
"/var/log/auth.log*",
"/var/log/secure*"
],
"exclude_files": [
".gz$"
],
"multiline": {
"pattern": """^\s""",
"match": "after"
},
"tags": [
"system-auth"
],
"processors": [
{
"add_locale": null
},
{
"rename": {
"fields": [
{
"from": "message",
"to": "event.original"
}
],
"ignore_missing": true,
"fail_on_error": false
}
},
{
"syslog": {
"field": "event.original",
"ignore_missing": true,
"ignore_failure": true
}
}
]
},
{
"id": "logfile-system.syslog-default-system",
"data_stream": {
"type": "logs",
"dataset": "system.syslog"
},
"paths": [
"/var/log/messages*",
"/var/log/syslog*",
"/var/log/system*"
],
"exclude_files": [
".gz$"
],
"multiline": {
"pattern": """^\s""",
"match": "after"
},
"processors": [
{
"add_locale": null
}
],
"ignore_older": "72h"
}
],
"meta": {
"package": {
"name": "system",
"version": "1.47.2"
}
}
},
{
"id": "winlog-system-default-system",
"revision": 1,
"name": "system-1",
"type": "winlog",
"data_stream": {
"namespace": "default"
},
"use_output": "default",
"package_policy_id": "default-system",
"streams": [
{
"id": "winlog-system.application-default-system",
"data_stream": {
"type": "logs",
"dataset": "system.application"
},
"name": "Application",
"condition": "${host.platform} == 'windows'",
"ignore_older": "72h"
},
{
"id": "winlog-system.security-default-system",
"data_stream": {
"type": "logs",
"dataset": "system.security"
},
"name": "Security",
"condition": "${host.platform} == 'windows'",
"ignore_older": "72h"
},
{
"id": "winlog-system.system-default-system",
"data_stream": {
"type": "logs",
"dataset": "system.system"
},
"name": "System",
"condition": "${host.platform} == 'windows'",
"ignore_older": "72h"
}
],
"meta": {
"package": {
"name": "system",
"version": "1.47.2"
}
}
},
{
"id": "system/metrics-system-default-system",
"revision": 1,
"name": "system-1",
"type": "system/metrics",
"data_stream": {
"namespace": "default"
},
"use_output": "default",
"package_policy_id": "default-system",
"streams": [
{
"id": "system/metrics-system.cpu-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.cpu"
},
"metricsets": [
"cpu"
],
"cpu.metrics": [
"percentages",
"normalized_percentages"
],
"period": "10s"
},
{
"id": "system/metrics-system.diskio-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.diskio"
},
"metricsets": [
"diskio"
],
"diskio.include_devices": null,
"period": "10s"
},
{
"id": "system/metrics-system.filesystem-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.filesystem"
},
"metricsets": [
"filesystem"
],
"period": "1m",
"processors": [
{
"drop_event.when.regexp": {
"system.filesystem.mount_point": "^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)"
}
}
]
},
{
"id": "system/metrics-system.fsstat-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.fsstat"
},
"metricsets": [
"fsstat"
],
"period": "1m",
"processors": [
{
"drop_event.when.regexp": {
"system.fsstat.mount_point": "^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)"
}
}
]
},
{
"id": "system/metrics-system.load-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.load"
},
"metricsets": [
"load"
],
"condition": "${host.platform} != 'windows'",
"period": "10s"
},
{
"id": "system/metrics-system.memory-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.memory"
},
"metricsets": [
"memory"
],
"period": "10s"
},
{
"id": "system/metrics-system.network-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.network"
},
"metricsets": [
"network"
],
"period": "10s",
"network.interfaces": null
},
{
"id": "system/metrics-system.process-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.process"
},
"metricsets": [
"process"
],
"period": "10s",
"process.include_top_n.by_cpu": 5,
"process.include_top_n.by_memory": 5,
"process.cmdline.cache.enabled": true,
"process.cgroups.enabled": false,
"process.include_cpu_ticks": false,
"processes": [
".*"
]
},
{
"id": "system/metrics-system.process.summary-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.process.summary"
},
"metricsets": [
"process_summary"
],
"period": "10s"
},
{
"id": "system/metrics-system.socket_summary-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.socket_summary"
},
"metricsets": [
"socket_summary"
],
"period": "10s"
},
{
"id": "system/metrics-system.uptime-default-system",
"data_stream": {
"type": "metrics",
"dataset": "system.uptime"
},
"metricsets": [
"uptime"
],
"period": "10s"
}
],
"meta": {
"package": {
"name": "system",
"version": "1.47.2"
}
}
},
{
"id": "httpjson-ti_otx-35ad24f6-00d1-4d06-8da0-9bb5bcceb994",
"revision": 1,
"name": "ti_otx-1",
"type": "httpjson",
"data_stream": {
"namespace": "default"
},
"use_output": "default",
"package_policy_id": "35ad24f6-00d1-4d06-8da0-9bb5bcceb994",
"streams": [
{
"id": "httpjson-ti_otx.threat-35ad24f6-00d1-4d06-8da0-9bb5bcceb994",
"data_stream": {
"type": "logs",
"dataset": "ti_otx.threat"
},
"config_version": "2",
"interval": "5m",
"request.method": "GET",
"request.tracer.filename": "../../logs/httpjson/http-request-trace-*.ndjson",
"request.url": "https://otx.alienvault.com/api/v1/indicators/export",
"request.timeout": "30s",
"request.transforms": [
{
"set": {
"target": "header.Content-Type",
"value": "application/json"
}
},
{
"set": {
"target": "header.X-OTX-API-KEY",
"value": "$co.elastic.secret{vk7apYsB2EJk2Ik-yREG}"
}
},
{
"set": {
"target": "url.params.modified_since",
"value": "[[.cursor.timestamp]]",
"default": """[[ formatDate (now (parseDuration "-400h")) "RFC3339" ]]"""
}
}
],
"response.split": {
"target": "body.results",
"keep_parent": true
},
"response.pagination": [
{
"set": {
"target": "url.value",
"value": "[[ .last_response.body.next ]]"
}
}
],
"cursor": {
"timestamp": {
"value": """[[ formatDate (now (parseDuration "-1h")) "RFC3339" ]]"""
}
},
"tags": [
"preserve_original_event",
"forwarded",
"otx-threat"
],
"publisher_pipeline.disable_host": true
}
],
"meta": {
"package": {
"name": "ti_otx",
"version": "1.20.1"
}
}
}
],
"secret_references": [
{
"id": "vk7apYsB2EJk2Ik-yREG"
}
],
"revision": 8,
"agent": {
"download": {
"sourceURI": "https://artifacts.elastic.co/downloads/"
},
"monitoring": {
"namespace": "default",
"use_output": "default",
"enabled": true,
"logs": true,
"metrics": true
},
"features": {},
"protection": {
"enabled": false,
"uninstall_token_hash": "duA+rtJG8vRKMg8HmXDGU8oTRwEHzn4PTl0EMUYClVI=",
"signing_key": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEc8HJYjJyoPKHWUHSDUwMo9fRNFEZV9nKjfrWisA93gWuR7qmPRPmfwFXeRKsQv3W7Y8warz87r2hPJqYBr049A=="
}
},
"signed": {
"data": "eyJpZCI6ImVsYXN0aWMtYWdlbnQtbWFuYWdlZC1lcCIsImFnZW50Ijp7ImZlYXR1cmVzIjp7fSwicHJvdGVjdGlvbiI6eyJlbmFibGVkIjpmYWxzZSwidW5pbnN0YWxsX3Rva2VuX2hhc2giOiJkdUErcnRKRzh2UktNZzhIbVhER1U4b1RSd0VIem40UFRsMEVNVVlDbFZJPSIsInNpZ25pbmdfa2V5IjoiTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFYzhISllqSnlvUEtIV1VIU0RVd01vOWZSTkZFWlY5bktqZnJXaXNBOTNnV3VSN3FtUFJQbWZ3RlhlUktzUXYzVzdZOHdhcno4N3IyaFBKcVlCcjA0OUE9PSJ9fSwiaW5wdXRzIjpbeyJpZCI6ImxvZ2ZpbGUtc3lzdGVtLWRlZmF1bHQtc3lzdGVtIiwibmFtZSI6InN5c3RlbS0xIiwicmV2aXNpb24iOjEsInR5cGUiOiJsb2dmaWxlIn0seyJpZCI6IndpbmxvZy1zeXN0ZW0tZGVmYXVsdC1zeXN0ZW0iLCJuYW1lIjoic3lzdGVtLTEiLCJyZXZpc2lvbiI6MSwidHlwZSI6IndpbmxvZyJ9LHsiaWQiOiJzeXN0ZW0vbWV0cmljcy1zeXN0ZW0tZGVmYXVsdC1zeXN0ZW0iLCJuYW1lIjoic3lzdGVtLTEiLCJyZXZpc2lvbiI6MSwidHlwZSI6InN5c3RlbS9tZXRyaWNzIn0seyJpZCI6Imh0dHBqc29uLXRpX290eC0zNWFkMjRmNi0wMGQxLTRkMDYtOGRhMC05YmI1YmNjZWI5OTQiLCJuYW1lIjoidGlfb3R4LTEiLCJyZXZpc2lvbiI6MSwidHlwZSI6Imh0dHBqc29uIn1dfQ==",
"signature": "MEUCIQC+tuu9/WTqnLz+hXuPu/l+oqNp7TKHLaBO+DTMLb8YpQIgRNdif0Two4Bsr9Qn9g3y1FZ3IwF2YZqVMMUaJQC2UE0="
},
"output_permissions": {
"default": {
"_elastic_agent_monitoring": {
"indices": [
{
"names": [
"logs-elastic_agent.apm_server-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.apm_server-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.auditbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.auditbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.cloud_defend-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.cloudbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.cloudbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.elastic_agent-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.endpoint_security-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.endpoint_security-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.filebeat_input-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.filebeat_input-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.filebeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.filebeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.fleet_server-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.fleet_server-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.heartbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.heartbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.metricbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.metricbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.osquerybeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.osquerybeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.packetbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-elastic_agent.packetbeat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.pf_elastic_collector-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.pf_elastic_symbolizer-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-elastic_agent.pf_host_agent-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
}
]
},
"_elastic_agent_checks": {
"cluster": [
"monitor"
]
},
"default-system": {
"indices": [
{
"names": [
"logs-system.auth-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-system.syslog-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-system.application-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-system.security-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"logs-system.system-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.cpu-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.diskio-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.filesystem-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.fsstat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.load-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.memory-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.network-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.process-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.process.summary-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.socket_summary-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
},
{
"names": [
"metrics-system.uptime-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
}
]
},
"35ad24f6-00d1-4d06-8da0-9bb5bcceb994": {
"indices": [
{
"names": [
"logs-ti_otx.threat-default"
],
"privileges": [
"auto_configure",
"create_doc"
]
}
]
}
}
},
"fleet": {
"hosts": [
"https://fleet-server:8220"
]
}
}
} |
|
OK it's looking like there is an issue in the secret replacement code on fleet server, maybe to do with the fact the secret is in an array @juliaElastic? @ebeahan is the |
|
Relevant chunk of the policy above As far as I understand, this is the expected value for Fleet to write to the policy object. Fleet Server will then decode the secret and fetch the underlying secrets value from the Some other relevant links from my digging around
On first glance, the Fleet Server code looks like it just does a fairly basic string replacement on any matches for the |
Correct, I added |
Yes, I think you are right, the replace code in fleet-server doesn't replace secret values if the vars are array, raised this issue to fix it: elastic/fleet-server#3083 |
|
Since this is a blocker to adoption of the secrets phase 1 work, I think we should prioritize this highly in sprint 21. That sprint is very full right now and needs to be trimmed down, but I think this should be something we look at as soon as we can. |
@kpollich happy to look at items that need to be removed/re-prio. Agree that this is important. Just to confirm, it only affects arrays? |
|
Turns out this was a pretty quick fix and @michel-laterman had some bandwidth in the current sprint. Here's his PR: elastic/fleet-server#3086. |
2 tasks
This was referenced Nov 28, 2023
5 tasks
6 tasks
harshitgupta-qasource
added
QA:Validated
|
We continue seeing issues testing in |
|
For the record, issue fixed here elastic/fleet-server#3206 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Integration policies and agent outputs often contain user's secrets, for example API keys or passwords. Currently, any user of fleet can see these values once they have been written. The goal of phase 1 of secrets storage is to remove the ability of Kibana and Fleet users to view secrets once they have been written.
A new system index will be created to store the secrets in, Kibana will only have write access to this index. Fleet server will be able to read the secrets and insert them into the policy before sending to the agent.
Implementation issues:
.fleet-secretsindex #154731.fleet-secretsindex to.secrets-fleetto prevent privileges conflict elasticsearch#96172.fleet-secretsindex internal elasticsearch#97728.fleet-secretsindex and replace secret references with secret values fleet-server#2485.fleet-policiesto secrets index when a field moves to being secret #162045End to end test cases (candidates for automation):
The text was updated successfully, but these errors were encountered: