Ability to list all secret path names from the SecretProvider

[ajcasagrande]

🚀 Feature Request

Relevant Package [REQUIRED]

go-mod-bootstrap SecretProvider

Description [REQUIRED]

Not sure if this would introduce a secerity concern (though it only removes a small amount of security through obscurity which isn't really security anyways).

It would be nice to be able to list the path names for all secrets in the SecretStore. The target use case I envision would be the ability to do something like this:

• User can assign the secret credentials for an Onvif camera by being given a list of all of the existing secret names/paths and selecting which one they want to assign to it.

This is currently not possible because you cannot get a list of all of the secret names/paths, you need to know the name/path beforehand.

Describe the solution you'd like

Something like this (name is just a suggestion):

type SecretProvider interface {
	...

	// ListAllSecretPaths returns the top level name/path for all secrets contained in the SecretStore.
	ListAllSecretPaths() ([]string, error)
}

Describe alternatives you've considered

Not really any alternatives or workarounds.

[lenny-goodell]

@bnevis-i , @jim-wang-intel , I know we could do this with InsecureSecrets since it is just reading the config and collecting all the Path values. Is there a way with Vault to list the current paths for a services secrets Store?

[bnevis-i]

There is no server-supported way to do this, in fact, there has been a PR against Vault that offered just this functionality that was closed without merging because it could possibly bypass the ACL system. There were also concerns that the various storage backends are not optimized for this kind of query. I could be very inefficient or impossible to do this for a service-based backend, but easy for a database backend.

That leaves client implementations. It would be impossible to start such a search at the root because the ACL system would forbid it. That means one would have to start at a known path and then iterate recursively on the client side. We would have to write all the code by hand. Because of the ACL system, there is no security concern doing this strictly from the client side.

So, we could implement EnumerateSecretsAtPath(basePath string) ([]string, error) to either one or N levels. N levels would of course take more code.

[ajcasagrande]

@bnevis-i Really what I would be after is to get the top level secrets for the current running EdgeX service. I am not familiar enough with EdgeX's Vault usage, but I assume each service is segregated out, maybe in their own "basePath"? In that case the EnumerateSecretsAtPath to 1 level is along the lines of what I was thinking.

Essentially if there are the following secrets FooCreds, Foo2Creds, and Foo3Creds, each with subkeys of username and password, added by doing a POST to /api/v2/secret with a payload like:

{
    "apiVersion":"v2",
    "path": "FooCreds",
    "secretData":[
        {
            "key":"username",
            "value":"..."
        },
        {
            "key":"password",
            "value":"..."
        }
    ]
}

I would just want the list of those 3 names ["FooCreds". "Foo2Creds", "Foo3Creds"]

[bnevis-i]

Yes, the root level (well, base of the service) is doable. I though GetSecrets without a secret list already did this? Or was it a list of keys that it takes?

[ajcasagrande]

Yes, the root level (well, base of the service) is doable. I though GetSecrets without a secret list already did this? Or was it a list of keys that it takes?

@bnevis-i yeah it takes a path and then list of keys, where the list of keys being empty means retrieve all sub-keys, unless there is a Vault specific implementation not exposed on the interface level.

	// GetSecret retrieves secrets from the service's SecretStore at the specified path.
	GetSecret(path string, keys ...string) (map[string]string, error)

Im running in non-secure mode right now, but that implementation at least does not let me do:

sdk.RunningService().SecretProvider.GetSecret("") // returns "Error, path () doesn't exist in secret store"

sdk.RunningService().SecretProvider.GetSecret("/") // returns "Error, path (/) doesn't exist in secret store"

[bnevis-i]

I think

ListSecretsAtPath(path string) ([]string, error)

Should do what you need, where

• ListSecretsAtPath("") will return the service's secrets at its HOME path in the secret store
• ListSecretsAtPath("subpath") will return secrets at HOME/subpath

Where HOME is defined in the configuration.toml as [SecretStore].Path (internally prefixed with /v1/secret/edgex/ for purposes of the http request)

[drkfmorton]

@ajcasagrande When reviewing paths for listing secrets, consul returns all sub-directories, see example one (1) below. In your mentioning above, you only want top-level paths returned as defined in example two (2), correct?

Example 1 (Actual list returned): Consul: Path used "edgex/appservices/2.0/app-rules-engine"

[
    "edgex/appservices/2.0/app-rules-engine/Clients/core-metadata/Host",
    "edgex/appservices/2.0/app-rules-engine/Clients/core-metadata/Port",
    "edgex/appservices/2.0/app-rules-engine/Clients/core-metadata/Protocol",
    "edgex/appservices/2.0/app-rules-engine/Database/BatchSize",
    "edgex/appservices/2.0/app-rules-engine/Database/Host",
    "edgex/appservices/2.0/app-rules-engine/Database/MaxIdle",
    "edgex/appservices/2.0/app-rules-engine/Database/Port",
    "edgex/appservices/2.0/app-rules-engine/Database/Timeout",
    "edgex/appservices/2.0/app-rules-engine/Database/Type",
    "edgex/appservices/2.0/app-rules-engine/HttpServer/HTTPSCertName"
]

Example 2 (proposed list returned - top level): Consul: Path used "edgex/appservices/2.0/app-rules-engine"

[
    "edgex/appservices/2.0/app-rules-engine/Clients/",
    "edgex/appservices/2.0/app-rules-engine/Database/",
    "edgex/appservices/2.0/app-rules-engine/HttpServer/"
]

Valult returns a list of keys as defined in Example three (3), which includes top-level keys and sub-directories. If this meets your expectations then I can complete the update.

{
    "request_id": "26e5426d-7184-801b-c106-eeaf6f085d64",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
        "keys": [
            "app-rules-engine/",
            "core-command/",
            "core-data/",
            "core-metadata/",
            "security-bootstrapper-redis/",
            "support-notifications/",
            "support-scheduler/"
        ]
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

[lenny-goodell]

Example 1 (Actual list returned): Consul: Path used "edgex/appservices/2.0/app-rules-engine"

For insecure secrets you don't need to go to Consul. Service may not be using Consul.You just need to look at the InescureSecrets map in the Service's loaded configuration.
https://github.com/edgexfoundry/go-mod-bootstrap/blob/main/config/types.go#L174-L181
Iterate thru the map and return the values for Path

[lenny-goodell]

Valult returns a list of keys as defined in Example three (3)

For Vault you have the services' SecretStore listed, not the Secrets that a service has.
The full path to these will look like this for Redis DB creds for Core Data: <vault base>/core-data/redisdb

Remember that this API is looking at the list of secret paths for a current service.

[drkfmorton]

Example 1 (Actual list returned): Consul: Path used "edgex/appservices/2.0/app-rules-engine"

For insecure secrets you don't need to go to Consul. Service may not be using Consul.You just need to look at the InescureSecrets map in the Service's loaded configuration. https://github.com/edgexfoundry/go-mod-bootstrap/blob/main/config/types.go#L174-L181 Iterate thru the map and return the values for Path

This is correct. I misspoke on my terminology usage of consul. I was testing getting insecure secrets with the API and all paths are returned.

[lenny-goodell]

This is correct. I misspoke on my terminology usage of consul. I was testing getting insecure secrets with the API and all paths are returned.

This a a per service thing. So the single URL for view InsecureSecrets for app-rules-engine would be edgex/appservices/2.0/app-rules-engine/Writable/InsecureSecrets.

[lenny-goodell]

I think

ListSecretsAtPath(path string) ([]string, error)

Should do what you need, where

• ListSecretsAtPath("") will return the service's secrets at its HOME path in the secret store
• ListSecretsAtPath("subpath") will return secrets at HOME/subpath

Where HOME is defined in the configuration.toml as [SecretStore].Path (internally prefixed with /v1/secret/edgex/ for purposes of the http request)

@bnevis-i it is much simpler since path is really secret name. i.e., we don't have sub-paths. What this issue is looking for is just the list of secret names (aka paths ) that are in the service's SecretStore.