Support user secrets: "secret" config type and Harness support

[jameinel]

To handle user secrets, with Juju (3.4(?) you can now have a config with a schema type of "secret". Juju will then ensure that it can hold a secret URL.

However, if I try to create a charm that uses this, when running the test suite I get:

Traceback (most recent call last):
  File "/home/jameinel/dev/charms/ubuntu-lite/test/test_charm.py", line 22, in test_on_start
    harness = testing.Harness(charm.Ubuntu)
  File "/home/jameinel/dev/charms/ubuntu-lite/venv/lib/python3.10/site-packages/ops/testing.py", line 251, in __init__
    self._backend = _TestingModelBackend(self._unit_name, self._meta, config_)
  File "/home/jameinel/dev/charms/ubuntu-lite/venv/lib/python3.10/site-packages/ops/testing.py", line 2079, in __init__
    self._config = _TestingConfig(config)
  File "/home/jameinel/dev/charms/ubuntu-lite/venv/lib/python3.10/site-packages/ops/testing.py", line 1966, in __init__
    self._config_set(key, value)
  File "/home/jameinel/dev/charms/ubuntu-lite/venv/lib/python3.10/site-packages/ops/testing.py", line 1996, in _config_set
    raise RuntimeError(
RuntimeError: Incorrectly formatted `options.yaml`: `type` needs to be one of [string, boolean, int, float], not secret.

I tested this with:

Installing collected packages: websocket-client, ops
  Attempting uninstall: ops
    Found existing installation: ops 1.5.5
    Uninstalling ops-1.5.5:
      Successfully uninstalled ops-1.5.5
Successfully installed ops-2.11.0 websocket-client-1.7.0

[jameinel]

So to trigger this behavior, I updated my charm's config to have:

+++ b/config.yaml
@@ -1 +1,5 @@
-options: {}
+options:
+  mysec:
+    type: secret
+    default: nil
+    description: "tell me your secrets"

It turns out that it only fails because I set the 'default:' key, which makes the Testing infrastructure call _config_set with the defaults, and then it validates the type.
However, in practice, you shouldn't set the config, but you will need to read it.

[jameinel]

and certainly the test suite will need to be able to set it :)

[tonyandrewmeyer]

This seems to be undocumented functionality (config.yaml, charmcraft.yaml, manage application, how to use secrets, release notes, secret reference). Is there documentation somewhere else?

[jameinel]

I know the Juju spec for it: https://docs.google.com/document/d/1gb-l-YTZtHyZk8UktEXVz5gFwyTJB163IMNA-4EmXnQ/edit?usp=sharing It was implemented back in Juju 3.3. It is entirely plausible that the rest of the documentation is not complete. I did put together a use case driving it here: canonical/istio-operators#380 (comment) John =:->

…

On Tue, Mar 26, 2024 at 4:30 PM Tony Meyer ***@***.***> wrote: This seems to be undocumented functionality (config.yaml <https://juju.is/docs/sdk/config-yaml>, charmcraft.yaml <https://juju.is/docs/sdk/charmcraft-yaml#heading--config>, manage application <https://juju.is/docs/juju/manage-applications#heading--configure-an-application>, how to use secrets <https://juju.is/docs/sdk/add-a-secret-to-a-charm>, release notes <https://juju.is/docs/juju/roadmap>, secret reference <https://juju.is/docs/juju/secret>). Is there documentation somewhere else? — Reply to this email directly, view it on GitHub <#1166 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABRQ7LRGUMRMX3WGGBWZNDY2HLF5AVCNFSM6AAAAABFI5E23CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRRGQYDQNBQG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tonyandrewmeyer]

We're going to get John's PR into ops 2.12 as an immediate unblocker, but we feel that there's more work to be done here.

It's likely that we need a new Harness method (add_user_secret) and the permissions checking probably needs to be updated as well. There might also be other adjustments that we haven't figured out yet as well.

We're expecting that we'll be able to do the more complete fix in 22.04.

Also: the Juju team are working on updating the documentation. We should double-check that at least the charmcraft.yaml and config.yaml pages are updated, since those can be done trivially.

[IronCore864]

I see the Jira ticket for this issue is assigned to me, so today I did some research today and tried juju add-secret.

It seems user secrets added by this command belong to the Model, as shown by juju secrets output as well as confirmed by this code here. It's not the same as secrets added by the secret-add hook tools, which belong to either the Applicaiton or a Unit. Did I get it right?

And it seems Harness can add secrets with owner set as the app or unit. I suppose we can add a method Harness.add_user_secret and set the owner as self.model.uuid, but we need to change a bunch of other methods like grant_secret, _ensure_secret_owner, secret_set, etc.

Did I get the whole picture?

Also it would be nice to merge Tony's charmcraft PR so that we can pack and test it.

[benhoyt]

Yeah, that sounds right, but maybe we can briefly go over this with Ian at the Juju standup today.

Thinking about it now, Harness.add_model_secret is not a great name, because it sounds like the secret is tied to the model (when it's actually app or unit). But yeah, adding add_user_secret is probably the way to go.

[benhoyt]

Per Juju recommendation, let's also consider renaming Harness.add_model_secret to add_charm_secret (with an alias for backwards-compatibility) and call the user-secret one add_user_secret. This fits the Juju team / docs terminology of "charm secret" vs "user secret".

Presumably add_user_secret would add the effective emulated grants automatically.

[DnPlas]

Hi @benhoyt regarding tests, what would be the recommended way of testing user secrets (adding, granting access, etc.) in an integration test environment? I see that we have add_secret, but that models application secrets, AFAICT, so in charm world I don't think we could do something like ops_test.model.add_secret (when we are using the pytest-operator).

[tonyandrewmeyer]

what would be the recommended way of testing user secrets (adding, granting access, etc.) in an integration test environment? I see that we have add_secret, but that models application secrets, AFAICT,

ops.testing (Harness) is for unit tests, not integration tests. At the moment, writing unit tests for user secrets with ops.testing is not well supported, but when this ticket is closed (probably next pulse) that will be fixed.

For integration tests, you should use pytest-operator and python-libjuju (which have the model.add_secret method that maps to juju add-secret as you said).

[DnPlas]

what would be the recommended way of testing user secrets (adding, granting access, etc.) in an integration test environment? I see that we have add_secret, but that models application secrets, AFAICT,

ops.testing (Harness) is for unit tests, not integration tests. At the moment, writing unit tests for user secrets with ops.testing is not well supported, but when this ticket is closed (probably next pulse) that will be fixed.

For integration tests, you should use pytest-operator and python-libjuju (which have the model.add_secret method that maps to juju add-secret as you said).

Thanks for clearing this out, I didn't know python-libjuju already had this.

[benhoyt]

This is now done with #1167 and #1176