[secret-changed] Not to lock ourselves out of our DB (DataPeer)

[juditnovak]

Steps to reproduce

Assume as below:

1. DB admin password changes (gets "committed" to the Juju Secret store)
2. Whenever the DB password changes, we may need to execute "some actions"
   Like updating the admin password on the database level
   Now we have a "chicken and egg" problem... As if either of those operations fail we lost the old/new password.
   Which is why additional actions (like updating the password in the DB) must go in the secret-changed event.
   These actions MUST go in the secret-changed event by convention (that's equally received by the owner as observers)
3. The secret-changed event handler (the single place where we"ve got the "memory" i.e. old value of the secret) fails and/or gets deferred
4. The charm tries to use the new secret value (whenever getting the secret that appears with the new value in the Juju Secret store) -- however the DB never succeeded to set the admin password yet, thus the new credentials aren't in place... While we have no trace of the old credentials anymore.

Expected behavior

As long as all actions on the charm's side, relating ot secret update aren't performed, the secret MUSTN'T be updated from the perspecive of the charm. It has to be "stuck" on the old value.

(At this point we may wanna consider so called callback functions to be linked to the secrets or not.)

Actual behavior

The above "horror" scenario applies

Versions

Operating system: doesn't matter

Juju CLI: 3.1+

Juju agent: corresponding

[github-actions]

https://warthogs.atlassian.net/browse/DPE-3324