fix(snap): Apply proxy's runtime config options after startup

[farshidtz]

Fixes #3863, see the issue for details.

This PR changes the configure hook to skip configuration options of security-proxy in defer-startup mode. To apply those options, the configure hook is called again after security-proxy-setup completes.

This PR changes the configure hook to skip runtime configuration options of security-proxy in defer-startup mode. Those config options should be applied after security-proxy-setup completes.

To achieve that:
a) The config option handling function is changed to ignore runtime config options of security-proxy in defer-startup install mode: canonical/edgex-snap-hooks#25
b) The configure executable is extended with an options subcommand which takes a service name as follows:

./configure options --service=security-proxy

The above command is called in a post-stop script after the security-proxy setup.

Prior to this change, the configure executable was called only as the snap configure hook. Future work may focus on file re-structuring of various executables under snap directory to add better visiblity into various helper executables.

PR Checklist

Please check if your PR fulfills the following requirements:

• I am not introducing a breaking change (if you are, flag in conventional commit message with BREAKING CHANGE: describing the break)
• I am not introducing a new dependency (add notes below if you are)
• I have added unit tests for the new feature or bug fix (if not, why?) - gadget testing is WIP
• I have fully tested (add details below) this the new feature or bug fix (if not, why?)
• I have opened a PR for the related docs change (if not, why?) - not required

Testing Instructions

The checkbox tests have coverage for the affected config options. Those tests were passing even before this fix because the config sets in those tests are executed after service startup.

To test the seeding, we should pass defaults from a Gadget snap:

defaults: 
  # edgexfoundry
  AZGf0KNnh8aqdkbGATNuRuxnt1GNRKkV:
    # Services that should auto start
    device-virtual: "on"
    support-scheduler: "on" 
    support-notifications: "on"
    
    env.security-proxy:
        # set a default user
        user: "admin,1,ES256"
        public-key: |
          -----BEGIN PUBLIC KEY-----
          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAETuwPTr5tL/NxiTkUWh2r93i6yaQf
          XiFV8kNjemjuTpOTc6yZu+zgNyt1pXpHmPmtIZ8X4eE0QGCIjlnfAz0dQA==
          -----END PUBLIC KEY-----

The OS must boot without error and the config options must be applied.

To seed this snap into an Ubuntu Core image to test the above, use the build from this branch under channel: latest/edge/fix-snap-proxy-config

New Dependency Instructions (If applicable)

[lenny-goodell]

@farshidtz , please amend you commit with a it signed-off to fix the DCO issue and you also must complete the PR check list. Provide reason for any not checked.

[bnevis-i]

LGTM. Should present the feature at the next security WG meeting.

[farshidtz]

Thanks for the reviews. I've amended the commit to add the signature.

Please note that the PR is a draft. I will continue testing and update the description before making it ready.

[tonyespy]

Nicely done, code changes look good to me. That said, I'd like a quick review from @anonymouse64 to ensure that there are no issues with re-running the configure hook from a script like this...

[anonymouse64]

As mentioned in our meeting, there are no issues from snapd's perspective with running the configure hook script/program, since you're really just running some code here. Snapd doesn't care that the code being executed was also executed for the configure hook at all.

That being said, I didn't test this PR so I can't say whether it works, but the intention of the feature is very cool and I'm happy to see it come to fruition 😄

[stolowski]

As discussed on mattermost, I think this is problematic. If this is triggered during installation, it's going to be racing with "seed" change and may cause conflict error ("seed" change in progress...).

The conflict arises because your configure hook code tries to start services with snapctl and since it is triggered "externally" (in this case by systemd) snapd will not consider this to be part of the seeding (install) change.

[farshidtz]

Thanks a lot for all the reviews, in particular to @anonymouse64 and @stolowski for rigorously testing the changes.

I've changed the logic to apply specific service-specific configurations after proxy setup, instead of calling the configure executable to perform a full config (as expected from the configure hook). We'll have to do some refactoring in future to better separate the "configure hook" from "configuring a service" without replicating code.

The PR description has been updated with details. This PR remains in draft until canonical/edgex-snap-hooks#25 is merged and included.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[siggiskulason]

I've reviewed the code with no comments - it looks fine to me. I'm proceeding to run checkbox/taf tests and then create the gadget and image.

• TAF functional tests: 5 failures (482 pass). The failures are not related to this PR and latest/edge results are identical.

• TAF integration tests: All pass

• Created a Ubuntu Core image as per instructions in Google Doc, with gadget-yaml specifying the public key for a new user. Then created a JWT token by copying the private key to the image with ​scp -P8022 private.pem [username]@localhost:
  ​, then ssh-ing onto the box and doing

• TOKEN=edgexfoundry.secrets-config proxy jwt --algorithm ES256 --private_key /var/snap/edgexfoundry/current/private.pem --id 1 --expiration=1h (with "1") being the user ID used in the gadget.yaml

• edgexfoundry.curl -k -X GET https://localhost:8443/core-data/api/v2/ping? -H "Authorization: Bearer $TOKEN"
  (Note that I could obviously have run edgexfoundry.secrets-config proxy jwt on my desktop instead to generate the token)

and this worked fine.

[siggiskulason]

LGTM - Approved - with tests passing as per my comment.

[farshidtz]

Thanks a lot @siggiskulason for the review and testing.

• Then created a JWT token by copying the private key to the image with ​scp -P8022 private.pem [username]@localhost:
  ​, then ssh-ing onto the box and doing
• TOKEN=edgexfoundry.secrets-config proxy jwt --algorithm ES256 --private_key /var/snap/edgexfoundry/current/private.pem --id 1 --expiration=1h (with "1") being the user ID used in the gadget.yaml
  (Note that I could obviously have run edgexfoundry.secrets-config proxy jwt on my desktop instead to generate the token)

Or you could generate the token on your machine using openssl as instructed here (3. alternative approach): https://github.com/edgexfoundry/edgex-go/tree/main/snap#jwt-tokens