Delayed start support services failed

[chr1shung]

🐞 Bug Report

Affected Services [REQUIRED]

The issue is located in: security-spiffe-token-provider

Is this a regression?

No

Description and Minimal Reproduction [REQUIRED]

Run support services with delayed-started configuration

🔥 Exception or Error

It seems like the service-key(support-notifications) sending to /api/v2/gettoken API
is not identical to service-name(notifications) parsed from SVID

support services:
level=INFO ts=2022-09-13T08:21:40.628715463Z app=support-notifications source=secret.go:165 msg="runtime token provider enabled"
level=INFO ts=2022-09-13T08:21:40.628728253Z app=support-notifications source=methods.go:138 msg="using Unix Domain Socket at unix:///tmp/edgex/secrets/spiffe/public/api.sock"
level=INFO ts=2022-09-13T08:21:40.728993534Z app=support-notifications source=methods.go:150 msg="workload got X509 source"
level=WARN ts=2022-09-13T08:21:40.757491841Z app=support-notifications source=secret.go:111 msg="Retryable failure while creating SecretClient: failed to get spiffe-token-provider gettoken api call, status code = 400 "

security-spiffe-token-provider:
...
level=DEBUG ts=2022-09-13T08:22:34.172212307Z app=security-spiffe-token-provider source=init.go:165 msg="receiving gettoken request"
level=DEBUG ts=2022-09-13T08:22:34.172257441Z app=security-spiffe-token-provider source=init.go:191 msg="extracting peer SVID from TLS peer certificates..."
level=DEBUG ts=2022-09-13T08:22:34.172266177Z app=security-spiffe-token-provider source=init.go:206 msg="verifying SVID format and server key..."
level=ERROR ts=2022-09-13T08:22:34.172444993Z app=security-spiffe-token-provider source=init.go:249 msg="SK: support-notifications, SN: notifications"
level=ERROR ts=2022-09-13T08:22:34.172451588Z app=security-spiffe-token-provider source=init.go:250 msg="unequal service key and servie name fo

🌍 Your Environment

Deployment Environment:
Ubuntu

EdgeX Version [REQUIRED]:
latest dev images

Anything else relevant?

[lenny-goodell]

@bnevis-i , @jim-wang-intel FYI

[bnevis-i]

PR #4159 addresses the issue if the mismatch of the service key in docker and the service key in code by adding a workaround.

@hahattan failed to mention that the edge-compose scripts, by default, don't support running of edgex-notifications and edgex-scheduler out of the box. However, there is relatively simple boilerplate code at https://github.com/edgexfoundry/edgex-compose/blob/main/compose-builder/add-runtime-token-config-template.yml that gets dynamically added when one does a "make run delayed-start ds-somedeviceservice". Because services are in the base dockerfiles, adding this support would either mean calling https://github.com/edgexfoundry/edgex-compose/blob/main/compose-builder/gen_runtime_token_config_compose_ext.sh for support-notifications and support-scheduler unconditionally, but then that would force support- services to run in delayed start mode.

After consult with @lenny-intel we decided not to go this extra step and just fix the bug as stated.