Improve service initialization process

[FelixTing]

There is an issue with the service initialization in secure mode if the external MQTT trigger is selected and auth mode is usernamepassword, clientcert, or cacert.

Example:
Edit the docker-compose.yml to add the custom ASC service's service key to EdgeX service secretstore-setup's ADD_SECRETSTORE_TOKENS environment variable. Assuming that the custom ASC service is app-external-mqtt-trigger.

  secretstore-setup:
    container_name: edgex-security-secretstore-setup
    depends_on:
    - security-bootstrapper
    - vault
    environment:
      ADD_KNOWN_SECRETS: redisdb[app-rules-engine],redisdb[device-rest],redisdb[device-virtual]
      ADD_SECRETSTORE_TOKENS: 'app-external-mqtt-trigger'

The app-external-mqtt-trigger ASC service's configuration contains the setting

[Trigger]
Type="external-mqtt"
  [Trigger.externalmqtt]
  Url = "tls://test.mosquitto.org:8884"
  SubscribeTopics="test/#"
  ClientId ="app-external-mqtt-trigger"
  Qos            = 0
  KeepAlive      = 10
  Retained       = false
  AutoReconnect  = true
  ConnectTimeout = "30s"
  SkipCertVerify = false
  AuthMode = "clientcert"
  SecretPath = "mqtt"

When starting the app-external-mqtt-trigger ASC service, it will fail on the trigger initialization and then shutdown.

level=INFO ts=2022-02-25T05:23:15.421383Z app=app- source=triggerfactory.go:54 msg="External MQTT trigger selected"
level=INFO ts=2022-02-25T05:23:15.421395Z app=app- source=mqtt.go:74 msg="Initializing MQTT Trigger"
level=ERROR ts=2022-02-25T05:23:15.422618Z app=app- source=service.go:175 msg="unable to create secure MQTT Client: Error found on handling secrets from underlying data-store: Received a '404' response from the secret store"
level=ERROR ts=2022-02-25T05:23:15.422648Z app=app- source=main.go:64 msg="MakeItRun returned error: failed to initialize Trigger"

Because the required secrets did not exist in Secret Store at the time the ASC service starts.

IMHO, we can resolve this issue by making the following changes:

• Start the Web Server before the trigger initialization. https://github.com/edgexfoundry/app-functions-sdk-go/blob/main/internal/app/service.go#L166-L196
• Add retry mechanism for MqttFactory.Create()

The concept of the retry mechanism is to wait for a signal from the AddSecret handler to retry if the service cannot get required secrets from Secret Store. With this proposal, users can store secrets to Secret Store using the Application Service API after ASC service is started so that the trigger initialization process can continue.

A draft PR will be submitted later on.