Core Command message bus Time out

[marcpfuller]

🐞 Bug Report

Affected Services [REQUIRED]

The issue is located in: core-command

Is this a regression?

This issue looks to be new, since enabling core command via message bus

Description and Minimal Reproduction [REQUIRED]

After retrieving Core commands via message bus, a time out error is being returned.

This can be reproduced with the following steps :
cd edgex-compose/compose-builder
make run no-secty ds-virtual
cd app-functions-sdk/app-service-template
make build
./app-new-service

🔥 Exception or Error

 
```
marcphil@marcphil-mobl:~/iotgDev/edgex/myrepos/app-functions-sdk-go/app-service-template$ ./app-new-service | grep -e "commands to choose from" -e "timed out"
level=DEBUG ts=2023-01-23T17:57:45.449180293Z app=app-new-service source=sample.go:126 msg="Device Random-Integer-Device has 16 commands to choose from. (in pipeline 'default-pipeline')"
level=DEBUG ts=2023-01-23T17:57:45.45224207Z app=app-new-service source=sample.go:126 msg="Device Random-Integer-Device has 16 commands to choose from. (in pipeline 'default-pipeline')"
level=DEBUG ts=2023-01-23T17:57:45.453824001Z app=app-new-service source=sample.go:126 msg="Device Random-Integer-Device has 16 commands to choose from. (in pipeline 'default-pipeline')"
level=DEBUG ts=2023-01-23T17:57:45.712331076Z app=app-new-service source=sample.go:126 msg="Device Random-Integer-Device has 16 commands to choose from. (in pipeline 'default-pipeline')"
level=DEBUG ts=2023-01-23T17:57:48.988303635Z app=app-new-service source=sample.go:126 msg="Device Random-UnsignedInteger-Device has 16 commands to choose from. (in pipeline 'default-pipeline')"
level=DEBUG ts=2023-01-23T17:57:49.111055123Z app=app-new-service source=sample.go:126 msg="Device Random-UnsignedInteger-Device has 16 commands to choose from. (in pipeline 'default-pipeline')"
level=ERROR ts=2023-01-23T17:57:53.998398992Z app=app-new-service source=runtime.go:322 msg="Pipeline (default-pipeline) function edgexfoundry/edgex-go#2 resulted in error: failed to get list of commands for Random-UnsignedInteger-Device device: timed out waiting response in pipeline 'default-pipeline' (X-Correlation-ID=30e7cfee-eb04-48ec-b196-9ade2ad1e48a)"
level=ERROR ts=2023-01-23T17:57:53.998472232Z app=app-new-service source=triggermessageprocessor.go:154 msg="message error in pipeline default-pipeline (30e7cfee-eb04-48ec-b196-9ade2ad1e48a): failed to get list of commands for Random-UnsignedInteger-Device device: timed out waiting response in pipeline 'default-pipeline'"
level=ERROR ts=2023-01-23T17:57:53.998512915Z app=app-new-service source=messaging.go:186 msg="MessageBus Trigger: Failed to process message on pipeline(s): 1 error occurred:\n\t* failed to get list of commands for Random-UnsignedInteger-Device device: timed out waiting response in pipeline 'default-pipeline'\n\n"
level=ERROR ts=2023-01-23T17:58:05.488586127Z app=app-new-service source=runtime.go:322 msg="Pipeline (default-pipeline) function edgexfoundry/edgex-go#2 resulted in error: failed to get list of commands for Random-Boolean-Device device: timed out waiting response in pipeline 'default-pipeline' (X-Correlation-ID=f83db392-cefa-43ab-953e-16d608342f81)"
level=ERROR ts=2023-01-23T17:58:05.48863551Z app=app-new-service source=triggermessageprocessor.go:154 msg="message error in pipeline default-pipeline (f83db392-cefa-43ab-953e-16d608342f81): failed to get list of commands for Random-Boolean-Device device: timed out waiting response in pipeline 'default-pipeline'"
```

🌍 Your Environment

Deployment Environment:
windows/ubuntu wsl

EdgeX Version [REQUIRED]:
3.0

Anything else relevant?

[lenny-goodell]

@cloudxxx8 , FYI. Note that the query request are working initially and then start timing out.

[lenny-goodell]

@hahattan , @cloudxxx8 , note that I am not seeing this issue with my new Request API. Once those PR are merged, I think we can close this issue.

[cloudxxx8]

@marcpfuller this issue should be gone after #4310 is merged. could you please see whether we can close this issue?

[lenny-goodell]

This looks to be an issue in App Sdk handling due to load. Works fine for first few request when load is high from Device Virtual and then starts failing. If load is light like from Device Simple, we don't see the time out errors.

[lenny-goodell]

addressed by #4404