Bug: Intermittent panic on agent startup

[dlarson04]

Describe the bug.

I am occasionally seeing a panic starting anax in container

I1009 12:21:07.971763      21 node.go:137] Exchange RPC retrieving device myorg/edge-auto-ubuntu-2004-amd64-1_horizon2 from exchange
I1009 12:21:07.993679      21 node.go:170] Exchange RPC retrieved device myorg/edge-auto-ubuntu-2004-amd64-1_horizon2 from exchange Name: edge-auto-ubuntu-2004-amd64-1_horizon2, Owner: myorg/admin, NodeType: device, ClusterNamespace: , HAGroup: , Pattern , LastHeartbeat: 2024-10-09T12:21:07.849127744Z[UTC], MsgEndPoint: , Arch: amd64, HeartbeatIntv: {0 0 0}
I1009 12:21:07.993776      21 cache.go:224] Update exchange cache NODE_DEF_CACHE/myorg/edge-auto-ubuntu-2004-amd64-1_horizon2 with Name: edge-auto-ubuntu-2004-amd64-1_horizon2, Owner: myorg/admin, NodeType: device, ClusterNamespace: , IsNamespaceScoped: false, HAGroup: , Pattern: , SoftwareVersions: map[], LastHeartbeat: 2024-10-09T12:21:07.849127744Z[UTC], RegisteredServices: [], MsgEndPoint: , Arch: amd64, UserInput: [], HeartbeatIntv: {0 0 0}
E1009 12:21:07.994364      21 cutil.go:822] unable to open Certificate file , error open : no such file or directory
I1009 12:21:07.994669      21 device.go:307] Succeeded updating device record to Org: myorg, Token: <set>, Name: edge-auto-ubuntu-2004-amd64-1_horizon2, NodeType: device, TokenLastValidTime: 1728476467, TokenValid: true, Pattern: , Config: State: configuring, Time: 1728476467, SoftwareVersions: map[config:1.0.0 horizon:2.32.0-1584]
I1009 12:21:08.002622      21 worker.go:480] CommandDispatcher: Running subworker NodePolicyWatcher
I1009 12:21:08.002872      21 node.go:229] AgreementWorker checking the node policy changes.
I1009 12:21:08.003149      21 policy.go:115] Exchange RPC getting node policy for myorg/edge-auto-ubuntu-2004-amd64-1_horizon2.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x1675909]

goroutine 5 [running]:
github.com/open-horizon/anax/exchange.GetNodePolicy({0x0, 0x0}, {0xc0000d0c00, 0x2c})
        /tmp/anax-gopath/src/github.com/open-horizon/anax/exchange/policy.go:125 +0x249
github.com/open-horizon/anax/agreement.(*AgreementWorker).checkNodePolicyChanges.GetHTTPNodePolicyHandler.func1({0xc0000d0c00?, 0x5?})
        /tmp/anax-gopath/src/github.com/open-horizon/anax/exchange/handlers.go:257 +0x2c
github.com/open-horizon/anax/exchangesync.GetProcessedExchangeNodePolicy(0xc0001f2000, 0xc0000d9ec0, 0xc0000d9ea8, 0x55310f?)
        /tmp/anax-gopath/src/github.com/open-horizon/anax/exchangesync/node_policy.go:95 +0xd5
github.com/open-horizon/anax/exchangesync.SyncNodePolicyWithExchange(0x0?, 0x0?, 0x0?, 0x0?)
        /tmp/anax-gopath/src/github.com/open-horizon/anax/exchangesync/node_policy.go:31 +0x158
github.com/open-horizon/anax/agreement.(*AgreementWorker).checkNodePolicyChanges(0xc000204900)
        /tmp/anax-gopath/src/github.com/open-horizon/anax/agreement/node.go:245 +0x329
github.com/open-horizon/anax/agreement.(*AgreementWorker).reconcileNodePolicy(...)
        /tmp/anax-gopath/src/github.com/open-horizon/anax/agreement/agreement.go:355
github.com/open-horizon/anax/worker.(*BaseWorker).DispatchSubworker.func1()
        /tmp/anax-gopath/src/github.com/open-horizon/anax/worker/worker.go:482 +0x289
created by github.com/open-horizon/anax/worker.(*BaseWorker).DispatchSubworker in goroutine 26
        /tmp/anax-gopath/src/github.com/open-horizon/anax/worker/worker.go:469 +0x106
Anax run number 1 ended with exit code 0.
Respawning anax...

coming from this line in exchange/policy.go

        targetURL := fmt.Sprintf("%vorgs/%v/nodes/%v/policy", ec.GetExchangeURL(), GetOrg(deviceId), GetId(deviceId))

The start up sequence was changed recently so the agent reads its node policy early to resolve another issue so this may have been introduced by that change

Describe the steps to reproduce the behavior.

No response

Expected behavior.

OH agent should never panic

Screenshots.

No response

Operating Environment

Linux

Additional Information

No response