bug: intermittent failure of TestPrivacyGroupEndorsementConcurrent

[hosie]

What happened?

[2024-12-20T08:39:49.642Z] DEBUG Stopped node-name=aliceNode
--- FAIL: TestPrivacyGroupEndorsementConcurrent (8.71s)
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    component_test.go:1764: 
        	Error Trace:	/Users/johnhosie/foo/paladin/bugfix/paladin/core/go/componenttest/component_test.go:1764
        	Error:      	Condition never satisfied
        	Test:       	TestPrivacyGroupEndorsementConcurrent
        	Messages:   	Transaction did not receive a receipt
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    utils_for_test.go:100: Using transaction latency threshold of 2s
    component_test.go:1822: 
        	Error Trace:	/Users/johnhosie/foo/paladin/bugfix/paladin/core/go/componenttest/component_test.go:1822
        	Error:      	Not equal: 
        	            	expected: "bob_value_0_0"
        	            	actual  : ""
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-bob_value_0_0
        	            	+
        	Test:       	TestPrivacyGroupEndorsementConcurrent
FAIL
FAIL	github.com/kaleido-io/paladin/core/componenttest	9.130s
FAIL

What did you expect to happen?

ok github.com/kaleido-io/paladin/core/componenttest 6.818s

How can we reproduce it (as minimally and precisely as possible)?

while :
do
go clean -testcache
go test -timeout 60s -tags testdbpostgres -run ^TestPrivacyGroupEndorsementConcurrent$ github.com/kaleido-io/paladin/core/componenttest >> log.log
done

Anything else we need to know?

re-running the test in a bash loop of 100, I see it pass 97 times and fail with this error 3 times

OS version

No response

[hosie]

This is a concurrency test so very prone to intermittent failures. Very likely there is a real issue in there that could be causing problems in other scenarios where it might be even more expensive to debug so this is certainly one that is worth getting a handle on.

[hosie]

Same test failed on this PR run #487 with a different symptom

2024-12-19T23:15:08.9244061Z [2024-12-19T23:15:00.692Z]  INFO Logs diverted to build/testbed.component-test.log
2024-12-19T23:15:08.9244532Z     utils_for_test.go:100: Using transaction latency threshold of 2s
2024-12-19T23:15:08.9244764Z     utils_for_test.go:527: 
2024-12-19T23:15:08.9245678Z         	Error Trace:	/home/runner/work/paladin/paladin/core/go/componenttest/utils_for_test.go:527
2024-12-19T23:15:08.9247130Z         	            				/home/runner/work/paladin/paladin/core/go/componenttest/component_test.go:1663
2024-12-19T23:15:08.9247485Z         	Error:      	Condition never satisfied
2024-12-19T23:15:08.9247950Z         	Test:       	TestPrivacyGroupEndorsementConcurrent
2024-12-19T23:15:08.9248620Z         	Messages:   	Deploy transaction did not receive a receipt
2024-12-19T23:15:08.9248857Z     utils_for_test.go:537: 
2024-12-19T23:15:08.9249832Z         	Error Trace:	/home/runner/work/paladin/paladin/core/go/componenttest/utils_for_test.go:537
2024-12-19T23:15:08.9251440Z         	            				/home/runner/work/paladin/paladin/core/go/componenttest/component_test.go:1663
2024-12-19T23:15:08.9251821Z         	Error:      	Expected value not to be nil.
2024-12-19T23:15:08.9252263Z         	Test:       	TestPrivacyGroupEndorsementConcurrent
2024-12-19T23:15:08.9252544Z --- FAIL: TestPrivacyGroupEndorsementConcurrent (8.26s)
2024-12-19T23:15:08.9252673Z FAIL
2024-12-19T23:15:08.9253017Z coverage: 59.8% of statements in github.com/kaleido-io/paladin/core/...
2024-12-19T23:15:08.9253322Z FAIL	github.com/kaleido-io/paladin/core/componenttest	61.891s