go/common/workerpool: Fix memory leak when workerpool is stopped early

[peternose]

When workerpool si stopped, the job channel might still contain jobs which haven't been processed. Therefore, the channel never closes and leaves one go routine hanging.

Test

Tested with the following code. Before the code printed 1/101, afterwards 1/1.

package main

import (
	"fmt"
	"runtime"
	"time"

	"github.com/oasisprotocol/oasis-core/go/common/workerpool"
)

func main() {
	fmt.Println(runtime.NumGoroutine())
	for i := 0; i < 100; i++ {
		leak()
	}
	time.Sleep(time.Minute)
	fmt.Println(runtime.NumGoroutine())
}

func leak() {
	pool := workerpool.New("p2p/rpc")
	pool.Resize(10)
	defer pool.Stop()

	for i := 0; i < 100; i++ {
		pool.Submit(func() {
			time.Sleep(time.Millisecond)
		})
	}
}

[codecov]

Codecov Report

Merging #5008 (a888835) into master (978ea30) will decrease coverage by 0.45%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #5008      +/-   ##
==========================================
- Coverage   67.14%   66.68%   -0.46%     
==========================================
  Files         478      478              
  Lines       51799    51800       +1     
==========================================
- Hits        34780    34544     -236     
- Misses      12756    12983     +227     
- Partials     4263     4273      +10     

Impacted Files	Coverage Δ
go/common/workerpool/workerpool.go	77.27% <0.00%> (-1.19%)	⬇️
go/ias/http/http.go	20.33% <0.00%> (-44.07%)	⬇️
go/worker/compute/executor/committee/status.go	66.66% <0.00%> (-16.67%)	⬇️
go/consensus/tendermint/apps/staking/auth.go	62.96% <0.00%> (-14.82%)	⬇️
go/keymanager/api/policy_sgx.go	33.33% <0.00%> (-13.34%)	⬇️
...onsensus/tendermint/apps/beacon/state/state_vrf.go	73.33% <0.00%> (-13.34%)	⬇️
go/worker/compute/executor/api/api.go	25.80% <0.00%> (-12.91%)	⬇️
go/runtime/host/sandbox/sandbox.go	70.00% <0.00%> (-6.56%)	⬇️
go/oasis-node/cmd/ias/auth.go	71.87% <0.00%> (-6.25%)	⬇️
go/beacon/api/grpc.go	28.00% <0.00%> (-5.15%)	⬇️
... and 45 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[kostko]

Please also backport.