sync: TestWaitGroupMisuse2 test is flaky

[kostya-sh]

Using tip (ca91de7) TestWaitGroupMisuse2 test fails approximately 19 times out of 20 on my PC:

ok      strings 0.104s
--- FAIL: TestWaitGroupMisuse2 (1.41s)
    waitgroup_test.go:110: Should panic
    waitgroup_test.go:80: Unexpected panic: <nil>
FAIL
FAIL    sync    1.558s
ok      sync/atomic 0.

[bradfitz]

Could you describe your PC?

[bradfitz]

How many CPUs? Which? Which OS, OS version, arch (32-bit or 64-bit)?

Dmitry, maybe 1e6 should be bigger in the test?

/cc @dvyukov

[kostya-sh]

OS: 64-bit Debian 8.1 running on VMWare player 7.1 (host OS Windows 8).

Hardware: 4 CPUs (Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz)

[kostya-sh]

I've just tested it under Windows 8 OS (the same hardware). The test is still flaky though success rate is higher.

[dvyukov]

@kostya-sh What is the iteration count in the test at which it stops being flaky?

[bradfitz]

And how long does the test take to run at said iteration count?

[marete]

It fails reliably on my Linux laptop in an LXC container running Ubuntu 14.10 (3/3 times):

ok strings 0.156s
--- FAIL: TestWaitGroupMisuse2 (3.25s)
waitgroup_test.go:110: Should panic
waitgroup_test.go:80: Unexpected panic:
FAIL
FAIL sync 3.410s
ok sync/atomic 0.897s

My CPU's details (this is a Nehalem class mobile CPU):

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 37
Model name: Intel(R) Core(TM) i5 CPU M 460 @ 2.53GHz
Stepping: 5
CPU MHz: 2528.000
CPU max MHz: 2528.0000
CPU min MHz: 1197.0000
BogoMIPS: 5056.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 3072K
NUMA node0 CPU(s): 0-3

My OS's details:

Ubuntu Linux 14.10 (x86_64), running kernel 4.0.6 (Linus' mainline) in an unprivileged LXC container within an Ubuntu 15.04 (x86_64) host:

marebri@utopic:~/devel/go.git/src$ uname -a
Linux utopic 4.0.6 #1 SMP Wed Jun 24 01:11:35 EAT 2015 x86_64 x86_64 x86_64 GNU/Linux

Other information:

I reproduced this on the Ubuntu 15.04 (x86_64) host (failed 1/3 times):

ok strings 0.179s
--- FAIL: TestWaitGroupMisuse2 (3.32s)
waitgroup_test.go:110: Should panic
waitgroup_test.go:80: Unexpected panic:
FAIL
FAIL sync 3.493s
ok sync/atomic 1.018s

vs the passing 2 tests:

ok strings 0.236s
ok sync 2.478s
ok sync/atomic 0.821s
ok syscall 0.131s

ok strings 0.312s
ok sync 0.452s
ok sync/atomic 1.025s
ok syscall 0.151s

EDIT: Both host and LXC container are on the current git tip: d0ed87d

[mikioh]

We can see this on the dragonfly buildbot: http://build.golang.org/log/fd18334c684f5cec5c7d4f939c39f26ec7c30741 by 03a48eb.

[kostya-sh]

Even with 8e7 iterations the test is still flaky on x86_64 Debian VMWare VM. It takes about 100 seconds to fail with this number of iterations.

It could take any time between 0.5 sec to 50 seconds for the test to succeed.

[iworker]

I reproduced this on Ubuntu 15.04 vivid (4 times)

--- FAIL: TestWaitGroupMisuse2 (2.77s)
    waitgroup_test.go:110: Should panic
    waitgroup_test.go:80: Unexpected panic: <nil>
FAIL
FAIL    sync    2.917s
....
2015/06/29 13:50:33 Failed: exit status 1

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz
Stepping:              4
CPU MHz:               1207.269
CPU max MHz:           3900.0000
CPU min MHz:           1200.0000
BogoMIPS:              7400.11
Virtualisation:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              10240K
NUMA node0 CPU(s):     0-7

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 15.04
Release:    15.04
Codename:   vivid

Linux *** 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

After all 1e6 iterations there was no one expected panic.

[xoba]

i've also seen the failure on a m3.xlarge aws ec2 instance (4 vCPU, 15 GiB), running ami-ee793a86 (ubuntu), golang commit a76c1a5:

ok cmd/nm 1.763s
ok cmd/objdump 4.847s
ok cmd/pack 3.671s
ok cmd/pprof/internal/profile 0.030s
ok cmd/vet 4.370s

GOMAXPROCS=2 runtime -cpu=1,2,4

ok runtime 36.327s

sync -cpu=10

--- FAIL: TestWaitGroupMisuse2 (1.58s)
waitgroup_test.go:110: Should panic
waitgroup_test.go:80: Unexpected panic:
FAIL
FAIL sync 1.682s
2015/06/29 10:28:14 Failed: exit status 1

[iworker]

@dvyukov with iteration count 1e9 this test didn't fails:

ok      sync    6.109s

But I think it's not good choice to increase iterations count. Maybe better to introduce some better way to get panic with more probability. Actually, maybe iterations count must depends on CPU's count. For example, 8 CPU's – 1e9, 2 CPU's – 1e6.

[dvyukov]

Maybe better to introduce some better way to get panic with more probability.

I don't see any way to increase panic probability.

We probably need to not run it in short mode. But that will make test effectively useless...
@bradfitz What do you think if we whitelist it for short mode only on some subset of builders?

[bradfitz]

It sounds like the test just sucks. I think it should be deleted if it can't made to be reliable.

[bradfitz]

But if you want to make it whitelist per builder, we'll need to finish #11346

[bradfitz]

This is also failing on my personal Linux server. Physical hardware, Ubuntu vivid 3.16.0-39-generic, 8 CPUs in /proc/cpuinfo:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 30
model name      : Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz
stepping        : 5     
microcode       : 0x3
cpu MHz         : 1200.000
cache size      : 8192 KB
physical id     : 0     
siblings        : 8
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes   
fpu_exception   : yes
cpuid level     : 11    
wp              : yes   
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips        : 5617.29
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

[bradfitz]

Disabling this for now in https://go-review.googlesource.com/11721

[bradfitz]

@dvyukov, do you care to make this test reliable before I delete it?

[dvyukov]

The test looks good to me. It just shows that scheduler sucks and doesn't execute runnable goroutines.

1.4:

$ stress -p 1 ./sync.test -test.run=TestWaitGroupMisuse2
...
7312 runs so far, 0 failures

1.7:

$ stress -p 1 ./sync.test -test.run=TestWaitGroupMisuse2
...
92 runs so far, 63 failures

Part of the problem is the next argument of runtime.runqput. But there may be other problems.

[bradfitz]

/cc @aclements because the scheduler reportedly sucks.

[dvyukov]

Just in case, this deflakes the test:
https://go-review.googlesource.com/#/c/33272/

[gopherbot]

CL https://golang.org/cl/36841 mentions this issue.