runtime: "found pointer to free object" on Windows

[bcmills]

This came to my attention via a windows-amd64-longtest failure in a TryBot (https://storage.googleapis.com/go-build-log/ef7da83e/windows-amd64-longtest_1a18709a.log).

2021-03-09T20:35:41-48ddf70/windows-amd64-longtest
2021-03-09T18:35:29-48895d0/windows-amd64-longtest
2021-03-05T18:47:09-c082f9f/windows-amd64-longtest

The symptom resembles #44614 (@randall77 @mdempsky), but that was marked fixed before the first known Windows failure occurred.

CC @mknyszek @aclements @jeremyfaller

[bcmills]

Marking as release-blocker at least until we understand whether this is a regression.

[mdempsky]

Could this be related to golang.org/cl/297389? All of those failure commits were after that CL was merged. /cc @zx2c4

[zx2c4]

https://build.golang.org/log/dcb6544b3a999b160ede916b98d49fad0d55465a

Plan 9 isn't running any Windows code, so I don't think it's related.

[mdempsky]

Fair point. Though there could also be separate issues affecting different platforms. The plan9-arm failures in #44903 date back to mid 2020, whereas the windows failures are all more recent. That makes me think there must have been something that affected Windows recently. E.g., I'm not saying that CL is incorrect, but maybe it's now tickling an underlying compiler/runtime issue that's been affecting plan9-arm and solaris too.

[zx2c4]

Hmm... could be. Something like this should mirror minimally what's going on with defers and unsafe.Pointer and the GC in that commit:

package main

import (
	"unsafe"
)

func main() {
	a := make([]unsafe.Pointer, 2)
	defer func() { println(a[1]) }()
	a[0] = unsafe.Pointer(&struct{ _ int }{})
	a[1] = unsafe.Pointer(&struct{ _ int }{})
	println(a[0])
}

[zx2c4]

runtime: marked free object in span 0x2dff28935b0, elemsize=24 freeindex=32 (bad use of unsafe.Pointer? try -d=checkptr)

These all say elemsize=32. Does that mean it's the result of an allocation of structs size 24 in a list? That seems to be the case, because the code goes onto say:

        for i := uintptr(0); i < s.nelems; i++ {
                addr := s.base() + i*s.elemsize

So in that case, what struct of size 24 is being allocated in a list? It can't be those unsafe.Pointers, where elemsize=8.

[mdempsky]

I think you meant they all say elemsize=24?

If so, yeah, the error message is that there's an object that was allocated on heap of size 24 bytes. Also, that GC previously ran and didn't find any references to the object and marked it dead; but a later run found a pointer to it again.

This could happen if a user converts a pointer to uintptr and then back to unsafe.Pointer in an unsafe way (hence the checkptr recommendation in the error message). But of course the compiler/runtime could be getting something wrong too.

[mdempsky]

If I understand correctly, the calls to updateProcThreadAttribute are making Windows (i.e., non-Go code) store pointers into the []unsafe.Pointer slice allocated in newProcThreadAttributeList?

If so, I suspect that's the issue then. It's not enough for the heap bits to be allocated as pointer memory. Mutators also need to know to use write barriers when storing pointers into memory during certain GC phases. Windows wouldn't know to do that.

[mdempsky]

I also suspect the 24-byte object that's being lost and found again is the fd slice allocated in StartProcess.

[alexbrainman]

If I understand correctly, the calls to updateProcThreadAttribute are making Windows (i.e., non-Go code) store pointers into the []unsafe.Pointer slice allocated in newProcThreadAttributeList?

updateProcThreadAttribute is an Windows API that does not document it's implementation

https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-updateprocthreadattribute

The doco says Updates the specified attribute in a list of attributes for process and thread creation.. It does not says that it stores pointers to our objects in that memory. Jason disassembled the UpdateProcThreadAttribute to discover how the function is implemented.

I am still against submitted CL

https://go-review.googlesource.com/c/go/+/297389/

because it is too complicated.

All we need to do here is to make sure that fd slice lives until the end of StartProcess.

So something like what I suggested at https://go-review.googlesource.com/c/go/+/297391/1/src/syscall/exec_windows.go#253 should work.

Alex

[zx2c4]

If I understand correctly, the calls to updateProcThreadAttribute are making Windows (i.e., non-Go code) store pointers into the []unsafe.Pointer slice allocated in newProcThreadAttributeList?

If so, I suspect that's the issue then. It's not enough for the heap bits to be allocated as pointer memory. Mutators also need to know to use write barriers when storing pointers into memory during certain GC phases. Windows wouldn't know to do that.

Ahh, I wasn't aware of that (and apparently nobody mentioned that on the rather lengthy CL). I'll send two new CLs to fix this the straight forward way for go/src/syscall and a more usable way for x/sys/windows.

[gopherbot]

Change https://golang.org/cl/300349 mentions this issue: syscall: use runtime.KeepAlive for ProcThreadAttributeList arguments

[gopherbot]

Change https://golang.org/cl/300369 mentions this issue: windows: use Go-managed pointer list for ProcThreadAttributeList

[zx2c4]

Those two CLs should address the matter.

[zx2c4]

This should actually still be open as https://golang.org/cl/300369 hasn't been merged yet.