runtime: pointer to struct field can point beyond struct allocation

[rsc]

If a non-zero-size struct contains a final zero-size field f, the address &x.f may point beyond the allocation for the struct. This could cause a memory leak or a crash in the garbage collector (invalid pointer found). This is a more general version of #9384.

[dvyukov]

We can move all zero-size fields to the beginning of structs.

[randall77]

That would work for any user-defined structs. But we could just as easily convert &(zero-sized thing) to a pointer to the zero object in the compiler.

I'm a bit more concerned about generically built structs, like reflect.call and reflect.funcLayout.

[rsc]

It seems like there are two options.

1. Fix the code manipulating the struct memory never to point past the end.
2. Add more memory to the struct so that all the existing manipulation code becomes correct.

I believe that since zero-size types are excluded from the problem, we've narrowed the problem to very rare cases. The best solution is probably the one that introduces the fewest new corner cases where bugs can hide. There are many fewer places that decide struct layout and allocation than there are things that walk structs, which means we should favor choice 2. (Another point in favor of choice 2: choice 1 may require changes to code we don't even control; what if something like goprotobuf broke these rules?)

In #9384, Keith proposed a new type flag to indicate that the memory footprint ends in a zero-length field, so that allocation would add just a little more room after that. That would probably work, but given the previous argument, I think that's more subtle than this problem warrants. I want something that's dead simple. I worry that if new(T) sometimes (but almost never) allocates > T.size bytes, that will create a space for bugs.

I propose the dumber solution that if a non-zero-sized struct ends in a zero-length field, we add 1 byte of padding to end of the struct before aligning the final struct size. That is, given:

type T1 struct {
    X byte
    Y [0]byte
}

type T2 struct {
    X uint32
    Y struct{}
}

T1 would have size 2 (= 1 + 0 + 1) and T2 would have size 8 (= 4 + 0 + 1 + padding to 4-byte alignment for field X).

In cases where this padding is a problem, it can be avoided by manual rearrangement or removal of the zero-length fields, as Keith did in the fix for #9384.

I think all the things making structs at runtime have to build gcProgs, and there are only two mentions of gcProg in package reflect: bucketOf and funcLayout. bucketOf is already fixed. funcLayout could just add regSize to the total size any time the total size is not 0. This would also make safe the use of frame+retOffset when functions have arguments but no results.

[randall77]

That sounds like a reasonable plan to me.
I curious about whether we need to do anything for chan struct{}. I don't
think we actually materialize a pointer to the struct elements in this
case, but I'm not sure. I think we set the elem pointer to point back to
the channel header.

On Mon, Dec 22, 2014 at 10:06 PM, Russ Cox [email protected] wrote:

It seems like there are two options.

1. Fix the code manipulating the struct memory never to point past the
   end.
2. Add more memory to the struct so that all the existing manipulation
   code becomes correct.

I believe that since zero-size types are excluded from the problem, we've
narrowed the problem to very rare cases. The best solution is probably the
one that introduces the fewest new corner cases where bugs can hide. There
are many fewer places that decide struct layout and allocation than there
are things that walk structs, which means we should favor choice 2.
(Another point in favor of choice 2: choice 1 may require changes to code
we don't even control; what if something like goprotobuf broke these rules?)

In #9384 #9384, Keith proposed a new
type flag to indicate that the memory footprint ends in a zero-length
field, so that allocation would add just a little more room after that.
That would probably work, but given the previous argument, I think that's
more subtle than this problem warrants. I want something that's dead
simple. I worry that if new(T) sometimes (but almost never) allocates >
T.size bytes, that will create a space for bugs.

I propose the dumber solution that if a non-zero-sized struct ends in a
zero-length field, we add 1 byte of padding to end of the struct before
aligning the final struct size. That is, given:

type T1 struct {
X byte
Y [0]byte
}

type T2 struct {
X uint32
Y struct{}
}

T1 would have size 2 (= 1 + 0 + 1) and T2 would have size 8 (= 4 + 0 + 1 +
padding to 4-byte alignment for field X).

In cases where this padding is a problem, it can be avoided by manual
rearrangement or removal of the zero-length fields, as Keith did in the fix
for #9384 #9384.

I think all the things making structs at runtime have to build gcProgs,
and there are only two mentions of gcProg in package reflect: bucketOf and
funcLayout. bucketOf is already fixed. funcLayout could just add regSize to
the total size any time the total size is not 0. This would also make safe
the use of frame+retOffset when functions have arguments but no results.

—
Reply to this email directly or view it on GitHub
#9401 (comment).

[randall77]

This looks a bit tricky. We have several structs like the following in the runtime:

type interfacetype struct {
typ _type
mhdr []imethod
m [0]imethod
}

The [0] is really a placeholder for a variable-length array to follow.

Then to get the address of the nth imethod in interfacetype i, we do:
add(unsafe.Pointer(i), unsafe.Sizeof(interfacetype{}) + n * unsafe.Sizeof(imethod{}))

That math is all wrong if we add a byte and then round the size of interfacetype.
(Why don't we use &i.mhdr[n] instead of that math? I'm not sure. Maybe just to avoid the bounds check? Could do &i.m + n * ... also.)

Other structs which use the [0] at the end trick include m, g, stackmap, itab, and uncommontype.

It is possible that users depend on this trick as well.
I'm thinking the add-a-byte at malloc time for objects with trailing zero fields is the safer way to go. We already round up to size classes so I don't think any code would be surprised by the allocated size being slightly larger than the requested size.

[gopherbot]

CL https://golang.org/cl/12864 mentions this issue.

[gopherbot]

CL https://golang.org/cl/15660 mentions this issue.

[gopherbot]

CL https://golang.org/cl/18542 mentions this issue.

[gopherbot]

CL https://golang.org/cl/33475 mentions this issue.