omitempty has no effect when node complexity exceeds inlining threshold

[saj]

Thank you for publishing this lovely library. :)

This bug smells similar to #376. It is so similar that I did double check I was on trunk before sending this bug report.

% dlv exec $(command -v msgp)
Type 'help' for list of commands.
(dlv) sources
[...]
/Users/saj/Documents/src/go/pkg/mod/github.com/tinylib/[email protected]/gen/decode.go
/Users/saj/Documents/src/go/pkg/mod/github.com/tinylib/[email protected]/gen/elem.go
/Users/saj/Documents/src/go/pkg/mod/github.com/tinylib/[email protected]/gen/encode.go
[...]

---

The following input works as expected.

package main

//go:generate msgp

type Flibbity struct {
	A Flibs `msg:"a,omitempty"`
	B Flibs `msg:"b,omitempty"`
}

type Flibs []Flib

type Flib struct {
	X int8 `msg:"x"`
	Y int8 `msg:"y"`
}

Flibbity.MarshalMsg includes the omitempty tests.

	// check for omitted fields
	zb0001Len := uint32(2)
	var zb0001Mask uint8 /* 2 bits */
	_ = zb0001Mask
	if z.A == nil {
		zb0001Len--
		zb0001Mask |= 0x1
	}
	if z.B == nil {
		zb0001Len--
		zb0001Mask |= 0x2
	}
	// variable map header, size zb0001Len

---

The following input demonstrates the problem.

package main

//go:generate msgp

type Flobbity struct {
	A Flobs `msg:"a,omitempty"`
	B Flobs `msg:"b,omitempty"`
}

type Flobs []Flob

type Flob struct {
	X Numberwang `msg:"x"`
	Y int8       `msg:"y"`
}

type Numberwang int8

Flobbity.MarshalMsg is missing the omitempty tests.

	// check for omitted fields
	zb0001Len := uint32(2)
	var zb0001Mask uint8 /* 2 bits */
	_ = zb0001Mask
	// variable map header, size zb0001Len

Curiously, if we remove either one of A or B from the type definition of Flobbity, the second example also begins to work as expected.

Here is a repository with the sources above.
https://github.com/saj/tinylib-msgp-381

[saj]

type Flobbity struct {
	A Flobs `msg:"a,omitempty"`
	B Flobs `msg:"b,omitempty"`
}

type Flobs []Flob

It would seem that the problem is that the Flobs type alias (this is a slice!) is only recognised if Flobs is inlined.

When we bung in the Numberwang, we exceed the inlining complexity threshold by coincidence, and the slice typing information appears to be lost.

I will try to rephrase the bug subject line.

[klauspost]

Looked a bit back and forth, and it seems like the simplest solution is also the best - don't consider the cost of the children when deciding if these aliases types should be inlined.

Proposed fix in #382