proposal: reflect: add method Type.Underlying()

[cosmos72]

Introduction

The reflect package provides at runtime most - but not all - the possible operations on Go types.

As such, it is extremely useful for advanced Go programs that manipulate types, allowing in many cases to forgo code generation and/or ugly workarounds that use interface{} instead of concrete types. It's also immensely useful for existing Go interpreters, which use it heavily.

Some of the missing features are currently very difficult to implement. Some examples:

• creating new types with methods: @crawshaw reflect: NamedOf #16522
• some edge cases of StructOf: reflect: StructOf doesn't generate wrapper methods for embedded fields #15924 reflect: StructOf should support embedding types with non-exported methods #20015 reflect: StructOf forbids anonymous fields without name #24781 reflect: permit unexported fields in StructOf #25401 reflect: can't call methods on StructOf with embedded interface field #38783

Other features, while relatively easier to implement, are still complicated - for example creating new named and/or recursive types without methods: @TheCount proposal: reflect: allow creation of recursive types #39528

One of the missing features is "getting the underlying Type of a Type":
it's a relatively smaller feature and, if I had to guess, comparably easier
to implement. Yet it would be very useful for advanced usages: especially, but not only, for Go interpreters.

To give a practical example, my interpreter gomacro contains a whole package xreflect (~7000 lines of code) to emulate the missing features of reflect.
Some hundreds of them are dedicated to work around the absence of reflect.Type.Underlying

Proposal

The proposal itself is very simple:
this issue proposes an additional method of reflect.Type:

type Type interface {
   // ...existing methods...

   // Underlying returns a type's underlying type.
   Underlying() Type
}

About the implementation:

it could be implemented similarly to the function reflect.PrtTo:

• first, look in an additional field underlying typeOff inside the struct reflect.rtype and, if non-zero, return it
• second, look for known types. For example, the underlying type of a basic type or an unnamed type is the type itself.
• third, check a dedicated cache underlyingMap
• fourth, create it by cloning the rtype and removing the name and (for non-interfaces) also the methods
  Limitation: an underlying struct type may have methods due to embedded fields, but they could be ignored - at least in a first version.

[ianlancetaylor]

I don't think that a new underlying field is practical. We've put a great deal of effort over the years into reducing the amount of space required for type reflection information. An underlying field would seem to force us to duplicate the rtype (though not the uncommontype) for every user defined type. That would be a heavy cost for a functionality that few people are asking for. Of course this doesn't mean that we couldn't use some different implementation approach.

Can you explain when this feature is needed? In general the reflect package tries to provide the functionality that is available in the language. As far as I can tell the language doesn't give you any way to discover the underlying type of some value. You can convert to the underlying type if you happen to know it, but you can't discover the underlying type. Am I missing something?

[cosmos72]

I don't think that a new underlying field is practical. [...]

The reason is clear, thanks. It can be omitted, and the other the steps would still suffice.

Can you explain when this feature is needed? In general the reflect package tries to provide the functionality that is available in the language. As far as I can tell the language doesn't give you any way to discover the underlying type of some value. You can convert to the underlying type if you happen to know it, but you can't discover the underlying type. Am I missing something?

My answer is simple: Go specs do provide a way to use (not discover) the underlying type U of some type T - by defining another type. Example:

type T /*...*/ // underlying type U not directly known/available to user code

type T2 T // underlying type of T2 will be U, thus equivalent to "type T2 U"

Appropriately, the main use case of Type.Underlying() is to properly implement the proposal reflect.Named #39528. To be more precise, the only reasonable semantic for the proposed functions

func Bind(named Type, underlying Type, methods []Method) /*...*/

or

func SetUnderlying(named Type, underlying Type)

is: the incomplete 'named' type is completed, by setting its underlying type to 'underlying', which must not be a defined type.

This is exactly the same semantics as the method SetUnderlying() of go/types/Named and, in order to be able to use it with arbitrary types, callers need a way to get the underlying type of an arbitrary (thus possibly defined) type.

More concretely, the reflect equivalent of the definition type T /*...*/' type T2 T above is:

var T reflect.Type = /* ... */

var T2 = reflect.NewNamed("", "T2")
T2.SetUnderlying(T.Underlying())

Clearly, one could argue that the method Type.Underlying() could be unexported and only used internally by the implementation of Type.SetUnderlying().

But reflect is about discovering at runtime the properties of a type:
if the argument "the reflect package tries to provide the functionality that is available in the language" is interpreted in its strictest sense, then reflect should not provide an API to discover at runtime the fields and methods of a Value or Type - namely, the following should not exists:

type Type interface {
    Field(int) StructField
    NumField() int
    Method(int) Method
    NumMethod() int
}

and analogously for Value, because Go language specs do not provide a way to discover fields and methods: a caller can only use them by name, which means it must already know their names.

Instead, such discovery functionality is fundamental at runtime - it allows to write general code that discovers the fields and methods of an arbitrary type or value, and uses them.

My opinion is that the underlying type of a type is a property, analogous to the fields or methods of a type, and that reflect should analogously allow to discover it at runtime.
Thus Type.Underlying() should be exported, and not merely an unexported feature used by Type.SetUnderlying()

[ianlancetaylor]

That's a fair point about types like NumField and Field. But Underlying still seems different to me, in that you can already determine all information about the underlying type through the existing mechanisms. The only thing you can't easily do is get a reflect.Type for the underlying type--though you can still do that in most cases using the other constructors.

Is there a use for Underlying other than to create a new named type?

[rsc]

To give a practical example, my interpreter gomacro contains a whole package xreflect (~7000 lines of code) to emulate the missing features of reflect.

What are those missing features?

If the missing features are only NamedOf and StructOf, we should probably be discussing whether and how to implement them inside reflect, not what APIs to provide to make them possible outside reflect. Inside reflect, we can do anything we want, without defining new API.

It's unclear why either NamedOf or StructOf needs Underlying when implemented inside reflect. (And in any event, they'd have access to unexported fields and the like.)

I'm just trying to understand better specific cases where Underlying would be used other than implementing more parts of reflect.

[cosmos72]

What are those missing features?

Here's a list of the missing features of reflect that I had to emulate, ordered from the hardest to the easiest:

• reflect.InterfaceOf: there is no way to create new interface types at runtime. The workaround is really complicated, because it needs not only an alternative representation for values of such interface types, but also needs to emulate conversions from/to such interface types, and type assertions.
• reflect.NamedOf (1/3): there is no way to add methods to types created at runtime, thus no way for them to implement interfaces (either compiled interfaces, or created at runtime - see previous point)
• reflect.NamedOf (2/3): there is no way to create recursive types at runtime, thus some parts of them must be approximated with interface{} or similar. For example, type IntList { Head int; Tail *IntList } is actually implemented as type IntList { Head int; Tail interface{} }
• reflect.NamedOf (3/3): there is no way to create new defined types at runtime
• reflect.StructOf does not support certain cases. Last time I checked, it did not allow unexported fields and had limitations on embedded fields (gomacro can be compiled with Go >= 1.9, so I did not check whether more recent versions of reflect package lifted some of the limitations)

If the missing features are only NamedOf and StructOf, we should probably be discussing whether and how to implement them inside reflect, not what APIs to provide to make them possible outside reflect. Inside reflect, we can do anything we want, without defining new API.

The the API I am proposing (reflect.Type.Underlying()) is not intended to provide a mechanism to implement some reflect features outside the package. It's just something that's useful for reflect internal implementation and, if exported, also a useful functionality for advanced users.

It's unclear why either NamedOf or StructOf needs Underlying when implemented inside reflect. (And in any event, they'd have access to unexported fields and the like.)

I'm just trying to understand better specific cases where Underlying would be used other than implementing more parts of reflect.

Well, the semantic of NamedOf and its companion func SetUnderlying(named Type, underlying Type), if they are ever implemented, must be defined precisely.
NamedOf is quite straightforward (except for some corner cases: should it accept the name of an already existing type?), while there are two main options for SetUnderlying:

1. named underlying type is set exactly to underlying, which must not be a defined type. This is what go/types.Named.SetUnderlying does, and requires a function similar to the proposed reflect.Type.Underlying to get the underlying type of an arbitrary type and pass it as SetUnderlying second argument.
2. named underlying type is set to underlying's underlying type. This does not require an exported method reflect.Type.Underlying, but does require some design inside reflectthat achieves the same effect. I have a draft implementation of NamedOf and SetUnderlying along these lines, and it amounts to ignoring the name, package and methods of underlying, while using its other properties.

[ianlancetaylor]

If we don't need Underlying outside of the reflect package, then we shouldn't add it. The reflect package can use an internal underlying method if that seems useful, but that doesn't need a proposal.

[cosmos72]

Well, in the end it is a design choice. If Type.Underlying is not added, it means that the proposed functions
func SetUnderlying(named Type, underlying Type)
or
func Bind(named Type, underlying Type /*...*/)
see proposal: reflect: allow creation of recursive types #39528 - if implemented, must have the semantic 2. listed above:

2. named underlying type is set to underlying's underlying type. This does not require an exported method reflect.Type.Underlying, but does require some design inside reflect that achieves the same effect.

[ianlancetaylor]

SetUnderlying and Bind don't yet exist. It would be premature to add Underlying by itself. We might wind up not adding SetUnderlying and Bind at all.

If that is the main reason for this proposal, then it should be folded into some other proposal, and not stand by itself. Thanks.

[TheCount]

@cosmos72 FWIW, my alternative proposal #39717 does have Underlying in a slightly different context.

[rsc]

Based on the discussion above, this seems like a likely decline.
(There's no need for this by itself, and there are other larger proposals to resolve first.)

[cosmos72]

I agree it's an addition mostly useful together with proposal: reflect: allow creation of recursive types #39528.
Since #39528 has been closed, we can close this too.

We will see if the final proposal to allow creating recursive reflect.Type (if any such proposal is accepted) needs Type.Underlying() or not - and in case I can open another proposal suggesting Type.Underlying() again.

[gonzojive]

I'm only loosely following the discussion above. Below is my use case that may be relevant - I can't tell. For the definitions type T U; type U struct{...}, is it possible to get the object reflect.TypeOf(U{}) from a reflect.TypeOf(T{})? (Edit: I see per the first few comments that this isn't directly possible today.)

More on my use case: I'm implementing a CSV parsing package that has a function RegisterCoder(t reflect.Type, decoder interface{}) where decoder is expected to look like func(string, *T) error or func(string, T) error. There are default implementations of coders for the basic types like int64, etc. I would like for any type T with an underlying type U to use U's registered coder, if one exists and no explicit coder is defined for T.

[cosmos72]

The definitions

type U = struct{...} // type alias
type T U;

create two types: the named type T which has the underlying type struct {...}. And U is just an alias for struct {...}

Given T, the package reflect does not currently provide any mechanism to get U or struct {...} - this exactly the addition I am proposing.

To be precise, there is a partial workaround: one could detect that reflect.TypeOf(T{}) has kind = reflect.Struct, extract its fields one by one, then pass them to reflect.StructOf() to rebuild the type struct {...} - but this both ad-hoc (i.e. relies on examining the type internals and doing different things for different reflect.Kinds) and has many pitfalls, because reflect.StructOf() has limited support for unexported fields and embedded fields.

Thus your use case would significantly benefit from this proposal - without it, from a type T you cannot easily get its underlying type struct {...} and then check if there's a registered code for it. The only direct thing you can do with reflect is asking if a type T is convertible to another type U using reflect.Type.ConvertibleTo() - but that would require looping on all the coders registered into your API, until you find a compatible one: possibly an expensive operation.

[gonzojive]

Thanks @cosmos72 for the explanation.

Regarding the above comment by @ianthehat:

I don't think that a new underlying field is practical. We've put a great deal of effort over the years into reducing the amount of space required for type reflection information. An underlying field would seem to force us to duplicate the rtype (though not the uncommontype) for every user defined type. That would be a heavy cost for a functionality that few people are asking for. Of course this doesn't mean that we couldn't use some different implementation approach.

I don't know Go's internals at all. From afar it seems like an extra word of storage for each type or less if cleverly encoded. Maybe you can elaborate on the storage cost if it is a concern.

Can you explain when this feature is needed? In general the reflect package tries to provide the functionality that is available in the language. As far as I can tell the language doesn't give you any way to discover the underlying type of some value. You can convert to the underlying type if you happen to know it, but you can't discover the underlying type. Am I missing something?

In practice, reflect is often used when the language isn't expressive enough. We use the reflect package to see what the programmer usually sees. In a sense, the language already offers a way to discover the underlying type of T - by reading its definition.

[cosmos72]

I don't know Go's internals at all. From afar it seems like an extra word of storage for each type or less if cleverly encoded. Maybe you can elaborate on the storage cost if it is a concern.

The need for additional storage can be removed.

My implementation outline above has four steps: the first three are basically three different caches: useful, but only the third one is strictly necessary - to guarantee that reflect.Type is canonical i.e. there is only one instance of each type.

The additional field is the first of such caches, and it can be safely omitted. This is important because, as @ianlancetaylor pointed out, Go tries extremely hard to minimize the space needed by reflect.Type

We use the reflect package to see what the programmer usually sees. In a sense, the language already offers a way to discover the underlying type of T - by reading its definition.

That's a much clearer explanation than mine. Thanks!

[gonzojive]

Thus your use case would significantly benefit from this proposal - without it, from a type T you cannot easily get its underlying type struct {...} and then check if there's a registered code for it. The only direct thing you can do with reflect is asking if a type T is convertible to another type U using reflect.Type.ConvertibleTo() - but that would require looping on all the coders registered into your API, until you find a compatible one: possibly an expensive operation.

Agreed. I tried using ConvertibleTo(), and it's not great semantically. In the case of encoding a type Distance with underlying type float64, it is reasonable to use float64's registered encoder if none is registered for Distance. However, lots of types have an underlying type float64, and it would be odd to use the encoder for, say, Angle, to decode Distance. The two are convertible to each other but are quite different from a type hierarchy perspective. This is where Underlying() is needed in the encoding use case.

[rsc]

No change in consensus, so declined.

[gonzojive]

Is there a workaround for this since the reflect API will not be expanded?

…

On Wed, Jul 8, 2020 at 10:07 AM Russ Cox ***@***.***> wrote: No change in consensus, so declined. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#39574 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUO54KSVCKMLUUF4A5OQDR2SRTLANCNFSM4N46GNXA> .

[cosmos72]

There is a partial workaround, i.e. create manually the underlying type with the existing reflect API, as in the code below.

There is no way to get the underlying type of a named interface type,
and certain cases involving structs with embedded or unexported fields are not supported.

package reflect_underlying

import (
	r "reflect"
	"unsafe"
)

var basic = []r.Type{
	r.Bool:          r.TypeOf(false),
	r.Int:           r.TypeOf(int(0)),
	r.Int8:          r.TypeOf(int8(0)),
	r.Int16:         r.TypeOf(int16(0)),
	r.Int32:         r.TypeOf(int32(0)),
	r.Int64:         r.TypeOf(int64(0)),
	r.Uint:          r.TypeOf(uint(0)),
	r.Uint8:         r.TypeOf(uint8(0)),
	r.Uint16:        r.TypeOf(uint16(0)),
	r.Uint32:        r.TypeOf(uint32(0)),
	r.Uint64:        r.TypeOf(uint64(0)),
	r.Uintptr:       r.TypeOf(uintptr(0)),
	r.Float32:       r.TypeOf(float32(0)),
	r.Float64:       r.TypeOf(float64(0)),
	r.Complex64:     r.TypeOf(complex64(0)),
	r.Complex128:    r.TypeOf(complex128(0)),
	r.String:        r.TypeOf(""),
	r.UnsafePointer: r.TypeOf((unsafe.Pointer(nil))),
}

func Underlying(t r.Type) (ret r.Type) {
	if t.Name() == "" {
		// t is an unnamed type. the underlying type is t itself
		return t
	}
	kind := t.Kind()
	if ret = basic[kind]; ret != nil {
		return ret
	}
	switch kind {
	case r.Array:
		ret = r.ArrayOf(t.Len(), t.Elem())
	case r.Chan:
		ret = r.ChanOf(t.ChanDir(), t.Elem())
	case r.Map:
		ret = r.MapOf(t.Key(), t.Elem())
	case r.Func:
		n_in := t.NumIn()
		n_out := t.NumOut()
		in := make([]r.Type, n_in)
		out := make([]r.Type, n_out)
		for i := 0; i < n_in; i++ {
			in[i] = t.In(i)
		}
		for i := 0; i < n_out; i++ {
			out[i] = t.Out(i)
		}
		ret = r.FuncOf(in, out, t.IsVariadic())
	case r.Interface:
		// not supported
	case r.Ptr:
		ret = r.PtrTo(t.Elem())
	case r.Slice:
		ret = r.SliceOf(t.Elem())
	case r.Struct:
		// only partially supported: embedded fields
		// and unexported fields may cause panic in r.StructOf()
		defer func() {
			// if a panic happens, return t unmodified
			if recover() != nil && ret == nil {
				ret = t
			}
		}()
		n := t.NumField()
		fields := make([]r.StructField, n)
		for i := 0; i < n; i++ {
			fields[i] = t.Field(i)
		}
		ret = r.StructOf(fields)
	}
	return ret
}

[ianlancetaylor]

@gonzojive It's very unlikely that we would ever support a mechanism for finding U given type T U. Even if did implement the Underlying method, for T it would return the underlying type of U, not U itself. If you want to build on methods that way, the normal Go approach is to use struct embedding, as in type T struct { U }.

[gonzojive]

Thanks @ianlancetaylor. Is there a philosophy behind not intending to support such a mechanism?

My supposition is that Go users should not make much of the chain of named types between a type and its underlying type. For example, type Distance float64; type Mass float64 should not be taken to mean "Distance can be converted to float64, but it shouldn't be treated as a Mass."

[ianlancetaylor]

In Go there are two main reasons to use a defined type: to prevent accidentally combining values of two different types, and to define methods on a type. When you write type A B, the newly defined type A does not have any of the methods of B. And of course it then requires an explicit conversion to convert between values of the two types (disregarding the case where they are interface types). In neither of those cases is there any special relationship between A and B. In fact in general writing type A B acts exactly the same as writing type A <underlying type of B>. So there is no obvious reason why the reflect package should permit recovering the fact that A was defined in terms of B rather than in terms of B's underlying type. And, as far as I can see, implementing such a feature would add a cost to all type descriptors for defined types, so we would need a significant benefit that outweighs that cost.