proposal: expanding interface function signature matching

[rfliam]

Shortly

Rather than interfaces matching strictly on the function signature, they would also consider if a signature's arguments, or returns implement an interface for a match.

Or in other words, make this code example compile and work:

package main

import (
    "fmt"
    "io/ioutil"
)

type SomeReader struct {
}

func (sr *SomeReader) Read(p []byte) (in int, err *ErrorSomeReader) {
     return 0, &ErrorSomeReader{}
}

type ErrorSomeReader struct {
}

func (esr *ErrorSomeReader) Error() string {
     return "SomeReaderError"
}

func main() {
       sr := &SomeReader{}
        _, err := ioutil.ReadAll(sr)
    fmt.Println("Error", err)
}

In today's go an interface matches a function only if the full signature matches exactly. So the above fails with the error:

cannot use sr (type *SomeReader) as type io.Reader in argument to ioutil.ReadAll:
    *SomeReader does not implement io.Reader (wrong type for Read method)
        have Read([]byte) (int, *ErrorSomeReader)
        want Read([]byte) (int, error)

However, *ErrorSomeReader implements error and thus *SomeReader seems like it should be an io.Reader, but is not.

This example is usefull to understand how this would work, but not why it should be added.

The Rationale

The real reason to add this arises when you have interfaces defined relative to other interfaces. Consider some code for doing verification of access with users and groups:

package verify

type User interface {
    Group() Group
}

type Group interface {
    Name() string
    PermittedPaths() []string
}

func VerifyAccess(u User, path string) (string, bool) {
    for _, g := range u.Groups() {
        for _, p := range g.PermittedPaths() {
            if p == path {
                return p.Name(), true
            }
        }
    }
    return "", false
}

Because verify.User is defined relative to verify.Group, any use of package verify must also now define itself with a direct tie into verify. Even when a verify.User isn't what they want. Callers probably have a much richer defintion of the group returned by a user. This breaks one of the nicest things about Go's interfaces: that packages can simply define what they need of an object without placing restrictions on a caller.

With this change any type with a method Group() returning any type with the methods Name() string, PermittedPaths() []string can be used by verify.VerifyAccess. A user of this package does not need to define it's almost undoubtedly richer user and group with the restricted view required by package verify.

With this change packages ask for behaviors of a more complicated nature without tying implementers to a limited view of the world. Thus we can write code which captures more generic and common sets of operations. Code becomes automatically more composable (in the functional sense) as any object which implements desired behaviors is permitted. And we don't loose any type safety. Finally it may encourage returning more "bare" types and less spurious interfaces.

This isn't totally new in realm of language design either. I believe Hakell's TypeClasses allow similar expressions.*

*I am not a Haskell expert by any stretch of the imagination.

The Downsides

Just some obvious ones:

1. It is less immediately obvious if a type implements an interface.
2. This might be difficult to understand? I certainly had a bit of trouble reaching a succinct explanation of it.
3. Compiler complexity considerations. I have no idea how hard this would be to actually implement.

Timeline

This would obviously probably fall into a Go 2.0, though such a change would explicitly not break any currently compiling code.

[ianlancetaylor]

You need to discuss implementation. Quite apart from whether this change is desirable or not, it is difficult to implement.

[minux]

It's easy to implement if we explicitly label implemented interface for each type, but that's not how Go works. You need to consider how to generate wrapper functions for yet unknown interface methods.

[bradfitz]

Changing the type system is out of scope for Go 1, and we're not accepting proposals for a hypothetical Go 2 yet.

I've flagged this for Go 2 (many people have type system desires), but there are no plans for a Go 2 at the current time.

As this is a proposal, I'll let @adg handle resolution of this issue.

[rfliam]

@ianlancetaylor You need to discuss implementation. Quite apart from whether this change is desirable or not, it is difficult to implement.

@minux It's easy to implement if we explicitly label implemented interface
for each type, but that's not how Go works.

You need to consider how to generate wrapper functions for yet
unknown interface methods.

A Naive Implementation

A naive and trivial implementation is to do this the same way we would as software engineers, an adapter pattern. We might write an adapter this way in go:

package main

import (
    "fmt"
    "io/ioutil"
)

type SomeReader struct {
}

func (sr *SomeReader) Read(p []byte) (n int, err *ErrorSomeReader) {
    return 0, &ErrorSomeReader{}
}

type ErrorSomeReader struct {
}

func (esr *ErrorSomeReader) Error() string {
    return "SomeReaderError"
}

type SomeReaderAdapter struct {
    SomeReader *SomeReader
}

func (sra SomeReaderAdapter) Read(p []byte) (n int, err error) {
    return sra.SomeReader.Read(p)
}

func main() {
    sr := &SomeReader{}
    _, err := ioutil.ReadAll(SomeReaderAdapter{sr})
    fmt.Println("Error", err)
}

If instead the compiler generated the adapting types for us a lot of "boiler plate" for this would simply disappear. Technically the act of generating these adapters isn't that difficult. Finding a truly efficient implementation may be.

Detecting when to generate these is very close to the same problem as detecting when to "cast" or "create" an interface for an object.

@bradfitz Changing the type system is out of scope for Go 1, and we're not accepting proposals for a hypothetical Go 2 yet.

Another way to view this is not a fundamental change to the type system, but the automated generation of adapters. And as I mentioned, this does not break compatibility with Go1 code (unlike most Go2 suggestions). I of course realize that it is a larger change then Go1 has done to date, and could be treated simply "too large".

[minux]

How could the compiler statically determine the set of "adapters" (we call them "wrapper" methods) to generate? For example, if *ErrorSomeReader implements N methods, there will be 2^N interfaces that it could satisfy, and note that the user can type assert any such interface to another at runtime and in general, the compiler can't determine which interfaces are needed, so the wrappers can't be statically generated.

[rfliam]

There are 2^N interfaces a type could satisfy, but there are a fixed number of interfaces in a system. In a given codebase you don't have to consider all possible interfaces, just the interfaces defined in the system. At most you have to generate | T | * | I | wrappers, with T being the set of types and I the set of interfaces. For the entire 750kloc go source code base there are ~2500 interfaces, so its practical to brute force search and generate that space (but certainly not efficient).

I admit I hadn't considered the runtime type assertion aspect since I avoid it like the plague. As I mentioned above its not impossible to generate a wrapper for every interface in a system a type could satisfy, just inefficient. However, the same code/methodology can be used to generate those warper functions/types at runtime that are type asserted instead of assigned to yes?

[minux]

How could the compiler (when compiling a single package) know the full
set of interface used in a system?

And runtime code generation is out of the question because of the security
implications (and also some platforms simply forbid runtime code generation,
e.g. iOS)

Not to mention that there are proposals to add reflect.MakeInterface (#4146),
so even if we could somehow let compiler generate wrappers for actual
interfaces present in a program, we can't add reflect.MakeInterface.

[adg]

Officially leaving this open as a "Go2" feature request, as it has come up before.
It is not a suitable change for Go1.

On a personal note, I am not inspired by the examples in this proposal. In particular, the example showing returning an error is a classic example of bad style (you should always return error values, not values of concrete types that implement error). This weakens the proposal as a whole.

[rfliam]

@adg On a personal note, I am not inspired by the examples in this proposal. In particular, the example showing returning an error is a classic example of bad style (you should always return error values, not values of concrete types that implement error). This weakens the proposal as a whole.

That is a good point, perhaps I should alter the example not to use error. It was simply a very simple illustration of the idea. That said, the standard library seems to strongly prefer baretypes for everything but error, and as I mentioned this would encourage that practice.

Here is another, perhaps more compelling example based how we use container/heap today.

package main

import (
    "container/heap"
    "fmt"
)

// An IntHeap is a min-heap of ints.
type IntHeap []int

func (h IntHeap) Len() int            { return len(h) }
func (h IntHeap) Less(i, j int) bool  { return h[i] < h[j] }
func (h IntHeap) Swap(i, j int)       { h[i], h[j] = h[j], h[i] }
func (h *IntHeap) Push(x interface{}) { *h = append(*h, x.(int)) }
func (h *IntHeap) Pop() interface{} {
    old := *h
    n := len(old)
    x := old[n-1]
    *h = old[0 : n-1]
    return x
}

func main() {
    var h IntHeap
    heap.Init(h)
    heap.Push(h, 1)
    heap.Push(h, 2)
    fmt.Println("Hello world: ", heap.Pop(h))

    // TYPE SAFTEY HAS ABANDONED ME!
    h.Push("1")
    fmt.Println("Hello world: ", heap.Pop(h))
}

YIKES! While a viable implementation all of our beautiful type safety just went out the window.

Now compare this with changing the interface matching.

package main

import (
    "container/heap"
    "fmt"
)

// An IntHeap is a min-heap of ints.
type IntHeap []int

func (h IntHeap) Len() int           { return len(h) }
func (h IntHeap) Less(i, j int) bool { return h[i] < h[j] }
func (h IntHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }
func (h *IntHeap) Push(x int)        { *h = append(*h, x) }
func (h *IntHeap) Pop() int {
    old := *h
    n := len(old)
    x := old[n-1]
    *h = old[0 : n-1]
    return x
}

func main() {
    var h IntHeap
    heap.Init(h)
    heap.Push(h, 1)
    heap.Push(h, 2)
    fmt.Println("Hello world: ", heap.Pop(h))

    // NO PROBLEM, WON'T COMPILE!
    // But since int is an interface{}, we can compile the this once the below line is commented out.
    h.Push("1")
    fmt.Println("Hello world: ", heap.Pop(h))
}

We are not getting generics or anything like that, but we are at least a little more typesafe.

I will get to answering minux's questions asap.

[minux]

Even if we somehow implements this proposal, I don't think heap.Push(h, "1") will be flagged as a compile error (because heap.Push takes an interface{} as the 2nd argument.) Now I realize the implementation difficulty is even bigger than I previously described. Because the method could take multiple arguments and return multiple arguments, each one is subject to interface conversion! Even if we could devise a way to get around the problem of generating all the wrapper for each interface conversion (we need at least two, eface and iface case), for N-argument methods, we still need 2^N wrapper functions.

[rsc]

On a procedural note, I don't want to use the issue tracker and the proposal process to plan Go 2. We have enough work keeping up with Go 1, and we haven't even begun to think about considering possibly planning Go 2. I am closing this issue so that it does not appear in pending proposals, but I'm leaving the Go2 and Proposal labels so we can find the issue if we ever do get to Go 2.

As for the actual content of the proposal, Ian noted that it is difficult to implement. But on top of that, it has a serious semantic problem. Andrew noticed the concrete problem with the error example, but it's not just a style nit. It's actually a complete showstopper.

Let's suppose we have:

type I interface {
    F(X) Y
}

type T whatever

func (T) F(X1) Y1 { ... }

var t T
var x X
var i I = t // proposal magic happens here
var y Y = i.F(x) // or maybe here

It must be that those last two lines of code would behave as the one line:

var y Y = Y(t.F(X1(x)))

I've written the conversions from X to X1 and from Y1 to Y explicitly but of course Go does not require them to be written in code like this. The conversions imply that X1 and Y must be interface types. X and Y1 may or may not be.

Suppose Y is interface{} and Y1 is int and t.F returns 0. Then y ends up being interface{}(0), which seems about right.

Suppose Y is error and Y1 is _MyError and t.F returns (_MyError)(nil). Then y ends up being a non-nil error containing (*MyError)(nil). (See the FAQ entry Andrew mentioned if this does not ring alarm bells.) So when T satisfies I in this case, it cannot return a successful result. It always returns an error. One could of course introduce an exception for typed nil pointers converting to interface{}, but why is the type lost in that one case? Why is a zero pointer different from a zero int? And we surely can't change the way existing interface conversions work. Why is the magic due to interface conversion different from an explicit interface conversion?

The same narrowing of possibility happens for the arguments, and again it might be a problem. If the interface says F(*MyError) but T's method is F(error), again you have the problem that F may need to be called with a nil error and cannot be. It's less of a concern in the argument position perhaps. But it is still likel a real problem.

This issue with nil does not happen if you require that all of X, X1, Y, Y1 are interface types. Then the conversion from nil of one interface type to nil of another interface type works as expected. But that likely restricts the utility of the change. It does not allow the Read example.

I think this is the showstopper.

But then I think there are also other semantic issues and circularities in even deciding whether one recursively defined interface is to be considered to implement a second one. I'm not going to say more about this problem but I think it's there.