testing: show diffs for incorrect output from Example tests

[komuw]

What version of Go are you using (go version)?

$ go version
go version go1.15.1 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/home/Library/Caches/go-build"
GOENV="/Users/home/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/home/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/home/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/home/mystuff/kama/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/gc/sfs6hvtd1r392kn91n3jp17m0000gn/T/go-build416873732=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Created test example (https://golang.org/pkg/testing/#hdr-Examples, https://blog.golang.org/examples).
A reproducer of the issue is at https://play.golang.org/p/969hijP9sMx

What did you expect to see?

That example test is failing because there is a mismatch between what is been output to stdOut and what the test runner expects the output to be.
Unfortunately for test examples(unlike normal tests), I'm not able to programmatically get the test output so that I can examine(via diffing or otherwise) it and compare it to the expected result.

What did you see instead?

I do not have a way to access(programmatically) the output of the test.

[komuw]

The only way I was able to achieve what I want is via the following bad hack;

diff --git a/src/testing/run_example.go b/src/testing/run_example.go
index 10bde49e5b..b1c86f54d2 100644
--- a/src/testing/run_example.go
+++ b/src/testing/run_example.go
@@ -16,6 +16,8 @@ import (
        "os"
        "strings"
        "time"
+
+       "github.com/google/go-cmp/cmp"
 )
 func runExample(eg InternalExample) (ok bool) {
@@ -54,6 +56,12 @@ func runExample(eg InternalExample) (ok bool) {
                os.Stdout = stdout
                out := <-outC
+               got := strings.TrimSpace(out)
+               want := strings.TrimSpace(eg.Output)
+               diff := cmp.Diff(want, got)
+               fmt.Println("diff:")
+               fmt.Println(diff)
+
                err := recover()
                ok = eg.processRunResult(out, timeSpent, err)
        }()

[dmitshur]

I do not have a way to access(programmatically) the output of the test.

Have you considered the -json flag of go test? For example:

{"Time":"2020-10-14T20:22:16.736222-04:00","Action":"output","Package":"m.test","Test":"ExampleHello","Output":"Hello \n"}
...
{"Time":"2020-10-14T20:22:16.736237-04:00","Action":"output","Package":"m.test","Test":"ExampleHello","Output":"Hello\n"}

[komuw]

Have you considered the -json flag of go test?

I guess that would work somehow. But I would have to run go test using another program that parses that output and compares the got and wanted parts of the output.

[bcmills]

As another alternative, you could use os/exec to run os.Args[0] as a subprocess, to either run the generated ExampleHello test or invoke the original ExampleHello function directly. (See https://play.golang.org/p/T4fDo8D5M0H, but note that it doesn't work in the Playground due to #41339.)

You could either extract the want: output by parsing the output of the subprocess, or use go/doc.Examples to extract it programmatically.

[bcmills]

Come to think of it, it would be pretty straightforward to encapsulate the approach from the above comment as an external library. The API could look something like:

// RegisterExamples registers the given example functions so that they can be run by test functions.
//
// If this process is a subprocess started by ExampleOutput, RegisterExample
// runs the requested example function and then exits with status 0.
func RegisterExamples(m *testing.M, examples ...func())

// ExampleOutput returns the output of the given example function,
// obtained by executing the test executable as a subprocess.
//
// The function must have been passed to RegisterExamples in TestMain.
//
// If the current platform cannot execute subprocesses, RunExample skips t.
func RunExample(t testing.TB, example func()) (stderr, stdout string, err error)

// ExampleOutput returns the output of example, which must be an Example function
// recognized by 'go test'.
func ExampleOutput(t testing.TB, example func()) (want string)

[rsc]

There is a trap here, namely adding infinite new API to package testing as individual needs come up.
We need testing to do a limited amount of things. It's already getting hard to remember everything.

Looking at https://play.golang.org/p/969hijP9sMx, the issue I see is not that testing needs new API but instead that the example test output needs to do a better job highlighting the differences, such as by showing a diff. Maybe we should do that instead?

[komuw]

Looking at https://play.golang.org/p/969hijP9sMx, the issue I see is not that testing needs new API but instead that the example test output needs to do a better job highlighting the differences, such as by showing a diff. Maybe we should do that instead?

That would be excellent.

[rsc]

Retitled for the suggestion I made last week, namely show diffs.
Does anyone object to that?

[mvdan]

What diff package would we use for this? gofmt -d just relies on the system's diff program, but I don't think that's a good idea for testing.

@josharian has been working on https://github.com/pkg/diff, so perhaps that could be vendored as an internal package.

[rsc]

We can find a reasonable diff package. I have a few sitting around too.

[rsc]

Based on the discussion, this seems like a likely accept.

[rsc]

No change in consensus, so accepted.

[gopherbot]

Change https://golang.org/cl/301589 mentions this issue: testing: add diffing to clearly discern Example test failure output

[mvdan]

I've just filed #51254, which is a pretty similar proposal, FYI :)