Formatting matcher output

[onsi]

Originally from onsi/ginkgo#73

This may already be implemented, I just haven't found it.

When I have an expectation like this:

Expect(something).ToNot(BeNil())

If it fails, it logs a deep inspection of something. I find that in many cases this leads to overwhelming output. For example, one of my structures includes a time.Time. This is (surprisingly) over 250 of output due to a massive DST lookup table that's included in its loc field. Some of my data structures include http or file handles that can generate thousands of lines of output (sometimes recursively, leading to hundreds of "too deep for me, man" messages).

I had assumed that I could implement Stringer or GoStringer to control the logging output, but Ginkgo doesn't seem to use these. Is there any way to get control over how complex structures are displayed?

[onsi]

Gomega's formatter does not simply print out Stringer/GoStringer because the representations for either of these are often insufficient to correctly diagnose problems.

It's actually funny that you picked time.Time as an example. It happens to exemplify this problem perfectly. Here's a test that passes on OS X, but fails on Linux )(:

package bar_test

import (
    "encoding/json"
    . "github.com/onsi/ginkgo"
    . "github.com/onsi/gomega"

        "fmt"
    "testing"
    "time"
)

func TestBar(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Bar Suite")
}

type Temporal struct {
    T time.Time
}

var _ = Describe("JSON", func() {
    It("should work", func() {
        original := Temporal{T: time.Now()}

        encoded, err := json.Marshal(original)
        Ω(err).ShouldNot(HaveOccurred())

        var hydrated Temporal

        err = json.Unmarshal(encoded, &hydrated)
        Ω(err).ShouldNot(HaveOccurred())

                fmt.Println(original, hydrated)
        Ω(hydrated).Should(Equal(original))
    })
})

All this is doing is JSON encoding a struct that contains a time.Time and then decoding it and asserting the original equals the decoded object. Apparently there's a wacky bug (feature?) in Go on Linux where the round trip results in an object that does not reflect.DeepEqual the original. Here's the output of the fmt.Println(original, hydrated) on linux:

{2014-05-24 19:40:30.57606596 +0000 UTC} {2014-05-24 19:40:30.57606596 +0000 UTC}

At first blush the values appear to be identical. If this is what Gomega showed you there'd be much confusion and no way to understand what was going wrong! Here's Gomega's formatted output:

  Expected
      <temporal_test.Temporal>: {
          T: {
              sec: 63536557230,
              nsec: 0x225611a8,
              loc: {name: "UTC", zone: nil, tx: nil, cacheStart: 0, cacheEnd: 0, cacheZone: nil},
          },
      }
  to equal
      <temporal_test.Temporal>: {
          T: {
              sec: 63536557230,
              nsec: 0x225611a8,
              loc: {
                  name: "Local",
                  zone: [
                      {name: "UTC", offset: 0, isDST: false},
                  ],
                  tx: [
                      {
                          when: -9223372036854775808,
                          index: 0,
                          isstd: false,
                          isutc: false,
                      },
                  ],
                  cacheStart: -9223372036854775808,
                  cacheEnd: 9223372036854775807,
                  cacheZone: {name: "UTC", offset: 0, isDST: false},
              },
          },
      }

As you can see the decoded object is missing the complex zone and tx fields... kinda crazy!

---

With all this said, I agree that Gomegas output can be overwhelmingly verbose at times. There isn't currently a way to change this but I would like to add one. I'm envisioning that you can instruct Gomega's format package to enter different verbosity modes. It will default to the current (high-verbosity) mode but you'll be able to do something like:

func TestBar(t *testing.T) {
       format.SetVerbosityLevel(format.Low)
    RegisterFailHandler(Fail)
    RunSpecs(t, "Bar Suite")
}

which would lead to much terser, but hopefully still useful for most cases, output.

Thoughts?

[onsi]

I've struggle a bit with what to do here. I think what I ended up with on 05c2b32 is a reasonable compromise.

You can now modify the (global) maximum recursion depth for the format package by setting format.MaxDepth -- this can help make deeply nested structs less verbose.

You can also opt into displaying Stringer and GoStringer strings instead of the recursive formatting by setting format.UseStringerRepresentation to true.

These are global and affect all output. I think between the two knobs you can get better control over your output.

Thoughts?

[bradfeehan]

Would it be possible to show the more verbose representation, but only if the less-verbose representations are identical? (vice-versa for inequality matches)

[khaf]

Sorry to revive such a long dead thread, but I think keeping the conversation in one thread is useful for the future readers.
There is a use case that I think is not covered by the current custom depth and stringer workarounds:
I have an object in my code that if dumped, it will generate a ~190MB text file! In general, I'd like to see the usual output (actually, for structs and maps, I'd rather see only a diff and not the whole dump, but I digress)
For this specific object, I'd rather implement a GomegaStringer interface to format it. In this case, I don't want to touch the format.MaxDepth and format.UseStringerRepresentation globally; I'd just rather implement the additional GomegaString() function in a test file as a helper only for this object. Do you think this a viable feature request?

[onsi]

I like the idea of a GomegaString convention. Would you be up for submitting a PR?

[khaf]

Absolutely.

[onsi]

Thanks! If you could also update the docs on the gh-pages branch that would be great too.

…

On Mar 17, 2021, at 8:31 AM, Khosrow Afroozeh ***@***.***> wrote:  Absolutely. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[khaf]

#427
I will make a PR for documentation after the review of this PR.

[khaf]

Now that I think of it, maybe it wouldn't be a bad idea to trim a big object's dump if it is bigger than a certain size - say around 2K characters - since it probably will not be useful anyway the way it is.

[onsi]

Agreed - I was thinking along similar lines. You up for adding a bit more to the PR (I'm currently heads down working on Ginkgo v2 and would appreciate the help making this a more pleasant user experience).

How about a new top-level global in format like format.MaxLength set to 4000 characters or something. If a representation exceeds MaxLength we truncate at MaxLength then inject this:

the very very long object here that goes on forever and ever but then gets trunc....

Gomega truncated this representation as it exceeds `format.MaxLength`. 
Consider having the object provide a custom `GomegaStringer` representation
or adjust the parameters in Gomega's `format` package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

to the end of the representation. (Unless the object has implemented GomegaStringer in which case we always emit the whole thing.

This way the user gets immediate helpful feedback in their test and they can go off and learn more. Thoughts?

[khaf]

100% agreed, even for the message! (I already even had the message, though yours is better English than mine)
Will make the PR in a bit.

[onsi]

thanks so much!

…

On Mar 17, 2021, at 10:54 AM, Khosrow Afroozeh ***@***.***> wrote:  100% agreed, even for the message! (I already even had the message, though yours is better English than mine) Will make the PR in a bit. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[khaf]

PR updated. I went the lazy way in updating the code, trying to change as little as I could.
While a test framework is not performance critical, I believe the format package should use string.Builder in most places, or even better, use io.Writer and avoid allocating memory as much as it can.

[onsi]

thanks @khaf !

[onsi]

oh - if you're still up for updating the docs, please do! otherwise let me know and i can take care of it later.

[khaf]

Here you go #429
Thank you for your help!