Pretty-printing inputs and outputs

[owickstrom]

Testing Problem

I'm using jqwik with much delight to test Apache Beam pipelines, i.e. integration tests using PBT. The inputs and outputs are generally large data structures, and bugs are found with sequences of inputs that trigger certain behaviour (e.g. a sequence of input elements interspersed with clock commands that cause time to pass in the test).

When jqwik finds a failing example it prints the minimal example input (after shrinking) along with the original input, I guess using toString? In any case, with these large structures, often being lists of large structures, the output is very hard to digest.

In my case, it can look something like the following, but with the lines being way longer:

sample = [[ElementReceived(timestampedValue=TimestampedValue(SuperLargeThing(...), 1970-01-01T00:00:00.001Z), ProcessingTimePassed(duration=PT1800S)], ...]
original-sample = [... far more SuperLargeThings ... ]

Suggested Solution

I'd like to have structures pretty-printed over multiple lines in order to make them more readable and easy troubleshooting. I can't suggest any universal pretty-printing solution for the JVM, so I'm thinking that this should be up to the user. Use toString by default, and let the user override with a custom printer.

In Haskell Hedgehog (which I'm most familiar with), there's an alternative to forAll called forAllWith, that takes a function to convert your generated value to a string. In jqwik, this is handled with method arguments and annotations, so I'm not sure what would be a suitable counterpart. Maybe there's some convention on the JVM (or in Java, specifically) that could be used?

I'm leaving the solution part of this issue intentionally vague.

Discussion

Another concern, which might be a separate issue altogether, is pretty-printing in assertions. The same pretty-printing solution used for examples would be nice to use in AssertJ or whatever assertions used. Ideally, I'd like to have pretty-printed data structures and diffs (like git diff). Example from an article I wrote:

I'm not sure if there exists any library for doing pretty-printed diffs that could be used both for printing examples and in assertions.

Looking forward to getting some feedback on this!

[jlink]

@owickstrom Can you give a specific example of how a structure of your choice should look when being pretty-printed? I'm especially interested in line breaks between different parameters and line breaks within a single parameter.

[owickstrom]

Sure! I can't show you the actual code as it's for a proprietary project, but I can sketch something out.

If we have a test looking something like this:

public void testExample(ForAll("inputs") List<Input<SuperLargeThing>> inputs, @ForAll Integer someNumber) { ... }

Then the Jqwik output might contain:

Sample
------

  inputs: [
    ElementReceived(
      timestampedValue=TimestampedValue(
        value=SuperLargeThing(
          foo=1,
          bar="",
          baz=null
        ),
        timestamp=1970-01-01T00:00:00.001Z
      )
    ),
    ProcessingTimePassed(
      duration=PT1800S
    ),
    ...
  ]
  someNumber: 0

Original Sample
---------------

(similar to above)

Note that I'd prefer every field/value pair to be on a separate line. If the pretty printer could compact small values on a single line then that's fine, but I rather have more lines than longer lines.

Also note that I used another formatting for "Sample" and "Original Sample" headings. That might not be in line with jqwik's style, so ignore that if you don't like it.

[jlink]

@owickstrom Thanks. IMO there's some generic stuff that could be improved and there's the idea of pluggable output formatting. I'll be pondering a bit and get back here.

[owickstrom]

@jlink Sounds good! 👍

[jlink]

Here's a first draft of what I could do...

Step 1: Make reporting configurable through jqwik.properties

reportingFormat=COMPACT|DETAILED

Step 2: Introduce DETAILED format, similar to

                              |-------------------jqwik-------------------
tries = 1                     | # of calls to property
...
seed = 8531870486618337326    | random seed to reproduce generated values

Sample
-------
aString (java.lang.String): "this is a string"
myObject (my.package.MyObject): <toString() or registered (multiline) printer>

Original Sample
----------------
aString (java.lang.String): "this is another string"
myObject (my.package.MyObject): <toString() or registered (multiline) printer>

Mind that in Java there's additional compiler configuration necessary (-parameters) to have real parameter names available. Otherwise they'll show up as arg0, arg1 etc.

Step 3: Allow registration of printers through Java's SPI mechanism with a few already registered default printers for lists, sets, arrays. Interface could be like

interface SamplePrinter<T> {
    boolean canPrintFor(TypeUsage targetType);
    List<String> print(T object);
}

What do you think?

[jlink]

To account for the difference of compact and detailed printing I modify the suggested type:

interface SamplePrintingFormat {
  boolean canPrint(TypeUsage targetType);
  String printCompact(Object object, Function<Object, String> compactPrinter);
  List<String> printDetailed(Object object, Function<Object, List<String> detailedPrinter);
}

The compactPrinter and detailedPrinter functions are needed to print containedObjects.

I suggest an optional annotation @SamplePrinting(SamplePrintingMode)
and an addition configuration parameter:

defaultSamplePrintingMode = COMPACT|DETAILED|AUTO

Mode AUTO would use compact mode under a certain threshold, e.g. 60 chars.

[jlink]

To make SamplePrintingFormat simpler I tend to just join the lines of detailed format with comma.
So here's my current suggestion:

interface SamplePrintingFormat {
  boolean canPrint(TypeUsage targetType);
  /**
   * @param printer Use to format contained objects
  **/
  List<String> print(Object object, Function<Object, List<String> printer);
}

I'll probably start working on it soon.

[jlink]

Some progress made. Consider the following property:

@Property(afterFailure = AfterFailureMode.RANDOM_SEED)
void reportFalsifiedSamples(
	@ForAll int anInt,
	@ForAll List<Integer> listOfInts,
	@ForAll @Size(min = 3) Map<@AlphaChars @StringLength(3) String, Integer> aMap
) {
	Assertions.assertThat(anInt).isLessThan(10);
}

will now produce the following report:

SampleReportingExamples:reportFalsifiedSamples = 

java.lang.AssertionError: 
Expecting:
 <10>
to be less than:
 <10> 

                              |-------------------jqwik-------------------
tries = 1                     | # of calls to property
checks = 1                    | # of not rejected calls
generation = RANDOMIZED       | parameters are randomly generated
after-failure = RANDOM_SEED   | use a new random seed
edge-cases#mode = MIXIN       | edge cases are mixed in
edge-cases#total = 0          | # of all combined edge cases
edge-cases#tried = 0          | # of edge cases tried in current run
seed = 5636506477930535961    | random seed to reproduce generated values

Sample
------
  anInt: 10
  listOfInts: []
  aMap: {"aaa"=0, "aba"=0, "AAA"=0}

Original Sample
---------------
  anInt: 40
  listOfInts:
    [
      3472291, -117543, -102, 997, 280, -922, -1294, 27, -13279, 64, -2147483648, 291, -582, -741, 
      20, 4103137, 61097, 6136
    ]
  aMap:
    {
      "pAr"=-8, 
      "oAz"=-5879760, 
      "Zzk"=6533, 
      "ULA"=-52, 
      ...
      "hZn"=49000, 
      "ZtX"=785
    }

[jlink]

More progress. Reporting arrays:

@Property(afterFailure = AfterFailureMode.RANDOM_SEED)
void reportFalsifiedArrays(
	@ForAll int anInt,
	@ForAll int[] arrayOfInts,
	@ForAll @Size(min = 2) @AlphaChars @StringLength(3) String[] arrayOfStrings
) {
	Assertions.assertThat(anInt).isLessThan(10);
}

Sample
------
  anInt: 10
  arrayOfInts: int[] []
  arrayOfStrings: String[] ["aaa", "aaa"]

Original Sample
---------------
  anInt: 195
  arrayOfInts: int[] [14114, 2120155713, 27, 2147483647, 2563]
  arrayOfStrings:
    String[] [
      "oJJ", "aza", "zrZ", "aAx", "Zaz", "SAz", "zlL", "ApK", "jAz", "oXA", "zam", "wZt", "XAa", 
      "XGA", "eZi"
    ]

Of course, available in "1.3.1-SNAPSHOT"

[owickstrom]

Looking very good! 😊 Skickat från ProtonMail mobile

…

-------- Originalmeddelande -------- På 14 juni 2020 13:18, Johannes Link skrev:

More progress. Reporting arrays: @Property ( afterFailure = AfterFailureMode . RANDOM_SEED ) void reportFalsifiedArrays( @forall int anInt, @forall int [] arrayOfInts, @forall @SiZe ( min = 2 ) @AlphaChars @stringlength ( 3 ) String [] arrayOfStrings ) { Assertions . assertThat(anInt) . isLessThan( 10 ); } Sample ------ anInt: 10 arrayOfInts: int[] [] arrayOfStrings: String[] ["aaa", "aaa"] Original Sample --------------- anInt: 195 arrayOfInts: int[] [14114, 2120155713, 27, 2147483647, 2563] arrayOfStrings: String[] [ "oJJ", "aza", "zrZ", "aAx", "Zaz", "SAz", "zlL", "ApK", "jAz", "oXA", "zam", "wZt", "XAa", "XGA", "eZi" ] Of course, available in "1.3.1-SNAPSHOT" — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#85 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AALFQCFP4EE6GWIQPC7EJMDRWSWYRANCNFSM4KKBBMDQ).

[jlink]

More progress. One can register sample formats for any class:

Consider the following property:

@Property(afterFailure = AfterFailureMode.RANDOM_SEED)
@Report(Reporting.GENERATED)
void reportWithFormat(@ForAll("dates") LocalDate localDate) {
	Assertions.assertThat(localDate).isBefore(LocalDate.of(2000, 1, 1));
}

@Provide
Arbitrary<LocalDate> dates() {
	Arbitrary<Integer> years = Arbitraries.integers().between(1900, 2100);
	Arbitrary<Integer> months = Arbitraries.integers().between(1, 12);
	Arbitrary<Integer> days = Arbitraries.integers().between(1, 28);

	return Combinators.combine(years, months, days).as(LocalDate::of);
}

And register this format class:

public static class LocalDateFormat implements SampleReportingFormat {

	@Override
	public boolean appliesTo(final Object value) {
		return value instanceof LocalDate;
	}

	@Override
	public Object report(final Object value) {
		LocalDate date = (LocalDate) value;
		Map<String, Object> valueMap = new HashMap<>();
		valueMap.put("year", date.getYear());
		valueMap.put("month", date.getMonth());
		valueMap.put("day", date.getDayOfMonth());
		return valueMap;
	}

	@Override
	public Optional<String> label(final Object value) {
		return Optional.of("LocalDate ");
	}
}

Then the output will be:

Sample
------
  localDate: LocalDate {"month"=JANUARY, "year"=2000, "day"=1}

Original Sample
---------------
  localDate: LocalDate {"month"=APRIL, "year"=2025, "day"=27}

[jlink]

here's the relevant section in the user guide: https://jqwik.net/docs/snapshot/user-guide.html#failure-reporting

[jlink]

Available in "1.3.1-SNAPSHOT"

[owickstrom]

Very cool, looking forward to trying this in our project at work!