Why does --raw cause some benchmarks to fail?

[RyanGlScott]

A nontrivial amount of benchmarks are failing when being written to binary format (.crit files, in our lingo), with one of two errors:

• Not a valid Unicode code point!: Originates in binary
• bag tag <num>: Originates in criterion

Either way, it's probably criterion being buggy. @rrnewton, do you think it's worth it to disable writing to binary format until we can figure out why this happens?

[rrnewton]

Good call re: disabling.

I think it is probably easier to fix Criterion's csv output (to include missing regressions) than it is to debug these problems with the binary output.

[rrnewton]

This is relevant to haskell/criterion#88

We should probably collect a list of all the places we've found crashes/corruption do to these Report binary instances.

[RyanGlScott]

Good idea. I'll kick off a build with binary enabled and compile a fresh list of packages that fail with one of the aforementioned errors.

[RyanGlScott]

These fail with user error (Not a valid Unicode code point!):

• buffer-builder-0.2.4.1 (url)
• conduit-1.2.6.6 (unfused)
• IntervalMap-0.5.0.1 (bench-set)
• loop-0.3.0 (bench-foldl-and-iorefs-are-slow)
• postgresql-binary-0.7.9 (encoding)
• streaming-commons-0.1.15.4 (count-chars)
• tz-0.1.1.0 (bench_greg)

These fail with bad tag:

• binary-typed-1.0 (criterion-overview: bad tag 255)
• inline-r-0.8.0.1 (bench-qq: bad tag 77)
• IntervalMap-0.5.0.1 (bench-rb-impl: bad tag 188)
• lifted-async-0.8.0.1 (benchmark-lifted-async-threaded: bad tag 255)
• loop-0.3.0 (bench-folds: bad tag 16)
• loop-0.3.0 (bench-traverse-w32: bad tag 193)
• tz-0.1.1.0 (bench_c: bad tag 255)
• tz-0.1.1.0 (bench_conv: bad tag 182)

[rrnewton]

We have successfully avoided this by not using binary reports, right?

Close?

[RyanGlScott]

Yep, this is no longer relevant to our needs.