Efficient binary serialization

[abooij]

Currently we are serializing and deserializing using bytestring builders and attoparsec, respectively. But that seems like overkill, as our grammars are much simpler than what those libraries support.

Michael Snoyman has written an article and library for serialization in Haskell that might be relevant to our use case.

[abooij]

This is now implemented as the store (see also the release announcement).

[tclv]

So I took a jab at implementing this. In the following screenshot you can see the performance differences. The performance is encoding/decoding 10000 random Wire Packages (with 20-200 bytes payload evenly distributed in microseconds). I put map id in there as well to give an idea of the cost of traversing the entire structure and forcing it to normal form.

[tclv]

The Storable interface was implemented like this for the benchmark:

toInt :: Integral a => a -> Int
toInt = fromInteger . toInteger
{-# INLINE toInt #-}

instance Store WirePackage where
  size = VarSize $ toInt . wirePackageSize
  poke (WirePackage sen size op pl) = do
    poke sen 
    poke op 
    poke size 
    let (sourceFp, sourceOffset, sourceLength) = B.toForeignPtr pl
    pokeFromForeignPtr sourceFp sourceOffset sourceLength
  peek = do
    sen <- peek
    op <- peek
    size <- peek
    let payloadSize = toInt size - 8
    pl <- peekToPlainForeignPtr "Data.ByteString.ByteString" payloadSize
    return $ WirePackage sen size op (B.PS pl 0 payloadSize)
  {-# INLINE size #-}
  {-# INLINE peek #-}
  {-# INLINE poke #-}

This implementation is largely derived from the implementation of ByteString in Store itself.
Before I can write a PR implementing this in Sudbury itself, I have a couple of things that need to be discussed:

• It might be nice to derive Generic on WirePackages and other applicable data structures as that allows a variety of useful classes to be used (like NFData). Probably more relevant for things that actually get exposed for the users.

• The implementation uses internals from ByteString.Internal. This module is "unsafe", and thus breaks the Safe pragmas. Is there a obvious solution to this? I'm not too familiar with the Safe pragmas.

• What does the Wayland protocol require with regards to Endianness? The people from Store were talking about implementing store in such a way that it will always write to Little Endian ByteString format (from both LE and BE machines). Is this desirable?

• Any ideas for more interesting tests than decodeEx . encode == id

• Is there a more idiomatic approach to toInt 😞

[abooij]

This is really nice.

I'll respond quickly now and more thoroughly later, so that I'm not holding you up:

• I'd like to err on the side of caution wrt Generic and NFData. When you need to do a full forcing of your objects explicitly, you're probably solving the wrong problem (I'd prefer an approach like the one taken here). But of course such instances are acceptable for benchmarks and tests. Note that you can add a "deriving instance" after the fact using the StandaloneDeriving language extension.
• The obvious solution is to stop using the safety pragmas as these are sort of deprecated anyway. Feel free to remove any safety pragma that gets in your way.
• The wayland wire protocol specifies that encoding is based on the host's encoding. I believe this is what store does anyway (but it'd be worth checking+documenting).
• Well perhaps we could hard-code some sample packages, and package streams. I think the easiest way to record them is to use sudbury itself.
• fromIntegral is perhaps what you are looking for.

[tclv]

• StandaloneDeriving is indeed what I was looking for, thanks!
• Okay I'll be liberal in removing them
• I think they do indeed. There's currently this issue: Support selecting endianness for numeric types mgsloan/store#31, where indeed HostEndianness is assumed.
• Alright, I think that might be worth doing in a separate PR (related to No clear haskell API #2 I guess)
• Crap

I'll get started on the PR