There are a lot of fiddly details about transiting between different
data encodings. And the QR spec is a bit vague on some parts.
I've detailed the problems at the sister project
zxing-cpp/zxing-cpp#62

I've decided to patch this in a works-for-me kind of way:
unmarked encodings are treated as bianry and if that's not
good enough the higher layers have the source material
and can try decoding it themselves.

I think I'd generated my expected output from zxing itself.
This was generated from the source file with
curl https://sampleswap.org//samples-ghost/DRUM%20LOOPS%20and%20BREAKS/161%20to%20180%20bpm/128[kb]161_amenvar3.aif.mp3 | head -c 856 | tee ../tests/amen-01.bin
856 is the packet size qrencode decided to chunk this file into when I first split it up.