Provide the ability to ignore irrelevant events when converting from Bytes to MIDI

[joaodarocha1]

Method: BytesToMidiEventConverter.ConvertMultiple(byte[] bytes, int offset, int length)

When converting a stream of device messages to Melanchall.DryWetMidi.Core.MidiEvent we need the ability to select only the relevant events. A device such as a Yamaha Genos will send too many irrelevant system messages. DryWetMidi will fail to read them and throw too many exceptions.

It would be great to get only the relevant messages types such as EventStatusBytes.Channel.NoteOn and EventStatusBytes.Channel.NoteOff. Once we have the statusByte we could skip the event and move on to the next one.

[melanchall]

A device such as a Yamaha Genos will send too many irrelevant system messages. DryWetMidi will fail to read them and throw too many exceptions

Can you please explain this in more detail? DWM should be able to read all MIDI events allowed by MIDI spec. I suppose your device sends sys ex events, but DWM should successfully read them.

Please show your code, exceptions and all data that can be useful to understand your issue and what's going on.

[joaodarocha1]

Upon further debugging, I've noticed that there is a NotEnoughBytesPolicy property on the BytesToMidiEventConverter class. Setting this property to NotEnoughBytesPolicy.Ignore has reduced significantly the number of exceptions thrown by the converter. I am still getting some UnexpectedRunningStatusException though.

Please find below a sample of my test code working correctly with the NotEnoughBytesPolicy property set to Ignore.

//NoteOn event
byte[] data = Convert.FromBase64String("AZBDL7o7tIIUAQEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==");
var bytesToMidiEventConverter = new BytesToMidiEventConverter();
var events = bytesToMidiEventConverter.ConvertMultiple(data, 1, 3);

//NormalSysEx event
data = Convert.FromBase64String("AfBDfgIxADiTt0hhAQEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==");
bytesToMidiEventConverter = new BytesToMidiEventConverter();
bytesToMidiEventConverter.NotEnoughBytesPolicy = NotEnoughBytesPolicy.Ignore;
events = bytesToMidiEventConverter.ConvertMultiple(data, 1, 6);

Thanks for you help!

[melanchall]

Well, regarding NormalSysEx. Bahavior of the DryWetMIDI is absolutely correct. If we look what bytes you have:

Sys ex event structure is: status byte, then, length of data (VLQ format), then data. The highlighted byte represents the length of an event. It's 67. But you specify reading from byte 1 using 6 bytes. 6 is not 67 required to read the event, so we have the exception.

If we write this:

bytesToMidiEventConverter.ConvertMultiple(data, 1, 69);

we'll get sys ex event successfully read.

But yes, if we write this:

bytesToMidiEventConverter.ConvertMultiple(data, 1, data.Length - 1);

we'll get UnexpectedRunningStatusException, but it's the next error. I'm pretty sure the issue is with the data. But I also see that DWM actually is not clear where in the bytes array exception occurred so it would be great to improve the library regarding this.

[joaodarocha1]

Please find below a sample of MIDI messages from the Yamaha Workstation that throws UnexpectedRunningStatusException.

byte[] data = Convert.FromBase64String("ATEA91Ej7J0vGgEAGgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==");
var bytesToMidiEventConverter = new BytesToMidiEventConverter();
bytesToMidiEventConverter.NotEnoughBytesPolicy = NotEnoughBytesPolicy.Ignore;
var events = bytesToMidiEventConverter.ConvertMultiple(data, 1, 3);

data = Convert.FromBase64String("ATYe98icBy7eGwEAGwEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==");
bytesToMidiEventConverter = new BytesToMidiEventConverter();
bytesToMidiEventConverter.NotEnoughBytesPolicy = NotEnoughBytesPolicy.Ignore;
var evaents = bytesToMidiEventConverter.Convert(data, 1, 3);

[melanchall]

For the first data:

So status byte is 49 (0x31) which means we have running status here (because the number is less than 0x80). But we have not real status byte before so DWM doesn't know the type of the current event. It sees running status, but it doesn't see status byte specified before. So this piece of data definitelycan't be read. I suppose your device sends data before bytes you've provided me and in that data there is the real status byte. This isolated piece of data is corrupted without context.

[joaodarocha1]

I am attaching a log of messages that leads to the UnexpectedRunningStatusException exception.

midi sequence.txt

[joaodarocha1]

Here is the dump of the data that leads to the exception
dump.txt
.

[melanchall]

Well, this dump is almost the same you've sent me before as base64. My asking was about full dump with all bytes as they coming from the device. What are these blocks of 1024 bytes coming from? Can you please send me the data BEFORE this chunking on blocks?

As for dump.txt, of course, the problem the same as with base64 – the data (bytes along with offsets and lengths) is broken, both on sys ex message and the last one. My assumption is the chunking made wrong, causing the data corruption.

[joaodarocha1]

I am running the code on an android device. The blocks of 1024 bytes are created by android MIDI API.

https://developer.android.com/reference/android/media/midi/package-summary

I am using the following device to convert MIDI to USB

http://www.funs-tech.com/h-pd-17.html?fromColId=104

[melanchall]

@joaodarocha1 Do you mean MidiReceiver class receives the MIDI data by blocks of 1024 bytes? If so, are you sure you gave me all captured blocks as they came from the device (so no blocks missing)?

[joaodarocha1]

This is the log that I got from another tool. Is it of any help to you?
midi raw log.txt

[melanchall]

@joaodarocha1 Thanks for the last log. Well, it's my fault, I forgot a small thing – BytesToMidiEventConverter performs conversions according to Standard MIDI Files (SMF) specification which states:

is used to specify a MIDI system exclusive message, either as one unit or in
packets, or as an "escape" to specify any arbitrary bytes to be transmitted. A normal complete
system exclusive message is stored in a MIDI File in this way:

F0 <length> <bytes to be transmitted after F0>

And that's why DryWetMIDI tries to read sys ex event length although there is no length there since the data got from a device, not from MIDI file. Right now there is no way to convert sys ex bytes without length with BytesToMidiEventConverter. Probably I need to improve converters providing a kind of UseSmfFormat property which is true by default, but can be set to false.

But as for last data

31 22 F7: data()

It's strange anyway. It looks like it's the end of previous sys ex

F0 43 7E 02 31 22: SysEx()

But why last bytes are sent seperately along with 0xF7... I don't know about such things.

[melanchall]

@joaodarocha1 Well, I've published prerelease version of the library – 6.1.2-prerelease1. Now you can set BytesFormat property on converters to BytesFormat.Device which switches how sys ex (and some other events) are read/written.

[joaodarocha1]

Awesome! I will test it.

[melanchall]

🚀 6.1.2 version is released now!

Prerelease NuGet packages will be unlisted soon, so please update the package references to the new version.

Thanks for using the library!

[joaodarocha1]

Sorry for the late reply. I was not working on this project for a while.

It is working now. I've added a video of the app working connect to the Yamaha keyboard.

VID-20240229-WA0003.mp4