rp2040 PICO arduino tinyUSB midi example midi_test crashes when simultaneously receiving and sending messages at (not too fast) speed

[crp500]

Operating System

Linux

Arduino IDE version

2.0.3

Board

PICO rp2040 from RP

ArduinoCore version

PICO/RP2040 community 2.7.0

TinyUSB Library version

release

Sketch & Compiled output (as attached txt files)

modified examples/adafruit/midi/midi_test
Added callback for handling incoming clock ticks
Output 1 byte sysex every 40 ms instead of noteon/off every 266 ms
Turned off midi thru
midi_test.txt
midi_test.ino.bin.txt

What happened ?

The sketch works if i run it without sending any midi events. ie) just letting the pico send out a sysex every 40ms.
The sketch works fine if I comment out the sendSysex in the sketch (dont send anything from PICO) and send the PICO clock ticks every 80ms.
The sketch crashes when i do both at the same time.
Sometimes it crashes within a minute...sometimes an hour.
The crash stops the PICO sending data and also stops the PICO receiving data.
A full reboot of the PICO is required.

How to reproduce ?

Upload the sketch to a RPI PICO rp2040 using arduino ide and community core.
Connect a daw or other midi producing program to the PICO and send it clock ticks at 30 bpm - 24ppq = about every 80ms.

I wrote a simple html file for testing that can be dropped onto a webMIDI compliant browser.
If you want to use that for testing then copy the below html to a file called maybe test.html and drop that onto chromium/brave/chrome/edge (probably wont work in firefox).
Note: the javascript in the html looks for a midi device called "PICO MIDI 1". Change the name if your device is named differently.

With the pico running launch or refresh the script running in the browser.
Wait for a few minutes or longer for a crash.

Here is a link to a text file called test1.txt containing the html i used for testing the PICO.

test1.txt

Rename test1.txt to test1.html and drop the file on a webmidi compliant browser to test the pico.

Debug Log

No response

Screenshots

[crp500]

I have now tried this with windows 10 and an macOS and I got the same results.
The example sketch crashes the pico when sending and receiving messages simultaneously at 36 messages a second.

[LeSuedois]

Hello,
I can confirm that this can happen when you send "to many" messages to the RP2040.
It's not related to the Host Computer or OS. I all ready crashed my code under win 10, 11, MacOS and several Linux distributions.
The solution for me (I usually use Max 8 to do some app prototyping) is to limit the MIDI troughput to ~30Hz.
What I observed is that you can send a 8kbyte configuration JSON from your computer over SYSEX without problems, but sending 50 CC's in a second results in instant crash.
IDK why this is so and for instance I am living nice with this limitation..

[crp500]

I tried this again with 1.18.3 and got the same results.
I sent midi messages (noteon/noteoff,clock,CC) to the pico at a rate of 600 per second without a problem. Runs for hours.
I sent midi messages (noteon/noteoff,clock,CC) from the pico at a rate of 600 per second without a problem. Runs for hours.
When i send messages in both directions at the same time (at a cumulative rate of 50 per second) it crashes (sometimes an hour or 2 later). If i send 600 out and 600 in per second it crashes within seconds.

[hathach]

Connect a daw or other midi producing program to the PICO and send it clock ticks at 30 bpm - 24ppq = about every 80ms.

late response, though since I am not familiar with MIDI, I need help with this step of reproducing. Please be specific on which app and which step you use to send clock ticks. It would be better if you are using an app that could run on ubuntu since that is my PC.

[LeSuedois]

@hathach I have some python code (3.10, works with pygame) on my Rpi at home. I will take a look tomorrow when i come home. But I don't think the pygame supports midi ticks
@crp500 maybe midi start/stop/continu with a more accurate time on the embedded side could be a solution for you?

[crp500]

Connect a daw or other midi producing program to the PICO and send it clock ticks at 30 bpm - 24ppq = about every 80ms.

late response, though since I am not familiar with MIDI, I need help with this step of reproducing. Please be specific on which app and which step you use to send clock ticks. It would be better if you are using an app that could run on ubuntu since that is my PC.

In the bug report above i linked 2 files.

1. midi_test.txt is a modified version of the midi_test sketch that sends out a midi event every 40 ms
2. test1.txt is a html file that can be dropped onto a webmidi compliant browser to use to send and receive midi from the pico.

midi_test.txt needs renaming to midi_test.ino and test1.txt needs renaming to test1.html
test1.html will work in ubuntu or windows or on a mac.
It looks for a midi device called "Pico MIDI 1" which is what ubuntu names the Pico by default when compiled as a midi device.

It dosen't really matter what the midi events are. There is nothing special about clock ticks. I have tried noteon, noteoff, cc, clock ticks, small sysex messages, all types of midi messages. All with the same result.

As I said,
I can send any type of midi event at extremely high speeds (i tested up to 600 events per second) without any problems as long as it is only in one direction - either sent from computer to pico, OR, sent from pico to computer.

However, if i try to send AND receive at the same time at even a slow rate of 50 messages combined per second then the pico eventually crashes. If i send AND receive at a rate of 600 cumulative per second the pico crashes within seconds.

It dosent matter what type of midi messages are involved.

[MickGyver]

I'm able to reproduce this error, it seems that if I disable the Serial printing, it takes a while longer to crash.

In a project I'm working on, I get a crash quite quickly if I send a MIDI sysex message to the USB MIDI port just after I print something to Serial. If I comment out the serial printing, the project doesn't crash.

Something related to buffers maybe?

[MickGyver]

NOTE/UPDATE: Disregard my example below, with Serial.flush() after the serial print, the example works flawlessly so it is not related to the original bug report.

Ok, I made a simple example that prints to serial and then sends a MIDI sysex message every 100ms, this gives me a crash after just a few seconds every time. If I disable the serial printing, there seems to be no crash. Here is the code you can try (you just need to monitor the serial port):

#include <Arduino.h>
#include <Adafruit_TinyUSB.h>
#include <MIDI.h>

Adafruit_USBD_MIDI usb_midi;
MIDI_CREATE_INSTANCE(Adafruit_USBD_MIDI, usb_midi, MIDI);

// Sysex message 
uint8_t sysex[9] = {0x3E, 0x00, 0x7F, 0x60, 0x00, 0x03, 0x0C, 0x0A, 0x19};
// Buffer for serial printing
char strBuffer[40];
// Message counter
uint32_t count = 0;

void setup()
{
#if defined(ARDUINO_ARCH_MBED) && defined(ARDUINO_ARCH_RP2040)
  // Manual begin() is required on core without built-in support for TinyUSB such as mbed rp2040
  TinyUSB_Device_Init(0);
#endif

  pinMode(LED_BUILTIN, OUTPUT);

  MIDI.begin(MIDI_CHANNEL_OMNI);

  Serial.begin(115200);

  while( !TinyUSBDevice.mounted() ) delay(1);
}

void loop()
{
  static uint32_t start_ms = 0;
  if ( millis() - start_ms > 100 )
  {
    start_ms += 100;

    sprintf(strBuffer, "%s %d", "Message count:", ++count);
    // Disable the serial printing and there will be no crash, with serial printing a crash happens after just a few seconds
    Serial.println(strBuffer);

    MIDI.sendSysEx(9, sysex);
  }
  // read any new MIDI messages
  MIDI.read();  
}

[myreauks]

I can confirm that the same crash is happening to me and indeed it can be fixed by commenting out all Serial prints.

Thanks for the solution MickGyver!
Would be nice to get a fix though to be able to use Serial prints while sending MIDI messages over usb.

[Copper280z]

I believe I've encountered the same bug while doing binary data transfer with 2 CDC serial ports. I'm using the first port for unidirectional debug prints (MCU->Host), and the second port for bidirectional binary communication. My test case has 8-16 byte Host->MCU commands being sent every few milliseconds, and the MCU responds with ~40 byte responses as soon as it parses a full command, with a rough guess of 150 bytes of debug prints per response. depending on the host command rate, the mcu will hang within 40-300 command-response iterations. I am not using the second core at all, and am not using FreeRTOS.

Using a black magic debug probe, I stopped execution once the pico hung and found that it was stuck in

(void) osal_mutex_lock(mutex, OSAL_TIMEOUT_WAIT_FOREVER);

at line 121 in tusb.c, which calls

mutex_enter_timeout_ms(mutex_hdl, msec);

which is defined in mutex.h in the pico sdk.

[LeSuedois]

@Copper280z Thank you for the precise debug!
I (once again) encountered this bug on an other project, since I really like MIDI and RP2040. And well, this really bothers me...
I tried following :
changed at line 121 in tusb.c from :
(void) osal_mutex_lock(mutex, OSAL_TIMEOUT_WAIT_FOREVER);
to
(void) osal_mutex_lock(mutex, OSAL_TIMEOUT_NORMAL);

Well since then it just works fine! Can anyone tell me what I did and when I will get problems? 🙈
It even works with plenty of Serial.print()'s. I know this is not really a solution since the source of the problem is the mutex waiting forever.. And I don't know why he is not executing after the resource is unlocked (as far as I understand).

But in my case I am not saving lives and thus this works for me! since I am not counting midi clicks I even don't care if I miss some messages..

[doctea]

Same OSAL_TIMEOUT_NORMAL workaround seems to fix my problems in #293 too -- would be great to figure out what the underlying cause is and what the real fix should be?

[ricardok-sp]

Same happens to me. The OSAL_TIMEOUT_NORMAL workaround works sometimes: the same device with the same firmware works and then doesn't work, I couldn't find a pattern yet testing with a Pico and a Yamaha P45 digital piano (whose MIDI sends a lot of msgs and ticks all the time).

[tttapa]

though since I am not familiar with MIDI, I need help with this step of reproducing

Hi @hathach, here's a minimal example with instructions for reproducing the issue on Ubuntu:

1. Install https://github.com/earlephilhower/arduino-pico (I used version 3.7.0) and select the Raspberry Pi Pico in the Tools > Board menu in the Arduino IDE.
2. Select "Adafruit TinyUSB" in the Tools > USB Stack menu (FQBN: rp2040:rp2040:rpipico:usbstack=tinyusb).
3. Upload the Arduino code at the bottom of this post to a Pi Pico.
4. Download MIDI-veryshort-challenge.tar.gz and extract it. It contains a .syx file with a couple of kilobytes of random MIDI messages.
5. Run the amidi command below to send and receive the MIDI .syx file.
6. You'll see that after a couple of tries, the Pico's LED stops blinking, and it no longer responds, requiring a reset.
7. You'll also see that even if the Pico doesn't crash, the MIDI data it receives does not match the data that was sent. I'm not sure if this issue is related.

amidi -l # list available MIDI devices
port=hw:1,0,0 # change accordingly
# send and receive MIDI data to/from the Pico         (sometimes this command hangs)
amidi -p $port -r MIDI-veryshort-response.syx -ca -t 1 -s MIDI-veryshort-challenge.syx
# compare the received data with what we sent         (other times this comparison fails)
cmp MIDI-veryshort-response.syx MIDI-veryshort-challenge.syx

#include <Adafruit_TinyUSB.h>

Adafruit_USBD_MIDI usb_midi;

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
  usb_midi.begin();
  while(!TinyUSBDevice.mounted())
    delay(1);
}

void loop() {
  // Simply echo back any packets we receive
  uint8_t packet[4];
  if (usb_midi.readPacket(packet)) {
    uint8_t retry = 3;
    while (!usb_midi.writePacket(packet) && retry-- > 0)
      delay(1);
  }
  // Blink an LED so we can see when it crashes
  static auto timer = millis();
  static bool state = false;
  if (millis() - timer >= 500) {
    digitalWrite(LED_BUILTIN, state = !state);
    timer += 500;
  }
}

[alexriegler12]

Please fix it.

[fred-rabelo]

I am having the same problem with the Adafruit Feather RP2040 board.
Is anyone any closer of finding the problem and a solution?
This makes TinyUSB and MIDI kinda not compatible IMHO.

[doctea]

Is anyone any closer of finding the problem and a solution?

Making this simple change fixes the problem for me #293 (comment)

[fred-rabelo]

Is anyone any closer of finding the problem and a solution?

Making this simple change fixes the problem for me #293 (comment)

Yes, it seems to be working, thanks!

[palthedog]

@fred-rabelo I've just took a look at #293 and I believe that changing OSAL_TIMEOUT_FOREVER to OSAL_TIMEOUT_NORMAL workaround isn't a jproper fix.

I've posted a (long) comment on #293 about it. If you are interested in potential issues of the workaround, check it out.