Slow initialization on subsequent run

[williamleong]

Hi, is anyone else facing a problem with slow initialization for a subsequent run?

When I first initialize, the setting and getting parameters, memories, TOC, etc from the Crazyflies is quite fast, maybe 1-2 seconds or less for each Crazyflie. However if I have to do this for a subsequent run, it can take more than 30 seconds for each Crazyflie.

Sometimes this is fixed by unplugging and replugging the Crazyradio. An alternative that worked so far was to reboot the computer. I am running Ubuntu 20.04 with 20 Crazyflies if it helps.

[whoenig]

I have not seen this myself, but we currently operate much smaller teams. Do you know if this also occurs when you fly 5 Crazyflies or so, or only for the 20? Do you use a single radio, only?

[williamleong]

Thanks for getting back. We use 4 radios and assign 5 Crazyflies to each radio. We have seen it happen with 10 Crazyflies and 2 radios as well. I do not remember the case with 5 Crazyflies, will have to check.

[williamleong]

If I use backend:=cpp, it typically gets stuck during broadcast (e.g. [all]: radiobroadcast://*/100/2M), but if I terminate and change to backend:=cflib, it is able to initialize all 20 correctly. This is with the Crazyradio PA.

[whoenig]

I have seen these issues when the URI is wrong and sometimes randomly for smaller teams. Sounds like in this case we somehow wait for some packet to arrive that never comes. I am pretty sure this is a firmware bug in the safelink, but so far I have not found a good reproducible for bitcraze.

[williamleong]

That's interesting to know. Do you think that disabling safelink will cause issues? I am not sure if there are strong assumptions on reliable communications in the framework.

[whoenig]

In Crazyswarm2 it's impossible to disable it. cflib has it also enabled, but is a bit slower, which seems to reduce the likelyhood of it occurring. Crazyswarm1 didn't rely on safelink (which created a whole set of other problems, like missing logging output).

You could try to artificially slow down the communication (https://github.com/bitcraze/crazyflie-link-cpp/blob/master/src/CrazyradioThread.cpp#L131). Other than that, we would probably need a good way to reproduce the issue.

[williamleong]

Yes, I wasn't able to reliably reproduce it either. However I did manage to get it to be stuck once, recompiled with your suggested changes, and it worked. Will need further testing to see if it is the solution or I just got lucky.

[williamleong]

One possible reason why using a delay instead of std::this_thread::yield(); could work is because queue_send_mutex_ is being locked in a while loop. It is possible that yield() returns immediately if there is no other thread demanding use of the CPU, therefore queue_send_mutex_ is being locked constantly in the while loop, resulting in a deadlock and preventing the lock from being acquired in Connection::send and messages from being pushed into queue_send_.

[Kaijun101]

@whoenig I have a question regarding the backends. So from my understanding, there are 2 backends: cpp and Cflib, which are built on C++ and python respectively. cpp utilise broadcast and cflib is unicast. Therefore, what's the advantages of using cpp, other than speed. And also in which cases, do we decide which backend to use?

[whoenig]

The backends have slightly different feature sets, see https://imrclab.github.io/crazyswarm2/overview.html#support-functionality-with-backends. Broadcasts are extremely important for larger teams when flying in a mocap, so the cflib backend is not really usable in that case.

Generally, we were hoping to retire the cpp backend eventually, since relying on cflib moves a lot of the maintenance/testing directly to the robot vendor. Currently, the cpp backend is significantly faster and broadcasts are needed for larger teams.