Lighthouse calibration lost in flight

[matejkarasek]

A strange issue: the calibration of one of the base stations got lost/changed in flight, causing the Kalman filter to reset (and keep resetting).
Happened in Command Based Flight via cfclient, where I just hit take off and then let it hover (battery endurance tests).
After this, even when on ground, I would get Kalman filter diverge and reset every few seconds.
I found out that when one (particular) base station was switched off, then all was fine.

Loading a saved system config in the Lighthouse tab fixed the issue and I could use all 4 base stations again.

The console messages are below, the affected base station is number 3.
Suspicious are the received calibrations from addresses that differ in just one character...

Platform: flapper (Bolt 1.0)
4 LH basestations V2.0
FW: 2022.3 with only minor changes (platform config)
CFclient: 2022.3 with minor changes (forcing to land when battery is low)

If useful, I have some logfiles as well.

SYS: Flapper (Bolt 1.0) is up and running!
SYS: Build 0:0f2f0f07d2c2 (2023.02) MODIFIED
SYS: I am 0x20373347344D500E00560028 and I have 1024KB of flash!
CFGBLK: v1, verification [OK]
DECK_INFO: CONFIG_DECK_FORCE=bcUSD:bcLedRing:bcFlapperDeck found
DECK_INFO: compile-time forced driver bcUSD added
DECK_INFO: compile-time forced driver bcLedRing added
DECK_INFO: compile-time forced driver bcFlapperDeck added
DECK_CORE: 4 deck(s) found
DECK_CORE: Calling INIT on driver bcLighthouse4 for deck 0
DECK_CORE: Calling INIT on driver bcUSD for deck 1
uSD: mount SD-Card [FAIL].
DECK_CORE: Calling INIT on driver bcLedRing for deck 2
DECK_CORE: Calling INIT on driver bcFlapperDeck for deck 3
IMU: BMI088: Using SPI interface.
IMU: BMI088 Gyro connection [OK].
IMU: BMI088 Accel connection [OK]
IMU: BMP388 I2C connection [OK]
ESTIMATOR: Using Kalman (2) estimator
CONTROLLER: Controller type forced
CONTROLLER: Using PID (1) controller
PWR_DIST: Using Flapper power distribution | PCB revD (2022) or newer
MTR-DRV: Using brushless motor driver
SYS: About to run tests in system.c.
SYS: NRF51 version: 2023.02
EEPROM: I2C connection [OK].
STORAGE: Storage check [OK].
IMU: BMI088 gyro self-test [OK]
DECK_CORE: Deck 1 test [OK].
DECK_CORE: Deck 3 test [OK].
SYS: Self test passed!
STAB: Wait for sensor calibration...
SYS: Free heap: 17208 bytes
LHFL: Lighthouse bootloader version: 2
STAB: Ready to fly.
ESTKALMAN: State out of bounds, resetting
LHFL: Bitstream CRC32: 112BC794 [PASS]
LHFL: Firmware version 6 verified, booting deck!
ESTKALMAN: WARNING: Kalman prediction rate low (97)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (97)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (97)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
LH: Got calibration from FEC56AC9 on channel 3
ESTKALMAN: WARNING: Kalman prediction rate low (98)
LH: Got calibration from FEC560C9 on channel 3
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)

[removed low rate warnings]

ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (97)
ESTKALMAN: WARNING: Kalman prediction rate low (97)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
LH: Got calibration from FEE560C9 on channel 3
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
LH: Got calibration from FEC560C9 on channel 3
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)

[removed low rate warnings]

ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
LH: Got calibration from BA29F8C8 on channel 4
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (96)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)

[removed low rate warnings]

ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (96)
ESTKALMAN: WARNING: Kalman prediction rate low (95)
LH: Got calibration from BA29D8C8 on channel 4
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (97)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)

[removed low rate warnings]

ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
LH: Got calibration from FEC563C9 on channel 3
ESTKALMAN: WARNING: Kalman prediction rate low (98)
LH: Got calibration from FEC560C9 on channel 3
ESTKALMAN: State out of bounds, resetting
ESTKALMAN: State out of bounds, resetting
ESTKALMAN: State out of bounds, resetting
ESTKALMAN: State out of bounds, resetting
ESTKALMAN: State out of bounds, resetting
ESTKALMAN: State out of bounds, resetting
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)
ESTKALMAN: WARNING: Kalman prediction rate low (98)

[matejkarasek]

Here the logfiles:
https://flappercz.sharepoint.com/:f:/s/Flapper/Elzqm-5-i-NJqNH6khjdJqYBmwBh1ZSZDmudvnnEJ7cIvg?e=rVndjq

[knmcguire]

hi! thanks for sharing. This week is our quartely meeting but I've put it on our triage list of next week

[krichardsson]

In the current implementation, we decode the calibration data whenever we receive it, also if we already have calibration data. One option could be to ignore calibration data if it is already present, but that would create some complications when you introduce a new base station or move to a new system.

There is a CRC in the transmitted calibration data and my assumption is that that bit errors in the transmission should have been found. Maybe there is a bug in the check?

[matejkarasek]

Thanks for checking, yes, seems like that...

[knmcguire]

We just had a triage meeting on this, but the bottom line is, this needs to be investigated more closely as we don't know why this happens.

[knmcguire]

Some more thoughts on this:

• We should once we received the calibration data once during runtime, stop the retrieval there.
• Perhaps the persistent memory write is broken
• It could be related to the check sum as well

[krichardsson]

The problem is most likely that the checksum is never checked. See

crazyflie-firmware/src/utils/src/lighthouse/ootx_decoder.c

Line 78 in 43a45c7

// TODO: Check CRC!