Expose and display on the yarpmotorgui the reason why a joint is in hardware fault

[S-Dafarra]

Problem

Often during demos in general, it may happen that a joint goes in hardware fault. In this case, the classical workflow would consist in looking for the logger, searching for the last yarprobotinterface entry, searching for the last message regarding the part that went in hardware fault.

Usually, during long testing sessions, the entries on the yarplogger could be close to 100. This complicates the finding of the correct yarprobotinterface entry, and it might be quite easy to get confused with previous runs of the yarprobotinterface. Hence, a common trick is to order the entries by "Last heard", but at this point, if there are other modules running and printing in the logger, we start playing whack-a-mole trying to catch the correct entry.

Once we have opened the correct log entry, then we have to scroll through potentially thousands of lines (see #558) to find the reason why that specific joint went in hardware fault.

Things get even more complicated if other joints in the same part are also in hardware fault because of different reasons.

All this may take easily tens of minutes, and during a potentially stressful situation, so a typical result is that we give up and we simply try to restart the joint. This is clearly not ideal since it will not help at all in avoiding this problem to occur again in the future.

Proposed solution

A possible solution to this problem would be to have the reason why a joint went in hardware fault directly inside the yarpmotorgui. The idea would be that in case a joint goes in VOCAB_CM_HW_FAULT, a small text box is displayed on the partItem displaying the reason for the hardware fault, retrieved directly from the board.

cc @traversaro @randaz81 @pattacini @marcoaccame @lornat75 @drdanz @Nicogene @maggia80

[pattacini]

Reporting here some comments collected internally.

From @traversaro

As the yarpmotorgui only communicates to the board via the YARP interfaces such as IControlMode, the first step to do is to eventually have interfaces for this information in a suitable interface, for example having a virtual bool IControlMode::getLastErrorMsg(int axis, std::string &err) method. fyi @randaz81

From @marcoaccame

At the lower layer we could implement a new message triggered by embObjMotionControl::getLastErrorMsg() which directly asks to the board the reason of the HF plus maybe some other info.

[valegagge]

Hi evrybody,
this is i great idea! So finally we start to have some smart diagnostic info!

I'd like just to point out that in some cases the last error is not the origin of the fault. So we need to be careful to give to the user the correct information. :)

cheers
Valentina

[S-Dafarra]

If we ask directly the board, we should be able to get the exact reason of the hardware fault. Hence, I guess that we just need to discuss the name of the method of IControlMode to retrieve this info 🤔

[randaz81]

Implemented in Yarp by PR robotology/yarp#2764

Notes:

• I haven't modified the IControlMode interface because it would break the compatibility. A new optional interface IJointFault was created for this purpose. If the interface is not available, everything will still run with no message displayed.
• In FakeMotionControl.cpp example, see the method FakeMotionControl::getLastJointFaultRaw(int j, int& fault, std::string& message) to understand how to implement the feature on the real robot device embobjMotionControl

[S-Dafarra]

Holy cow, that was fast! Awesome!

[pattacini]

Great! We may then take on the implementation of the remaining part of the pipeline in the next sprint.

[mfussi66]

Hi @S-Dafarra
Here is a little demo showing the work so far, in which the passing of the hardware limit is triggered:

hfaultdetect-2021-12-06_13.27.54.mp4

The displayed message is the same one displayed in the yarplogger, but of course the textbox will be expanded to show it properly.

We had an internal discussion and we think that it might be useful to expose the the Motion Controller error category:

https://github.com/robotology/icub-firmware-shared/blob/master/eth/embobj/plus/comm-v2/icub/EoError.c#L176-L198

    {eoerror_value_MC_motor_external_fault,  "MC: exernal fault button pressed."},
    {eoerror_value_MC_motor_overcurrent,     "MC: overcurrent. The motor has been turned off to prevent it from being damaged by an impulsive spike of current. par16 = ID of joint."},
    {eoerror_value_MC_motor_i2t_limit,       "MC: i2t limit exceeded. The motor has been turned off to prevent it from being damaged by overheating due to a continuous high current. par16 = ID of joint."},
    {eoerror_value_MC_motor_hallsensors,     "MC: 2FOC hall sensors fault. Invalid sequence in motor Hall effect sensors, please check motor hall cable connections. par16 = ID of joint."},
    {eoerror_value_MC_motor_can_invalid_prot,"MC: 2FOC CAN invalid protocol. The EMS and 2FOC firmware versions are incompatible, please update. par16 = ID of joint."},
    {eoerror_value_MC_motor_can_generic,     "MC: 2FOC CAN generic error. Errors happened in the CAN bus between the EMS and the 2FOC board. par16 = ID of joint."},
    {eoerror_value_MC_motor_can_no_answer,   "MC: 2FOC CAN no answer. The communication between the EMS and the 2FOC board has been lost for more than 50 ms. par16 = ID of joint."},
    {eoerror_value_MC_axis_torque_sens,      "MC: torque sensor timeout. The joint is in a compliant interaction mode or torque control mode, and data from torque sensor have been unavailable for more than 100 ms. par16 = ID of joint."},
    {eoerror_value_MC_aea_abs_enc_invalid,   "MC: AEA encoder invalid data. Hardware problem in the magnetic position sensor of the joint caused invalid position readings. par16 = AEA port (msb) and ID of joint (lsb)."},
    {eoerror_value_MC_aea_abs_enc_timeout,   "MC: AEA encoder timeout. No answer from the magnetic position sensor of the joint (cable broken?). par16 = AEA port (msb) and ID of joint (lsb)."},
    {eoerror_value_MC_aea_abs_enc_spikes,    "MC: AEA encoder has spikes. There is impulsive noise in the measures of the magnetic position sensor of the joint. par16 = AEA port (msb) and ID of joint (lsb)."},
    {eoerror_value_MC_motor_qencoder_dirty,  "MC: 2FOC quadrature encoder dirty. The number of thicks in a complete revolution of the motor was lower than expected, the optical disks need to be cleaned. In par64 0xFF is the mask of raw encoder value. par16 = ID of joint."},
    {eoerror_value_MC_motor_qencoder_index,  "MC: 2FOC quadrature encoder index broken. The reference special thick was not detected during a complete revolution of the motor, please check motor encoder cables. In par64 0xFF is the mask of raw encoder value. par16 = ID of joint."},
    {eoerror_value_MC_motor_qencoder_phase,  "MC: 2FOC quadrature encoder phase broken. The motor encoder is not counting even if the motor is moving, please check motor encoder cables. In par64 0xFF is the mask of raw encoder value. par16 = ID of joint."},
    {eoerror_value_MC_generic_error,         "MC: generic motor error (see 64 bit mask parameter)."},
    {eoerror_value_MC_motor_wrong_state,     "MC: 2FOC wrong state. The 2FOC motor controller is in a control state different from required by the EMS. In par64 0xF0 is the mask of requested state, 0x0F is the mask of actual state. par16 = ID of joint."},
    {eoerror_value_MC_joint_hard_limit,      "MC: hard limit reached. The joint position is outside its hardware boundaries. par16 = ID of joint."},
    {eoerror_value_MC_motor_qencoder_phase_disappeared, "MC: qenc error has disappeared, warning counter has been reset."}

As a first iteration of this feature implementation we could support the above states, and expand them in the future.

For example, lower-level fault states can be triggered by the 2FOC boards:

typedef union
{
    struct
    {
       ...
        unsigned CAN_BufferOverRun      :1;
        unsigned SetpointExpired        :1;
       ...
        unsigned SiliconRevisionFault   :1;
        ...
    } bits;
        
    uint32_t bitmask;
        
} MotorFaultState;

https://github.com/robotology/icub-firmware/blob/master/emBODY/eBcode/arch-arm/embobj/plus/mc/Motor.h#L60
Their communication "upstream", however, would require deeper work on the firmware code, in order to implement a consistent forwarding mechanism.

[mfussi66]

PR opened in icub-main: robotology/icub-main#779

[maggia80]

cc @gsisinna @Uboldi80

[gsisinna]

useful feature, I'm waiting for PR and use in testing phase

[pattacini]

The PR has been merged.
However, to use the new feature, we still need to address:

• Update iCubGenova09 to the use of the latest NWS/NWC layers robots-configuration#319
• Add support for IJointFault interface to visualize why a joint is in HW fault on the Yarpmotorgui yarp#2781 (we're still judging whether to merge this PR or rely on the upcoming YARP-3.6)

[pattacini]

The new feature will be shipped via the upcoming Distro 2021.11.1 📦

[pattacini]

Distro 2021.11.1 released 🚀
See #570.