Skip to content

Commit

Permalink
Update faultstate (#312)
Browse files Browse the repository at this point in the history
* added basics of multi-bus support to CAN driver

* New CAN data structure
More flexible CAN Send structure

* - Compat fixes for applications with new driver data type and function signatures
- Compile fixes for driver
- Deprecated data types removed

* - Compat layer for legacy MC driver
- Minor bug fixes
- Notice of deprecation for legacy MC driver

* reconfigured enum and implemented lookup table

* implemented LUT and build fixes

* - Moved LUT into the C file to fix linker issue
- Moved LUT struct def into the C file
- fixed doc comments

* - changed file organization
- still need to make readfunc populate the idx field

* changed lookup table to be a sparse array, changed enum to fit

* implemented read functionality

* more appropriate error checking in CANbus_Send

* - fixed temperature CAN ID
- fixed comments
- fixed lookup table type

* Small fixes

* Started CANbus test file

* Added some tests for repeated blocking and non-blocking CANbus_Send

* Some changes to test

* Put CAN1 and CAN3 in loopback mode for testing

* Connected CAN1 and CAN3 to test read/writes

* Objectives for ReadTritium Rewrite

* 11/5 - AM

* 11/5 - PM

* Implemented error handling

* Modified Minion_Read_Input to read inputs and outputs and renamed to Minion_Read_Pin

* Added test for testing the ignition contactor task

* Updated code due to the Minion Read function rewrite

* Removed unneed error

* Removed unneeded error and shortened error name

* Fixed tritium error handling. Added velocity read and getter capabilities. In process of adding fault handlling capabilities

* added if-statement to skip T_NONE errors

* for gage

* Added fault state handling.

* Deleted old motor controller file and header file, and mentions of header file.

* fixed velocity conversions to multiplying fixed point value by ten instead of dividing by ten so that the value is accurately displayed in fixed point format. Added macros to initialize internally held variables Motor_RPM and Motor_Velocity (they are both initialized to 0)

* Implelemted motor controller restart function in ReadTritium, use this function in FaultState.c. Defined a new error code (T_INIT_FAIL) to shut off the motor controller in the event a restart/initialization fails.

* Made changes from design review, including using chargingDisable in the callback, removing extra header files, and taking out the line that set the OS error location bitmap.

* master version of CAN driver

* fixed restart, changed CAN driver slightly. Commented out old updateVelocity.

* erged updatevelocity_rewrite

* verified all FSM transitions in Test_SendTritium on renode simulation platform

* fixed SendTritium test float printing

* fixed can msg construction

* fixed can message memory accesses for reading and sending to tritium

fixed cruise velocity setpoint to work with rpm instead of m/s

* fixed map function and removed magic number usage, also fixed scope macro (#298)

* bye bye old sim

* Changed Fault State, added exception struct, general assertion function, and rewrote OS assert. Created variable mailbox and mutex and moved semaphore to fault state.

* updated EnterFaultState with prio based switch case. combined ArrayMotorKill with nonreocverableFaultHandler.

* wrote callback functions for priority 2

* wrote assert function for tritium

* fixed function pointer implementation, added h files, unused variables

* Moved Faultstate mutex post, fixed a conditional for callbacks, added a macro for initializing CurrException

* Renamed _assertError to assertExceptionError, had the exception callback take void for arguments, changed the hall sensor callback to use if statements instead of making a new exception inside the callback.

* Fixed a couple more merge conflicts.

* Removed delay in fault state main thread.

* Created test file for the updated fault state with exception levels with priority levels. Wrote test for exception priorities 1 and 2 in an isolated environment apart from other application tasks.

* Added assertOSError test.

* Attempted to add a Faultstate test for ReadTritium using the FaultThread leader board and motor simulator files.

* Added draft of ReadCarCAN Exception test to FaultThread test.

* Fixed build by adding header file and small mistakes.

* printf debug

* printf now working!

* Changed the fault state exception assertion mutex to a binary semaphore since different tasks are setting and handling exceptions (a mutex would have to be posted by the same task that acquired it)

* Added missing calls to delete tasks for the tests.

* Added a print statement to assertOSError, changed assert functions to only use assertDisplayError and not assertUpdateDisplayError because they really do act the same and we didn't really use assertDisplayError in other apps anynways.

* Added a bit more to the Fault State tests, even though CAN communication between machines on Renode tests is not working.

* Added Contactors_Init(), made CarSim loop sending messages for testing purposes.

* Addressing review comments: removed macro, volatilequalifier for exception and EnterFaultState in header since they aren't needed.

* Some small things changed while testing

* Changed exception prio field to use an enum instead of an int.

* Added/updated comments at the tops of the test files explaining what they are supposed to do.

* Moved tasks done by the bps-sim and motor-sim on Renode into the main task in the leaderboard test in an attempt to avoid the many issues previously plaguing this endeavor.

* Test now runs on LoopBack mode. However, the weird print issue still happens if you let it run through a second round.

* Updated comments and added a print statement to readTritium for motor restarts.

* Beginning work for new fault thread mechanism that spawns new tasks of high or low priority to deal with errors. Created task files, made priority and macro changes in Tasks, add comments for planning.

* Began new redesign of error handling in which everything is localized to each task and scheduler locking is used to ensure high-priority faults are handled first. This commit includes finishing the error assertion function for the display, as well as renaming variable/function names related to assertDisplayError to reflect our new specification of using the word 'error' to describe when we have issues.

* Actually, this commit is where names get changed to replace 'fault' with 'error'

* Added a currError variable to hold the current error code for inspection purposes, removed task files from the last fault state redesign attempt which we have since aborted.

* Wrote assertTritiumError function for ReadTritium, created new nonrecoverableErrorHandler in Tasks.c to cover standard high-priority error actions like killing contactors, displaying the fault, and entering an infinite while loop.

* Renamed Display_Fault function to Display_Error.

* Added parameters and function to nonrecoverableErrorHnadler so that it can display the fault screen.

* Split nonrecoverableErrorHandler so that arrayMotorKill to turn off the motor and array contactors can be called outside of the nonrecoverable error handler, began work on the ReadCarCAN error assertion function

* Finished a first compiling draft of the ReadCarCAN assert function.

* Renamed the display error variable from currError to displayError.

* Added assertTaskError function to serve as a single assertion function that takes care of locking the scheduler and displaying a fault before jumping to a callback. This allows us to avoid locking the scheduler in many places throughout the code.

* Reordered and added new parameters to assertTaskError function.

* Rewrote readTritium error assertion to use the main assertTaskError function and a callback handler instead of the separate assertion functions for each task that we were planning earlier. Also added an enum to define options for locking the scheduler and choosing recoverable/nonrecoverable when asserting an error.

* Typedef-ed the error assertion option enums to enforce usage, reorder assertion and handler functions in readTritium so that the program can compile.

* Began moving assertError function for display errors from display driver to updatedisplay app.

* Moved error handling from Display.c to UpdateDisplay.c, decided to set global error variables for tasks inside individual task error assertion functions instead of in the main assertion function to avoid needing to figure out which location an error is coming from to set the correct error variable.

* Modified test file, ReadTritium hall and nonhall sensor error tests appear to work.

* This is a commit where I tried to test assertDisplayError by using the SetCruiseState function and it didn't work because it returns an error but nothing checks it and the assertDisplayError function is static. Next I'm going try using PutNext to overflow the fifo because I think that was something we wanted to test, but in order to do that I'm going to have to make it nonstatic.

* Removed assertDisplayError from UpdateDisplay header file to make it a static function

* Actually the display error thing seems to work as intended. Nothing major changed.

* Deleted faultstate things.

* Small documentation changes.

* Reverted ReadCarCAN to where it was before fix_ignition_delay branched off to prepare for merging update_faultstate into fix_ignition_delay.

* Renamed assertDisplayError to assertUpdateDisplayError to reduce changes to files.

* Deleted the motorsim and carsim files for faultstate testing because the current test runs in loopback mode and doesn't use extra Renode machines.

* Small documentation changes and file cleanup.

* Update faultstate testfile to work with new faultstate.

* Bringing improvements to the fault thread test from fix_ignition_delay so that it has the most recent changes, including those to test ReadCarCAN.

* Small changes made while updating error handling for ReadCarCAN on fix_ignition_delay.

* Renamed the scheduler lock option, changed error_code_t to a typedef.

* Update Apps/Src/ReadTritium.c

Accepted suggestion to restructure assertTritiumError for less-nested if-elses

Co-authored-by: Sidharth N. Babu <[email protected]>

* Update Apps/Src/ReadTritium.c

Accepted suggestion to change the small function handler_Tritium_HallError to an inline function

Co-authored-by: Sidharth N. Babu <[email protected]>

* Changed _assertOSError to use the os_error_loc_t, attempted to realign the updatedisplay enum.

* Update Apps/Src/ReadTritium.c

Co-authored-by: Sidharth N. Babu <[email protected]>

* Reattempting to align the UpdateDisplay.h error enum.

* Aligned(?) the comments from UpdateDisplay.h. I think it should all look nice now

* Changed assertTaskError to take an error_code_t, and changed error_code_t to be an int16_t.

* Modified ReadCarCAN so it compiles

* Addressed PR review comments. Mostly small changes and one doc alteration.

* Additional PR review changes (renaming things)

* Apply suggestions from code review

Co-authored-by: Ishan Deshpande <[email protected]>

* Addressed review comment: assertUpdateDisplayError takes in an UpdateDisplayError_t instead of a DisplayError_t

* Fixed assertUpdateDisplayError using the wrong error type.

* Additional review edits: Changed error_code_t from an int16_t to a uint16_t and made more functions use it, including Display_Error, which previously took a uint8_t instead. Also added more comments to assertTritiumError.

* Updated Test_DisplayApps to compile with new fault handling system.

* Addressed review and other comments: added prints to error infinite loop, made all nonrecoverable errors automatically lock the scheduler, renamed arrayMotorKill to EmergencyContactorOpen.

* Deleted #define NUM_TRITIUM_ERRORS and MotorController_getTritiumError which weren't being used.

* Added a comment about always locking the scheduler on nonrecoverable faults.

* Added additional comments to the test file.

* Removed redundant calls to assertUpdateDisplayError. The only ones left are those in UpdateDisplay_PutNext (UPDATEDISPLAY_ERR_FIFO_PUT, UpdateDisplay_PopNext(UPDATEDISPLAY_ERR_FIFO_POP, UPDATEDISPLAY_ERR_DRIVER from display_send), UpdateDisplay_SetComponent(UPDATEDISPLAY_ERR_PARSE_COMP). The rest of the errors that get returned from functions aren't necessary and can be removed in a future issue ticket.

* Changed fault handling for UpdateDisplay to clear the queue and reset the display every time instead of erroring out once a restart threshold has been reached.

* Made print statements and comments a bit clearer/cleaner.

* Display fault is now 4 hex digits

* Fixes during hardware testing to a) stop the display error from being overwritten by initializing display and update display earlier in the test and b) prevent the UpdateDisplay task from running into FIFO_POP errors by setting the message queue semaphore to zero when resetting the queue. This was also added to assertTaskError to ensure the error doesn't get overwritten, but this really shouldn't happen so we may want to take it out.

* Removed unnecessary value assignments from the UpdateDisplay error enum.

* Apply suggestions from code review

Removing unused macros and correcting documentation

Co-authored-by: Diya Rajon <[email protected]>

* Changed Display fault enum from positive to default values, added a delay in UpdateDisplay_Init since errors wouldn't be displayed unless we waited more than 215 ms before sending them.

* Name change

* Addressed review comments- changed assertTaskError to throwTaskError, changed a bit of throwTaskError logic, removed some comments.

* Renamed assertTaskError to throwTaskError in the test file.

* Changed a print statement so that it still makes sense when sending an error of NONE.

* Removed MOTOR_STOPPED macro.

* Removed the ERR_NO_CHANGE in UpdateDisplay and all checks for if things haven't changed since we weren't ever checking them anyways.

* Update Tests/Test_FaultThread_Exceptions.c

Accepted deletion of unnecessary tab

Co-authored-by: Ishan Deshpande <[email protected]>

* Moved ReadCarCAN Display_Evac into a callback for the BPSTrip message, changed the recoverability option to nonrecoverable.

* Fixed Doxygen comments.

* Removed references to OS_LOC in new faultstate code.

* Removed one more reference to OS LOC.

* Replaced accidentally-deleted Doxygen comment.

---------

Authored-by: Madeleine Lee <[email protected]>
  • Loading branch information
KnockbackNemo authored Oct 28, 2023
1 parent b741fb4 commit 5ae3439
Show file tree
Hide file tree
Showing 21 changed files with 863 additions and 505 deletions.
19 changes: 0 additions & 19 deletions Apps/Inc/FaultState.h

This file was deleted.

12 changes: 1 addition & 11 deletions Apps/Inc/ReadTritium.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
/**
* @copyright Copyright (c) 2018-2023 UT Longhorn Racing Solar
/* Copyright (c) 2021 UT Longhorn Racing Solar
* @file ReadTritium.h
* @brief
*
Expand All @@ -15,8 +14,6 @@
#include "common.h"
#include "Tasks.h"

#define CAR_STOPPED 0

/**
* Motor Error States
* Read messages from motor in ReadTritium and trigger appropriate error messages as needed based on bits
Expand All @@ -35,16 +32,9 @@ typedef enum{
T_INIT_FAIL = (1<<9), //motor controller fails to restart or initialize
T_NONE = 0x00,
} tritium_error_code_t;
#define NUM_TRITIUM_ERRORS 9 //9 errors, and 1 entry for no error

/**
* @brief Returns the current error status of the tritium controller
*/
tritium_error_code_t MotorController_getTritiumError(void);

float Motor_RPM_Get();
float Motor_Velocity_Get();
void MotorController_Restart();

#endif

Expand Down
78 changes: 43 additions & 35 deletions Apps/Inc/Tasks.h
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@

/**
* Priority Definitions
*/
#define TASK_FAULT_STATE_PRIO 1
*/
#define TASK_INIT_PRIO 2
#define TASK_READ_TRITIUM_PRIO 3
#define TASK_SEND_TRITIUM_PRIO 4
Expand All @@ -44,7 +43,6 @@
#define DEFAULT_STACK_SIZE 256
#define WATERMARK_STACK_LIMIT DEFAULT_STACK_SIZE/2

#define TASK_FAULT_STATE_STACK_SIZE DEFAULT_STACK_SIZE
#define TASK_INIT_STACK_SIZE DEFAULT_STACK_SIZE
#define TASK_SEND_TRITIUM_STACK_SIZE DEFAULT_STACK_SIZE
#define TASK_READ_CAR_CAN_STACK_SIZE DEFAULT_STACK_SIZE
Expand All @@ -55,12 +53,14 @@
#define TASK_DEBUG_DUMP_STACK_SIZE DEFAULT_STACK_SIZE
#define TASK_COMMAND_LINE_STACK_SIZE DEFAULT_STACK_SIZE

/**
* Task error variable type
*/
typedef uint16_t error_code_t;

/**
* Task Prototypes
*/
void Task_FaultState(void* p_arg);

void Task_Init(void* p_arg);

void Task_SendTritium(void* p_arg);
Expand All @@ -79,10 +79,11 @@ void Task_DebugDump(void *p_arg);

void Task_CommandLine(void* p_arg);



/**
* TCBs
*/
extern OS_TCB FaultState_TCB;
extern OS_TCB Init_TCB;
extern OS_TCB SendTritium_TCB;
extern OS_TCB ReadCarCAN_TCB;
Expand All @@ -97,7 +98,6 @@ extern OS_TCB CommandLine_TCB;
/**
* Stacks
*/
extern CPU_STK FaultState_Stk[TASK_FAULT_STATE_STACK_SIZE];
extern CPU_STK Init_Stk[TASK_INIT_STACK_SIZE];
extern CPU_STK SendTritium_Stk[TASK_SEND_TRITIUM_STACK_SIZE];
extern CPU_STK ReadCarCAN_Stk[TASK_READ_CAR_CAN_STACK_SIZE];
Expand All @@ -113,37 +113,12 @@ extern CPU_STK CommandLine_Stk[TASK_COMMAND_LINE_STACK_SIZE];
*/
extern OS_Q CANBus_MsgQ;

/**
* Semaphores
*/
extern OS_SEM FaultState_Sem4;

/**
* @brief Initialize the task switch hook
* Registers the hook with the RTOS
*/
void TaskSwHook_Init(void);


/**
* Global Variables
*/

/**
* Fault Enum
*
* Different fault states that need to be handled by the FaultState task
*/
typedef enum{
FAULT_NONE = 0x00, // No fault
FAULT_OS = 0x01, // for OS faults
FAULT_UNREACH = 0x02, // for unreachable conditions
FAULT_TRITIUM = 0x04, // for errors sent from the tritium
FAULT_READBPS = 0x08, // for unsuccessfully reading from BPS CAN
FAULT_DISPLAY = 0x10, // for display faults
FAULT_BPS = 0x20, // for if BPS trips
} fault_bitmap_t;

/**
* Task trace
*
Expand All @@ -160,10 +135,43 @@ typedef struct {

extern task_trace_t PrevTasks;

// Store error codes that are set in task error assertion functions
extern error_code_t Error_ReadTritium;
extern error_code_t Error_ReadCarCAN;
extern error_code_t Error_UpdateDisplay;

/**
* Error variables
*/
extern fault_bitmap_t FaultBitmap;
* Error-handling option enums
*/

// Scheduler lock parameter option for asserting a task error
typedef enum {
OPT_NO_LOCK_SCHED,
OPT_LOCK_SCHED
} error_scheduler_lock_opt_t;

// Recoverable/nonrecoverable parameter option for asserting a task error
typedef enum {
OPT_RECOV,
OPT_NONRECOV
} error_recov_opt_t;

/**
* @brief For use in error handling: opens array and motor precharge bypass contactor
* and turns on additional brakelight to signal that a critical error happened.
*/
void EmergencyContactorOpen();

/**
* @brief Assert a task error by setting the location variable and optionally locking the scheduler,
* displaying a fault screen (if nonrecoverable), jumping to a callback function, and entering an infinite loop.
* Called by task-specific error-assertion functions that are also responsible for setting the error variable.
* @param errorCode the enum for the specific error that happened
* @param errorCallback a callback function to a handler for that specific error,
* @param lockSched whether or not to lock the scheduler to ensure the error is handled immediately
* @param nonrecoverable whether or not to kill the motor, display the fault screen, and enter an infinite while loop
*/
void throwTaskError(error_code_t errorCode, callback_t errorCallback, error_scheduler_lock_opt_t lockSched, error_recov_opt_t nonrecoverable);

/**
* @brief Assert Error if OS function call fails
Expand Down
18 changes: 12 additions & 6 deletions Apps/Inc/UpdateDisplay.h
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@
* Error types
*/
typedef enum{
UPDATEDISPLAY_ERR_NONE = 0,
UPDATEDISPLAY_ERR_FIFO_PUT =-1, // Error putting command in fifo
UPDATEDISPLAY_ERR_FIFO_POP =-2, // Error popping command from fifo
UPDATEDISPLAY_ERR_PARSE_COMP =-3, // Error parsing component/val in SetComponent
UPDATEDISPLAY_ERR_NO_CHANGE =-4, // No change in component value
UPDATEDISPLAY_ERR_NONE,
UPDATEDISPLAY_ERR_FIFO_PUT, // Error putting command in fifo
UPDATEDISPLAY_ERR_FIFO_POP, // Error popping command from fifo
UPDATEDISPLAY_ERR_PARSE_COMP, // Error parsing component/val in SetComponent
UPDATEDISPLAY_ERR_DRIVER // Driver call returned an error
} UpdateDisplayError_t;

/**
Expand Down Expand Up @@ -130,6 +130,12 @@ UpdateDisplayError_t UpdateDisplay_SetRegenState(TriState_t state);
*/
UpdateDisplayError_t UpdateDisplay_SetCruiseState(TriState_t state);

#endif
/**
* @brief Clears the display message queue and sets the message counter semaphore value to 0
* @param none
* @returns none
*/
void UpdateDisplay_ClearQueue(void);

#endif
/* @} */
22 changes: 0 additions & 22 deletions Apps/Src/DebugDump.c
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,6 @@
#include "Tasks.h"
#include "SendTritium.h"

// global variables
extern fault_bitmap_t FaultBitmap;

static const char *MINIONPIN_STRING[] = {
FOREACH_PIN(GENERATE_STRING)
Expand All @@ -30,15 +28,6 @@ static const char *GEAR_STRING[] = {
/*----------------------------------------------*/
#define FAULT_BITMAP_NUM 6

static const char *FAULT_BITMAP_STRING[] = {
"FAULT_NONE",
"FAULT_OS",
"FAULT_UNREACH",
"FAULT_TRITIUM",
"FAULT_READBPS",
"FAULT_DISPLAY",
"FAULT_BPS"
};
/*----------------------------------------------*/

void Task_DebugDump(void* p_arg) {
Expand Down Expand Up @@ -76,17 +65,6 @@ void Task_DebugDump(void* p_arg) {
printf("Current Gear: %s\n\r", GEAR_STRING[get_gear()]);
print_float("Current Setpoint: ", get_currentSetpoint());

// fault bitmap
printf("Fault Bitmap: ");
if(FaultBitmap == FAULT_NONE){
printf("%s", FAULT_BITMAP_STRING[0]);
}else{
for(int i = 0; i < FAULT_BITMAP_NUM; i++){
if(FaultBitmap & (1 << i)){
printf("%s ", FAULT_BITMAP_STRING[i]);
}
}
}
printf("\n\r");

// Delay of 5 seconds
Expand Down
Loading

0 comments on commit 5ae3439

Please sign in to comment.