Convert to user selected camera matching (#1556)

docs/source/docs/contributing/design-descriptions/camera-matching.md

@@ -0,0 +1,112 @@
+# Camera Matching
+
+Diagrams generated by the [PlantUML UML editor](https://www.plantuml.com/plantuml/). Copy the image URLs below and decode in the editor to make changes.
+
+## Initial Setup
+
+When PhotonVision first starts, settings are loaded from disk and [VisionSources](https://javadocs.photonvision.org/org/photonvision/vision/processes/VisionSource.html) are created for every serialized & active [Camera Configuration](https://javadocs.photonvision.org/org/photonvision/common/configuration/CameraConfiguration.html)
+
+![](https://www.plantuml.com/plantuml/png/VP5FQnin4CNl-XI3JotK-DAJAI6fIw6GfOMbFkKoramSqTKVfF6MVFkETfKsei6trVpUldbwkYs2MIv-CeI29omCcn5d9XXPn8LpsG0MAErWaggTTGc3m6P05nRizQD7HrTS3336IxOC0mOySrwqS_5lIeT8bubxgVTNN9jRhpYCXvXNP8lLpokxsWvZNcwtlQaNsSDzH8B773sGAxzC7MvlDFSUxeXWKie4DeP7futelC8z73AZCDnPSJD35xKOh5F5DR31IU3d-1aiUive06PTlSRTm_V4eH4uFJ-4Aamn2xmxFMyJojDx0x2AjtNn-WSJ73_UltRyzC_o2mjRQH1IZecpE4t5WPOmX_5R7sPof_NyVvwghNbK-LVL1sbErTneFLqxNxF27pdEZZXNs8gjbJFrhHdYLxMredrx1Obm70QZvnUBtKxdJE2NnosxNVj3qIYO1GB_Rb3DEZAlQxKPowMuS7u8oIMUNE0F84-PaOgvvK0NF_q1)
+
+## UI Workflow
+
+A [background thread](https://javadocs.photonvision.org/org/photonvision/common/util/TimedTaskManager.html) will periodically query CSCore and Libcamera for what cameras we currently see connected. This list is provided to the web UI for display.
+
+![](https://www.plantuml.com/plantuml/svg/POvDJyCm343l-HLMxnFt7j14uJ099AHkSCvSCopoCSLE-FjaxQW8kpbwpy_PYjgasJk3qJb2vHW4kZrxcc1lvGjURB0dIXrO0LLlpBakCFBP1eNkZQLkm1XpGchS8hvLXt68YMQ6WdLiyJCVqNfATZRSxwkLtka8XzriP3P6rM_kww4U7hac2oK8z0qJ5KOIKwJYvLOFJo5VUafm61zWYOjPwEPQ6M88X4fJuyoPzKD_IyEuMwrLk8rLhOrbxk4rooVWwbmvE1Rz9rbKBdJ7OHakInzy4hEbC6NlVW00)
+
+![](images/matching_ui.png)
+
+This UI allows users to "Activate" a camera that's never been seen before, or activate a CameraConfiguration we've seen before but was disabled. Allowing camera configurations to be saved but not loaded by default lets us support temporarily disabling/unplugging a camera without flooding log files.
+
+Since our backend logic intentionally does not protect users from plugging camera B into the port that camera A was active on, the UI shall show a warning but vision processing will (attempt to) continue like normal.
+
+### Activate New Camera
+
+When a new camera (ie, one we can't match by-path to a deserialized CameraConfiguration) is activated, we'll create a spin up a new Vision Module for it
+
+![](https://www.plantuml.com/plantuml/svg/VL7B3jCm4BpxArOzWKIK2wSALOKWf4gDG8fQBhquzggrY1_u4TI_PvCufGRKK-ATySpiU1yYzp7fWJdwAg4SDn4stx67qs43F41I9NHMGLa3dKrU8BJSy2lwcJa6_LzgQsKQ_g9g_K8rgvMCfckiNo0H1FsMy57rWclqV6OCw-b5e1o4iQIg7MNVmaSfeRz3CkfdGZ0am6YUmOuR5UyWRYX-X7M-XSOZZmX5_i2uY6ga-RG5uqE4K_S9SYAWORLRTjZ2LuSc8-HzCHFHMH_XJN-l78-tjmpomjNakDn02UVtnrKHZPnDckvGcZng-DU7kBCFCH-imk1PdDRzy2VoPumeuYhcl7L87UDKIj795q-CRzwEIgAVmDpaqNA9igoCINpgBDUhyvj42-UsPNHU9UgQvgIXvvSCTRtUe7UAt4Sm-2k395OWus9BiGM6eCprOfnoE2Y3xo3UF78Ps1wDJ7hu3G00)
+
+### Deactivate Camera
+
+Deactivating a camera will release the native resources it owns, and return the CameraConfiguration to the pool of currently disabled cameras we can re-enable later.
+
+![](https://www.plantuml.com/plantuml/svg/ROxBJiCm44Nt_efHLtIH7yWYgWWB8gYeI0eRDXDdW97yABQd5N-FvV1G8bOUppdNrxkOC2InHftooPfFw19idcc4OxS1Z22yH4ySsJlelGHDi4U7RnIAUOxsNtNl9p4hrQxKjczzeC9qr7bSudiUDLeAM0ppSrDAk6foRmqtX3hn6HD16GXcvSMDdo2EFuJ0vOtATexO77aawxDdo_TKNbLLCvVNq1eV_vwuwbxXs5zllwNV_Xe6mZ3vYrkeRTzjvvv6k8Q3n7TmT86OC541LG6tmt20Xpkr8pU9DLy0)
+
+### Reactivate a CameraConfig
+
+When a new camera (ie, one we can't match by-path to a deserialized CameraConfiguration) is activated, we'll create and spin up a new Vision Module for it.
+
+![](https://www.plantuml.com/plantuml/svg/VP9BYnD158NtzIikirAmoSPL8s5YH1n8SB1duYQRsrtNcGlrAElHadzlLVgXXP9LACvNvvmwwViGqSUabN3vbmTsQ2BSVQSUdX_k00CahgKJ1xO6EflyG714Wo_ah-GOz7_HevL9KOrgVSDrTgk9VRUtVfA6C5XFjNpWVa1D7g-4Maut2ir5X4ZSR7Ft5huH3f57Z0II0_QA94msPzDV81d-cGWCQX82LOJdxYCuwoEmWHH8G9cWsIPkuSlJqoFyG5R9ao0ZXIXIZcbXxwaax4eKGVNm8DO2OrWpvWvN-sOxFRw5huxCh41_EPkrp9l-qZYChsy5m0GtKt2vGH9Exm-BOobMGlRTGnsoxlTlJc5BJYPNgWgOuUNL7_vK_aIHXhYOEMyT-SWKCbLDyzbduj7RaINv8ix_py6Y95bF9YJzjTcyiixmJag85ax7eyZdnMApsSdYeQ-VGDXibXijT15z14E_5b6CbJ9EiRdsG26mUJaRnuuK6te7yTKJoY3koSYarMy0)
+
+# Camera Matching Requirements
+
+## Definitions
+- VALID USB PATH: a path in the form `/dev/v4l/by-path/[UUID]`
+- VIDEO DEVICE PATH: a CSCore-provided identifier derived from the V4L path `/dev/video[N]` on Linux, or an opaque string on Windows
+- UNIQUE NAME: an identifier that is unique within the set of all deserialized CameraConfigurations and unmatched USB cameras
+  - I don't love this, it means that a USB camera matched to a VisionModule will share a UNIQUE NAME, right?
+- DESERIALIZED CAMERA CONFIGURATIONS: The set of camera configurations loaded from disk and provided to the VisionSourceManager. This configuration data structure includes the UNIQUE NAME
+- CURRENTLY ACTIVE CAMERAS: The set of VisionModules currently active and processing vision data, and associated metadata
+
+## Startup:
+
+- GIVEN An emtpy set of deserialized Camera Configurations
+<br>WHEN PhotonVision starts
+<br>THEN no VisionModules will be started
+
+- GIVEN A valid set of deserialized Camera Configurations
+<br>WHEN PhotonVision starts
+<br>THEN VisionModules will be started FOR EACH un-DISABLED config
+
+- GIVEN A valid set of deserialized Camera Configurations
+<br>WHEN PhotonVision starts
+<br>THEN VisionModules will NOT be started FOR EACH DISABLED config
+
+- GIVEN A CameraConfiguration with a VALID USB PATH
+<br>WHEN a VisionModule is created
+<br>THEN The VisionModule shall open the camera using the USB path
+
+- GIVEN A CameraConfiguration without a valid USB path
+<br>WHEN a VisionModule is created
+<br>THEN The VisionModule shall open the camera using the VIDEO DEVICE PATH
+
+## Camera (re)enumeration:
+
+- GIVEN a NEW USB CAMERA is avaliable for enumeration
+<br>WHEN a USB camera is discovered by VisionSourceManager
+<br>AND the USB camera's VIDEO DEVICE PATH is not in the set of DESERIALIZED CAMERA CONFIGURATIONS
+<br>THEN a UNIQUE NAME will be assigned to the camera info
+
+- GIVEN a NEW USB CAMERA is avaliable for enumeration
+<br>WHEN a USB camera is discovered by VisionSourceManager
+<br>AND the USB camera's VIDEO DEVICE PATH is in the set of DESERIALIZED CAMERA CONFIGURATIONS
+<br>THEN a UNIQUE NAME equal to the matching DESERIALIZED CAMERA CONFIGURATION will be assigned to the camera info
+  - This is a weird case. How -should- we handle this? see above
+
+## Creating from a new camera
+
+- Given: A UNIQUE NAME from a NEW USB CAMERA
+<br>WHEN I request a new VisionModule is created for this NEW USB CAMREA
+<br>AND the camera has a VALID USB PATH
+<br>AND the camera's VALID USB PATH is not in use by any CURRENTLY ACTIVE CAMERAS
+<br>THEN a NEW VisionModule will be started for the NEW USB CAMERA using the VALID USB PATH
+
+- Given: A UNIQUE NAME from a NEW USB CAMERA
+<br>WHEN I request a new VisionModule is created for this NEW USB CAMREA
+<br>AND the camera does not have a VALID USB PATH
+<br>AND the camera's VIDEO DEVICE PATH is not in use by any CURRENTLY ACTIVE CAMERAS
+<br>THEN a NEW VisionModule will be started for the NEW USB CAMERA using the VIDEO DEVICE PATH
+
+## Deactivate
+
+- Given: A UNIQUE NAME from a CURRENTLY ACTIVE CAMERA
+<br>WHEN I request the VisionModule be DEACTIVATED
+<br>THEN the VisionModule will be stopped for the given CURRENTLY ACTIVE CAMERA
+<br>AND the CameraConfiguration DISABLED flag will be set to TRUE
+
+## Reactivate
+
+- Given: A UNIQUE NAME from a DESERIALIZED CAMERA CONFIGURATIONS
+<br>WHEN I request the VisionModule be ACTIVATED
+<br>AND the CameraConfiguration's DISABLED flag is TRUE
+<br>THEN a VisionModule will be created and started for the camera

docs/source/docs/contributing/design-descriptions/index.md

@@ -4,4 +4,5 @@
 :maxdepth: 1
 image-rotation
 time-sync
+camera-matching
 ```

photon-client/package.json

@@ -20,11 +20,13 @@
     "@msgpack/msgpack": "^3.0.0-beta2",
     "axios": "^1.6.3",
     "jspdf": "^2.5.1",
+    "lodash": "^4.17.21",
     "pinia": "^2.1.4",
     "three": "^0.160.0",
     "vue": "^2.7.14",
     "vue-router": "^3.6.5",
     "vue-virtual-scroll-list": "^2.3.5",
+    "vue2-helpers": "^2.1.1",
     "vuetify": "^2.7.1"
   },
   "devDependencies": {

photon-client/src/App.vue

@@ -40,6 +40,9 @@ if (!is_demo) {
       if (data.calibrationData !== undefined) {
         useStateStore().updateCalibrationStateValuesFromWebsocket(data.calibrationData);
       }
+      if (data.visionSourceManager !== undefined) {
+        useStateStore().updateDiscoveredCameras(data.visionSourceManager);
+      }
     },
     () => {
       useStateStore().$patch({ backendConnected: false });

photon-client/src/assets/styles/variables.scss

@@ -3,6 +3,11 @@
 $default-font: "Prompt", sans-serif !default;
 $body-font-family: $default-font;
 $heading-font-family: $default-font;
+$body-background: #282c34;
+
+body {
+  background: $body-background;
+}
 
 .v-application {
   font-family: $default-font !important;

photon-client/src/components/app/photon-camera-stream.vue

@@ -1,20 +1,20 @@
 <script setup lang="ts">
 import { computed, inject, ref, onBeforeUnmount } from "vue";
-import { useCameraSettingsStore } from "@/stores/settings/CameraSettingsStore";
 import { useStateStore } from "@/stores/StateStore";
 import loadingImage from "@/assets/images/loading.svg";
 import type { StyleValue } from "vue/types/jsx";
 import PvIcon from "@/components/common/pv-icon.vue";
+import type { UiCameraConfiguration } from "@/types/SettingTypes";
 
 const props = defineProps<{
   streamType: "Raw" | "Processed";
   id: string;
+  cameraSettings: UiCameraConfiguration;
 }>();
 
 const emptyStreamSrc = "//:0";
 const streamSrc = computed<string>(() => {
-  const port =
-    useCameraSettingsStore().currentCameraSettings.stream[props.streamType === "Raw" ? "inputPort" : "outputPort"];
+  const port = props.cameraSettings.stream[props.streamType === "Raw" ? "inputPort" : "outputPort"];
 
   if (!useStateStore().backendConnected || port === 0) {
     return emptyStreamSrc;
@@ -32,8 +32,12 @@ const streamStyle = computed<StyleValue>(() => {
 });
 
 const containerStyle = computed<StyleValue>(() => {
-  const resolution = useCameraSettingsStore().currentVideoFormat.resolution;
-  const rotation = useCameraSettingsStore().currentPipelineSettings.inputImageRotationMode;
+  if (props.cameraSettings.validVideoFormats.length === 0) {
+    return { aspectRatio: "1/1" };
+  }
+  const resolution =
+    props.cameraSettings.validVideoFormats[props.cameraSettings.pipelineSettings.cameraVideoModeIndex].resolution;
+  const rotation = props.cameraSettings.pipelineSettings.inputImageRotationMode;
   if (rotation === 1 || rotation === 3) {
     return {
       aspectRatio: `${resolution.height}/${resolution.width}`
@@ -54,9 +58,9 @@ const overlayStyle = computed<StyleValue>(() => {
 
 const handleCaptureClick = () => {
   if (props.streamType === "Raw") {
-    useCameraSettingsStore().saveInputSnapshot();
+    props.cameraSettings.pipelineSettings[props.cameraSettings.currentPipelineIndex].saveInputSnapshot();
   } else {
-    useCameraSettingsStore().saveOutputSnapshot();
+    props.cameraSettings.pipelineSettings[props.cameraSettings.currentPipelineIndex].saveOutputSnapshot();
   }
 };
 const handlePopoutClick = () => {
@@ -69,6 +73,16 @@ const handleFullscreenRequest = () => {
 };
 
 const mjpgStream: any = ref(null);
+
+const handleStreamError = () => {
+  if (streamSrc.value && streamSrc.value !== emptyStreamSrc) {
+    console.error("Error loading stream:", streamSrc.value, " Trying again.");
+    setTimeout(() => {
+      mjpgStream.value.src = streamSrc.value;
+    }, 100);
+  }
+};
+
 onBeforeUnmount(() => {
   if (!mjpgStream.value) return;
   mjpgStream.value["src"] = emptyStreamSrc;
@@ -79,14 +93,14 @@ onBeforeUnmount(() => {
   <div class="stream-container" :style="containerStyle">
     <img :src="loadingImage" class="stream-loading" />
     <img
-      v-show="streamSrc !== emptyStreamSrc"
       :id="id"
       ref="mjpgStream"
       class="stream-video"
       crossorigin="anonymous"
       :src="streamSrc"
       :alt="streamDesc"
       :style="streamStyle"
+      @error="handleStreamError"
     />
     <div class="stream-overlay" :style="overlayStyle">
       <pv-icon

photon-client/src/components/app/photon-sidebar.vue

@@ -2,6 +2,9 @@
 import { computed, getCurrentInstance } from "vue";
 import { useSettingsStore } from "@/stores/settings/GeneralSettingsStore";
 import { useStateStore } from "@/stores/StateStore";
+import { useCameraSettingsStore } from "@/stores/settings/CameraSettingsStore";
+import { PlaceholderCameraSettings } from "@/types/SettingTypes";
+import { useRoute } from "vue2-helpers/vue-router";
 
 const compact = computed<boolean>({
   get: () => {
@@ -14,6 +17,12 @@ const compact = computed<boolean>({
 
 // Vuetify2 doesn't yet support the useDisplay API so this is required to access the prop when using the Composition API
 const mdAndUp = computed<boolean>(() => getCurrentInstance()?.proxy.$vuetify.breakpoint.mdAndUp || false);
+
+const needsCamerasConfigured = computed<boolean>(() => {
+  return (
+    useCameraSettingsStore().cameras.length === 0 || useCameraSettingsStore().cameras[0] === PlaceholderCameraSettings
+  );
+});
 </script>
 
 <template>
@@ -35,20 +44,32 @@ const mdAndUp = computed<boolean>(() => getCurrentInstance()?.proxy.$vuetify.bre
           <v-list-item-title>Dashboard</v-list-item-title>
         </v-list-item-content>
       </v-list-item>
+      <v-list-item link to="/settings">
+        <v-list-item-icon>
+          <v-icon>mdi-cog</v-icon>
+        </v-list-item-icon>
+        <v-list-item-content>
+          <v-list-item-title>Settings</v-list-item-title>
+        </v-list-item-content>
+      </v-list-item>
       <v-list-item ref="camerasTabOpener" link to="/cameras">
         <v-list-item-icon>
           <v-icon>mdi-camera</v-icon>
         </v-list-item-icon>
         <v-list-item-content>
-          <v-list-item-title>Cameras</v-list-item-title>
+          <v-list-item-title>Camera</v-list-item-title>
         </v-list-item-content>
       </v-list-item>
-      <v-list-item link to="/settings">
+      <v-list-item
+        link
+        to="/cameraConfigs"
+        :class="{ cameraicon: needsCamerasConfigured && useRoute().path !== '/cameraConfigs' }"
+      >
         <v-list-item-icon>
-          <v-icon>mdi-cog</v-icon>
+          <v-icon :class="{ 'red--text': needsCamerasConfigured }">mdi-swap-horizontal-bold</v-icon>
         </v-list-item-icon>
         <v-list-item-content>
-          <v-list-item-title>Settings</v-list-item-title>
+          <v-list-item-title :class="{ 'red--text': needsCamerasConfigured }">Camera Matching</v-list-item-title>
         </v-list-item-content>
       </v-list-item>
       <v-list-item link to="/docs">
@@ -119,4 +140,18 @@ const mdAndUp = computed<boolean>(() => getCurrentInstance()?.proxy.$vuetify.bre
   height: 70px;
   object-fit: contain;
 }
+
+.cameraicon {
+  animation: pulse 2s infinite;
+}
+
+@keyframes pulse {
+  0%,
+  100% {
+    transform: scale(0.95);
+  }
+  50% {
+    transform: scale(1.05);
+  }
+}
 </style>

photon-client/src/components/cameras/CameraCalibrationCard.vue

@@ -20,6 +20,7 @@ const settingsValid = ref(true);
 
 const getUniqueVideoFormatsByResolution = (): VideoFormat[] => {
   const uniqueResolutions: VideoFormat[] = [];
+  if (useCameraSettingsStore().currentCameraSettings.validVideoFormats.length === 0) return uniqueResolutions;
   useCameraSettingsStore().currentCameraSettings.validVideoFormats.forEach((format) => {
     const index = uniqueResolutions.findIndex((v) => resolutionsAreEqual(v.resolution, format.resolution));
     const contains = index != -1;
@@ -248,7 +249,7 @@ const setSelectedVideoFormat = (format: VideoFormat) => {
           </v-simple-table>
         </v-row>
         <v-divider />
-        <v-row style="display: flex; flex-direction: column" class="mt-4">
+        <v-row v-if="useCameraSettingsStore().isConnected" style="display: flex; flex-direction: column" class="mt-4">
           <v-card-subtitle v-show="!isCalibrating" class="pl-3 pa-0 ma-0"> Configure New Calibration</v-card-subtitle>
           <v-form ref="form" v-model="settingsValid" class="pl-4 mb-10 pr-5">
             <!-- TODO: the default videoFormatIndex is 0, but the list of unique video mode indexes might not include 0. getUniqueVideoResolutionStrings indexing is also different from the normal video mode indexing -->