Feature/sf

src/av2/evaluation/scene_flow/SUBMISSION_FORMAT.md

@@ -0,0 +1,42 @@
+# 3D Scene Flow Submission Format
+
+The evaluation expects a zip archive of [Apache Feather](https://arrow.apache.org/docs/python/feather.html) files --- one for each example. The unzipped directory must have the format:
+
+```terminal
+- <test_log_1>/
+  - <test_timestamp_ns_1>.feather
+  - <test_timestamp_ns_2>.feather
+  - ...
+- <test_log_2>/
+- ...
+```
+
+The evaluation is run on a subset of the test set. Use the utility function `get_eval_subset` to get the `SceneFlowDataloader` indices to submit.  Each feather file should contain your flow predictions for the subset of points returned by `get_eval_mask` in the format:
+
+- `flow_tx_m` (float16): x-component of the flow (in meters) in the first sweeps's ego-vehicle reference frame.
+- `flow_ty_m` (float16): y-component of the flow (in meters) in the first sweeps's ego-vehicle reference frame.
+- `flow_tz_m` (float16): z-component of the flow (in meters) in the first sweeps's ego-vehicle reference frame.
+- `is_dynamic` (bool): Predicted dynamic/static labels for each point. A point is considered dynamic if its ground truth flow has a $\ell^2$-norm greater then $0.05 \textit{ m}$ once ego-motion has been removed.
+
+
+For example, the first log in the test set is `0c6e62d7-bdfa-3061-8d3d-03b13aa21f68` and the first timestamp is `315971435999927221`, so there should be a folder and file in the archive of the form: `0c6e62d7-bdfa-3061-8d3d-03b13aa21f68/315971435999927221.feather`. That fill should look like this:
+```python
+       flow_tx_m  flow_ty_m  flow_tz_m
+0      -0.699219   0.002869   0.020233
+1      -0.699219   0.002790   0.020493
+2      -0.699219   0.002357   0.020004
+3      -0.701172   0.001650   0.013390
+4      -0.699219   0.002552   0.020187
+...          ...        ...        ...
+68406  -0.703613  -0.001801   0.002373
+68407  -0.704102  -0.000905   0.002567
+68408  -0.704590  -0.001390   0.000397
+68409  -0.704102  -0.001608   0.002283
+68410  -0.704102  -0.001619   0.002207
+```
+The file `example_submission.py` contains a basic example of how to output the submission files. The script `make_submission_archive.py` will create the zip archive for you and validate the submission format. Then you can submit the outputted file to the competition leaderboard!
+
+
+# Local Evaluation
+
+Before evaluating on the _test_ set, you will want to evaluate your model on the _validation_ set. To do this, first run `make_annotation_files.py` to create a set of files containing the minimum ground truth flow information needed to run the evaluation. Then, once you have your output saved in the feather files described above, run `eval.py` to compute all of the leaderboard metrics.

src/av2/evaluation/scene_flow/constants.py

@@ -0,0 +1,104 @@
+"""Constants for scene flow evaluation."""
+
+from __future__ import annotations
+
+from enum import Enum, unique
+from typing import Final
+
+import av2.evaluation.scene_flow.eval as eval
+from av2.datasets.sensor.constants import AnnotationCategories
+
+SCENE_FLOW_DYNAMIC_THRESHOLD: Final = 0.05
+SWEEP_PAIR_TIME_DELTA: Final = 0.1
+
+CATEGORY_TO_INDEX: Final = {**{"NONE": 0}, **{k.value: i + 1 for i, k in enumerate(AnnotationCategories)}}
+
+FLOW_METRICS: Final = {
+    "Accuracy Relax": eval.compute_accuracy_relax,
+    "Accuracy Strict": eval.compute_accuracy_strict,
+    "Angle Error": eval.compute_angle_error,
+    "EPE": eval.compute_end_point_error,
+}
+SEGMENTATION_METRICS: Final = {
+    "TP": eval.compute_true_positives,
+    "TN": eval.compute_true_negatives,
+    "FP": eval.compute_false_positives,
+    "FN": eval.compute_false_negatives,
+}
+
+
+@unique
+class InanimateCategories(str, Enum):
+    """Annotation categories representing inanimate objects that aren't vehicles."""
+
+    BOLLARD = "BOLLARD"
+    CONSTRUCTION_BARREL = "CONSTRUCTION_BARREL"
+    CONSTRUCTION_CONE = "CONSTRUCTION_CONE"
+    MOBILE_PEDESTRIAN_CROSSING_SIGN = "MOBILE_PEDESTRIAN_CROSSING_SIGN"
+    SIGN = "SIGN"
+    STOP_SIGN = "STOP_SIGN"
+
+
+@unique
+class LeggedCategories(str, Enum):
+    """Annotation categories representing objects that move using legs."""
+
+    ANIMAL = "ANIMAL"
+    DOG = "DOG"
+    OFFICIAL_SIGNALER = "OFFICIAL_SIGNALER"
+    PEDESTRIAN = "PEDESTRIAN"
+
+
+@unique
+class SmallVehicleCategories(str, Enum):
+    """Annotation categories representing small vehicles."""
+
+    BICYCLE = "BICYCLE"
+    BICYCLIST = "BICYCLIST"
+    MOTORCYCLE = "MOTORCYCLE"
+    MOTORCYCLIST = "MOTORCYCLIST"
+    STROLLER = "STROLLER"
+    WHEELCHAIR = "WHEELCHAIR"
+    WHEELED_DEVICE = "WHEELED_DEVICE"
+    WHEELED_RIDER = "WHEELED_RIDER"
+
+
+@unique
+class VehicleCategories(str, Enum):
+    """Annotation categories representing regular vehicles."""
+
+    ARTICULATED_BUS = "ARTICULATED_BUS"
+    BOX_TRUCK = "BOX_TRUCK"
+    BUS = "BUS"
+    LARGE_VEHICLE = "LARGE_VEHICLE"
+    MESSAGE_BOARD_TRAILER = "MESSAGE_BOARD_TRAILER"
+    RAILED_VEHICLE = "RAILED_VEHICLE"
+    REGULAR_VEHICLE = "REGULAR_VEHICLE"
+    SCHOOL_BUS = "SCHOOL_BUS"
+    TRAFFIC_LIGHT_TRAILER = "TRAFFIC_LIGHT_TRAILER"
+    TRUCK = "TRUCK"
+    TRUCK_CAB = "TRUCK_CAB"
+    VEHICULAR_TRAILER = "VEHICULAR_TRAILER"
+
+
+@unique
+class MetricBreakdownCategories(str, Enum):
+    """Meta-categories for the scene flow task."""
+
+    ALL = "All"
+    BACKGROUND = "Background"
+    FOREGROUND = "Foreground"
+
+
+NO_CLASS_BREAKDOWN: Final = {MetricBreakdownCategories.ALL: list(range(31))}
+FOREGROUND_BACKGROUND_BREAKDOWN: Final = {
+    MetricBreakdownCategories.BACKGROUND: [0],
+    MetricBreakdownCategories.FOREGROUND: [
+        CATEGORY_TO_INDEX[k.value]
+        for k in (
+            list(InanimateCategories) + list(LeggedCategories) + list(SmallVehicleCategories) + list(VehicleCategories)
+        )
+    ],
+}
+
+FLOW_COLUMNS: Final = ("flow_tx_m", "flow_ty_m", "flow_tz_m")