Merged commit includes the following changes:

427369385  by Waymo Research:

    Internal change

--
427363374  by Waymo Research:

    Add command line tools for semantic segmentation metrics
    This CL adds a command line tool for generating prediction file and a tool for calculating metrics.

--
427352112  by Waymo Research:

    Add a colab tutorial for the 3D semantic segmentation labels.

--
427343731  by Waymo Research:

    Update bibtex citation for motion dataset

--
427337830  by Waymo Research:

    Update for perception dataset v1.3 release

--
427301282  by Waymo Research:

    Tutorial for using and visualizing human keypoints.

--
427299085  by Waymo Research:

    Keypoint util to crop image and keypoints to a bounding box.

--
427283846  by Waymo Research:

    Fix a typo

--
427005379  by Waymo Research:

    Add semantic segmentation metrics calculation utils.
    This cl adds util function to calculate mean iou for semantic segmentation.

--
426999326  by Waymo Research:

    Functions to draw 2D and 3D keypoints in colab.

--
426440925  by Waymo Research:

    Add segmentation label type.

--
426274400  by Waymo Research:

    Functions to build wireframes from human keypoints for visualization purposes.

--
425752522  by Waymo Research:

    N/A

--
425741762  by Waymo Research:

    Append TP, FP, FN to the Precision-Recall tensor.

--
425734533  by Waymo Research:

    An util library to simplify visualization of keypoint keypoints.

--
425734181  by Waymo Research:

    Add pytype annotations to frame_utils.

--
425657425  by Waymo Research:

    Changes ParseScenarioAndPredictionsFromTensors to work if the first agent(s) are invalid, but later agents are valid.

--
424706428  by Waymo Research:

    Add a soft mAP metric to the open motion dataset.

--
424641292  by Waymo Research:

    Fix pip docker image building scripts.

--
424161283  by Waymo Research:

    Changes std::vector to use reserve to ensure empty objects are not created.

--
420333827  by Waymo Research:

    Print better ShardName for BreakdownGeneratorAllButSign

--
418706218  by Waymo Research:

    Added proto fields to store camera -> lidar object association.

--
418094418  by Waymo Research:

    Added a new type for FOREHEAD keypoints.

--
413021210  by Waymo Research:

    Migrate to protobuf default fast c++ extension

--
411919991  by Waymo Research:

    Migrate to protobuf default fast c++ extension

--
410384101  by Waymo Research:

    Force garbage collection in reset_states() of detection evaluator.

--
408748477  by Waymo Research:

    Add an extra namespace for human keypoint protos.

--
407649419  by Waymo Research:

    Proto messages and fields to store human keypoints for the Perception dataset.

--
398333246  by Waymo Research:

    Fix memory overbound reading which caused tsan test failure

--

PiperOrigin-RevId: 427369385

--
429336959  by Waymo Research:

    Internal change

--
429197557  by Waymo Research:

    Internal change

--
429143515  by Waymo Research:

    Colab tutorial for WOD occupancy flow challenge.

--
429142549  by Waymo Research:

    Utilities for WOD occupancy flow challenge.

--
427909488  by Waymo Research:

    Internal change.

--
427906270  by Waymo Research:

    OKS metric for human keypoints.

--
427902082  by Waymo Research:

    Internal change.

--
429712750  by Waymo Research:

    Add licence headers for new keypoint modules.

--
429712662  by Waymo Research:

    Add examples into colab how to use metrics.

--
429712618  by Waymo Research:

    Tools to convert keypoint protos into tensors.

--
429712128  by Waymo Research:

    Add per class IOU for 3D semantic segmentation.
    Also fix some lints.

--
429694437  by Waymo Research:

    Split keypoint utils into data and drawing utils.

--
429693761  by Waymo Research:

    WOD occupancy flow challenge: add instructions for running a Jupyter kernel locally.

--
429686904  by Waymo Research:

    Improve wording issues for human keypoints.

--
429685905  by Waymo Research:

    Tools to create all supported metrics for keypoints.

--
429685685  by Waymo Research:

    Metric Percetage of Correct Keypoints (PCK).

--
429685523  by Waymo Research:

    MPJPE metric for human keypoints.

--
429685363  by Waymo Research:

    Refactor keypoint OKS metric by extracting a base class for all metrics.

--
429667913  by Waymo Research:

    Update the 3d semseg types and tutorial colab.

--
429662948  by Waymo Research:

    internal updates

--
429651913  by Waymo Research:

    Update readme

--
429617238  by Waymo Research:

    Minor readability changes for the open motion dataset metrics.

--
429466469  by Waymo Research:

    Add semantic segmentation submission proto and update dummy files.

--
429416918  by Waymo Research:

    WOD occupancy flow challenge: create a separate submission file for each test shard.

    This ensures that the output protobufs stay under the 2GB limit.  It also reduces the memory requirement for running the colab.

--
430520427  by Waymo Research:

    Add license headers to files which are missing it.

--
429784107  by Waymo Research:

    Update a comment

--

33277383  by Waymo Research:

    Internal change

--
433261081  by Waymo Research:

    Fix typos and minor issues in the code related to keypoints.

--
433132388  by Waymo Research:

    Clarify comments in occupancy flow modules.

--
432596992  by Waymo Research:

    Support compression and uncompression of segmentation labels.

--
432340005  by Waymo Research:

    Update command for creating submission tar.gz file.

--
432263484  by Waymo Research:

    Use list of scenario_ids in WOD occupancy flow tutorial notebook.

--
432248767  by Waymo Research:

    Internal change only.

--
432096622  by Waymo Research:

    internal change

--
432048666  by Waymo Research:

    Internal change.

--
432043703  by Waymo Research:

    Update semantic segmentation proto to allow multiple range images in each frame

--
431090436  by Waymo Research:

    Specify effective validation set for WOD occupancy flow challenge.

    Simplify submission proto messages.

--
433359745  by Waymo Research:

    Add arxiv link for the occupancy flow paper to the colab.

--
433341994  by Waymo Research:

    Internal updates

--
433377526  by Waymo Research:

    Normalize sdc yaw when generating occupancy flow ground-truth.

--
433505401  by Waymo Research:

    Minor fixes to the tutorial colabs. Updated the 3d semseg image.

--
433499289  by Waymo Research:

    Minor fixes to the keypoint tutorial.

--
433517877  by Waymo Research:

    Fix json for the keypoint tutorial.

--
433555762  by Waymo Research:

    Fix json for occupancy flow tutorial notebook.

--
433541722  by Waymo Research:

    Update the 3d semseg visualization image. Added a legend.

--

README.md

@@ -1,10 +1,27 @@
 # Waymo Open Dataset
 
-The Waymo Open Dataset was first launched in August 2019 with a perception dataset comprising high resolution sensor data and labels for 1,950 segments. We have released the Waymo Open Dataset publicly to aid the research community in making advancements in machine perception and autonomous driving technology.
+We have released the Waymo Open Dataset publicly to aid the research community in making advancements in machine perception and autonomous driving technology.
+
+The Waymo Open Dataset is composed of two datasets - the Perception dataset with high resolution sensor data and labels for 1,950 scenes, and the Motion dataset with object trajectories and corresponding 3D maps for 103,354 scenes.
+
+
+## March 2022 Update
+
+We released v1.3 of the Perception dataset and the 2022 challenges. We have updated this repository to add support for the new labels and the challenges.
+ - Added 3D semantic segmentation labels, tutorial, and metrics.
+ - Added 2D and 3D keypoint labels, tutorial, and metrics.
+ - Added correspondence between 2D and 3D labels.
+ - Added tutorial and utilities for Occupancy Flow Prediction Challenge.
+ - Added the soft mAP metric for Motion Prediction Challenge.
+
+Coming soon:
+ - Add scenes with camera imagery, as a test set for the Camera-Only 3D Detection Challenge.
+ - Add a new metric for the Camera-Only 3D Detection Challenge
+
 
 ## September 2021 Update
 
-We released v1.1 of the motion dataset to include lane connectivity information. To read more on the technical details, please read [lane_neighbors_and_boundaries.md](docs/lane_neighbors_and_boundaries.md).
+We released v1.1 of the Motion dataset to include lane connectivity information. To read more on the technical details, please read [lane_neighbors_and_boundaries.md](docs/lane_neighbors_and_boundaries.md).
  - Added lane connections. Each lane has a list of lane IDs that enter or exit the lane.
  - Added lane boundaries.  Each lane has a list of left and right boundary features associated with the lane and the segment of the lane where the boundary is active.
  - Added lane neighbors. Each lane has a list of left and right neighboring lanes.  These are lanes an agent may make a lane change into.
@@ -13,7 +30,7 @@ We released v1.1 of the motion dataset to include lane connectivity information.
 
 ## March 2021 Update
 
-We expanded the Waymo Open Dataset to also include a motion dataset comprising object trajectories and corresponding 3D maps for over 100,000 segments. We have updated this repository to add support for this new dataset. Please refer to the [Quick Start](docs/quick_start.md).
+We expanded the Waymo Open Dataset to also include a Motion dataset comprising object trajectories and corresponding 3D maps for over 100,000 segments. We have updated this repository to add support for this new dataset. Please refer to the [Quick Start](docs/quick_start.md).
 
 Additionally, we added instructions and examples for the real-time detection challenges. Please follow these [Instructions](waymo_open_dataset/latency/README.md).
 
@@ -37,12 +54,23 @@ This code repository (excluding third_party) is licensed under the Apache Licens
 The Waymo Open Dataset itself is licensed under separate terms. Please visit [https://waymo.com/open/terms/](https://waymo.com/open/terms/) for details.  Code located at third_party/camera is licensed under a BSD 3-clause copyright license + an additional limited patent license applicable only when the code is used to process data from the Waymo Open Dataset as authorized by and in compliance with the Waymo Dataset License Agreement for Non-Commercial Use.  See third_party/camera for details.
 
 ## Citation
-@inproceedings{sun2020scalability,
-  title={Scalability in perception for autonomous driving: Waymo open dataset},
-  author={Sun, Pei and Kretzschmar, Henrik and Dotiwalla, Xerxes and Chouard, Aurelien and Patnaik, Vijaysai and Tsui, Paul and Guo, James and Zhou, Yin and Chai, Yuning and Caine, Benjamin and others},
-  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
-  pages={2446--2454},
-  year={2020}
+### for Perception dataset
+@InProceedings{Sun_2020_CVPR,
+  author = {Sun, Pei and Kretzschmar, Henrik and Dotiwalla, Xerxes and Chouard, Aurelien and Patnaik, Vijaysai and Tsui, Paul and Guo, James and Zhou, Yin and Chai, Yuning and Caine, Benjamin and Vasudevan, Vijay and Han, Wei and Ngiam, Jiquan and Zhao, Hang and Timofeev, Aleksei and Ettinger, Scott and Krivokon, Maxim and Gao, Amy and Joshi, Aditya and Zhang, Yu and Shlens, Jonathon and Chen, Zhifeng and Anguelov, Dragomir},
+  title = {Scalability in Perception for Autonomous Driving: Waymo Open Dataset},
+  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+  month = {June},
+  year = {2020}
+}
+
+### for Motion dataset
+@InProceedings{Ettinger_2021_ICCV,
+  author={Ettinger, Scott and Cheng, Shuyang and Caine, Benjamin and Liu, Chenxi and Zhao, Hang and Pradhan, Sabeek and Chai, Yuning and Sapp, Ben and Qi, Charles R. and Zhou, Yin and Yang, Zoey and Chouard, Aur\'elien and Sun, Pei and Ngiam, Jiquan and Vasudevan, Vijay and McCauley, Alexander and Shlens, Jonathon and Anguelov, Dragomir},
+  title={Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset},
+  booktitle= Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
+  month={October},
+  year={2021},
+  pages={9710-9719}
 }
 
 ## Dataset Metadata

configure.sh

@@ -28,7 +28,7 @@ else
   PYTHON="python${PYTHON_VERSION}.${PYTHON_MINOR_VERSION}"
 fi
 PIP="$PYTHON -m pip"
-update-alternatives --install /usr/bin/python3 python3 "/usr/bin/$PYTHON" 1
+update-alternatives --install /usr/bin/python3 python3 "$(which $PYTHON)" 1
 
 function write_to_bazelrc() {
   echo "$1" >> .bazelrc

pip_pkg_scripts/build.Dockerfile

@@ -11,17 +11,36 @@ RUN bash bazel-4.0.0-installer-linux-x86_64.sh
 RUN apt-get remove --purge -y python3.5 python3.6
 RUN rm -f /etc/apt/sources.list.d/jonathonf-ubuntu-python-3_6-xenial.list
 RUN apt-key del F06FC659
-RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA6932366A755776
-RUN echo "deb http://ppa.launchpad.net/deadsnakes/ppa/ubuntu xenial main" > /etc/apt/sources.list.d/deadsnakes-ppa-xenial.list
-RUN apt-get update && apt-get install -y python3.6 python3.7 python3.8 python3.8-distutils python3.9 python3.9-distutils
+
+# Deadsnakes PPA no longer supports 16.04
+# https://github.com/deadsnakes/issues/issues/195
+# So we build all python versions here
+RUN mkdir /tmp/python
+RUN apt-get update
+RUN apt-get install -y apt-utils
+RUN apt-get install -y build-essential checkinstall libreadline-gplv2-dev libncursesw5-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libssl-dev zlib1g-dev openssl libffi-dev
+
+RUN for v in 3.6.15 3.7.12 3.8.12 3.9.10; do \
+    wget "https://www.python.org/ftp/python/$v/Python-${v}.tar.xz" && \
+    tar xvf "Python-${v}.tar.xz" -C /tmp/python && \
+    cd "/tmp/python/Python-${v}" && \
+    ./configure && \
+    make -j8 altinstall; \
+  done
+
+
+RUN apt-get install -y libopenexr-dev
 RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
 RUN for python in python3.6 python3.7 python3.8 python3.9; do \
       $python get-pip.py && \
       $python -m pip install --upgrade pip setuptools auditwheel && \
       $python -m pip install --upgrade grpcio>=1.24.3; \
+      $python -m pip install --upgrade matplotlib plotly immutabledict; \
+      $python -m pip install --upgrade OpenEXR tensorflow_graphics; \
     done
 
 VOLUME /tmp/artifacts
+VOLUME /root
 COPY . /tmp/repo
 WORKDIR /tmp/repo
 

pip_pkg_scripts/build.sh

@@ -33,12 +33,32 @@ else
   PYTHON="python${PYTHON_VERSION}.${PYTHON_MINOR_VERSION}"
 fi
 
+# Force Keras release compatibility with TensorFlow releases.
+# https://github.com/keras-team/keras#release-and-compatibility
+case "$TF_VERSION" in
+  2.5.*)
+    export KERAS_VERSION='2.5.0rc0'
+    ;;
+  2.6.*)
+    export KERAS_VERSION='2.6.0'
+    ;;
+  2.7.*)
+    export KERAS_VERSION='2.7.0'
+    ;;
+  2.8.*)
+    export KERAS_VERSION='2.8.0'
+    ;;
+  *)
+    export KERAS_VERSION='nightly'
+esac
+
 curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
 $PYTHON get-pip.py --user
 
 PIP="$PYTHON -m pip"
 
 ${PIP} install --upgrade setuptools --user
+${PIP} install --upgrade keras=="$KERAS_VERSION" --user
 ${PIP} install --upgrade tensorflow=="${TF_VERSION}" --user
 
 ./configure.sh

pip_pkg_scripts/setup.py.in

@@ -18,9 +18,12 @@ from setuptools import setup
 from setuptools.command.install import install
 from setuptools.dist import Distribution
 
-__version__ = '1.4.1'
+__version__ = '1.4.3'
 REQUIRED_PACKAGES = [
     'tensorflow==' + '.'.join('TF_VERSION'.split('-')),
+    'plotly>=5.5.0',
+    'matplotlib>=3.2.2',
+    'immutabledict>=2.0.0'
 ]
 project_name = 'waymo-open-dataset-tf-TF_VERSION'
 

third_party/auditwheel.sh

@@ -14,12 +14,18 @@
 # limitations under the License.
 # ==============================================================================
 
-TF_SHARED_LIBRARY_NAME=$(grep -r TF_SHARED_LIBRARY_NAME .bazelrc | awk -F= '{print$2}')
+function parse_bazelrc_value() {
+  awk -F= '/'$1'/ {print(substr($2, 2, length($2)-2))}' .bazelrc
+}
+
+TF_SHARED_LIBRARY_NAME="$(parse_bazelrc_value TF_SHARED_LIBRARY_NAME)"
+TF_SHARED_LIBRARY_DIR="$(parse_bazelrc_value TF_SHARED_LIBRARY_DIR)"
 
 POLICY_JSON=$(find / -name policy.json)
 
-sed -i "s/libresolv.so.2\"/libresolv.so.2\", $TF_SHARED_LIBRARY_NAME/g" $POLICY_JSON
+sed -i "s/libresolv.so.2\"/libresolv.so.2\", \"$TF_SHARED_LIBRARY_NAME\"/g" $POLICY_JSON
 
 cat $POLICY_JSON
 
+export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$TF_SHARED_LIBRARY_DIR"
 auditwheel $@

third_party/camera/ops/camera_model_ops.cc

@@ -217,6 +217,7 @@ REGISTER_OP("WorldToImage")
     .Input("global_coordinate: T")
     .Output("image_coordinate: T")
     .SetShapeFn([](shape_inference::InferenceContext* c) {
+
       return Status::OK();
     })
     .Doc(R"doc(
@@ -246,6 +247,7 @@ REGISTER_OP("ImageToWorld")
     .Input("image_coordinate: T")
     .Output("global_coordinate: T")
     .SetShapeFn([](shape_inference::InferenceContext* c) {
+
       return Status::OK();
     })
     .Doc(R"doc(

tutorial/BUILD

@@ -12,6 +12,7 @@ filegroup(
     name = "framedata",
     srcs = [
         "3d_point_cloud.png",
+        "frame_with_keypoints.tfrecord",
         "frames",
     ],
 )
@@ -27,19 +28,34 @@ py_binary(
     ],
     python_version = "PY3",
     deps = [
+        # Implicit IPython dependency.
+        # Implicit matplotlib dependency.
+        # Implicit numpy dependency.
+        # Implicit tensorflow dependency.
+        # Implicit tensorflow_graphics/image:transformer dependency.
         "//waymo_open_dataset:dataset_proto_py_pb2",
         "//waymo_open_dataset:label_proto_py_pb2",
         "//waymo_open_dataset/metrics/ops:py_metrics_ops",
         "//waymo_open_dataset/metrics/python:config_util_py",
+        "//waymo_open_dataset/metrics/python:keypoint_metrics",
         "//waymo_open_dataset/protos:breakdown_proto_py_pb2",
         "//waymo_open_dataset/protos:map_proto_py_pb2",
         "//waymo_open_dataset/protos:metrics_proto_py_pb2",
         "//waymo_open_dataset/protos:motion_metrics_proto_py_pb2",
         "//waymo_open_dataset/protos:motion_submission_proto_py_pb2",
+        "//waymo_open_dataset/protos:occupancy_flow_metrics_proto_py_pb2",
+        "//waymo_open_dataset/protos:occupancy_flow_submission_proto_py_pb2",
         "//waymo_open_dataset/protos:scenario_proto_py_pb2",
         "//waymo_open_dataset/protos:submission_proto_py_pb2",
         "//waymo_open_dataset/utils:box_utils",
         "//waymo_open_dataset/utils:frame_utils",
+        "//waymo_open_dataset/utils:keypoint_data",
+        "//waymo_open_dataset/utils:keypoint_draw",
+        "//waymo_open_dataset/utils:occupancy_flow_data",
+        "//waymo_open_dataset/utils:occupancy_flow_grids",
+        "//waymo_open_dataset/utils:occupancy_flow_metrics",
+        "//waymo_open_dataset/utils:occupancy_flow_renderer",
+        "//waymo_open_dataset/utils:occupancy_flow_vis",
         "//waymo_open_dataset/utils:range_image_utils",
         "//waymo_open_dataset/utils:transform_utils",
     ],

tutorial/tutorial.ipynb

@@ -138,7 +138,7 @@
       },
       "source": [
         "(range_images, camera_projections,\n",
-        " range_image_top_pose) = frame_utils.parse_range_image_and_camera_projection(\n",
+        " _, range_image_top_pose) = frame_utils.parse_range_image_and_camera_projection(\n",
         "    frame)"
       ],
       "execution_count": 0,
@@ -793,4 +793,4 @@
       ]
     }
   ]
-}
+}