From 5de359f3429e1496761790770868296140161b66 Mon Sep 17 00:00:00 2001 From: Waymo Research Date: Wed, 22 Sep 2021 12:28:24 -0700 Subject: [PATCH] Merged commit includes the following changes: 398302195 by Waymo Research: Add a script to build all tf/py versions -- 398300973 by Waymo Research: Update for motion dataset v1.1 release -- PiperOrigin-RevId: 398302195 --- README.md | 9 +++++++++ docs/quick_start.md | 31 ++++++++--------------------- pip_pkg_scripts/README.md | 8 ++++---- pip_pkg_scripts/docker_build_all.sh | 29 +++++++++++++++++++++++++++ 4 files changed, 50 insertions(+), 27 deletions(-) create mode 100755 pip_pkg_scripts/docker_build_all.sh diff --git a/README.md b/README.md index c0613bb..3c43b91 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,15 @@ The Waymo Open Dataset was first launched in August 2019 with a perception dataset comprising high resolution sensor data and labels for 1,950 segments. We have released the Waymo Open Dataset publicly to aid the research community in making advancements in machine perception and autonomous driving technology. +## September 2021 Update + +We released v1.1 of the motion dataset to include lane connectivity information. To read more on the technical details, please read [lane_neighbors_and_boundaries.md](docs/lane_neighbors_and_boundaries.md). + - Added lane connections. Each lane has a list of lane IDs that enter or exit the lane. + - Added lane boundaries. Each lane has a list of left and right boundary features associated with the lane and the segment of the lane where the boundary is active. + - Added lane neighbors. Each lane has a list of left and right neighboring lanes. These are lanes an agent may make a lane change into. + - Improved timestamp precision. + - Improved stop sign Z values. + ## March 2021 Update We expanded the Waymo Open Dataset to also include a motion dataset comprising object trajectories and corresponding 3D maps for over 100,000 segments. We have updated this repository to add support for this new dataset. Please refer to the [Quick Start](docs/quick_start.md). diff --git a/docs/quick_start.md b/docs/quick_start.md index 96cd38b..ab3f276 100644 --- a/docs/quick_start.md +++ b/docs/quick_start.md @@ -11,42 +11,28 @@ As a first step, please visit https://www.waymo.com/open to gain access to the W ## Use pre-compiled pip/pip3 packages for Linux -We only pre-compiled the package for Python 3.6, 3.7, 3.8 for Linux. If you need the lib for a different python version, follow steps in pip_pkg_scripts to build pip package on your own. +We only pre-compiled the package for Python 3.7, 3.8, 3.9 for Linux. If you need the lib for a different python version, follow steps in pip_pkg_scripts to build pip package on your own. ``` bash pip3 install --upgrade pip ``` -### tf 2.4.0. - -``` bash -pip3 install waymo-open-dataset-tf-2-4-0 --user -``` - -### tf 2.3.0. +### tf 2.6.0. ``` bash -pip3 install waymo-open-dataset-tf-2-3-0 --user +pip3 install waymo-open-dataset-tf-2-6-0 --user ``` -### tf 2.2.0. +### tf 2.5.0. ``` bash -pip3 install waymo-open-dataset-tf-2-2-0 --user +pip3 install waymo-open-dataset-tf-2-5-0 --user ``` -### tf 2.1.0. -Note available for Python 3.8. - -``` bash -pip3 install waymo-open-dataset-tf-2-1-0 --user -``` - -### tf 2.0.0 -Note available for Python 3.8. +### tf 2.4.0. ``` bash -pip3 install waymo-open-dataset-tf-2-0-0 --user +pip3 install waymo-open-dataset-tf-2-4-0 --user ``` ## Compile in our docker container @@ -55,8 +41,7 @@ Follow instructions in [pip_pkg_scripts](../pip_pkg_scripts/README.md). ## Local compilation without docker system requirements * g++ 5 or higher. -* python 3.6, 3.7 with TensorFlow 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0 -* python 3.8 with TensorFlow 2.2.0, 2.3.0, 2.4.0 +* python 3.7, 3.8, 3.9 with TensorFlow 2.4.0, 2.5.0, 2.6.0 The code has two main parts. One is a utility written in C++ to compute the evaluation metrics. The other part is a set of diff --git a/pip_pkg_scripts/README.md b/pip_pkg_scripts/README.md index 4c61e73..8fe9d05 100644 --- a/pip_pkg_scripts/README.md +++ b/pip_pkg_scripts/README.md @@ -3,7 +3,7 @@ ```bash docker build --tag=open_dataset_pip -f pip_pkg_scripts/build.Dockerfile . mkdir /tmp/artifacts -docker run --mount type=bind,source=/tmp/artifacts,target=/tmp/artifacts -e "PYTHON_VERSION=3" -e "PYTHON_MINOR_VERSION=9" -e "PIP_MANYLINUX2010=1" -e "TF_VERSION=2.5.0" open_dataset_pip +docker run --mount type=bind,source=/tmp/artifacts,target=/tmp/artifacts -e "PYTHON_VERSION=3" -e "PYTHON_MINOR_VERSION=9" -e "PIP_MANYLINUX2010=1" -e "TF_VERSION=2.6.0" open_dataset_pip ``` This command will execute the `build.sh` inside the container, which clones the @@ -13,7 +13,7 @@ github repository, builds the library and outputs `.whl` packages under To manually run commands inside docker ```bash -docker run -it --entrypoint='/bin/bash' --mount type=bind,source=/tmp/artifacts,target=/tmp/artifacts -e "PYTHON_VERSION=3" -e "PYTHON_MINOR_VERSION=9" -e "PIP_MANYLINUX2010=1" -e "TF_VERSION=2.5.0" open_dataset_pip +docker run -it --entrypoint='/bin/bash' --mount type=bind,source=/tmp/artifacts,target=/tmp/artifacts -e "PYTHON_VERSION=3" -e "PYTHON_MINOR_VERSION=9" -e "PIP_MANYLINUX2010=1" -e "TF_VERSION=2.6.0" open_dataset_pip ``` ## Build a pip package locally without docker container @@ -24,7 +24,7 @@ First follow quick start to install all the depdencies such as bazel. Then run export PYTHON_VERSION=3 export PYTHON_MINOR_VERSION=9 export PIP_MANYLINUX2010=0 -# Only support 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0 -export TF_VERSION=2.5.0 +# Only support 2.4.0, 2.5.0, 2.6.0 +export TF_VERSION=2.6.0 ./pip_pkg_scripts/build.sh ``` diff --git a/pip_pkg_scripts/docker_build_all.sh b/pip_pkg_scripts/docker_build_all.sh new file mode 100755 index 0000000..d5e360b --- /dev/null +++ b/pip_pkg_scripts/docker_build_all.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# Copyright 2021 The Waymo Open Dataset Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== + +set -e -x + +docker build --tag=open_dataset_pip -f pip_pkg_scripts/build.Dockerfile . +mkdir /tmp/artifacts | true +mkdir /tmp/artifacts_all | true + +for pyv in 7 8 9; do + for tfv in 5 6; do + docker run --mount type=bind,source=/tmp/artifacts,target=/tmp/artifacts -e "PYTHON_VERSION=3" -e "PYTHON_MINOR_VERSION=${pyv}" -e "PIP_MANYLINUX2010=1" -e "TF_VERSION=2.${tfv}.0" open_dataset_pip + cp -f /tmp/artifacts/* /tmp/artifacts_all + done +done +