Detection metric release. #159

benjaminrwilson · 2020-09-04T21:24:02Z

Provides a set of metrics to be used for detection evaluation.

argoverse/evaluation/detection_utils.py

tests/test_eval_detection.py

johnwlambert · 2020-09-05T18:02:39Z

tests/test_eval_detection.py

+    assert dist_fn(df1, df2, DistFnType.TRANSLATION) == 75 ** (1 / 2)
+
+
+def test_scale_distance() -> None:


1-line docstring would be helpful to explain the scenario and goal of the test

Done. We should add documentation for the metrics somewhere. Where do you think that should go?

What sort of documentation were you thinking of? I think comments inline in the code are great, and then in the EvalAI Evaluation tab we should add the equivalent of https://evalai.cloudcv.org/web/challenges/challenge-page/453/evaluation.

Let's add a README in the evaluation folder.

johnwlambert · 2020-09-05T18:03:31Z

tests/test_eval_detection.py

+    assert metrics.AOE.Means == 0
+
+
+def _rotvec_to_quat(rotvec: R) -> R:


can we go ahead and make this a function in the argoverse.utils.transform along with its counterpart quat2rotmat? and add a unit test on it?

Or maybe I misunderstood -- what is a rotvec? axis-angle parameterization?

Happy to change this if it fits in with the other code better. Yeah: https://en.wikipedia.org/wiki/Axis%E2%80%93angle_representation#Rotation_vector. Let me know what you think.

This doesn't need to be it's own function in argoverse.utils.transform since it uses https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.transform.Rotation.html. The thing that should be done is the quaternion re-ordering, which is the [3, 0, 1, 2] part. But we can resolve that in the other comment. Leaving this open so we remember to change it

johnwlambert · 2020-09-05T18:09:40Z

argoverse/evaluation/detection_utils.py

+        scale_errors = 1 - (inter / union)
+        return scale_errors
+    elif metric == DistFnType.ORIENTATION:
+        dt_yaws = R.from_quat(np.vstack(dt_df["quaternion"].array)[:, [3, 0, 1, 2]]).as_euler("xyz")[:, 2]


instead of repeating the [3,0,1,2] here, i would prefer to make the quat2yaw (or a better name) an explicit function in argoverse.utils.transform. I like how you are using scipy for all of this. you could use this hacky function as a unit test on it:

I agree. Let's settle the rotvec portion and this simultaneously.

argoverse/evaluation/eval_detection.py

johnwlambert · 2020-09-05T18:20:07Z

I believe the Travis build is failing: https://travis-ci.org/github/argoai/argoverse-api

tqdm, pandas, and click are not included in our current dependencies, if I remember right.

argoverse/evaluation/detection_utils.py

janesjanes · 2020-09-07T04:34:19Z

Thanks for the push! and thanks john for going over it, I haven't look at it in details yet, but for the build failing seems like it's timeout at tox. If it pass locally in your machine then maybe we can try increase the timeout from 10 mins (30 mins works for me)

benjaminrwilson · 2020-09-08T12:48:19Z

I believe the Travis build is failing: https://travis-ci.org/github/argoai/argoverse-api

tqdm, pandas, and click are not included in our current dependencies, if I remember right.

I've removed tqdm for now --- the multiprocessing implementation was causing an issue when running locally. Using pandas and click would probably be worth having further discussion.

argoverse/evaluation/eval_detection.py

seanremy · 2020-09-13T01:58:42Z

I made a bunch of changes, I believe I've addressed all of the comments except the one about Pandas.

I know in our discussion we decided to limit Pandas to the post-processing stage, but after looking back over the code I think it might be simpler to leave it as is. What do you guys think?

argoverse/evaluation/eval_detection.py

argoverse/evaluation/detection_utils.py

argoverse/evaluation/eval_detection.py

argoverse/evaluation/detection_utils.py

argoverse/evaluation/eval_detection.py

argoverse/evaluation/detection_utils.py

argoverse/evaluation/eval_detection.py

argoverse/evaluation/detection_utils.py

argoverse/evaluation/eval_detection.py

argoverse/evaluation/detection_utils.py

johnwlambert · 2020-10-05T20:51:34Z

tests/test_eval_detection.py

+    metrics = assign(dts, gts, cfg)
+    # if these assign correctly, we should get an ATE of 0.1 for the first two
+    expected_result: float = 0.1
+    assert np.isclose(metrics[0, 4], expected_result)


do you mind explaining what [0,4] means in the metrics dict?
maybe something like

ATE_COL_IDX = 4 assert np.isclose(metrics[0, ATE_COL_IDX], expected_result) # instance 0 assert np.isclose(metrics[1, ATE_COL_IDX], expected_result) # instance 1

how is the 0,1,2 instance order determined?

johnwlambert · 2020-10-05T20:52:57Z

tests/test_eval_detection.py

+    # ensure the detections match at all thresholds, have 0 TP errors, and have AP = 1
+    assert (
+        cls_to_accum["VEHICLE"]
+        == np.array([[1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0], [1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0]])


Small nitpick -- might be a bit easier to parse if we make the values here expected_ATE, expected_ASE instead of 0, 1 etc, which i have a bit of a hard time following

Detection metric release.

b2bb060

johnwlambert suggested changes Sep 5, 2020

View reviewed changes

johnwlambert reviewed Sep 5, 2020

View reviewed changes

argoverse/evaluation/eval_detection.py Outdated Show resolved Hide resolved

johnwlambert suggested changes Sep 5, 2020

View reviewed changes

argoverse/evaluation/detection_utils.py Outdated Show resolved Hide resolved

Documentation and function parameter changes.

357e789

benjaminrwilson and others added 5 commits September 9, 2020 08:33

Cleaned up unused methods.

2de0952

Fix clipping.

690ff3d

add some logging

07ab68c

add test docstrings

ee9b37f

doc fixes

8cbb898

johnwlambert reviewed Sep 10, 2020

View reviewed changes

argoverse/evaluation/eval_detection.py Outdated Show resolved Hide resolved

seanremy added 4 commits September 12, 2020 21:30

fix pr comments

8b97bcc

replace click with argparse

c85b5a7

add missing types to function header

d28d01e

doc for dist_fn

8f08b67

janesjanes reviewed Sep 14, 2020

View reviewed changes

argoverse/evaluation/eval_detection.py Outdated Show resolved Hide resolved

janesjanes reviewed Sep 14, 2020

View reviewed changes

argoverse/evaluation/eval_detection.py Outdated Show resolved Hide resolved

Ben Wilson added 2 commits September 14, 2020 15:54

Fixed a few argument issues.

5f532fc

Added angle normalization method.

9707b4e

johnwlambert suggested changes Sep 14, 2020

View reviewed changes

Ben Wilson added 2 commits September 14, 2020 23:36

Temporary fix (potentially).

83b4ef2

Typing fixes + other enhancements.

fdf1e90

johnwlambert suggested changes Sep 17, 2020

View reviewed changes

janesjanes reviewed Sep 17, 2020

View reviewed changes

argoverse/evaluation/eval_detection.py Outdated Show resolved Hide resolved

Type fixes + matplotlib fix.

8509df4

johnwlambert reviewed Sep 17, 2020

View reviewed changes

argoverse/evaluation/detection_utils.py Outdated Show resolved Hide resolved

benjaminrwilson and others added 20 commits October 1, 2020 11:19

More type fixes.

7c7fff0

More type fixes.

dd99a56

Update travis build.

ed51c79

Attempt to fix environment issue.

d06e811

Updated pre-commit.

3674c93

Updated pre-commit.

db8baa6

Remove mypy from pre-commit.

1e4847d

Revert back to previous pre-commit.

5b5b9d2

Reverting travis configuration as well.

9afdf8f

Docstring update.

3609cf6

Docstring update.

1754a0c

Add back tox timeout.

16d49ab

Modified isort.

ad1619c

Another style test.

12f2056

Black update.

4673043

Black + isort fix.

bae3b7d

Ensure 120 length isort.

96b83bd

Added units tests and stubs.

64ecce3

Added new tests + updated docstrings.

c778106

add assign and accumulate tests, plus other small fixes

3bf475c

johnwlambert reviewed Oct 5, 2020

View reviewed changes

seanremy and others added 6 commits October 5, 2020 17:12

slight clarification of some unit tests

e1adfa6

Removed unused import.

24d1c5d

Restructured fpaths.

7301fd1

Small fixes.

95845ed

Removed unused import.

de65f3e

Uncommented unit test.

f77119c

johnwlambert approved these changes Oct 8, 2020

View reviewed changes

johnwlambert merged commit 1dfc03e into argoverse:master Oct 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detection metric release. #159

Detection metric release. #159

benjaminrwilson commented Sep 4, 2020

johnwlambert Sep 5, 2020

seanremy Sep 10, 2020

johnwlambert Sep 10, 2020

benjaminrwilson Sep 17, 2020

johnwlambert Sep 5, 2020

johnwlambert Sep 5, 2020

benjaminrwilson Sep 8, 2020

seanremy Sep 10, 2020 •

edited

Loading

johnwlambert Sep 5, 2020

benjaminrwilson Sep 8, 2020

johnwlambert commented Sep 5, 2020 •

edited

Loading

janesjanes commented Sep 7, 2020

benjaminrwilson commented Sep 8, 2020

seanremy commented Sep 13, 2020

johnwlambert Oct 5, 2020 •

edited

Loading

johnwlambert Oct 5, 2020

		assert dist_fn(df1, df2, DistFnType.TRANSLATION) == 75 ** (1 / 2)


		def test_scale_distance() -> None:

		assert metrics.AOE.Means == 0


		def _rotvec_to_quat(rotvec: R) -> R:

Detection metric release. #159

Detection metric release. #159

Conversation

benjaminrwilson commented Sep 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seanremy Sep 10, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnwlambert commented Sep 5, 2020 • edited Loading

janesjanes commented Sep 7, 2020

benjaminrwilson commented Sep 8, 2020

seanremy commented Sep 13, 2020

johnwlambert Oct 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seanremy Sep 10, 2020 •

edited

Loading

johnwlambert commented Sep 5, 2020 •

edited

Loading

johnwlambert Oct 5, 2020 •

edited

Loading