Waymo validation issue

[ammaryasirnaich]

Hi there, thanks helping me out previously with the Waymo dataset creation. I did managed to create the dataset and have successfully tested the training for pointpiller model as an example, but in the evaluation phase i an get error KeyError: 'pred_instances_3d', in my evaluation configuration and its something like this

val_evaluator = dict(
    type='WaymoMetric',
    ann_file="data/waymo/kitti_format/waymo_infos_val.pkl",
    waymo_bin_file= "data/waymo/waymo_format/gt.bin",
    data_root='data/waymo/waymo_format',
    backend_args=backend_args,
    convert_kitti_format=True,
    idx2metainfo='data/waymo/waymo_format/idx2metainfo.pkl'
    )

I have tried to follow the instruction on this path

The error is out it
11/28 00:21:27 - mmengine - INFO - Epoch(test) [39900/39987] eta: 0:00:07 time: 0.0922 data_time: 0.0021 memory: 368
11/28 00:21:31 - mmengine - INFO - Epoch(test) [39950/39987] eta: 0:00:03 time: 0.0963 data_time: 0.0030 memory: 368

Converting prediction to KITTI format
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 39987/39987, 111.9 task/s, elapsed: 357s, ETA:     0s
Result is saved to /tmp/tmpki56cozu/results/pred_instances_3d.pkl.
Start converting ...
[                                                  ] 0/39987, elapsed: 0s, ETA:Traceback (most recent call last):
  File "tools/test.py", line 151, in <module>
    main()
  File "tools/test.py", line 147, in main
    runner.test()
  File "/homesdatasets7/mmdet/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1823, in test
    metrics = self.test_loop.run()  # type: ignore
  File "/homesdatasets7/mmdet/lib/python3.8/site-packages/mmengine/runner/loops.py", line 438, in run
    metrics = self.evaluator.evaluate(len(self.dataloader.dataset))
  File "/homesdatasets7/mmdet/lib/python3.8/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate
    _results = metric.evaluate(size)
  File "/homesdatasets7/mmdet/lib/python3.8/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate
    _metrics = self.compute_metrics(results)  # type: ignore
  File "/mmdetection3d/mmdet3d/evaluation/metrics/waymo_metric.py", line 182, in compute_metrics
    result_dict, tmp_dir = self.format_results(
  File "/mmdetection3d/mmdet3d/evaluation/metrics/waymo_metric.py", line 391, in format_results
    converter.convert()
  File "/mmdetection3d/mmdet3d/evaluation/functional/waymo_utils/prediction_to_waymo.py", line 368, in convert
    convert_func(i)
  File "/mmdetection3d/mmdet3d/evaluation/functional/waymo_utils/prediction_to_waymo.py", line 285, in convert_one_fast
    if len(self.results[res_index]['pred_instances_3d']) > 0:
KeyError: 'pred_instances_3d'

I have also tried removing the idx2metainfo and using the normal one

val_evaluator = dict(
    type='WaymoMetric',
    ann_file="data/waymo/kitti_format/waymo_infos_val.pkl",
    waymo_bin_file= "data/waymo/waymo_format/gt.bin",
    data_root='data/waymo/waymo_format',
    backend_args=backend_args,
    convert_kitti_format=False)

and i use to get the below erro

11/29 15:30:08 - mmengine - INFO - Epoch(test) [2350/2500]    eta: 0:04:16  time: 1.2338  data_time: 0.0519  memory: 12030  
11/29 15:31:10 - mmengine - INFO - Epoch(test) [2400/2500]    eta: 0:02:49  time: 1.2408  data_time: 0.0340  memory: 12138  
11/29 15:32:10 - mmengine - INFO - Epoch(test) [2450/2500]    eta: 0:01:24  time: 1.1888  data_time: 0.0437  memory: 12139  
11/29 15:33:19 - mmengine - INFO - Epoch(test) [2500/2500]    eta: 0:00:00  time: 1.3789  data_time: 0.0367  memory: 12162  
Traceback (most recent call last):
  File "tools/test.py", line 151, in <module>
    main()
  File "tools/test.py", line 147, in main
    runner.test()
  File "/python3.8/site-packages/mmengine/runner/runner.py", line 1823, in test
    metrics = self.test_loop.run()  # type: ignore
  File "/python3.8/site-packages/mmengine/runner/loops.py", line 438, in run
    metrics = self.evaluator.evaluate(len(self.dataloader.dataset))
  File "/python3.8/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate
    _results = metric.evaluate(size)
  File "/python3.8/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate
    _metrics = self.compute_metrics(results)  # type: ignore
  File "/mmdetection3d/mmdet3d/evaluation/metrics/waymo_metric.py", line 182, in compute_metrics
    result_dict, tmp_dir = self.format_results(
  File "/mmdetection3d/mmdet3d/evaluation/metrics/waymo_metric.py", line 381, in format_results
    converter = Prediction2Waymo(
  File "/mmdetection3d/mmdet3d/evaluation/functional/waymo_utils/prediction_to_waymo.py", line 108, in __init__
    self.get_file_names()
  File "/mmdetection3d/mmdet3d/evaluation/functional/waymo_utils/prediction_to_waymo.py", line 117, in get_file_names
    if 'path_mapping' in self.backend_args:
TypeError: argument of type 'NoneType' is not iterable

Did you encounter such errors while you where running your models ? Would really appreciate your help with this if possible!

[DYZhang09]

Sorry for the late reply. I didn't encounter such errors before. You can go to the official repo of mmdetection3d for help.