Benchmarks
Benchmarking the robustness of state-of-the-art methods in all seven scenarios of the nuScenes-R and Waymo-R.
| Method | Modality | Performance | LiDAR | Camera | |||||||
| P_c | mP_R | R | Stuck | FOV | Object | Stuck | Missing | Occlusion | Calib | ||
| nuScenes-R (mAP / NDS) | |||||||||||
| CenterPoint | L | 56.8/65.0 | 23.4/46.3 | 0.41/0.71 | 26.1/47.5 | 15.6/43.0 | 28.4/48.5 | - | - | - | - |
| DETR3D | C | 34.9/43.4 | 17.6/31.5 | 0.50/0.73 | - | - | - | 17.3/32.3 | 14.5/29.9 | 14.3/29.0 | 24.2/35.0 |
| PointAugmenting | LC | 46.9/55.6 | 33.7/48.3 | 0.72/0.87 | 25.3/43.5 | 13.3/37.7 | 21.3/39.4 | 42.1/52.8 | 37.0/49.8 | 40.7/52.2 | 43.6/53.8 |
| MVX-Net | LC | 61.0/66.1 | 38.4/53.6 | 0.63/0.81 | 35.2/51.4 | 17.6/43.1 | 34.0/51.1 | 48.3/58.8 | 32.7/50.6 | 45.5/57.6 | 50.8/59.9 |
| TransFusion | LC | 66.9/70.9 | 52.8/63.1 | 0.79/0.89 | 33.4/52.3 | 20.3/45.8 | 34.6/53.6 | 65.9/70.2 | 64.9/69.7 | 65.5/70.0 | 66.5/70.7 |
| Waymo-R(L2 mAP / L2 mAPH) | |||||||||||
| CenterPoint | L | 66.0/63.4 | 30.6/29.4 | 0.46/0.46 | 29.5/28.3 | 30.3/29.1 | 32.1/30.9 | - | - | - | - |
| DETR3D | C | 16.2/15.7 | 10.1/9.8 | 0.62/0.62 | - | - | - | 13.0/12.6 | 8.4/8.2 | 10.9/10.5 | 8.0/7.8 |
| PointAugmenting | LC | 52.5/50.7 | 39.6/38.3 | 0.75/0.76 | 24.7/23.9 | 24.3/23.4 | 26.2/25.3 | 51.7/50.0 | 50.4/48.6 | 50.3/48.6 | 49.8/48.1 |
| MVX-Net | LC | 59.7/54.1 | 44.3/40.1 | 0.74/0.74 | 27.5/24.9 | 28.8/25.6 | 28.7/26.0 | 58.2/52.7 | 55.9/50.5 | 56.4/51.1 | 54.9/49.6 |
| TransFusion | LC | 66.7/64.1 | 51.2/49.1 | 0.77/0.77 | 30.2/29.0 | 30.2/29.0 | 32.7/31.3 | 66.5/63.9 | 66.1/63.5 | 66.2/63.6 | 66.3/63.7 |