Pith. sign in

REVIEW 59 references

CoreNet: Conflict Resolution Network for Point-Pixel Misalignment and Sub-Task Suppression of 3D LiDAR-Camera Object Detection

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.06550 v1 pith:J4RTYCMQ submitted 2025-01-11 cs.CV

Yiheng Li , Yang Yang , Zhen Lei This is my paper · ORCID
classification cs.CV
keywords corenetsuppressionfeaturemisalignmentobjectpoint-pixelqueryspace
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Fusing multi-modality inputs from different sensors is an effective way to improve the performance of 3D object detection. However, current methods overlook two important conflicts: point-pixel misalignment and sub-task suppression. The former means a pixel feature from the opaque object is projected to multiple point features of the same ray in the world space, and the latter means the classification prediction and bounding box regression may cause mutual suppression. In this paper, we propose a novel method named Conflict Resolution Network (CoreNet) to address the aforementioned issues. Specifically, we first propose a dual-stream transformation module to tackle point-pixel misalignment. It consists of ray-based and point-based 2D-to-BEV transformations. Both of them achieve approximately unique mapping from the image space to the world space. Moreover, we introduce a task-specific predictor to tackle sub-task suppression. It uses the dual-branch structure which adopts class-specific query and Bbox-specific query to corresponding sub-tasks. Each task-specific query is constructed of task-specific feature and general feature, which allows the heads to adaptively select information of interest based on different sub-tasks. Experiments on the large-scale nuScenes dataset demonstrate the superiority of our proposed CoreNet, by achieving 75.6\% NDS and 73.3\% mAP on the nuScenes test set without test-time augmentation and model ensemble techniques. The ample ablation study also demonstrates the effectiveness of each component. The code is released on https://github.com/liyih/CoreNet.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 14 canonical work pages

  1. [1]

    Huang, S

    Z. Huang, S. Sun, J. Zhao, L. Mao, Multi-modal policy fusion for end- to-end autonomous driving, Information Fusion 98 (2023) 101834. https://doi.org/10.1016/j.inffus.2023.101834

    arXiv 2023

  2. [2]

    Fernandes, A

    D. Fernandes, A. Silva, R. N´ evoa, C. Sim˜ oes, D. Gonzalez, M. Guevara, P. Novais, J. Monteiro, P. Melo-Pinto, Point-cloud based 3d object de- tection and classification methods for self-driving applications: A survey and taxonomy, Information Fusion 68 (2021) 161–191. https://doi.org/10.1016/j.inffus.2020.11.002

    work page doi:10.1016/j.inffus.2020.11.002 2021

  3. [3]

    Zhang, L

    X. Zhang, L. Wang, J. Chen, C. Fang, L. Yang, Z. Song, G. Yang, Y. Wang, X. Zhang, J. Li, Dual radar: A multi-modal dataset with dual 4d radar for autononous driving, arXiv preprint arXiv:2310.07602 (2023). https://doi.org/10.48550/arXiv.2310.07602

    arXiv doi:10.48550/arxiv.2310.07602 2023

  4. [4]

    X. Wang, Z. Sun, A. Chehri, G. Jeon, Y. Song, Deep learning and multi- modal fusion for real-time multi-object tracking: Algorithms, challenges, datasets, and comparative study, Information Fusion 105 (2024) 102247. https://doi.org/10.1016/j.inffus.2024.102247 24

    arXiv 2024

  5. [5]

    J. Mao, S. Shi, X. Wang, H. Li, 3d object detection for autonomous driving: A comprehensive survey, International Journal of Computer Vision (2023) 1–55. https://doi.org/10.1007/s11263-023-01790-1

    work page doi:10.1007/s11263-023-01790-1 2023

  6. [6]

    Wu, Fusion-based modeling of an intelligent algorithm for enhanced object detection using a deep learning approach on radar and camera data, Information Fusion (2024) 102647

    Y. Wu, Fusion-based modeling of an intelligent algorithm for enhanced object detection using a deep learning approach on radar and camera data, Information Fusion (2024) 102647. https://doi.org/10.1016/j.inffus.2024.102647

    arXiv 2024

  7. [7]

    M. Hao, Z. Zhang, L. Li, K. Dong, L. Cheng, P. Tiwari, X. Ning, Coarse to fine-based image–point cloud fusion network for 3d object detection, Information Fusion 112 (2024) 102551. https://doi.org/10.1016/j.inffus.2024.102551

    arXiv 2024

  8. [8]

    Jiang, D

    X. Jiang, D. Wang, K. Bi, S. Wang, M. Zhang, Mshp3d: Multi-stage cross-modal fusion based on hybrid perception for indoor 3d object de- tection, Information Fusion 112 (2024) 102591. https://doi.org/10.1016/j.inffus.2024.102591

    arXiv 2024

Show all 59 references
  1. [9]

    L. Yang, X. Zhang, J. Li, C. Wang, Z. Song, T. Zhao, Z. Song, L. Wang, M. Zhou, Y. Shen, et al., V2x-radar: A multi-modal dataset with 4d radar for cooperative perception, arXiv preprint arXiv:2411.10962 (2024). https://doi.org/10.48550/arXiv.2411.10962

    2024 doi

  2. [10]

    Z. Song, G. Zhang, L. Liu, L. Yang, S. Xu, C. Jia, F. Jia, L. Wang, Robofusion: Towards robust multi-modal 3d obiect detection via sam, arXiv preprint arXiv:2401.03907 (2024). https://doi.org/10.48550/arXiv.2401.03907

    2024 doi arXiv

  3. [11]

    S. Xu, F. Li, Z. Song, J. Fang, S. Wang, Z.-X. Yang, Multi-sem fusion: multimodal semantic fusion for 3d object detection, IEEE Transactions on Geoscience and Remote Sensing (2024). https://doi.org/10.1109/TGRS.2024.3387732

    2024

  4. [12]

    Jiang, S

    S. Jiang, S. Xu, L. Liu, Z. Song, Y. Bo, Z.-X. Yang, et al., Sparsein- teraction: Sparse semantic guidance for radar and camera 3d object detection, in: ACM Multimedia 2024, 2024. https://doi.org/10.1145/3664647.3681565 25

    2024

  5. [13]

    Y. Li, Y. Yang, Z. Lei, Rctrans: Radar-camera transformer via radar densifier and sequential decoder for 3d object detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2025

    2025

  6. [14]

    Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. L. Rus, S. Han, Bev- fusion: Multi-task multi-sensor fusion with unified bird’s-eye view rep- resentation, in: 2023 IEEE international conference on robotics and automation (ICRA), IEEE, 2023, pp. 2774–2781. https://doi.org/10.110...

    2023

  7. [15]

    J. Yin, J. Shen, R. Chen, W. Li, R. Yang, P. Frossard, W. Wang, Is- fusion: Instance-scene collaborative fusion for multimodal 3d object de- tection, arXiv preprint arXiv:2403.15241 (2024). https://doi.org/10.48550/arXiv.2403.15241

    2024 doi arXiv

  8. [16]

    Philion, S

    J. Philion, S. Fidler, Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d, in: Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, Springer, 2020, pp. 194–210. https://doi...

    2020 doi

  9. [17]

    Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, J. Dai, Bev- former: Learning bird’s-eye-view representation from multi-camera im- ages via spatiotemporal transformers, in: European conference on com- puter vision, Springer, 2022, pp. 1–18. https://doi.org/10.1007/978-...

    2022 doi

  10. [19]

    Y. Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Y. Shi, J. Sun, Z. Li, Bevdepth: Acquisition of reliable depth for multi-view 3d object detection, in: Pro- ceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 1477–1485. https://doi.org/10.1609/aaai.v37i2.25233 26

    2023 doi

  11. [21]

    Huang, G

    J. Huang, G. Huang, Z. Zhu, Y. Ye, D. Du, Bevdet: High-performance multi-camera 3d object detection in bird-eye-view, arXiv preprint arXiv:2112.11790 (2021). https://doi.org/10.48550/arXiv.2112.11790

    2021 doi arXiv

  12. [22]

    J. Hou, Z. Liu, Z. Zou, X. Ye, X. Bai, et al., Query-based temporal fusion with explicit motion for 3d object detection, Advances in Neural Information Processing Systems 36 (2024)

    2024

  13. [25]

    Huang, Y

    J. Huang, Y. Ye, Z. Liang, Y. Shan, D. Du, Detecting as labeling: Rethinking lidar-camera fusion in 3d object detection, arXiv preprint arXiv:2311.07152 (2023). https://doi.org/10.48550/arXiv.2311.07152

    2023 doi arXiv

  14. [27]

    T. Wang, X. Zhu, J. Pang, D. Lin, Fcos3d: Fully convolutional one- stage monocular 3d object detection, in: Proceedings of the IEEE/CVF 27 International Conference on Computer Vision, 2021, pp. 913–922. https://doi.org/10.1109/ICCVW54120.2021.00107

    2021

  15. [30]

    Y. Liu, T. Wang, X. Zhang, J. Sun, Petr: Position embedding transfor- mation for multi-view 3d object detection, in: European Conference on Computer Vision, Springer, 2022, pp. 531–548. https://doi.org/10.1007/978-3-031-19812-0_31

    2022 doi

  16. [32]

    S. Doll, R. Schulz, L. Schneider, V. Benzin, M. Enzweiler, H. P. Lensch, Spatialdetr: Robust scalable transformer-based 3d object detection from multi-view camera images with global cross-sensor attention, in: Euro- pean Conference on Computer Vision, Springer, 2022, pp. 230–2...

    2022 doi

  17. [33]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). https://doi.org/10.48550/arXiv.1706.03762

    2017 doi arXiv

  18. [34]

    Carion, F

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: European conference 28 on computer vision, Springer, 2020, pp. 213–229. https://doi.org/10.1007/978-3-030-58452-8_13

    2020 doi

  19. [36]

    Z. Li, F. Wang, N. Wang, Lidar r-cnn: An efficient and universal 3d object detector, in: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2021, pp. 7546–7555. https://doi.org/10.1109/CVPR46437.2021.00746

    2021

  20. [37]

    C. R. Qi, H. Su, K. Mo, L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652– 660. https://doi.org/10.1109/CVPR.2017.16

    2017 doi

  21. [38]

    C. R. Qi, L. Yi, H. Su, L. J. Guibas, Pointnet++: Deep hierarchical feature learning on point sets in a metric space, Advances in neural information processing systems 30 (2017). https://doi.org/10.48550/arXiv.1706.02413

    2017 doi arXiv

  22. [39]

    S. Shi, X. Wang, H. Li, Pointrcnn: 3d object proposal generation and detection from point cloud, in: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, 2019, pp. 770–779. https://doi.org/10.1109/CVPR.2019.00086

    2019

  23. [40]

    C. R. Qi, W. Liu, C. Wu, H. Su, L. J. Guibas, Frustum pointnets for 3d object detection from rgb-d data, in: Proceedings of the IEEE con- ference on computer vision and pattern recognition, 2018, pp. 918–927. https://doi.org/10.1109/CVPR.2018.00102

    2018

  24. [41]

    Y. Chen, Y. Li, X. Zhang, J. Sun, J. Jia, Focal sparse convolutional networks for 3d object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5428–5437. https://doi.org/10.1109/CVPR52688.2022.00535 29

    2022

  25. [43]

    Y. Zhou, O. Tuzel, Voxelnet: End-to-end learning for point cloud based 3d object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4490–4499. https://doi.org/10.1109/CVPR.2018.00472

    2018

  26. [44]

    Y. Yan, Y. Mao, B. Li, Second: Sparsely embedded convolutional de- tection, Sensors 18 (10) (2018) 3337. https://doi.org/10.3390/s18103337

    2018 doi

  27. [46]

    A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, O. Beijbom, Point- pillars: Fast encoders for object detection from point clouds, in: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 12697–12705. https://doi.org/10.1109/CVPR.2019.01298

    2019

  28. [48]

    J. Bi, H. Wei, G. Zhang, K. Yang, Z. Song, Dyfusion: Cross-attention 3d object detection with dynamic fusion, IEEE Latin America Transactions 22 (2) (2024) 106–112. https://doi.org/10.1109/TLA.2024.10412035

    2024

  29. [49]

    Z. Song, G. Zhang, J. Xie, L. Liu, C. Jia, S. Xu, Z. Wang, Voxelnextfu- sion: A simple, unified and effective voxel fusion framework for multi- modal 3d object detection, arXiv preprint arXiv:2401.02702 (2024). https://doi.org/10.1109/TGRS.2023.3331893 30

    2024 arXiv

  30. [50]

    S. Xu, D. Zhou, J. Fang, J. Yin, Z. Bin, L. Zhang, Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection, in: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), IEEE, 2021, pp. 3047–3054. https://doi.org/10.1109/ITSC48978....

    2021

  31. [52]

    T. Yin, X. Zhou, P. Kr¨ ahenb¨ uhl, Multimodal virtual point 3d detection, Advances in Neural Information Processing Systems 34 (2021) 16494– 16507. https://doi.org/10.48550/arXiv.2111.06881

    2021 doi arXiv

  32. [53]

    V. A. Sindagi, Y. Zhou, O. Tuzel, Mvx-net: Multimodal voxelnet for 3d object detection, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 7276–7282. https://doi.org/10.1109/ICRA.2019.8794195

    2019

  33. [54]

    Z. Chen, Z. Li, S. Zhang, L. Fang, Q. Jiang, F. Zhao, Deformable feature aggregation for dynamic multi-modal 3d object detection, in: European conference on computer vision, Springer, 2022, pp. 628–644. https://doi.org/10.1007/978-3-031-20074-8_36

    2022 doi

  34. [55]

    Z. Yang, J. Chen, Z. Miao, W. Li, X. Zhu, L. Zhang, Deepinteraction: 3d object detection via modality interaction, Advances in Neural Infor- mation Processing Systems 35 (2022) 1992–2005. https://doi.org/10.48550/arXiv.2208.11112

    2022 doi arXiv

  35. [56]

    Liang, H

    T. Liang, H. Xie, K. Yu, Z. Xia, Z. Lin, Y. Wang, T. Tang, B. Wang, Z. Tang, Bevfusion: A simple and robust lidar-camera fusion framework, Advances in Neural Information Processing Systems 35 (2022) 10421– 10434. https://doi.org/10.48550/arXiv.2205.13790

    2022 doi arXiv

  36. [60]

    Z. Song, L. Yang, S. Xu, L. Liu, D. Xu, C. Jia, F. Jia, L. Wang, Graph- bev: Towards robust bev feature alignment for multi-modal 3d object detection, arXiv preprint arXiv:2403.11848 (2024). https://doi.org/10.48550/arXiv.2403.11848

    2024 doi arXiv

  37. [61]

    Zhuang, Z

    J. Zhuang, Z. Qin, H. Yu, X. Chen, Task-specific context decoupling for object detection, arXiv preprint arXiv:2303.01047 (2023). https://doi.org/10.48550/arXiv.2303.01047

    2023 doi arXiv

  38. [62]

    Z. Song, L. Liu, F. Jia, Y. Luo, C. Jia, G. Zhang, L. Yang, L. Wang, Robustness-aware 3d object detection in autonomous driving: A review and outlook, IEEE Transactions on Intelligent Transportation Systems (2024). https://doi.org/10.1109/TITS.2024.3439557

    2024

  39. [63]

    L. Wang, X. Zhang, Z. Song, J. Bi, G. Zhang, H. Wei, L. Tang, L. Yang, J. Li, C. Jia, et al., Multi-modal 3d object detection in autonomous driving: A survey and taxonomy, IEEE Transactions on Intelligent Ve- hicles 8 (7) (2023) 3781–3798. https://doi.org/10.1109/TIV.2023.3264658

    2023

  40. [65]

    Z. Song, C. Jia, L. Yang, H. Wei, L. Liu, Graphalign++: An accurate feature alignment by graph matching for multi-modal 3d object detec- tion, IEEE Transactions on Circuits and Systems for Video Technology (2023). https://doi.org/10.1109/TCSVT.2023.3306361

    2023

  41. [66]

    Z. Chen, Z. Li, S. Zhang, L. Fang, Q. Jiang, F. Zhao, B. Zhou, H. Zhao, Autoalign: pixel-instance feature aggregation for multi-modal 3d object detection, arXiv preprint arXiv:2201.06493 (2022). https://doi.org/10.24963/ijcai.2022/116

    2022 arXiv

  42. [67]

    Z. Song, F. Jia, H. Pan, Y. Luo, C. Jia, G. Zhang, L. Liu, Y. Ji, L. Yang, L. Wang, Contrastalign: Toward robust bev feature alignment via con- trastive learning for multi-modal 3d object detection, arXiv preprint arXiv:2405.16873 (2024). https://doi.org/10.48550/arXiv.2405.16873

    2024 doi arXiv

  43. [73]

    Z. Liu, J. Hou, X. Wang, X. Ye, J. Wang, H. Zhao, X. Bai, Lion: Lin- ear group rnn for 3d object detection in point clouds, arXiv preprint arXiv:2407.18232 (2024). https://doi.org/10.48550/arXiv.2407.18232

    2024 doi arXiv

  44. [74]

    Y. Li, Y. Chen, X. Qi, Z. Li, J. Sun, J. Jia, Unifying voxel-based repre- sentation with transformer for 3d object detection, Advances in Neural Information Processing Systems 35 (2022) 18442–18455. https://doi.org/10.48550/arXiv.2206.00630

    2022 doi arXiv

  45. [75]

    Q. Cai, Y. Pan, T. Yao, C.-W. Ngo, T. Mei, Objectfusion: Multi-modal 3d object detection with object-centric fusion, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 18067–18076. https://doi.org/10.1109/ICCV51070.2023.01656

    2023

  46. [77]

    Contributors, MMDetection3D: OpenMMLab next-generation platform for general 3D object detection, https://github.com/ open-mmlab/mmdetection3d (2020)

    M. Contributors, MMDetection3D: OpenMMLab next-generation platform for general 3D object detection, https://github.com/ open-mmlab/mmdetection3d (2020)

    2020

  47. [78]

    Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10012–10022. https://doi.org/10.1109/ICCV48922.2021.00986 34

    2021

  48. [79]

    B. Zhu, Z. Jiang, X. Zhou, Z. Li, G. Yu, Class-balanced group- ing and sampling for point cloud 3d object detection, arXiv preprint arXiv:1908.09492 (2019). https://doi.org/10.48550/arXiv.1908.09492

    2019 doi arXiv

  49. [80]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. https://doi.org/10.1109/CVPR.2016.90

    2016 doi

  50. [81]

    Y. Lee, J. Park, Centermask: Real-time anchor-free instance segmenta- tion, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 13906–13915. https://doi.org/10.1109/CVPR42600.2020.01392

    2020

  51. [82]

    Y. Dong, C. Kang, J. Zhang, Z. Zhu, Y. Wang, X. Yang, H. Su, X. Wei, J. Zhu, Benchmarking robustness of 3d object detection to common corruptions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1022–1032. https://doi.org/10.110...

    2023

Pith tools