Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Towards Omnidirectional Reasoning with 360-R1: A Dataset, Benchmark, and GRPO-based Method

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces the first open 360-degree VQA dataset and a GRPO-based method that improves a 7B vision-language model on it.

desk verdict Genuinely new open-source 360-degree VQA dataset and benchmark, but the claimed +6% gain is untrustworthy because the same model family generates the references, computes the rewards, and scores the final results. read the letter →

arxiv 2505.14197 v1 pith:KER2J5HK submitted 2025-05-20 cs.CV

Xinshen Zhang , Zhen Ye , Xu Zheng This is my paper
classification cs.CV
keywords omnidirectionalimages360-degreevisualquestionansweringpanoramicbenchmarkgrouprelativepolicyoptimizationrule-basedreinforcementlearningspatialreasoningpolar-regiondistortionmultimodallargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current multimodal language models handle ordinary 2D images well but fail on 360-degree panoramic images, especially in the distorted polar regions, and that this calls for dedicated data and training. It presents OmniVQA, an open dataset of 4,852 visual question-answer pairs built from 1,213 equirectangular indoor panoramas, with questions about object identification, object attributes, and spatial relationships in polar regions. It also introduces OmniVQABench, a 200-image evaluation benchmark, and 360-R1, a post-training method that couples group relative policy optimization (GRPO) with three rule-based rewards: reasoning-process similarity, answer semantic accuracy, and structured format compliance. The central empirical claim is that 360-R1 improves a 7-billion-parameter vision-language model on this benchmark by roughly 6 percent, with the largest gains in answer quality. A reader should care because the work supplies a public yardstick and a training recipe for reasoning about the full spherical surroundings, a perceptual demand that ordinary image benchmarks do not capture.

What carries the argument

The load-bearing mechanism is a triple of hand-designed reward functions folded into GRPO: a reasoning process similarity reward that scores the generated chain-of-thought against a reference via an LLM judge, an answer semantic accuracy reward that does the same for the final answer, and a structured format compliance reward that is a binary check on tag presence, order, and nesting. GRPO normalizes each response reward within a sampled group as $\hat{r}_i = (r_i - \mathrm{mean}(r))/\mathrm{std}(r)$, uses the normalized value as the advantage for every token in that response, and optimizes a clipped surrogate objective with a KL penalty. The dataset construction relies on a two-step chain-of-thought pipeline: a fine-tuned vision model produces detailed descriptions, a reasoning model produces multi-step reasoning, and an instruction-tuned model summarizes it into a concise answer, with an iterative sentence-embedding similarity filter deciding which pairs enter the training set and manual review handling the rest. The reward design plus the group-relative update rule is what carries the claimed improvement.

What would settle it

One concrete check is to replace the model-written references on a held-out subset of the benchmark with independently human-verified answers and re-score; if 360-R1 no longer beats its base model, the claimed gain is grading bias. A second check is to run 360-R1 on an outdoor 360-degree set, since all training data are indoor; a collapse to baseline performance would indicate the method learned the benchmark rather than omnidirectional understanding.

Watch

Extended reading notes

Core claim

The paper claims that omnidirectional visual question answering is a distinct, under-measured capability and that it can be substantially improved by the combination of a purpose-built dataset and rule-based reinforcement learning. On OmniVQABench, the base model scores are low across all tested vision-language models, and 360-R1 raises the base 7B model on every metric reported: BERTScore-F1 from 0.8675 to 0.8839, sentence-embedding F1 from 0.6252 to 0.6388, one LLM-judged F1 from 0.4308 to 0.4996, and a second LLM-judged F1 from 0.5055 to 0.5472; the abstract summarizes these as a 6 percent improvement. The paper attributes the gains to group-relative advantage estimation and to reward functions that jointly supervise the reasoning trace, the final answer, and the output format, with ablation results showing that a balanced weighting of reasoning and answer rewards matters and that the larger 7B model benefits more than the 3B model.

Load-bearing premise

The whole result rests on the assumption that the model-generated reference reasoning and answers used both for training rewards and for evaluation are correct and unbiased; if those references are wrong or written in a distinctive style, the reported 6 percent improvement could be an artifact of matching that style.

Editorial extensions

If this is right

  • The OmniVQA dataset and OmniVQABench give the community a public, reproducible way to measure how well vision-language models handle polar-region distortion, occlusion, and spatial layout in 360-degree scenes.
  • 360-R1 shows that rule-based reinforcement learning with format and semantic rewards can improve a 7B model on this benchmark without collecting any new image data.
  • The largest reported gains are in answer accuracy (one LLM-judged answer subscore rises from 0.4220 to 0.5140), which suggests the method mainly improves the final response rather than only the reasoning trace.
  • The ablation shows reward weighting is a first-order design choice: over-weighting the answer reward lowers one LLM-judged F1 score by about 7 percent, while over-weighting reasoning lowers sentence-embedding similarity, so the balanced default matters.
  • Training ran on four GPUs for about 2.8 days, indicating that the recipe is computationally accessible to many research groups.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same reward triple could plausibly transfer to other distortion-heavy 360-degree tasks such as depth estimation or semantic segmentation; a direct check would be to train on those tasks and see whether gains of the same size appear.
  • Because the benchmark's reference answers and its graders come from the same model lineage, a human-labeled held-out set would reveal whether 360-R1 is genuinely better at panoramic reasoning or simply better at imitating the reference generator's style.
  • Since all training images are indoor, the reported gain may not survive outdoor or dynamic omnidirectional scenes; adding a small outdoor test set would test whether the method learned a general distortion-handling skill rather than a room-specific shortcut.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces OmniVQA, claimed as the first open-source omnidirectional visual question answering dataset, built from 1,213 equirectangular-projection images of the Stanford 2D-3D-S dataset with 4,852 QA pairs spanning object identification, attribute analysis, and spatial relationship reasoning. It also constructs OmniVQABench, a 200-image benchmark, evaluates several state-of-the-art MLLMs on it, and proposes 360-R1, a GRPO-based post-training method with three rewards: reasoning process similarity, answer semantic accuracy, and structured format compliance, with similarity scores computed by DeepSeek-V3. Experiments report consistent improvements of 360-R1 over Qwen2.5-VL-7B-Instruct, including QwenScore-F1 +6.88% and DeepSeekScore-F1 +4.17%.

Significance. If the benchmark references were independent and human-verified, the paper would provide a useful resource and a straightforward recipe for improving MLLM reasoning in 360-degree scenes. Strengths include the modular data generation pipeline, the iterative refinement loop with SentenceBERT-Score, the evaluation of multiple model families, and the learning curves in Appendix B. The current significance is limited, however, by the circular relationship between reference generation, reward computation, and evaluation, by the lack of any disjointness statement between training and benchmark images, and by the absence of statistical validation and released artifacts.

major comments (4)
  1. [§5.1, §4, §6] The reasoning and answer rewards in §5.1 compute similarity between generated outputs and dataset references using DeepSeek-V3, while Table 3's QwenScore and DeepSeekScore evaluate against the same references with the same LLM-as-judge paradigm. Since the references were produced by Qwen2.5-VL, DeepSeek-R1, and Qwen2.5-14B and only DeepSeek-V3-reviewed in §4, the reported +6.88% QwenScore-F1 and +4.17% DeepSeekScore-F1 gains may reflect improved alignment with the reference generator's stylistic preferences rather than genuine improvement in 360-degree understanding. Please add a human-verified evaluation subset or use an independent judge model family not involved in training or reference generation, and report both.
  2. [§3.1, §4] The training set (1,213 images from 2D-3D-S, §3.1) and the benchmark (200 images from 2D-3D-S, §4) are drawn from the same source, and the paper never states that the 200 benchmark images are disjoint from the 1,213 training images. If any overlap exists, the reported improvements are inflated by image-level leakage. Please clarify the exact split and remove overlapping images from the training set.
  3. [Tables 3–5] All conclusions rest on point estimates over a 200-image benchmark with no confidence intervals, repeated runs, or significance tests. Several improvements are small in absolute terms, such as BERTScore-F1 +1.64% and SentenceBERTScore-F1 +1.36% in Table 3, so the central '+6%' claim is not statistically secured. Please provide per-sample scores, bootstrap confidence intervals, or multi-seed means.
  4. [§4] The text states that GPT-4o descriptions are manually verified, but the reasoning and answer references are only reviewed by DeepSeek-V3. Given that these references are simultaneously the RL training targets and the evaluation ground truth, the lack of human verification of the final 200-sample reasoning and answer pairs is a load-bearing gap. Please report human agreement rates or at least a human-checked subset.
minor comments (6)
  1. [§4] The paragraph beginning 'To construct a reliable benchmark for multi-modal omnidirectional VQA...' is duplicated verbatim within the same section.
  2. [Abstract, §1] The claims of 'first dataset' and 'first benchmark' should be qualified as 'first open-source', since §2 cites the existing VQA 360° dataset [13].
  3. [§3.1] The phrase 'spatio-temporal reasoning' is inaccurate for a static-image dataset; the tasks described are spatial only.
  4. [§5.1] The rewards are described as 'rule-based' but two of them rely on DeepSeek-V3 LLM judgments; only the format compliance reward is truly rule-based.
  5. [§6, Appendix] No code, data, or release URL is provided despite the open-source claims; an availability statement with links is needed for reproducibility.
  6. [Throughout] There are several typographical issues, including 'we we introduce' in §3.2, 'Results Anlysis' in §6.1, 'over 1 epochs' in §6, and inconsistent capitalization of 'QwenScore' and 'Qwenscore'.

Circularity Check

2 steps flagged · score 7.0 of 10

The reported +6% gain is the same reference-similarity objective used as the GRPO reward, evaluated by the same DeepSeek-family models that produced and reviewed the benchmark references, so the central claim is largely circular.

  1. fitted input called prediction [Section 5.1 (Reward Function Design) and Section 6 (Experimental Setup; Table 3)]
    "Reasoning Process Similarity Reward evaluates the semantic and logical alignment between the generated reasoning and a reference COT reasoning. ... computes a similarity score using the DeepSeek-V3 [19] model via prompt-based evaluation. ... Answer Semantic Accuracy Reward measures the semantic similarity between the generated and reference answers. ... DeepSeekScore uses DeepSeek-chat [19] to output a score between 0.0 and 1.0."

    The first two rewards are exactly the objective optimized by GRPO, and the Section 6 evaluation metrics compare generated outputs to the same kind of reference reasoning/answers using the same DeepSeek-family judge. Table 3's headline improvements (+6.88% QwenScore-F1, +4.17% DeepSeekScore-F1) therefore measure how much better the policy imitates the benchmark's own LLM-generated reference distribution. BERTScore and SentenceBERTScore are also similarity-to-reference metrics, so none of the four metrics is an external check; the reported superiority is, by the paper's own reward definitions, closeness to the target distribution that the policy was rewarded to match.

  2. other [Section 4 (Benchmark Construction), contrasted with Section 6's 'human annotations' claim]
    "To generate fine-grained visual descriptions for the 200 selected images, GPT-4o [47] is prompted ... All descriptions are manually verified ... The DeepSeek-R1 model takes these descriptions as input and generates structured CoT reasoning. ... we then use Qwen2.5-14B to generate concise answers ... DeepSeek-V3 is subsequently employed to review the reasoning-answer pairs."

    Only the image descriptions are manually verified; the benchmark's reasoning and answer 'ground truth' are produced by DeepSeek-R1 and Qwen2.5-14B and reviewed by DeepSeek-V3. These are the same model families that act as the RL reward judge and as the final evaluator (DeepSeek-chat/QwenLLM). Section 6 calls the references 'human annotations,' but the stated pipeline provides no human verification of the reasoning/answer pairs. Correctness on this benchmark is thus defined as agreement with in-family, machine-generated references, not with independently established 360-degree understanding.

full rationale

The central experimental claim of the paper is that 360-R1 improves omnidirectional reasoning by about +6% on OmniVQABench (Table 3). The structural problem is that the RL reward and the evaluation metric are the same operation: both score generated reasoning and answers by semantic similarity to reference reasoning/answers using a DeepSeek-family LLM. Since the benchmark references were themselves generated by DeepSeek-R1/Qwen2.5-14B and vetted by DeepSeek-V3, the evaluation is not independent of the training signal. Optimizing the reward is expected to raise the reported scores; the +6% gain is therefore partly a measure of how well the policy learned the reference-generation style rather than an external validation of spatial understanding. The paper's Section 7 limitation statement (4.8K QA pairs, indoor-only) is a scale/diversity caveat and does not repair this structural circularity. No load-bearing self-citation chain was found: the many prior panoramic papers by the authors are background material, not the argument. The format-compliance reward and the dataset artifact itself retain independent content, so the paper is not wholly circular, but the headline improvement reduces substantially to reward/evaluation identity, warranting a score of 7.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The central claims rest on the accuracy of model-generated references and on LLM judges, rather than on human-verified ground truth or external benchmarks. The reward weights are tuned on a small ablation, and the dataset selection threshold is hand-chosen. No new physical entities are introduced.

free parameters (2)
  • reward weights (format:reasoning:answer) = 0.1:0.45:0.45
    Chosen from an ablation (Table 5) over three configurations; the default balances reasoning and answer rewards. This hand-tuned design choice affects the final method.
  • SentenceBERT-Score selection threshold = 0.8
    Used in the iterative refinement loop to decide which generated reasoning-answer pairs are added to the training set without manual review (Section 3.2). Changing the threshold changes dataset composition.
assumptions (2)
  • domain assumption The reference reasoning and answers generated by Qwen2.5-VL, DeepSeek-R1, Qwen2.5-14B, and GPT-4o are accurate enough to serve as ground truth for both RL rewards and benchmark scoring.
    Invoked throughout Section 3.2 and Section 4; only a subset of samples is manually corrected, so a large part of the dataset quality rests on unverified model outputs.
  • domain assumption DeepSeek-V3 and Qwen LLM judges produce valid semantic similarity scores that reflect true answer quality.
    Used as reward signals in Section 5.1 and as evaluation metrics in Section 6 (QwenScore, DeepSeekScore). No human correlation study is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Omnidirectional Reasoning with 360-R1: A Dataset, Benchmark, and GRPO-based Method." pith.science (2026). https://pith.science/paper/KER2J5HK

@misc{pith2026250514197,
  author       = {Pith},
  title        = {Pith review of: Towards Omnidirectional Reasoning with 360-R1: A Dataset, Benchmark, and GRPO-based Method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KER2J5HK}},
  note         = {Machine review of arXiv:2505.14197}
}
read the original abstract

Omnidirectional images (ODIs), with their 360{\deg} field of view, provide unparalleled spatial awareness for immersive applications like augmented reality and embodied AI. However, the capability of existing multi-modal large language models (MLLMs) to comprehend and reason about such panoramic scenes remains underexplored. This paper addresses this gap by introducing OmniVQA, the first dataset and conducting the first benchmark for omnidirectional visual question answering. Our evaluation of state-of-the-art MLLMs reveals significant limitations in handling omnidirectional visual question answering, highlighting persistent challenges in object localization, feature extraction, and hallucination suppression within panoramic contexts. These results underscore the disconnect between current MLLM capabilities and the demands of omnidirectional visual understanding, which calls for dedicated architectural or training innovations tailored to 360{\deg} imagery. Building on the OmniVQA dataset and benchmark, we further introduce a rule-based reinforcement learning method, 360-R1, based on Qwen2.5-VL-Instruct. Concretely, we modify the group relative policy optimization (GRPO) by proposing three novel reward functions: (1) reasoning process similarity reward, (2) answer semantic accuracy reward, and (3) structured format compliance reward. Extensive experiments on our OmniVQA demonstrate the superiority of our proposed method in omnidirectional space (+6% improvement).

Figures

Figures reproduced from arXiv: 2505.14197 by the authors.

Figure 1
Figure 1. Comparison of occlusion reasoning in 360° VQA. Given a panoramic scene and a spatial question about whether an object in the upper pole region is partially occluded, four models provide different levels of reasoning. 360-R1 demonstrates the most precise and comprehensive reasoning, identifying relevant spatial elements and producing a correct answer. QwenVL2.5-7B gives the correct answer but its explanation is parti… view at source ↗
Figure 2
Figure 2. Error Cases in Omnidirectional Captioning. Three common errors by multimodal LLMs on 360° images: (a) misidentified objects; (b) incorrect object attributes or context; (c) hallucinated content unrelated to the image. 1 Introduction Driven by the rapid advancement of Augmented Reality (AR), Virtual Reality (VR), and Embodied AI systems—as well as the growing demand for immersive visual experiences—omnidirectional im… view at source ↗
Figure 3
Figure 3. Overview of the OmniVQA Dataset Construction and 360-R1 Framework. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Iterative Refinement Pipeline. Developed based on the Stanford 2D-3D-S dataset [3], which covers over 6,000 m² of indoor scenes, OmniVQA dataset contains 1,213 panoramic images in equirectangular projection (ERP) format at a resolution of 1440×720, preserving angular f…
Figure 5
Figure 5. Figure 5: Benchmark Construction Pipeline [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Radar Chart of Reasoning Scores for OmniVQA Benchmark Models. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Radar Chart of Answer Accuracy for OmniVQA Benchmark Models. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Radar Chart of F1 (Harmonic Mean of Reasoning and Answer) for OmniVQA Benchmark [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Format and Tag Reward Across Training Steps. [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Answer Similarity Reward Over Training Steps. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Reasoning Similarity Reward Over Training Steps. [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Generated Completion Length Over Training Steps. [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What if? Emulative Simulation with World Models for Situated Reasoning

    cs.CV 2026-03 conditional novelty 6.5 of 10

    WanderDream supplies 15.8K panoramic mental-exploration videos and 158K QA pairs showing that world-model imagination measurably improves situated spatial reasoning without active exploration.

  2. Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Survey organizing panoramic scene analysis literature by architectural design and training paradigm, identifying the absence of methods achieving both strict spherical equivariance and full reuse of perspective-pretra...

  3. Omnidirectional Spatial Modeling from Correlated Panoramas

    cs.CV 2025-09 conditional novelty 5.0 of 10

    The authors create a cross-frame panoramic VQA benchmark from 3D scene data and show that GRPO fine-tuning of Qwen2.5-VL raises its score on that benchmark.

Reference graph

Works this paper leans on

96 extracted references · 55 canonical work pages · cited by 3 Pith papers

  1. [1]

    Alayrac et al

    J.-B. Alayrac et al. Flamingo: A visual language model for few-shot learning. In arXiv Preprint,

  2. [2]

    Antol, A

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015

    2015

  3. [3]

    Armeni, S

    I. Armeni, S. Sax, A. R. Zamir, and S. Savarese. Joint 2d-3d-semantic data for indoor scene understanding. In arXiv Preprint, 2017. arXiv:1702.01105

    arXiv 2017

  4. [4]

    Bai et al

    J. Bai et al. Qwen-vl: A frontier large vision-language model with versatile abilities. In arXiv Preprint, 2023. arXiv:2308.12966

    arXiv 2023

  5. [5]

    Bai et al

    J. Bai et al. Qwen2.5-vl: Vision-language agents with spatiotemporal reasoning capabilities. In arXiv Preprint, 2025. arXiv:2501.28001

    work page arXiv 2025

  6. [6]

    Caffagni et al

    D. Caffagni et al. The revolution of multimodal large language models: A survey, 2024

    2024

  7. [7]

    Interact360: Interactive identity-driven text to 360 ° panorama generation

    Zeyu Cai, Zhelong Huang, Xu Zheng, Yexin Liu, Chao Liu, Zeyu Wang, and Lin Wang. Interact360: Interactive identity-driven text to 360 ° panorama generation. In 2024 IEEE Conference on Artificial Intelligence (CAI), pages 728–736. IEEE, 2024

    2024

  8. [8]

    Z. Cao, Z. Wang, Y . Liu, Y .-P. Cao, Y . Shan, W. Zeng, and L. Wang. Learning high-quality navigation and zooming on omnidirectional images in virtual reality. In arXiv preprint arXiv:2405.00351, 2024

    work page Pith review arXiv 2024

Show all 96 references
  1. [9]

    Chang, A

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y . Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017

    2017

  2. [10]

    D. S. Chaplot, D. Gandhi, A. Gupta, and R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In In Neural Information Processing Systems (NeurIPS), 2020

    2020

  3. [11]

    Chen et al

    X. Chen et al. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    2025

  4. [12]

    Chen et al

    Z. Chen et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    2025

  5. [13]

    Chou, W.-L

    S.-H. Chou, W.-L. Chao, W.-S. Lai, M. Sun, and M.-H. Yang. Visual question answering on 360° images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1607–1616. IEEE, 2020

    2020

  6. [14]

    Coors, A

    B. Coors, A. P. Condurache, and A. Geiger. Spherenet: Learning spherical representations for detection and classification in omnidirectional images. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018

    2018

  7. [15]

    Danieau, T

    F. Danieau, T. Lopez, N. Mollet, B. Leroy, O. Dumas, and J.-F. Vial. Enabling embodiment and interaction in omnidirectional videos. In 2017 IEEE International Conference on Multimedia and Expo (ICME), pages 697–702, 2017

    2017

  8. [16]

    A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra. Visual dialog. CoRR, abs/1611.08669, 2016

    2016 arXiv

  9. [17]

    A. C. A. M. de Faria, F. de Castro Bastos, J. V . N. A. da Silva, V . L. Fabris, V . de Sousa Uchoa, D. G. de Aguiar Neto, and C. F. G. dos Santos. Visual question answering: A survey on techniques and common trends in recent literature. CoRR, abs/2305.11033, 2023

    2023 arXiv

  10. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  11. [19]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025. 10

    2025

  12. [20]

    Fang et al

    X. Fang et al. Creation-mmbench: Assessing context-aware creative intelligence in mllm, 2025

    2025

  13. [21]

    Y . Gao, J. Xu, and W. Huang. Creation-mmbench: Evaluating context-aware creative intelligence in mllms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024. To appear

    2024

  14. [22]

    Goyal, T

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    2017

  15. [23]

    Han et al

    J. Han et al. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  16. [24]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2021

    2021

  17. [25]

    Huang et al

    R. Huang et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In arXiv Preprint, 2023. arXiv:2304.12995

    2023 arXiv

  18. [26]

    Huang et al

    W. Huang et al. Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025

    2025

  19. [27]

    D. A. Hudson and C. D. Manning. GQA: a new dataset for compositional question answering over real-world images. CoRR, abs/1902.09506, 2019

    1902 arXiv

  20. [28]

    Mmunlearner: Reformulating multimodal machine unlearning in the era of multimodal large language models

    Jiahao Huo, Yibo Yan, Xu Zheng, Yuanhuiyi Lyu, Xin Zou, Zhihua Wei, and Xuming Hu. Mmunlearner: Reformulating multimodal machine unlearning in the era of multimodal large language models. arXiv preprint arXiv:2502.11051, 2025

    2025 arXiv

  21. [29]

    N. D. Huynh, M. R. Bouadjenek, S. Aryal, I. Razzak, and H. Hacid. Visual question answering: from early developments to recent advances - a survey. CoRR, abs/2501.03939, 2025

    2025 arXiv

  22. [30]

    Jiang, Z

    H. Jiang, Z. Sheng, S. Zhu, Z. Dong, and R. Huang. Unifuse: Unidirectional fusion for 360 ◦ panorama depth estimation. IEEE Robotics and Automation Letters, 2021

    2021

  23. [31]

    Krantz, E

    J. Krantz, E. Wijmans, A. Majundar, D. Batra, and S. Lee. Beyond the nav-graph: Vision and language navigation in continuous environments. In European Conference on Computer Vision (ECCV), 2020

    2020

  24. [32]

    A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 4392–4412, 2020

    2020

  25. [33]

    Li et al

    B. Li et al. Llava-onevision: Easy visual task transfer, 2024

    2024

  26. [34]

    J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    2023

  27. [35]

    J. Li, D. Li, C. Xiong, and S. C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. CoRR, abs/2201.12086, 2022

    2022 arXiv

  28. [36]

    J. Li, J. Su, C. Xia, and Y . Tian. Distortion-adaptive salient object detection in 360° omnidi- rectional images. In Proceedings of the IEEE Journal of Selected Topics in Signal Processing, volume 14, pages 38–48, 2020

    2020

  29. [37]

    X. Li, Y . Liu, and Y . Wang. Object detection in omnidirectional images based on spherical cnn. In Proceedings of the 2021 7th IEEE International Conference on Network Intelligence and Digital Content (IC-NIDC), pages 269–273. IEEE, 2021

    2021

  30. [38]

    Y . Li, Y . Guo, Z. Yan, X. Huang, D. Ye, and L. Ren. Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In 2022 Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, June 2022. 11

    2022

  31. [39]

    Z. Li, G. Qi, W. Hu, X. Ma, and Q. Guo. Application of augmented reality in campus navigation. In 2021 6th International Conference on Intelligent Computing and Signal Processing (ICSP), pages 889–893, 2021

    2021

  32. [40]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Visual instruction tuning with llava. InAdvances in Neural Information Processing Systems (NeurIPS), pages 1–16. Curran Associates, Inc., 2023

    2023

  33. [41]

    Lu et al

    H. Lu et al. Deepseek-vl: Towards real-world vision-language understanding, 2024

    2024

  34. [42]

    Lu et al

    P. Lu et al. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  35. [43]

    Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all

    Yuanhuiyi Lyu, Xu Zheng, Dahun Kim, and Lin Wang. Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all. arXiv preprint arXiv:2405.16108, 2024

    2024 arXiv

  36. [44]

    Unibind: Llm-augmented unified and balanced representation space to bind them all

    Yuanhuiyi Lyu, Xu Zheng, Jiazhou Zhou, and Lin Wang. Unibind: Llm-augmented unified and balanced representation space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26752–26762, 2024

    2024

  37. [45]

    Maaz et al

    M. Maaz et al. Video-chatgpt: Towards detailed video understanding via large vision and language models. In arXiv Preprint, 2023. arXiv:2306.05424

    2023 arXiv

  38. [46]

    Mirowski et al

    P. Mirowski et al. The streetlearn environment and dataset, 2019

    2019

  39. [47]

    Gpt-4o: Openai’s new multimodal flagship model, 2024

    OpenAI. Gpt-4o: Openai’s new multimodal flagship model, 2024. Accessed: 2025-05-14

    2024

  40. [48]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/ learning-to-reason-with-llms/ , 2024. Accessed: 2025-05-14

    2024

  41. [49]

    Qwen2.5 technical report, 2025

    Qwen Team. Qwen2.5 technical report, 2025

    2025

  42. [50]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, and A. Ramesh. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 8748–8763. PMLR, 2021

    2021

  43. [51]

    Ramesh et al

    A. Ramesh et al. Hierarchical text-conditional image generation with clip latents. In arXiv Preprint, 2022. arXiv:2204.06125

    2022 arXiv

  44. [52]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019

    2019

  45. [53]

    Schwenk, A

    D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022

    2022

  46. [54]

    A. R. Sekkat, Y . Dupuis, P. Vasseur, and P. Honeine. The omniscape dataset. In2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1603–1608, 2020

    2020

  47. [55]

    Shao et al

    Z. Shao et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  48. [56]

    Z. Shen, C. Lin, K. Liao, L. Nie, Z. Zheng, and Y . Zhao. Panoformer: Panorama transformer for indoor 360° depth estimation. In European Conference on Computer Vision, pages 195–211. Springer, 2022

    2022

  49. [57]

    Z. Shen, C. Lin, L. Nie, K. Liao, and Y . Zhao. Distortion-tolerant monocular depth estimation on omnidirectional images using dual-cubemap. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2021

    2021

  50. [58]

    Su and K

    Y .-C. Su and K. Grauman. Learning spherical convolution for fast features from 360° imagery. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NeurIPS), 2017

    2017

  51. [59]

    Thawakar et al

    O. Thawakar et al. Llamav-o1: Rethinking step-by-step visual reasoning in llms, 2025

    2025

  52. [60]

    Wang et al

    P. Wang et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024. 12

    2024

  53. [61]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903, 2022

    2022 arXiv

  54. [62]

    Wijmans, A

    E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD- PPO: learning near-perfect pointgoal navigators from 2.5 billion frames. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. ...

    2020

  55. [63]

    Wu et al

    C. Wu et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024

    2024

  56. [64]

    Wu et al

    S. Wu et al. Next-gpt: Any-to-any multimodal llm. In arXiv Preprint, 2023. arXiv:2309.05519

    2023 arXiv

  57. [65]

    Wu et al

    Z. Wu et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024

    2024

  58. [66]

    J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba. Recognizing scene viewpoint using panoramic place representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2012

    2012

  59. [67]

    Xu et al

    G. Xu et al. Llava-cot: Let vision language models reason step-by-step, 2024

    2024

  60. [68]

    K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y . Ben- gio. Show, attend and tell: Neural image caption generation with visual attention. CoRR, abs/1502.03044, 2015

    2015 arXiv

  61. [69]

    A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges

    Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges. arXiv preprint arXiv:2412.11936, 2024

    2024 arXiv

  62. [70]

    Yang et al

    A. Yang et al. Qwen2 technical report, 2024

    2024

  63. [71]

    L. Yang, L. Li, X. Xin, Y . Sun, Q. Song, and W. Wang. Large-scale person detection and localization using overhead fisheye cameras. In ICCV, 2023

    2023

  64. [72]

    Yang et al

    Y . Yang et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  65. [73]

    Yue et al

    X. Yue et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024

    2024

  66. [74]

    Yue et al

    X. Yue et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024

    2024 arXiv

  67. [75]

    H. Yun, Y . Yu, W. Yang, K. Lee, and G. Kim. Pano-avqa: Grounded audio-visual question answering on 360◦ videos. In ICCV, 2021

    2021

  68. [76]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training, 2023

    2023

  69. [77]

    Zhang, V

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi. Bertscore: Evaluating text generation with bert, 2020

    2020

  70. [78]

    Zhang, Y

    W. Zhang, Y . Liu, X. Zheng, and L. Wang. Goodsam: Bridging domain and capacity gaps via segment anything model for distortion-aware panoramic semantic segmentation. In CVPR, pages 28264–28273. IEEE, 2024

    2024

  71. [79]

    Zheng, T

    X. Zheng, T. Pan, Y . Luo, and L. Wang. Look at the neighbor: Distortion-aware unsupervised domain adaptation for panoramic semantic segmentation. In ICCV, pages 18641–18652. IEEE, 2023

    2023

  72. [80]

    Zheng, P

    X. Zheng, P. Zhou, A. V . Vasilakos, and L. Wang. Semantics, distortion, and style matter: Towards source-free UDA for panoramic segmentation. In CVPR, pages 27885–27895. IEEE, 2024. 13

    2024

  73. [81]

    Zheng, P

    X. Zheng, P. Y . Zhou, A. V . Vasilakos, and L. Wang. 360sfuda++: Towards source-free UDA for panoramic segmentation by learning reliable category prototypes. IEEE Trans. Pattern Anal. Mach. Intell., 47(2):1190–1204, 2025

    2025

  74. [82]

    Zheng, J

    X. Zheng, J. Zhu, Y . Liu, Z. Cao, C. Fu, and L. Wang. Both style and distortion matter: Dual-path unsupervised domain adaptation for panoramic semantic segmentation. In CVPR, pages 1285–1295. IEEE, 2023

    2023

  75. [83]

    Deep learning for event-based vision: A comprehensive survey and benchmarks

    Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang. Deep learning for event-based vision: A comprehensive survey and benchmarks. arXiv preprint arXiv:2302.08890, 2023

    2023 arXiv

  76. [84]

    Retrieval augmented generation and understanding in vision: A survey and new outlook

    Xu Zheng, Ziqiao Weng, Yuanhuiyi Lyu, Lutao Jiang, Haiwei Xue, Bin Ren, Danda Paudel, Nicu Sebe, Luc Van Gool, and Xuming Hu. Retrieval augmented generation and understanding in vision: A survey and new outlook. arXiv preprint arXiv:2503.18016, 2025

    2025 arXiv

  77. [85]

    Zhong, X

    D. Zhong, X. Zheng, C. Liao, Y . Lyu, J. Chen, S. Wu, L. Zhang, and X. Hu. Omnisam: Omnidirectional segment anything model for UDA in panoramic semantic segmentation. CoRR, abs/2503.07098, 2025

    2025

  78. [86]

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language un- derstanding with advanced large language models. In arXiv Preprint, 2023. arXiv:2304.10592. 14 Appendix A Prompt Templates This part collects all of the system and user prompts we use t...

    2023 arXiv

  79. [87]

    Structure content into 3-5 explicit reasoning steps

  80. [88]

    First,"

    Use transition words (e.g., "First," "Next," "Then") and thinking-style phrases (e.g., "Hmm," "Let’s see," "Okay")

  81. [89]

    Ensure logical flow: each step derives from the prior one

  82. [90]

    Preserve all original information

  83. [91]

    role":"system

    End with a clear conclusion. Question: {question} Original reasoning: {reasoning} Transformed COT reasoning: 15 A.3 CoT Summarisation (Answer Synthesis) Role in Experiment: This prompt guides theQwen2.5-14B-Instruct model to convert a structured Chain of Thought into a concise...

  84. [92]

    }, {"role

    Completeness If errors exist, provide a corrected version. If fully consistent, reply only: CONSISTENT."}, {"role":"user","content":"Question: {question} Reasoning: {reasoning} Current Answer: {answer} Please review item #{index}. If not consistent, provide corrected answer. I...

  85. [93]

    Score range: 0.0-1.0

  86. [94]

    Consider accuracy and context

  87. [95]

    Return ONLY the numeric score with 4 decimal places

  88. [96]

    }, {"role

    No additional text."}, {"role":"user","content":"Reference: {reference}\nCandidate: {candidate}\ nScore:"} A.7 Reasoning Similarity Scoring (GRPO Reward) Role in Experiment: This prompt is used during Group Relative Policy Optimization (GRPO) training to compute the semantic s...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.