OmniGAIA: Towards Native Omni-Modal AI Agents

Guanting Dong; Hao Wang; Haoxuan Li; Jiajie Jin; Jiarui Jin; Ji-Rong Wen; Shijian Wang; Wenxiang Jiao; Xiaoxi Li; Yinuo Wang

REVIEW 4 major objections 6 minor 2 cited by

OmniGAIA: Towards Native Omni-Modal AI Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OmniGAIA, a 360-task benchmark built from an omni-modal event graph, requires AI assistants to perceive video with audio and image+audio pairs, reason in multiple hops, and use external tools over several turns; on it, Gemini-3-Pro reaches

desk verdict A genuinely useful new benchmark, but the OmniAtlas improvement is likely inflated by the read_* tools/prompt confound; the training recipe needs a matched baseline before the numbers can be trusted. read the letter →

arxiv 2602.22897 v3 pith:5XI654B2 submitted 2026-02-26 cs.AI cs.CLcs.CVcs.LGcs.MM

Xiaoxi Li , Wenxiang Jiao , Jiarui Jin , Haoxuan Li , Hao Wang , Shijian Wang , Guanting Dong , Jiajie Jin

show 5 more authors

Yinuo Wang Yuan Lu Ji-Rong Wen Zhicheng Dou Zhouchen Lin

This is my paper · ORCID

classification cs.AIcs.CLcs.CVcs.LGcs.MM

keywords omni-modalagentsbenchmarkconstructionmulti-hopreasoningtooluseeventgraphpreferenceoptimization(DPO)perceptionvideoandaudiounderstanding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OmniGAIA is a benchmark of 360 tasks that for the first time combines omni-modal perception—video-with-audio and image+audio inputs—with multi-hop reasoning and multi-turn use of external tools (web search, browser, code), judged by open-form answers that are verifiable against real-world facts. The tasks are built from a novel omni-modal event graph that links entities and events across modalities, then expands the graph with web evidence and 'fuzzifies' nodes so each question forces the full reasoning path. The paper reports that the benchmark is hard: the best proprietary model reaches 62.5 Pass@1 while the strongest open-source model scores 13.3. It then introduces OmniAtlas, a training recipe—hindsight-guided tree exploration for trajectory synthesis, masked supervised fine-tuning, and OmniDPO for correcting the first error in failed trajectories—which lifts Qwen3-Omni to 20.8 and cuts tool-use and reasoning errors.

What carries the argument

The load-bearing object is the omni-modal event graph: a graph whose nodes are entities and events extracted from video and audio at time-stamped granularity, and whose edges are cross-modal relations. A reasoning agent expands this graph by searching the web, browsing pages, executing code, and retrieving related clips; then 'event fuzzification' hides specific nodes along long paths, forcing any solver to traverse the whole chain. On the training side, the key mechanism is OmniDPO plus masked trajectory SFT: SFT supervises only agent-generated reasoning and tool-call tokens while masking tool observations, and OmniDPO locates the first erroneous step of each failed trajectory and preferenc

What would settle it

Have five independent annotators attempt a random sample of, say, 50 OmniGAIA tasks using the same tools, blind to the paper's labels; if more than a small fraction cannot reach the labeled answer or produce divergent answers, the uniqueness assumption fails and all reported Pass@1 numbers are suspect.

Watch

Extended reading notes

Core claim

The central claim is that omni-modal agent capability is not a perception problem alone: it requires a closed loop of joint perception, multi-step planning, and tool-based verification, and no previous benchmark measures that combination. The paper establishes this by constructing OmniGAIA so that every task needs cross-modal grounding plus next-hop evidence from web tools to yield a unique open-form answer, and by showing that models trained only for perception collapse on it. It further claims that the OmniAtlas recipe materially improves open models on this axis—Qwen3-Omni from 13.3 to 20.8—and that the dominant failure modes in current agents are ineffective tool use and reasoning errors

Load-bearing premise

The whole yardstick presumes the 360 annotated answers are correct, unique, and answerable from the given media plus tools—checked by three human reviewers but without inter-annotator agreement, so an ambiguous or wrong label would shift every model's score and the ordering.

Editorial extensions

If this is right

If the benchmark is accepted, omni-modal agents should be evaluated on tool-integrated long-horizon tasks, not just perception, and native perception is the default, since delegating to perception tools costs accuracy and latency (Table 3).
More parameters alone do not fix agentic capability: a 560B model underperforms a 30B model on OmniGAIA, so tool-use policy is a distinct bottleneck.
The OmniAtlas recipe is transferable: on a 7B backbone it raises Pass@1 from 3.6 to 13.3, and on 30B from 13.3 to 20.8, with the largest gains coming from reduced tool misuse.
Hard tasks (multi-hop reasoning over long media) remain open even for the strongest model (38.5% vs 78.7% on easy), so the benchmark has headroom for future agentic RL and perception research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

I would expect the event-graph pipeline to double as a data generator: the same expansion-and-fuzzification construction could synthesize training trajectories for other omni-modal agent tasks, making OmniGAIA both a test and a seed.
The error cascade finding suggests a testable extension: forcing a minimal tool budget, or injecting a 'verify your hypothesis' instruction, may recover much of the hard-task gap without further training.
Because Pass@1 judgments mix exact match with an LLM judge, the reported gaps (13.3 vs 20.8) could be sensitive to judge leniency; an editorial check would be to re-judge all hard-task outputs with a second judge model.
The paper's training claims imply that failure localization matters more than data volume: OmniDPO targets the first error while the rest of the trajectory stays intact, so an ablation that applies DPO to entire failed trajectories instead of prefixes should show smaller gains—this is directly testable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The benchmark is the real contribution. OmniGAIA fills a gap that no one else has filled: tasks that truly require video+audio or image+audio perception, multi-hop reasoning, and multi-turn tool use, with open-form verifiable answers and a hard enough ceiling that even Gemini-3-Pro lands at 62.5 while open models sit in the teens. The event-graph construction pipeline is a thoughtful way to synthesize these tasks, and the paper is transparent about its process — detailed appendices, tool schemas, prompts, case studies including the OmniAtlas failure. I believe the benchmark itself is solid and useful.

The training recipe is another story. The headline claim that OmniAtlas lifts Qwen3-Omni from 13.3 to 20.8 is not causally identified. Section 5.1 says all models get the same external tools, but that explicitly covers web search/browser/code executor. OmniAtlas additionally gets read_image/read_audio/read_video and a system prompt that pushes it to call them when media is unclear. The untrained Qwen3-Omni baseline runs without those tools and without that prompt. So +7.5 conflates the SFT/DPO training, the larger toolset, and the prompt. The same confound runs through Table 4 and the error-analysis claims. A reader cannot tell how much of the gain is the tool/prompt itself — which matters, since giving any capable open model a perception-friendly toolset and telling it to use it might produce a comparable jump. Table 3's tool-based perception setting is a different manipulation, so it does not save this.

The missing train/test split is equally concerning. The paper never states that the 360 benchmark tasks are disjoint from the 2,156 trajectories used for OmniAtlas training. If the training set was constructed from the same source media or even the same QA pairs, some of the improvement could be memorization. This is simple to fix and should be required.

The ground-truth audit is weaker than claimed. Three graduate reviewers verified each task, but there is no inter-annotator agreement, no quantification of how many tasks were rejected or modified, and the LLM screening uses the same DeepSeek-V3.2 that later judges answers. None of this is fatal — I do not see signs of deliberate contamination — but it is unverified and should be audited in a revision.

Bottom line: the benchmark deserves wide use. The OmniAtlas recipe is plausible but currently unproven. I would send this to peer review, and I would insist on a matched baseline — same tools, same system prompt, no training — plus a clear statement of the train/test split and an inter-annotator agreement number. If those come through, this will be a solid dataset paper. As it stands, cite the benchmark, but treat the 13.3→20.8 claim with suspicion.

Referee Report

4 major / 6 minor

Summary. The paper presents OmniGAIA, a benchmark of 360 omni-modal agent tasks that require joint perception over video-with-audio or image+audio inputs, multi-hop reasoning, and multi-turn external tool use (web search, browser, code). Tasks are constructed via an omni-modal event-graph pipeline: raw media are mined for structured signals, an event graph is built and expanded with tools, and questions are generated by fuzzifying graph nodes/edges, followed by LLM screening and three-reviewer human verification. The paper also proposes OmniAtlas, a training recipe for open omni-modal models that combines hindsight-guided tree exploration, trajectory-level masked SFT, and OmniDPO for fine-grained error correction. Experiments show that OmniGAIA is difficult: Gemini-3-Pro reaches 62.5 Pass@1 while Qwen3-Omni reaches 13.3, and OmniAtlas improves Qwen3-Omni to 20.8. The paper further provides fine-grained error analysis and a controlled comparison of native vs. tool-based perception.

Significance. If the benchmark's ground truth is sound and the OmniAtlas gain is attributable to the proposed training recipe, this is a valuable contribution: it introduces a harder evaluation axis for omni-modal agents and a practical recipe that improves open-source models. The event-graph construction pipeline is a scalable and interesting methodology, and the paper ships code, a leaderboard, and human-verified tasks. The fine-grained error analysis and the three case studies are useful. However, the central claims currently rest on two unaddressed risks: the benchmark's train/test separation and ground-truth auditability, and the confounding of the OmniAtlas training gain with an expanded tool set and system prompt. These need to be resolved before the benchmark and recipe can be relied upon.

major comments (4)

[§5.1, Tables 2/4, App. B.2–B.3] The OmniAtlas gain (+7.5, Qwen-3-Omni 13.3→20.8) is not causally identified. §5.1 says all models receive the same external tools (web search, browser, code executor), but OmniAtlas is additionally given read_image/read_audio/read_video tools (App. B.3) and a system prompt instructing active perception (App. B.2). The base Qwen-3-Omni baseline is run without these tools and with the base prompt. Thus the gain conflates trajectory SFT/OmniDPO with a larger tool set and a perception-seeking instruction. Please provide a matched ablation: train/evaluate OmniAtlas without the read_* tools and with the base system prompt, or give baselines the same tool set/prompt, to isolate the training contribution. The same confound affects Table 4 and the error-analysis claims.
[§3.7, §4.4, App. A.1] No train/test split is defined. OmniGAIA is described as 360 tasks; Appendix A.1 trains on 2,156 synthesized trajectories and §4.4 refers to a 'training set'. The paper never states that these trajectories are disjoint from the 360 benchmark tasks. If the trajectories are derived from the same OmniGAIA tasks used for evaluation, all OmniAtlas numbers are obtained on training data and the comparison with baselines is invalid. Please specify the split, describe how training tasks are selected, and confirm that no evaluation task appears in the training set.
[§3.6] Ground-truth validity rests on three graduate reviewers and LLM screening, but no inter-annotator agreement or independent audit is reported. The screening models (DeepSeek-V3.2, Gemini-3-Pro) overlap with the models used for graph construction (§3.3–3.4), trajectory synthesis (§4.2), and answer judging (§5.1). Since every model score depends on answer correctness/uniqueness, please provide a verification protocol: e.g., a random sample of tasks re-annotated by different annotators, agreement statistics, and an assessment of screening bias (what types of tasks were removed).
[§5.1, App. A.2/B.1] The LLM-as-a-Judge (DeepSeek-V3.2) is also a construction/training model, and no human evaluation of judge accuracy is reported. The fallback extracts only the last 20 words of the model output, which may discard a correct answer embedded in a long justification. Please validate the judge against human labels on a subsample and report agreement; also test sensitivity to judge model choice.

minor comments (6)

[§3 intro] Typo: 'next-hope evidence' should be 'next-hop evidence'.
[Abstract/body/tables] Inconsistent model naming: 'Qwen3-Omni' vs. 'Qwen-3-Omni' across the abstract, body, and tables; standardize.
[Fig. 3] The word cloud is difficult to read; a cleaner visualization of query terms would be more informative.
[App. A.1] Training hyperparameters (learning rate, batch size, warmup, precision, etc.) are omitted; please provide them for reproducibility.
[Table 3] In the 'Audio and Visual Perception Models as Tools (Input No Media)' row, clarify whether Qwen-3 is the text-only model and how it consumes tool outputs.
[§5.3] The fine-grained error analysis relies on LLM-generated labels; no human agreement is reported, so these numbers should be treated as preliminary or supplemented with an agreement study.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark and training claims are empirical and self-contained.

full rationale

The paper's core claims are empirical rather than definitional. OmniGAIA tasks are produced through event-graph construction, agentic expansion, fuzzification, and human/LLM screening (§3.2–§3.6), and the reported Pass@1 numbers are measured model outputs, not quantities derived from the construction equations. OmniAtlas is trained with hindsight-guided tree exploration, masked SFT, and OmniDPO (§4.2–§4.4); the training process uses ground-truth-conditioned verification, which is standard supervised/RL-style training, not a fitted parameter renamed as a prediction. No uniqueness theorem is invoked, and no load-bearing argument reduces to a self-citation. The related-work section contains many self-citations, but the benchmark and training results do not depend on them. The overlap between the construction/review models (DeepSeek-V3.2, Gemini-3) and the LLM-as-a-Judge, and the fact that OmniAtlas additionally receives read_image/read_audio/read_video tools and a perception-seeking system prompt (App. B.2–B.3), are experimental-validity concerns rather than circular reductions; they affect causal interpretation of the 13.3→20.8 gain, but do not make the derivation equivalent to its inputs. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

Central claims depend on the benchmark's ground truth, which rests on unvalidated LLM pipelines and a small human review; the training recipe contributes standard hyperparameters.

free parameters (3)

candidate continuations k in hindsight-guided tree exploration = 3
Hand-chosen branch factor for trajectory synthesis (§4.2); affects trajectory diversity and quality but not fitted to a target.
video clip length cap = 60 s
Clips at most 60 seconds for fine-grained mining (§3.2); arbitrary granularity choice.
training epochs = 2 SFT + 2 DPO
Training schedule in §A.1; chosen without reported hyperparameter search.

assumptions (4)

domain assumption DeepSeek-V3.2 event graphs faithfully represent cross-modal entities/relations
§3.3: the graph is 'the backbone' and used to synthesize tasks; no validation of graph correctness beyond downstream human review.
domain assumption Human review of 360 QA pairs guarantees solvability and answer uniqueness
§3.6: three graduate reviewers verify each pair; no inter-rater agreement or audit provided.
domain assumption DeepSeek-V3.2 LLM judge reliably decides answer equivalence
§5.1: LLM-as-a-Judge fallback; judge model is from the same family used in construction; no judge accuracy measurement.
domain assumption LLM screening correctly filters tasks that do not require omni-modal perception or tools
§3.6: review committee filters based on criteria; could bias the benchmark toward LLM-solvable patterns.

invented entities (1)

Omni-modal event graph
purpose: Structured representation connecting cross-modal entities/events to synthesize multi-hop QA (§3.3-3.5)
Internal construction artifact; no external falsifiable handle, its validity is only asserted via human review.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OmniGAIA: Towards Native Omni-Modal AI Agents." pith.science (2026). https://pith.science/paper/5XI654B2

@misc{pith2026260222897,
  author       = {Pith},
  title        = {Pith review of: OmniGAIA: Towards Native Omni-Modal AI Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5XI654B2}},
  note         = {Machine review of arXiv:2602.22897}
}

read the original abstract

Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g., vision-language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models
cs.CV 2026-08 conditional novelty 6.0 of 10

A training-free two-stage token compression framework preserves 92.9% of Qwen2.5-Omni-7B's audio-visual understanding accuracy while using 6.8% of the original multimodal-token FLOPs.
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
cs.LG 2026-07 conditional novelty 5.5 of 10

Query-guided, modality-symmetric token compression preserves audio-visual evidence better than audio-guided budgets at 25–35% retention on Qwen2.5-Omni.

Reference graph

Works this paper leans on

6 extracted references · 1 linked inside Pith · cited by 2 Pith papers

[1]

Do not guess or hallucinate details from blurry or ambiguous regions

**CERTAINTY FIRST**: Only provide information you are absolutely certain about. Do not guess or hallucinate details from blurry or ambiguous regions. If you are unsure, do not include it
[2]

ocr": [ 19 {

**COMPREHENSIVE & FACTUAL**: Focus on extracting strictly factual, objective information. Ensure high detail and density of information to support future analysis. Describe all visible text, objects, and people in detail. The JSON object must contain the following fields: ‘‘‘json { "ocr": [ 19 { "text": "detected text string", "detailed_features": "detail...
[5]

asr": [ {

**COMPREHENSIVE & FACTUAL**: Concentrate on factual information from both speech and non-speech sounds. Ensure high detail and density of information to support future analysis. The JSON object must contain the following fields: ‘‘‘json { "asr": [ { "text": "transcribed text", "start": 0.0, "end": 2.5, "speaker": "speaker_1" } ], "speakers": { "speaker_1"...
[6]

Do not guess unclear speech or ambiguous sounds

**CERTAINTY FIRST**: Only provide information you are absolutely certain about. Do not guess unclear speech or ambiguous sounds. If you are unsure, do not include it
[7]

asr": [ // If a single person speaks for a long time, segment the speech into pieces, with each segment containing one piece of information {

**COMPREHENSIVE & FACTUAL**: Concentrate on factual information from both speech and non-speech sounds. Ensure high detail and density of information to support future analysis. The JSON object must contain the following fields: ‘‘‘json { "asr": [ // If a single person speaks for a long time, segment the speech into pieces, with each segment containing on...

2024
[2026]

look/listen-where-needed

URL https://arxiv.org/abs/2602.00846. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.Trans. Mach. Learn. Res., 2025, 2025a. URL https://openreview.net/ forum?id=zKv8qULV6n. Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shi...

arXiv 2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Do not guess or hallucinate details from blurry or ambiguous regions

**CERTAINTY FIRST**: Only provide information you are absolutely certain about. Do not guess or hallucinate details from blurry or ambiguous regions. If you are unsure, do not include it

[2] [2]

ocr": [ 19 {

**COMPREHENSIVE & FACTUAL**: Focus on extracting strictly factual, objective information. Ensure high detail and density of information to support future analysis. Describe all visible text, objects, and people in detail. The JSON object must contain the following fields: ‘‘‘json { "ocr": [ 19 { "text": "detected text string", "detailed_features": "detail...

[3] [5]

asr": [ {

**COMPREHENSIVE & FACTUAL**: Concentrate on factual information from both speech and non-speech sounds. Ensure high detail and density of information to support future analysis. The JSON object must contain the following fields: ‘‘‘json { "asr": [ { "text": "transcribed text", "start": 0.0, "end": 2.5, "speaker": "speaker_1" } ], "speakers": { "speaker_1"...

[4] [6]

Do not guess unclear speech or ambiguous sounds

**CERTAINTY FIRST**: Only provide information you are absolutely certain about. Do not guess unclear speech or ambiguous sounds. If you are unsure, do not include it

[5] [7]

asr": [ // If a single person speaks for a long time, segment the speech into pieces, with each segment containing one piece of information {

**COMPREHENSIVE & FACTUAL**: Concentrate on factual information from both speech and non-speech sounds. Ensure high detail and density of information to support future analysis. The JSON object must contain the following fields: ‘‘‘json { "asr": [ // If a single person speaks for a long time, segment the speech into pieces, with each segment containing on...

2024

[6] [2026]

look/listen-where-needed

URL https://arxiv.org/abs/2602.00846. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.Trans. Mach. Learn. Res., 2025, 2025a. URL https://openreview.net/ forum?id=zKv8qULV6n. Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shi...

arXiv 2025