Native Active Perception as Reasoning for Omni-Modal Understanding

Chi-Wing Fu; Jin Xu; Jinzheng He; Junyang Lin; Pheng-Ann Heng; Qize Yang; Ruiyang Xu; Yunfei Chu; Yuxuan Wang; Zhenghao Xing

REVIEW 4 major objections 5 minor 20 references

Native Active Perception as Reasoning for Omni-Modal Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OmniAgent turns long-video understanding into a query-driven loop that distills raw frames and audio into text memory, letting a 7B model beat a 10x larger one on LVBench.

desk verdict A solid systems paper with a genuinely interesting active-perception loop and a credible 7B-vs-72B result, but the LongVALE gain is contaminated by training on its own split and the central decoupling claim rests on an unmeasured observation-fidelity assumption that the paper should be pushed to quantify. read the letter →

arxiv 2606.19341 v2 pith:VWLJL656 submitted 2026-06-17 cs.CV cs.CLcs.SD

Zhenghao Xing , Ruiyang Xu , Yuxuan Wang , Jinzheng He , Ziyang Ma , Qize Yang , Yunfei Chu , Jin Xu

show 3 more authors

Junyang Lin Chi-Wing Fu Pheng-Ann Heng

This is my paper · ORCID

classification cs.CVcs.CLcs.SD

keywords activeperceptionlongvideounderstandingomni-modalagentPOMDPpersistenttextualmemorytest-timescalingreinforcementlearningtemporalgrounding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that long-video understanding does not require watching everything. It proposes an agent that runs an iterative Observe–Think–Act loop: conditioned on a query, it requests specific frames, audio segments, or clips, writes what it saw into a persistent text memory, and deletes the raw media before the next turn. The claim is that this strict distillation makes reasoning cost depend on the difficulty of the query, not the length of the video. To train such behavior, the paper introduces an agentic supervised fine-tuning stage that synthesizes successful interaction trajectories, then a reinforcement-learning stage whose turn-level entropy weighting (TAURA) credits pivotal 'discovery' turns. If the claim holds, a 7B agent can beat a 10 times larger passive model on long-video benchmarks while using about a quarter of the frames, and performance keeps improving when more interaction turns are allowed.

What carries the argument

The load-bearing mechanism is the Observation–Thought–Action (OTA) cycle operating inside a POMDP. The environment returns raw media (frames, audio, clips) as a transient percept; the policy distills it into a textual Observation that is appended to a persistent memory, then the raw percept is discarded before the next turn. This strict 'look, write, forget the pixels' loop is what decouples memory from duration. On the training side, TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage) rescales each trajectory-level advantage by the mean token entropy of each turn, so high-uncertainty discovery turns receive amplified credit instead of the uniform credit that vanilla GRPO gives all tu

What would settle it

Take a set of long-video questions whose answers depend on fine-grained details (exact wording of on-screen text, small object counts, subtle sounds). Run OmniAgent, then immediately ask a second round of questions whose answers are visible or audible only in the raw media that was purged after the observation step. If accuracy on those 'forgotten detail' questions is near chance while a passive model that kept the frames answers them well, the textual memory is not faithful and the decoupling claim fails.

Watch

Extended reading notes

Core claim

OmniAgent's central claim is that active perception can be native reasoning rather than a preprocessing step. The paper formulates video understanding as a Partially Observable Markov Decision Process in which the model itself chooses, at each turn, one of four actions—retrieve frames, retrieve audio, retrieve a synchronized clip, or answer—and then generates a structured triplet: an Observation (a textual distillation of what the retrieved media showed), a Thought (why the evidence does or does not resolve the query), and an Action. Crucially, after each observation is written, the environment purges the raw media from context so the model's internal state is a compact text memory whose siz

Load-bearing premise

The framework assumes the agent's short textual Observation, written before raw media is deleted, retains every detail that later turns might need; if that transcription is lossy, the purging step guarantees irreversible forgetting and the central decoupling claim collapses.

Editorial extensions

If this is right

If decoupling holds, long-video benchmarks that currently require dense frame processing can be approached with a constant per-turn media budget, making hour-long videos feasible on a single 7B model.
Positive test-time scaling means giving the agent more turns is a reliable compute–accuracy axis, analogous to spending more 'thinking time' on harder queries.
Agentic SFT with best-of-N trajectories and dual quality control (outcome verification plus reasoning-coherence audit) is a transferable recipe for bootstrapping agentic behavior in other multimodal settings.
TAURA's entropy rescaling suggests that credit in multi-turn agentic RL should be assigned at the level of decision forks, not uniformly across whole trajectories; vanilla GRPO can actively degrade performance on perception tasks (DailyOmni drop in the ablation).
The interpretable OTA traces make the model's evidence-gathering and termination decisions auditable, since each answer is grounded in explicitly recorded observations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the decoupling claim would probe observation fidelity: run OmniAgent on a video, then ask questions about details that appeared only in purged raw media and were not recorded in any Observation. The gap between OmniAgent and a passive model that kept the frames quantifies how lossy the textual bottleneck is; the paper currently offers qualitative trajectories but no such number.
The same OTA loop is modality-agnostic: any environment where raw data is expensive to keep (egocentric video, meeting audio, document scans, robot sensor streams) could inherit the decoupling argument, since the mechanism only assumes that a compact textual summary can substitute for raw percepts.
Entropy is a coarse proxy for decision importance; replacing or augmenting the turn-level entropy weight with expected information gain (how much the next action reduces uncertainty about the answer) could make TAURA's credit assignment sharper and is directly measurable in the same environment.
Because the persistent memory still grows linearly with turns, 'decoupled from duration' really means 'bounded by trajectory complexity'; for extremely long videos with many necessary discoveries, the text memory could itself become the bottleneck, suggesting a future compression or summarization of older observations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth a serious look. The core idea — a single omni-modal policy running an Observation-Thought-Action loop with strict purging of raw media into persistent text memory — is a real step beyond existing agentic video systems, and the two-stage training (Agentic SFT plus the TAURA entropy-rescaled advantage) hangs together. The ablations actually support the central claims: Agentic SFT beats standard SFT, TAURA beats vanilla GRPO, and the positive test-time scaling on VideoMME-Long is a nice empirical result. Code and checkpoints are public, and the runtime analysis is a plus.

The soft spots are real but not fatal. Most importantly, the SFT corpus uses the training split of LongVALE, and LongVALE is also in the evaluation suite. That makes the +33.4 gain on LongVALE uninterpretable as a measure of the method — it could just be memorization. The authors need to re-evaluate on a held-out split or drop that claim. Second, every headline table reports single runs without error bars or significance tests. For a paper built on a few-point differences, that matters. Third, the observation-fidelity concern from the stress test is legitimate: the instruction template explicitly warns the agent will forget any detail not recorded, yet nothing in the training objective rewards observation completeness. The qualitative examples even show the agent producing a wrong observation (the "10 stages" miscount in Kyo-yuzen) and only recovering because it later re-sensed. That means the decoupling claim can fail for exactly the queries the framework is designed for, and the cost of recovery may scale with video duration. To be fair, Table 5 suggests that on LVBench the agent's turn count grows only slowly with duration, so the decoupling does hold in practice on that benchmark. But the fidelity assumption remains unmeasured, and the authors should be asked to quantify it — e.g., by checking how often the final answer depends on details that were purged and had to be re-acquired.

The "first native" framing is a bit strong given the existing agentic video literature, but the specific combination here does seem new. The self-citation to Qwen2.5-Omni is appropriate.

Overall: this is a paper for anyone working on long-video understanding or multimodal agents. It deserves peer review, not a desk reject, but the reviewers should push for a clean LongVALE evaluation, error bars, and a direct fidelity measurement before acceptance.

Referee Report

4 major / 5 minor

Summary. The paper proposes OmniAgent, a 7B omni-modal agent that formulates video understanding as a POMDP-driven iterative Observation-Thought-Action cycle. At each turn the agent actively selects frames, audio, or clips, distills the raw percept into a textual observation, and purges the raw media so that the persistent context is text-only. Training proceeds in two stages: Agentic SFT, which bootstraps the policy from best-of-N synthesized trajectories with outcome verification and a GPT-4o rationality audit, and Agentic RL with TAURA, an entropy-weighted variant of GRPO that redistributes trajectory-level advantage to high-uncertainty turns. Experiments on ten benchmarks report state-of-the-art results among open-source models, including a 50.5% on LVBench that surpasses Qwen2.5-VL-72B (47.3%), along with positive test-time scaling on VideoMME-Long.

Significance. If the central claims hold, the paper makes a practically important contribution: it provides a concrete mechanism for long-video understanding whose context cost does not grow with video duration, and it demonstrates that active perception can be learned end-to-end in a single omni-modal model rather than through external tool orchestrators. The framework is well-motivated, the OTA formulation is clear, and the authors release code and model weights. TAURA is a novel, empirically motivated credit-assignment heuristic, and the paper includes useful ablations separating Agentic SFT from standard SFT and TAURA from vanilla GRPO. The strength of the contribution depends on three load-bearing assumptions: (i) the textual observation O_k faithfully captures all details that may be needed later, (ii) the training and evaluation splits for overlapping benchmarks are disjoint, and (iii) the reported gains are statistically reliable despite single-run evaluation.

major comments (4)

[Algorithm 1, §3.1, §3.2, Eq. (2)/(5)] The decoupling claim (reasoning complexity independent of raw video duration) rests on O_k being a faithful and sufficiently complete encoding of the purged E_{k-1}. The training objectives in Eq. (2) and Eq. (5) contain no term for observation completeness, and the GPT-4o rationality audit in §3.2 only checks that T_k is entailed by M_{k-1} and O_k, not that O_k records all details a later query might need. Since E_{k-1} is discarded, any omitted detail is recoverable only through new sensing actions, whose number could scale with video duration. The paper gives no quantitative fidelity measurement. Figure 7 illustrates the risk: the initial frame-based observation states '10 stages', which is later corrected only by an audio action. Please add a fidelity evaluation (e.g., recall of salient facts in O_k against ground-truth descriptions, or performance on follow-up questions answered fr
[§3.2, Tables 1 and 3] The SFT corpus is built from the training splits of five datasets, including LongVALE and VSI-Train-10k (the training split of VSI-Bench), and both LongVALE and VSI-Bench are also used for evaluation. The paper does not explicitly state that the evaluation splits are disjoint from the SFT/RL training splits in terms of videos and queries. Without this guarantee, large reported gains such as +33.4 IoU on LongVALE and 48.4 on VSI-Bench could be inflated by memorization rather than active perception. Please specify the exact split protocol for each SFT source that overlaps with evaluation, and confirm that no test video or question was seen during SFT or RL.
[Tables 1–5, Figure 2] All benchmark results are single-run reports without error bars, confidence intervals, or significance tests. The key comparisons are often small: in Table 4, TAURA vs. vanilla GRPO differs by +0.2, +0.7, +1.2, +1.8, and +2.6 across five benchmarks; in Table 5, accuracy fluctuates by about ±3 points across duration bins; and the +6.2% test-time scaling in Figure 2 is based on a single evaluation. These differences could be within noise. Please provide variance estimates or significance tests (e.g., multiple seeds, bootstrap CIs) for at least the central claims: TAURA > GRPO, positive test-time scaling, and duration invariance in Table 5.
[Table 3, §4.2] The temporal-grounding comparisons against Qwen2.5-Omni are dramatic (LongVALE 5.7→39.1, VUE-TR 8.0→46.1), but the paper does not describe the evaluation protocol for baselines. If Qwen2.5-Omni was evaluated as a passive model without the agentic instruction template and action interface, it would be substantially handicapped on a task that requires outputting time ranges. Please report the exact prompt and interface used for each baseline and, ideally, compare against a strong grounding-capable baseline (e.g., Vidi) under the same interface. Without this, the absolute gain cannot be attributed solely to active perception.

minor comments (5)

[Eq. (4)] The underbrace labeled w_{i,k} covers the entire fraction, but the text says normalizing H_{i,k} relative to the group mean. Clarify that the denominator is the mean entropy across all turns in the group, not across trajectories only.
[Figure 2] The x-axis is the maximum turn limit K, not the number of executed turns. Since average executed turns saturate at about 11.7, the figure demonstrates a truncation effect more than a scaling law. Consider plotting accuracy against executed turns and discussing the distinction.
[Table 1] The footnote VISTA result is based on LongV A is ambiguous. The VISTA row has dashes in VSI-Bench and LVBench, so it is unclear which column the footnote refers to.
[Abstract, §1] The claim first native omni-modal agent needs a precise definition of native (e.g., single model with no external perception modules) to distinguish it from methods like LongVT that also use tool calling.
[Appendix C.4] The runtime table reports Frames, Model (s), and Wall-clock (s) but does not define the difference between the latter two nor report variance. Clarify what is included in each column and report the number of samples and standard deviation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: core claims are empirically tested against external benchmarks; self-citations are to released models or independently re-derived results.

full rationale

The paper's derivation chain is not circular. The core novelty—an Observation-Thought-Action loop with strict context purging—is a mechanism whose success is assessed by external benchmark accuracy (VideoMME, LVBench, DailyOmni, etc.) under verifiable rewards defined in Eq. (2), not by the mechanism's own definitions. The 'decoupling' claim is supported by an empirical duration analysis (Table 5) and test-time scaling curve (Figure 2), both of which could have failed; they are not forced by Algorithm 1, which only guarantees constant media overhead, not constant reasoning turns. TAURA's entropy weighting is motivated by an independently labeled fork analysis in Appendix C (Gemini-2.5-Pro identifies forks; entropy is then measured), so it does not reduce to a self-citation. The only overlapping-author citations (Qwen2.5-Omni as base model; Wang et al. 2025b on high-entropy tokens) are either a released, externally reproducible checkpoint or are independently corroborated in the paper, so they are not load-bearing. The unmeasured observation-fidelity assumption (instruction template: 'Once media is omitted, you will forget any detail not recorded here') is a genuine correctness risk that could break the decoupling claim for fine-grained queries, but it is an empirical assumption, not an equation-level circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The central result rests on trained checkpoints and a set of thresholds/hyperparameters rather than a closed-form derivation; the most consequential free parameters are the SFT quality filters and the turn limits.

free parameters (4)

Coherence threshold in GPT-4o audit = 3/5
Chosen by hand in the dual-stage quality control to filter SFT trajectories; directly shapes the trained policy.
Outcome verification thresholds for continuous tasks = IoU ≥ 0.5; MRA ≥ 0.5
Defines what counts as a successful SFT trajectory for temporal grounding and size estimation.
Turn limits K = SFT [5,32], RL [5,10], eval up to 52
The test-time scaling result is a function of K; the model is trained with smaller turn budgets than those used in Figure 2.
Max visual tokens and context window = 1024 per image, 768 per frame, 64K context
Implementation choices that limit percept detail and thus constrain the fidelity of textual observations.

assumptions (5)

domain assumption Qwen2.5-Omni-7B provides sufficient base perception to extract query-relevant details from frames and audio.
The whole pipeline is built on this base model; if its perception is weak, active selection cannot fully compensate (see §3.2, §4.1).
domain assumption Textual observations are faithful enough to replace raw media in the persistent memory.
Algorithm 1 and §3.1 purge raw media and keep only O_k; the central decoupling claim depends on this lossy summary being sufficient.
domain assumption Mean token entropy of a turn is a reliable proxy for decision criticality.
TAURA relies on this; Appendix C reports a 79.2% correlation on a Gemini-labeled subset, but it is not a proven causal guarantee.
domain assumption Official train/test splits of the evaluated benchmarks are respected.
SFT uses training splits of LongVALE and VSI-Train-10k, but the paper does not specify which official eval splits are used in Tables 1 and 3.
standard math Standard POMDP/GRPO mathematics and verifiable rewards apply.
The RL objective follows standard GRPO with token-level loss; no nonstandard mathematical machinery is introduced.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Native Active Perception as Reasoning for Omni-Modal Understanding." pith.science (2026). https://pith.science/paper/VWLJL656

@misc{pith2026260619341,
  author       = {Pith},
  title        = {Pith review of: Native Active Perception as Reasoning for Omni-Modal Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWLJL656}},
  note         = {Machine review of arXiv:2606.19341}
}

abstract

Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5% vs. 47.3%).

Figures

Figures reproduced from arXiv: 2606.19341 by the authors.

**Figure 1.** The OmniAgent Framework for Native Active Perception. Unlike passive methods that process video frames uniformly, OmniAgent treats perception as an iterative reasoning process via an Observation-Thought-Action (OTA) cycle. Conditioned on a specific query, the agent executes on-demand actions to selectively gather audio-visual cues, distilling high-dimensional transient percepts into a persistent textual memory until… view at source ↗

**Figure 3.** Accuracy vs. Visual Frame Count on LVBench. OmniAgent-7B (red diamond, 50.5%) outperforms the 10× larger Qwen2.5-VL-72B (47.3%) while using ∼73% fewer frames (203 vs. 768). Marker shape distinguishes agentic (diamond) from passive (circle) models, and marker size represents parameter scale. Test-time Scaling Analysis. We analyze the scaling properties on VideoMME-Long by extending the maximum turn limit K from 6 to… view at source ↗

**Figure 4.** The complete agent instruction template used across all stages of the OmniAgent pipeline. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: Correlation between Entropy and Critical Reasoning Steps. (a) Histogram of the entropy difference ∆H = Hi,kfork − Hi. 79.2% of critical fork steps exhibit higher uncertainty than the trajectory mean. (b) A qualitative case study on the ”Company” query. A distinct entro…

**Figure 6.** Figure 6: Qualitative Example of Active Reasoning (MCQ). OmniAgent first narrows the search space by revisiting the early broadcast window where host intros and lower-thirds typically occur, then visually confirms the green dress/skirt host at multiple timestamps. When audio che…

**Figure 7.** Figure 7: Qualitative Example of Audio-visual Collaborative Reasoning (MCQ). OmniAgent performs targeted temporal queries to locate the Kyo-yuzen description, then cross-checks on-screen text with narration. It resolves the ambiguity about the number of stages by consulting the …

**Figure 8.** Figure 8: Qualitative Example of Temporal Grounding. OmniAgent combines coarse full-video scanning with targeted audio checks to propose a candidate gavel event, actively falsifies the early false lead via localized probing, then pivots to a later visually-agitated segment and c…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 4 linked inside Pith

[1]

For every user message containing multimodal content, the environmentdeletesthe dictionary entries of typeimage,video, oraudio

Media Removal:Once a sensing turn k is completed and its textual distillation Ok is recorded, the environment iterates through the previous chat history. For every user message containing multimodal content, the environmentdeletesthe dictionary entries of typeimage,video, oraudio
[2]

Video-LLaV A: Learning united visual representation by alignment before projection

3 Lin, B., Ye, Y ., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-LLaV A: Learning united visual representation by alignment before projection. InAnnual Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5971–5984, 2024. 1 Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tuning. InAdvances in Neural Informatio...

arXiv 2024
[3]

Temporal Bisection: Find ’start’ and ’end’ boundary frames where a state changes, then iteratively narrow the interval to locate the exact transition second or frame
[4]

Vide- oRFT: Incentivizing video reasoning capability in mllms via reinforced fine-tuning

6, 7 Wang, Q., Yu, Y ., Yuan, Y ., Mao, R., and Zhou, T. Vide- oRFT: Incentivizing video reasoning capability in mllms via reinforced fine-tuning. InAdvances in Neural Infor- mation Processing Systems (NeurIPS), 2025a. 6 Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X.-H., Yang, J., Zhang, Z., Liu, Y ., Yang, A., Zhao, A., Yue, Y ...
[5]

LifelongMemory: Lever- aging llms for answering queries in egocentric videos

3 Wang, Y ., Yang, Y ., and Ren, M. LifelongMemory: Lever- aging llms for answering queries in egocentric videos. CoRR, abs/2312.05269, 2023. 3 Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., and Bansal, M. VideoTree: Adaptive tree- based video representation for LLM reasoning on long videos. InConference on Computer Vision and Pa...

arXiv 2023
[6]

VCA: video curious agent for long video understanding

3 Yang, Z., Chen, D., Yu, X., Shen, M., and Gan, C. VCA: video curious agent for long video understanding. In International Conference on Computer Vision (ICCV), 2025c. 3 Yang, Z., Wang, S., Zhang, K., Wu, K., Leng, S., Zhang, Y ., Li, B., Qin, C., Lu, S., Li, X., and Bing, L. LongVT: Incentivizing ”thinking with long videos” via native tool calling. InCo...

arXiv 2026
[8]

Frames 10.0s-20.0s. Timestamps: [10.00s, 12.50s, 15.00s] [MEDIA OMITTED...]

Metadata-Preserving Rewrite:To maintain the semantic integrity of the reasoning trace, the deleted media objects are replaced with a text-based summary appended with the marker [MEDIA OMITTED - Refer to Observation Ok]: • ForVisual Sampling( aframes), the environment recompiles the exact timestamps of all purged frames from the prior message content (e.g....
[9]

- Scanning: Use wide ranges (e.g., start=0, end=duration, num={max_frames_len}) to discover the overall timeline and identify key milestones or potential scene cuts

Visual Search (get_frames): (Max {max_frames_len} frames). - Scanning: Use wide ranges (e.g., start=0, end=duration, num={max_frames_len}) to discover the overall timeline and identify key milestones or potential scene cuts. - Precision: Use narrow windows (1-2s) with high num for micro-details (logos, text, fast motions, or subtle object state changes)

Show all 20 references

[10]

Person_A at [20, 45]

Counting & Re-ID: Assign approximate spatial locations [y, x] (0-100 scale; [0,0] is top-left) to each unique instance (e.g., " Person_A at [20, 45]") in your observation. This spatial ID prevents re-counting the same object across different frames/steps
[12]

- Verbatim Logging: Identify speakers and transcribe speech near-verbatim

Audio Analysis (get_audio): (Max {max_audio_len}s). - Verbatim Logging: Identify speakers and transcribe speech near-verbatim. CRITICAL: Do not paraphrase or infer words to fit your hypothesis. - Acoustic Context: Identify critical off-screen or background sounds (e.g., footst...
[13]

Who moved first?

Multi-Modal Action Analysis (get_clip): (Max {max_clip_len}s). - Action & Temporal Dynamics: Analyze the nature of movement (speed, direction, continuity) and precise sequencing to solve "Who moved first?" or "Was the motion deliberate?". - Process Logic: Use when the continuo...
[14]

type": "get_frames

{"type": "get_frames", "start": float, "end": float, "num": int}
[15]

type": "get_audio

{"type": "get_audio", "start": float, "end": float}
[16]

type": "get_clip

{"type": "get_clip", "start": float, "end": float}
[17]

type": "answer

{"type": "answer", "content": "string"} - MCQ: Letter only (e.g., "A"). - TR: JSON array of one or more pairs, e.g., "[[10.5, 20.0], [35.0, 40.0]]". - NUM/SIZE: A single number string, e.g., "10.3". - FF: Detailed descriptive text. ============= STRICT EXECUTION PROTOCOL =====...
[18]

The policy is confident in this information gathering, resulting in low entropy (H≈0.397)

Routine Scanning:The agent performs standard scanning. The policy is confident in this information gathering, resulting in low entropy (H≈0.397)
[19]

This observation acts as a critical filter, ruling out these candidates

The Fork Step:A distinct entropy spike ( H≈0.927 ) occurs when the agent identifies that options A, B, and C are explicitly mentioned in the audio. This observation acts as a critical filter, ruling out these candidates. The spike reflects the agent’s pivotal decision to switc...
[20]

Nish Parkar

Resolution:Subsequent verification resolves the ambiguity, returning the agent to a lower entropy state ( H≈0.790 ). This confirms that entropy spikes serve as a reliable signal for identifying high-value reasoning steps that warrant amplified reinforcement. 16 Native Active P...
[2024]

Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

3 Fu, C., Dai, Y ., Luo, Y ., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y ., Zhang, M., et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InConference on Computer Vision and Pattern Recognition (CVPR), pp. 24...

2025 arXiv
[2025]

GPT-4o system card.arXiv preprint arXiv:2410.21276, 2024

6 OpenAI. GPT-4o system card.arXiv preprint arXiv:2410.21276, 2024. URL https://openai .com/index/hello-gpt-4o/. 3, 4, 6, 7 Rasheed, H., Zumri, M., Maaz, M., Yang, M., Khan, F. S., and Khan, S. H. Video-CoM: Interactive video reasoning via chain of manipulations. InConference ...

2024 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

For every user message containing multimodal content, the environmentdeletesthe dictionary entries of typeimage,video, oraudio

Media Removal:Once a sensing turn k is completed and its textual distillation Ok is recorded, the environment iterates through the previous chat history. For every user message containing multimodal content, the environmentdeletesthe dictionary entries of typeimage,video, oraudio

[2] [2]

Video-LLaV A: Learning united visual representation by alignment before projection

3 Lin, B., Ye, Y ., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-LLaV A: Learning united visual representation by alignment before projection. InAnnual Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5971–5984, 2024. 1 Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tuning. InAdvances in Neural Informatio...

arXiv 2024

[3] [3]

Temporal Bisection: Find ’start’ and ’end’ boundary frames where a state changes, then iteratively narrow the interval to locate the exact transition second or frame

[4] [4]

Vide- oRFT: Incentivizing video reasoning capability in mllms via reinforced fine-tuning

6, 7 Wang, Q., Yu, Y ., Yuan, Y ., Mao, R., and Zhou, T. Vide- oRFT: Incentivizing video reasoning capability in mllms via reinforced fine-tuning. InAdvances in Neural Infor- mation Processing Systems (NeurIPS), 2025a. 6 Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X.-H., Yang, J., Zhang, Z., Liu, Y ., Yang, A., Zhao, A., Yue, Y ...

[5] [5]

LifelongMemory: Lever- aging llms for answering queries in egocentric videos

3 Wang, Y ., Yang, Y ., and Ren, M. LifelongMemory: Lever- aging llms for answering queries in egocentric videos. CoRR, abs/2312.05269, 2023. 3 Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., and Bansal, M. VideoTree: Adaptive tree- based video representation for LLM reasoning on long videos. InConference on Computer Vision and Pa...

arXiv 2023

[6] [6]

VCA: video curious agent for long video understanding

3 Yang, Z., Chen, D., Yu, X., Shen, M., and Gan, C. VCA: video curious agent for long video understanding. In International Conference on Computer Vision (ICCV), 2025c. 3 Yang, Z., Wang, S., Zhang, K., Wu, K., Leng, S., Zhang, Y ., Li, B., Qin, C., Lu, S., Li, X., and Bing, L. LongVT: Incentivizing ”thinking with long videos” via native tool calling. InCo...

arXiv 2026

[7] [8]

Frames 10.0s-20.0s. Timestamps: [10.00s, 12.50s, 15.00s] [MEDIA OMITTED...]

Metadata-Preserving Rewrite:To maintain the semantic integrity of the reasoning trace, the deleted media objects are replaced with a text-based summary appended with the marker [MEDIA OMITTED - Refer to Observation Ok]: • ForVisual Sampling( aframes), the environment recompiles the exact timestamps of all purged frames from the prior message content (e.g....

[8] [9]

- Scanning: Use wide ranges (e.g., start=0, end=duration, num={max_frames_len}) to discover the overall timeline and identify key milestones or potential scene cuts

Visual Search (get_frames): (Max {max_frames_len} frames). - Scanning: Use wide ranges (e.g., start=0, end=duration, num={max_frames_len}) to discover the overall timeline and identify key milestones or potential scene cuts. - Precision: Use narrow windows (1-2s) with high num for micro-details (logos, text, fast motions, or subtle object state changes)