{"id":"6ce7a9e0-9081-47b2-9add-c75186420fdb","arxiv_id":"2608.13239","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Simple direct-answer fine-tuning matches or beats chain-of-thought reasoning models for social audio-visual QA, and a question-independent caption performs as well as full video and audio input.","lead":"An audit of the IntentBench social audio-visual QA benchmark finds that roughly 30% of questions are broken or answerable from text alone, and the authors release a cleaned version called IntentBench-Prime. It then shows that a plain supervised fine-tuning baseline beats chain-of-thought reasoning models on three benchmarks, and that a text caption can replace the video almost without loss.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5's Vanilla SFT row may not be trained on the same restricted S-IQ+EMER data as Caption/Question SFT, confounding the caption-parity finding.","rationale":"The reader's weakest_assumption identifies precisely the unresolved experimental condition: whether Table 5's Vanilla SFT was retrained on the same restricted data as Caption/Question SFT. This is load-bearing because it determines whether the caption-vs-video comparison is a valid modality ablation or a comparison confounded by training-data size and composition. The paper's wording is genuinely ambiguous: Section 5 first says the baselines are compared with the Vanilla SFT method from Section 4, then says training is identical apart from modality. The numerical mismatch (69.5 vs 70.4/69.0) suggests a different training run, but the text never confirms it. A concrete config/code check would settle the issue. I agree with the reader's conditional verdict: the finding should not be taken as definitive until this condition is clarified, but the un-finetuned comparison and the overall benchmark audit provide enough independent support that rejection is not warranted.","tokens_in":18076,"tokens_out":5069,"duration_ms":50998,"concrete_test":"Inspect the released code/config to determine the exact training-data manifest used for the 'Vanilla SFT' row in Table 5. If it is identical to the Section 4 full 20K-video set, retrain Vanilla SFT on the restricted Social-IQ 2.0 + EMER set (matching Caption SFT and Question SFT) and rerun Table 5; if the gap to Caption SFT remains within ~1 point, Finding 3 survives, otherwise the caption-parity claim is confounded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Finding 3 — that a generic textual caption yields QA performance on par with processing full multimodal data — rests entirely on Table 5, where Caption SFT (68.6) is compared to Vanilla SFT (69.5). Section 5 says 'We limit the training data to Social-IQ 2.0 and EMER' and 'Apart from the modality setup, training is identical between the three variants,' but it also says the baselines are 'compared with the Vanilla SFT method from Section 4.' In Section 4, Vanilla SFT is trained on the full HumanOmniV2-matching data: OmniInstruct, Video-R1, Social-IQ 2.0 training data, and 200 EMER entries, i.e., roughly 20K videos plus 10K images. The Table 5 score of 69.5 also differs from both Section 4 Vanilla SFT scores (70.4 LoRA, 69.0 full FT). If Table 5's Vanilla SFT is the Section 4 model, then it was trained on substantially more and different data than Caption SFT, so the observed parity is not a clean modality ablation: the video model had a training-data advantage, and the caption model still matched it. If Table 5's Vanilla SFT was retrained on the restricted S-IQ+EMER set, the paper should state this explicitly and report the training-data composition. This is the single most load-bearing ambiguity because Finding 3 is the paper's most surprising conclusion and would otherwise be an artifact of an unmatched experimental condition. The un-finetuned rows (caption 62.1 vs video 63.2) partially mitigate the concern, but they do not establish the finetuned claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper audits the IntentBench benchmark, introduces a curated version (IntentBench-Prime), and reports three findings: (1) IntentBench is noisy, with about 7% broken questions and about 23% of questions answerable from text alone; (2) a simple direct-answer supervised fine-tuning baseline (Vanilla SFT) matches or outperforms current chain-of-thought/GRPO-based reasoning methods on IntentBench-Prime, WorldSense, and Daily-Omni at a fraction of the training and inference cost; and (3) training a model to answer from a question-independent textual caption yields accuracy comparable to training on the full video+audio input, suggesting that current omni-models extract limited additional question-relevant information from the video and audio streams. The authors release the IntentBench-Prime exclusion list, the Vanilla SFT model, and code.","tokens_in":18441,"tokens_out":6645,"duration_ms":63998,"significance":"If the findings hold, they are practically important for the audio-visual question answering community: they challenge the necessity of expensive CoT/GRPO training for social AV-QA, advocate for a mandatory direct-SFT baseline, and expose a surprising ceiling on the use of video input by current MLLMs. The paper's strengths include a controlled setup for Finding 2 (same base model, same training data, conservative use of the original published numbers for HumanOmniV2), concrete efficiency measurements, and the practical design of the curated benchmark as an exclusion list that allows re-scoring of existing results. The main risk to the paper's central claims is the training-data ambiguity in the Table 5 comparison, which is load-bearing for Finding 3.","major_comments":[{"comment":"The caption-parity finding depends on whether the Vanilla SFT row in Table 5 was trained on the same restricted Social-IQ 2.0 + EMER data as Caption SFT and Question SFT. Section 5 says 'We limit the training data to Social-IQ 2.0 and EMER' and 'Apart from the modality setup, training is identical between the three variants,' but it also says the baselines are 'compared with the Vanilla SFT method from Section 4,' where Vanilla SFT is trained on the full 20K-video/10K-image mixture. The reported Table 5 value (69.5) differs from both Section 4 values (70.4 LoRA, 69.0 full FT), so the row cannot be unambiguously identified. If the Table 5 Vanilla SFT is the Section 4 model, the comparison is confounded by training-data volume and is not a clean modality ablation. Please state explicitly which training set was used for each row in Table 5, and if Vanilla SFT was retrained on the restricted data, report that run's training-data composition and hyperparameters.","section":"Section 5, Table 5"},{"comment":"The headline '7.1% broken questions' in Finding 1 is presented without the caveat that it is a lower bound. The curation procedure manually verifies questions in ranked order and stops after 300 consecutive non-broken questions, having verified 'more than half' of the S-IQ questions. The unverified remainder could still contain broken questions, so the 7.1% figure is not a measured rate on the full benchmark. Please report the number of questions actually manually inspected and explicitly describe the 7.1% statistic as a lower bound, or provide an estimate that accounts for the stopping rule.","section":"Section 3.2, broken-question curation"},{"comment":"Finding 3 is stated as 'a generic textual caption yields QA performance on par with processing the full multimodal data,' but the table-level support is a 0.9-point difference (Caption SFT 68.6 vs Vanilla SFT 69.5) on a single benchmark split, with no confidence intervals, significance test, or repeated-seed variation. Category-level differences are larger (e.g., How: 64.1 vs 67.0; Why: 68.2 vs 68.6), so the aggregate parity could be consistent with noise. Please provide variance estimates, multiple seeds, or a statistical test to support the parity claim, or soften the claim accordingly.","section":"Section 5, Table 5 and 'on par' claim"}],"minor_comments":[{"comment":"The name of the curated benchmark is written inconsistently as 'Intentbench-Prime' in the abstract and 'IntentBench-Prime' in the body; please unify the capitalization.","section":"Abstract and throughout"},{"comment":"The block containing 'FINDING 1' and Figure 1 appears to be duplicated in the introduction; one copy should be removed for clarity.","section":"Figure 1 and surrounding text"},{"comment":"The claim that 'there is no reasoning method with a comparable base-model and size that outperforms our Vanilla SFT baseline' should be made precise by listing the base model and parameter count of AVATAR and AffectOmni; as written, the reader cannot verify comparability.","section":"Section 4.2, Table 3"},{"comment":"The finding that the option-swapping makes questions easier for LLMs would be more convincing if the analysis reported per-model text-only accuracies rather than only the unanimous-agreement rate, since the latter is a strict criterion and may compress the effect.","section":"Section 3.1, Table 1"},{"comment":"The sentence 'We limit the training data to Social-IQ 2.0 and EMER to reduce the cost of the experiment' should specify which of the compared models the restriction applies to; this is the same ambiguity raised in the first major comment and should be resolved in the text.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful news first: this paper gives the social AV-QA community two things it actually needs. The IntentBench audit is careful and quantified—broken questions, text-answerability via an LLM ensemble, and the counterproductive option-swapping are all documented with examples, and the release of an exclusion list means existing results can be filtered without re-running. Finding 2 is also solid: Vanilla SFT uses the same base model and training data as HumanOmniV2, the comparison against original paper numbers is conservative, and the efficiency gap is stark. I buy that the field should adopt a direct-answer SFT baseline as standard.\n\nThe soft spot is exactly where the stress-test note lands. Finding 3 rests on Table 5, and the paper never states clearly whether the Vanilla SFT row there was retrained on the restricted Social-IQ 2.0 + EMER set or is carried over from Section 4, where it saw the full ~20K video / 10K image mixture. The score 69.5 matches neither Section 4 number (69.0 full FT, 70.4 LoRA), so it looks like a different run, but the text says only \"we limit the training data\" without specifying which rows that applies to. If the Table 5 Vanilla SFT is the full-data model, the caption parity is not a clean modality ablation—the caption model matched a model that saw far more data, which would actually strengthen the qualitative conclusion but needs to be said explicitly. If it was retrained on the restricted set, the paper should report that composition. Either way, this is fixable, but it is load-bearing for the most provocative claim.\n\nTwo minor points. There are no error bars or seed information, and the caption-vs-video gap is about one point, so the finetuned parity claim is weaker than the benchmark-audit and baseline findings. And the captioner shares Qwen2.5-Omni as a base model with the video model, which is worth noting even though the caption is question-independent and not socially tuned. That said, the un-finetuned rows (caption 62.1 vs video 63.2) independently point in the same direction, so I don't think Finding 3 is an artifact—it's just under-specified.\n\nThis paper deserves peer review. The benchmark cleaning and the Vanilla SFT result are valuable on their own, and Finding 3, once the training condition is clarified, will be a useful reference point. I'd send it to referees with a specific request: state the exact training data for every row in Table 5 and add variance information if any exists. The authors are transparent about limitations and the core claims are likely to hold.","headline":"A genuinely useful benchmark audit and a convincing cheap-baseline result, but the caption-parity claim hinges on a training-condition ambiguity in Table 5 that the authors need to resolve before it is taken as established.","tokens_in":18995,"tokens_out":2809,"would_cite":true,"duration_ms":29604,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chain-of-thought reasoning for social audio-visual question answering is expensive and ineffective: a plain direct-answer fine-tuned baseline matches or beats reasoning models, and a text caption alone performs as well as the full video…","keywords":["social audio-visual question answering","chain-of-thought reasoning","multimodal large language models","supervised fine-tuning","benchmark curation","IntentBench-Prime","text-only priors","caption-based video QA"],"falsifier":"Retrain the video-based Vanilla SFT model on exactly the Social-IQ 2.0 + EMER subset used for Caption SFT and Question SFT, with identical hyperparameters and training steps, and compare its IntentBench-Prime (Hard) accuracy to Caption SFT's 68.6%. If the video model beats the caption model by a clear margin (say, more than 3 points) under this matched-data condition, then the paper's claim that a generic caption is on par with the full multimodal input would be refuted; if the gap stays within one point, the claim holds. The retrained model should also be re-evaluated on WorldSense and Daily-Omni to confirm that the restricted data, not the video modality, is what drives the earlier baseline gains.","tokens_in":17850,"feed_emoji":"🎬","tokens_out":13952,"duration_ms":120261,"temperature":0.7,"pith_summary":"This paper claims that the field's emphasis on chain-of-thought reasoning for social audio-visual question answering is misplaced. A minimal baseline, Vanilla SFT—standard supervised fine-tuning that directly outputs the answer with no reasoning trace—matches or outperforms existing reasoning-based systems on IntentBench-Prime, WorldSense, and Daily-Omni, while training and inference are far cheaper. The paper also shows that the IntentBench benchmark itself is noisy: about 7% of its questions are broken and about 23% can be answered from text alone; the cleaned re-release, IntentBench-Prime, removes both classes. Finally, replacing the video and audio with a generic, question-independent caption yields accuracy essentially equal to the full multimodal input, and a question-only model still captures most of the gain. The practical upshot is that direct-answer SFT should be the mandatory baseline for any new method in this area, and that current omni-models are not yet extracting much question-specific social information from video.","feed_headline":"Plain fine-tuning beats reasoning models on social video QA","feed_subtitle":"A direct-answer baseline matches or beats reasoning models on three benchmarks. A plain caption matches the full video.","key_machinery":"The central instrument is the Vanilla SFT baseline: supervised fine-tuning (with LoRA on the language module) of Qwen2.5-Omni so that, given the video, audio, and question, it emits only the answer letter(s), with no chain-of-thought trace. It serves as the minimal control against which all CoT methods are judged. Around it sit two controlled ablations trained in the same way but with reduced input: Question SFT, which sees only the question and answer options, and Caption SFT, which sees a generic question-independent video caption generated by ASID-Captioner instead of the audiovisual stream. Comparing the three isolates three contributions: the benchmark's learnable language priors, the information contained in a condensed textual scene description, and whatever extra the raw video and audio provide. This same three-way design produces the paper's deployment proposal: precompute captions once per video, then answer many queries from text at low latency.","core_discovery":"The paper establishes three findings. Finding 1: IntentBench, the standard benchmark, is contaminated—four LLMs unanimously answer 22.9% of its Social-IQ 2.0 questions from the question and answer text alone, and manual ranking removes 7.1% of questions as broken; the released IntentBench-Prime excludes both, retaining a Hard split of 1,899 questions. Finding 2: Vanilla SFT, a LoRA fine-tune of Qwen2.5-Omni trained on the same data and protocol as HumanOmniV2 but with direct answers, achieves 70.4% accuracy on IntentBench-Prime Hard, 65.2% on Daily-Omni, and 48.8% on WorldSense, matching or beating HumanOmniV2, AVATAR, and AffectOmni in most columns, while using roughly 11x less training compute and 356x less decoding time. Finding 3: the modality ablations show that a question-only SFT model reaches 62.0% on IntentBench-Prime Hard, and a caption-only SFT model reaches 68.6%—within a point of the 69.5% video-based Vanilla SFT—so language priors and a generic caption cover nearly all of the benchmark performance. The authors argue that reasoning traces are therefore not the bottleneck, that new methods must be measured against direct-answer SFT, and that the bottleneck is the model's ability to extract question-specific information from the video and audio.","pith_inferences":["If the caption-parity result holds beyond this exact setup, it suggests the bottleneck is the interface between the video encoder and the language model: the visual stream is currently being reduced to scene-level features that carry little question-relevant social detail. A direct test would be to feed question-conditioned or socially annotated captions (gaze, gesture, prosody) and see whether ca","The 22.9% text-answerability estimate is conservative because it requires four LLMs to agree unanimously; a single strong LLM would likely flag a higher fraction as text-answerable, which would further shrink the 'Hard' split and raise the bar for what counts as a multimodal question.","Because ASID-Captioner and the QA models share the same base model, part of the caption advantage may come from shared tokenization and format alignment rather than from caption content; swapping in a captioner from a different model family would isolate that effect.","The paper measures accuracy and latency, but not the explanatory value of the reasoning trace; a complementary study would ask whether the 356x decode-time saving is worth giving up the human-inspectable rationales that CoT models produce."],"forward_implications":["New training methods for social AV-QA should be reported together with a direct-answer SFT baseline trained on the same data; otherwise accuracy gains cannot be attributed to the method rather than to plain fine-tuning.","Researchers can re-score their existing IntentBench results against IntentBench-Prime by applying the released exclusion list, avoiding new evaluation runs and making the cleaned benchmark immediately usable.","Benchmark builders should screen questions for text-only answerability and broken options before release; the paper's 22.9% text-answerable and 7.1% broken rates show how large the contamination can be.","A two-stage caption-then-answer architecture becomes a practical alternative: the caption is computed once per video, so query-time latency drops dramatically and one caption can serve many questions.","The results imply that progress in social AV-QA will come less from better reasoning schedules and more from models that can pull question-specific cues out of the video and audio streams."],"supporting_citations":[{"why":"Introduces HumanOmniV2 and the IntentBench benchmark; supplies the reasoning method, the training data protocol, and the primary in-domain comparison numbers.","marker":"[30]"},{"why":"Provides Qwen2.5-Omni, the base model for Vanilla SFT, Caption SFT, and Question SFT.","marker":"[29]"},{"why":"Defines WorldSense, the out-of-domain benchmark used to test whether direct-answer SFT generalizes beyond social training data.","marker":"[6]"},{"why":"Defines Daily-Omni, the second out-of-domain benchmark, including its context-understanding and reasoning subcategories.","marker":"[37]"},{"why":"Reports AffectOmni, a CoT reasoning system that Vanilla SFT matches or outperforms on average across the benchmarks.","marker":"[25]"},{"why":"Reports AVATAR, a CoT reasoning method used as an additional comparison point for the baseline.","marker":"[8]"},{"why":"Supplies ASID-Captioner, the model that generates the generic question-independent captions for the caption-vs-video comparison.","marker":"[12]"},{"why":"Social-IQ 2.0 is the source of most IntentBench questions and one of the two datasets used to train the Section 5 SFT variants.","marker":"[27]"},{"why":"EMER provides the emotion-recognition category of IntentBench and is the second dataset in the restricted training set.","marker":"[13]"},{"why":"Introduces GRPO, the reinforcement-learning optimizer that the compared reasoning models use and that Vanilla SFT deliberately omits.","marker":"[24]"}],"fun_headline_variants":["Vanilla SFT beats reasoning models on social video QA","Direct answers match or beat reasoning in social video QA","Language priors dominate social audio-visual QA","Simple SFT tops chain-of-thought on social video QA","Reasoning models fall short on social video QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The caption-vs-video comparison assumes that the video-based Vanilla SFT model in that experiment was trained on the same restricted dataset (Social-IQ 2.0 plus EMER) as the caption and question models; if it was instead the main Section 4 model trained on the full 20K-video set, the apparent parity is confounded by training-data size.","fun_headline_variants_meta":{"raw":{"variants":["Vanilla SFT beats reasoning models on social video QA","Direct answers match or beat reasoning in social video QA","Language priors dominate social audio-visual QA","Simple SFT tops chain-of-thought on social video QA","Reasoning models fall short on social video QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000777,"raw_usage":{"total_tokens":3493,"prompt_tokens":1059,"completion_tokens":2434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":2357}},"tokens_in":675,"tokens_out":2434,"duration_ms":16410,"temperature":1.0,"reasoning_tokens":2357,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:35:26.137651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the video-based Vanilla SFT model on exactly the Social-IQ 2.0 + EMER subset used for Caption SFT and Question SFT, with identical hyperparameters and training steps, and compare its IntentBench-Prime (Hard) accuracy to Caption SFT's 68.6%. If the video model beats the caption model by a clear margin (say, more than 3 points) under this matched-data condition, then the paper's claim that a generic caption is on par with the full multimodal input would be refuted; if the gap stays within one point, the claim holds. The retrained model should also be re-evaluated on WorldSense and Daily-Omni to confirm that the restricted data, not the video modality, is what drives the earlier baseline gains.","supporting_citations":[{"cited_title":"In: The Fourteenth Inter- national Conference on Learning Representations (2026),https://openreview","cited_arxiv_id":null,"evidence_quote":"Defines WorldSense, the out-of-domain benchmark used to test whether direct-answer SFT generalizes beyond social training data."},{"cited_title":"IEEE Transactions on Affective Computing pp","cited_arxiv_id":null,"evidence_quote":"Reports AffectOmni, a CoT reasoning system that Vanilla SFT matches or outperforms on average across the benchmarks."},{"cited_title":"In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026)","cited_arxiv_id":null,"evidence_quote":"Reports AVATAR, a CoT reasoning method used as an additional comparison point for the baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Social-IQ 2.0 is the source of most IntentBench questions and one of the two datasets used to train the Section 5 SFT variants."}],"review_version":1}