{"id":"92989cf2-0424-4112-a769-758ccb9de9c2","arxiv_id":"2412.17022","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper introduces FriendsQA, a 44.6K-question video QA dataset for long-form story understanding, generated by an LLM multi-agent pipeline with balanced fine-grained topics and evaluated on 10 models.","lead":"FriendsQA is a new video question answering benchmark built from 234 episodes of Friends, with 44,600 questions spread evenly across 14 story-topic categories. It is generated automatically by a multi-agent LLM pipeline and used to score 10 video QA models, which all perform well below the level needed for deep story understanding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quality claim rests on 4,790 manually checked first-season questions; the remaining ~39.8K are filtered only by two LLM reviewers using the same script-based text the generator saw, so full-corpus and video-grounded validity are not established.","rationale":"The reader identified the core load-bearing assumption: the quality of the full 44.6K-question dataset is inferred from manual verification of only the first season, with the remaining questions vetted solely by two LLM reviewers. I agree that this is the primary vulnerability. I add a second, related dimension: the generator, the reviewers, and even the manual 'revision' all operate on the same script-based representation, not on the raw video. This means the audit validates textual answerability more than visual answerability, which matters for a benchmark whose central construct is deep video understanding. I do not think this requires changing the reader's CONDITIONAL verdict, because both concerns can be settled by the same concrete audit: a stratified human check on the video itself. The proposed check also tests the fine-grained topic labels, which are self-assigned by the generator and are not validated by the paper's existing manual pass. If the audit passes, the benchmark can be used provisionally; if it fails, the released resource needs cleaning or the claims must be restricted to a verified subset.","tokens_in":24064,"tokens_out":6026,"duration_ms":62821,"concrete_test":"Sample about 1,500 questions stratified by all 10 seasons, by the 14 fine-grained topics, and by single-episode versus cross-episode type. Have independent annotators watch the corresponding video segments (not read the script) and check: (a) the question is answerable from the video alone; (b) exactly one choice is correct; (c) the provided ground truth is that choice; and (d) the assigned perception/inference attribution and topic label are correct. Report direct-retention and revised-retention rates with 95% confidence intervals per stratum. If the overall error rate is statistically indistinguishable from the first-season rates in Table 2 (within about 2-3%) and topic-label accuracy is high, the full-dataset claim is supported; otherwise the authors should release a cleaned version or restrict the benchmark claim to a verified subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that FriendsQA is a large-scale, high-quality DVU benchmark. The only human-quality evidence is the first-season subset FriendsQA-S1 (4,790 of 44.6K questions, about 10.7%, reported in Table 2). For the remaining questions across nine later seasons, acceptance depends entirely on two LLM reviewers, Gemini 1.5 Pro and Claude 3.5 Sonnet, who are given the same textual 'video information' (script plus bounding boxes) as the generator; they cannot catch errors that require visual verification, and their judgments are correlated with the generator. The S1 retention rates (89.85-94.44%) already imply that roughly 5-10% of automatically generated questions fail or require revision; if a similar or worse rate holds in later seasons or in the harder cross-episode split, several thousand questions have wrong ground truth or ambiguous options, which would bias per-topic accuracy, model rankings, and the difficulty analysis. Moreover, the manual revision in Table 2 also used the script-based video information rather than the raw video, so it validates answerability from the transcript, not necessarily from the visual content. A human baseline is absent, so 'models fall short on deep video understanding' cannot be cleanly separated from 'questions are not visually answerable.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FriendsQA, a large-scale deep video understanding dataset built from 234 episodes of the sitcom Friends. The authors propose StoryMind, a multi-agent LLM framework in which a Gemini 1.5 Pro generator produces multiple-choice questions conditioned on scripts enriched with detected character bounding boxes and timestamps, and two LLM reviewers (Gemini 1.5 Pro and Claude 3.5 Sonnet) filter questions for relevance and correctness. The dataset contains 44,600 questions, including 9,470 cross-episode questions, balanced across 14 fine-grained topics formed by crossing seven story topics (C, A, L, CA, CL, AL, CAL) with two attributions (perception, inference). A difficulty score is defined from length and content ratios, and questions are bucketed into easy, medium, and hard levels. Ten state-of-the-art VideoQA models are evaluated zero-shot, with reported accuracy substantially lower on FriendsQA than on factoid benchmarks (e.g., VideoChat2 44.05% vs. 61.70% on NExT-QA). The paper also reports manual verification of 4,790 first-season questions and compares model accuracy on that verified subset with the automatically filtered version.","tokens_in":24386,"tokens_out":4646,"duration_ms":42940,"significance":"If the quality of the full dataset holds, FriendsQA would be a valuable community resource: it is the first DVU dataset with deliberate fine-grained topic balance (Gini 0.927, entropy 3.794), it combines long videos with a large question count (98,874.8 K question-seconds), and it includes cross-episode questions and a per-question difficulty annotation. The StoryMind pipeline, with explicit topic feedback and dual-LLM review, is a practical recipe for lowering annotation cost, and the paper ships code and dataset links. The S1 manual check (94.44% single / 90.95% cross retention after revision) is a useful sanity check. However, the significance is conditional: full-corpus quality and visual answerability are not established, and the absence of a human baseline limits the interpretability of the reported model gaps. The dataset is likely to be useful for benchmarking long-form video QA, but the paper's central 'high-quality benchmark' claim needs stronger evidence.","major_comments":[{"comment":"The manual verification covers only FriendsQA-S1 (4,790 questions, about 10.7% of the 44.6K pool) and is used to extrapolate retention rates to all ten seasons. Since the first-season retention after revision is 94.44% for single-episode and 90.95% for cross-episode questions, a comparable error rate on the remaining roughly 39.8K questions implies thousands of questions with wrong, ambiguous, or non-unique ground truth. The paper should either manually audit a stratified sample across all seasons and both question types, or prominently restrict the quality claim to FriendsQA-S1 and provide error-rate bounds for the full set. Without this, the per-topic accuracies in Table 4 and the difficulty analysis in Figure 11 are not trustworthy as benchmark numbers.","section":"Dataset Quality (Table 2)"},{"comment":"Both reviewers receive exactly the same 'video information' as the generator: script text with inserted character bounding boxes and timestamps. They therefore cannot catch errors that require visual verification (e.g., whether a character is actually on screen, whether an action is visible), and their judgments are correlated with the generator since they read the same source. The manual revision in Table 2 also uses the script-based video information rather than the raw video. Consequently, the reported quality reflects answerability from transcripts, not necessarily visual answerability. The claim that VideoQA models 'fall short' on deep video understanding requires a human baseline on the same video-question pairs, or at minimum a visual-verification study on a sample. As written, the performance gap could partly reflect questions that are not visually grounded.","section":"Question Filtration (Figure 6, Appendix B.2)"},{"comment":"The difficulty score sigma_i = sigma_l / mu_l + sigma_c / mu_c is a heuristic ratio of whole-video to relevant-video length and instance counts, with no justification for the additive form or for the 9:3:1 bucketing into easy, medium, and hard levels. Figure 10(b) shows a downward trend in average accuracy as difficulty increases, but this is aggregated and does not establish that the measure is calibrated for individual questions; SeViLA shows no difficulty effect. The content factor counts characters and locations from the video information, not semantic content, so it may conflate long-range reasoning with simple sparsity. The authors should validate the measure with item-level analysis (e.g., accuracy vs. difficulty within topic and attribution, comparison with an LLM-based difficulty rating, or human difficulty judgments) before using it to draw conclusions such as 'P questions are harder than I questions'.","section":"Difficulty Measure (Eq. 1, Figure 10)"},{"comment":"All 10 models are evaluated zero-shot with official defaults, and no human or fine-tuned upper bound is reported. Table 4 therefore shows relative ordering among zero-shot models, but the headline claim that state-of-the-art models 'fall far short' on FriendsQA lacks a reference point. A human-accuracy estimate on a sample, using the same multiple-choice interface, would let the reader separate model limitations from dataset artifacts. This is particularly important given the reviewer-groundedness issue above: without a human baseline, the gap between 61.70% on NExT-QA and 44.05% on FriendsQA cannot be cleanly attributed to the dataset's deep-understanding requirements.","section":"Evaluation Setting"}],"minor_comments":[{"comment":"There are typographical errors: 'Manul' should be 'Manual' in the Table 2 header, and 'FirendsQA-M' should be 'FriendsQA-M' in the paragraph after Table 3.","section":"Table 2 and main text"},{"comment":"The fine-grained topic distribution figure appears to contain corrupted glyphs in the provided manuscript; the figure should be replaced with a legible rendering so that the balanced distribution claim can be verified visually.","section":"Dataset Statistics (Figure 8)"},{"comment":"The LLM-based categorization of existing datasets into 14 topics is reported without a human-agreement check; a small human annotation study on the categorization prompt would strengthen the comparison in Figure 2.","section":"Figure 2 and Appendix A"},{"comment":"The threshold used for balanced topic coverage is not stated; the paper should specify the target number of questions per fine-grained topic and how the 'same threshold' is set for single-episode and cross-episode questions.","section":"Question Generation (Figure 5)"},{"comment":"The claim that P questions have higher overall difficulty than I questions is based on averaged difficulty factors without confidence intervals or significance testing; adding variability estimates would make the comparison more convincing.","section":"Evaluation Result (Figure 10a)"},{"comment":"The choice of concatenating 4 consecutive episodes for cross-episode questions is attributed to the generator's context limit; this is an architectural constraint rather than a story-structure choice and should be acknowledged as a limitation in the cross-episode design.","section":"Data Preparation"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a vision or multimedia venue, and the dataset could be a useful community resource. The main risk is that the full 44.6K dataset will be released without a human audit, and downstream users will treat the LLM-filtered questions as gold-standard annotations. I would ask for a stratified human audit (or explicitly restricted quality claims) and a human baseline before publication. The name conflict with the existing transcript-based FriendsQA dataset (Yang and Choi 2019) is also worth clarifying in the novelty discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Friend, you asked about FriendsQA. It's a serious dataset paper with a real asset: a 44.6K-question benchmark for long-form story video QA, with a deliberately balanced spread across 14 fine-grained topic categories (character/action/location combinations × perception/inference) and a cross-episode split that requires synthesizing four episodes. That's genuinely new; prior DVU datasets are either small (HLVU, DeepMaven) or short-video (TVQA, MovieQA). The StoryMind pipeline — one generator, two LLM reviewers, iterative balancing, difficulty scoring — is reusable, and they ship code and data.\n\nThe good news: the authors did manually verify all 4,790 first-season questions, and the numbers (94.44% single, 90.95% cross retained after revision) are credible. The average model accuracy difference between the raw automatic set and the manually corrected set is 0.15–0.25%, which is a reasonable sanity check. The zero-shot evaluation of 10 models also shows the expected monotonic difficulty trend for 9 of 10, so the difficulty heuristic has at least some behavioral validity.\n\nThe soft spot is the one you and the stress-test flagged: only ~10.7% of the 44.6K questions get human eyes. Everything else passes through two LLM reviewers that are fed exactly the same textual 'video information' (script + character bounding boxes) as the generator. They can catch some errors, but they cannot detect questions that aren't visually answerable from the actual video. And the manual revision itself used the same script-based info, so even the verified questions are validated against the transcript, not the pixels. Without a human baseline on the full set, the headline 'models fall short on deep video understanding' is confounded with 'some questions may not be answerable from video alone.' That's a real limitation, but it's not fatal — it just means the benchmark should be presented as a provisional resource, not a settled ground truth.\n\nMinor quibbles: the difficulty ratio 9:3:1 is arbitrary, and the cross-episode questions are generated from concatenated scripts, which may over-rely on dialogue. Neither undercuts the core resource.\n\nWho should read this: anyone building or evaluating long-form video QA models. I'd accept it for peer review and push for a statistically sound audit plan (or full manual pass on a random sample), a human baseline, and a released commit hash with exact artifacts.","headline":"Useful new DVU benchmark, but the quality claim only holds for season 1; treat the rest as LLM-filtered until an audit shows otherwise.","tokens_in":24871,"tokens_out":2460,"would_cite":true,"duration_ms":23142,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces FriendsQA, a 44.6K-question benchmark for deep video understanding of story videos, and shows current video-QA models fall far short on it.","keywords":["deep video understanding","video question answering","story videos","fine-grained topics","multi-agent LLM","benchmark dataset","cross-episode reasoning","difficulty measurement"],"falsifier":"Sample, say, 300 questions at random from seasons 2-10, have a human verify that (a) the question is answerable from the episode and (b) exactly one choice is correct. If the human retention rate falls well below the 90-94% observed on season 1, the reported model accuracies would need to be re-estimated on a corrected subset.","tokens_in":23870,"feed_emoji":"🎬","tokens_out":6225,"duration_ms":47936,"temperature":0.7,"pith_summary":"This paper claims that the field's deep-video-understanding (DVU) benchmarks are too narrow to test story comprehension, and that a new dataset called FriendsQA fixes that. FriendsQA supplies 44.6K multiple-choice questions across 14 fine-grained story-topic categories, generated automatically by a multi-agent LLM pipeline called StoryMind and filtered by two independent LLM reviewers. The paper reports that ten current video-QA models score far lower on FriendsQA than on factoid benchmarks—for example, VideoChat2 drops from 61.70% on NExT-QA to 44.05%—suggesting the dataset measures something existing tests miss. If the dataset is as clean as claimed, it gives the community a balanced, per-topic diagnostic for story video understanding and a pipeline that can scale to other shows.","feed_headline":"Best video-AI model scores ~47% on new deep story benchmark","feed_subtitle":"A 44K-question benchmark reveals video-AI models miss long-range storylines in a favorite sitcom.","key_machinery":"The mechanism is StoryMind, a multi-agent pipeline that generates and filters the dataset. A generator LLM receives script, timeline, character detection bounding boxes (from a shot-based instance search) and writes questions with an assigned fine-grained topic; a feedback loop iterates until each of the 14 topics reaches the same quota, producing a balanced distribution. Two independent LLM reviewers then drop any question whose answer they do not both agree on and cannot verify against the script-video information. Each question also carries a difficulty score $\\sigma^i = \\sigma_l^i/\\mu_l + \\sigma_c^i/\\mu_c$, where $\\sigma_l^i$ is the ratio of whole-video length to relevant-video length and $\\sigma_c^i$ the ratio of whole-video to relevant-video instance count; this score is intended to let users separate 'hard' from 'easy' questions. The 14 fine-grained topics (C, A, L, CA, CL, AL, CAL times P and I) are the unit of analysis that lets the paper pinpoint where models fail, e.g., character recognition.","core_discovery":"The central claim is that FriendsQA is a large-scale, balanced benchmark for deep video understanding of story videos, and that current state-of-the-art models fail it. The dataset is derived from 234 episodes of Friends, averaging 1,358 seconds each, and contains 44.6K questions split evenly across 14 fine-grained topics—the seven story topics character (C), action (A), location (L), and their combinations, each posed in both perception (P) and inference (I) form. Roughly 9,470 questions span four consecutive episodes, requiring cross-episode reasoning. The paper's evaluation of ten VLM and MLLM models shows accuracy consistently near or below chance for many topic types, with character-related perception questions being the hardest; the best single-episode result is VideoLLaMA2 at 47.12%. The paper interprets these results as evidence that story-level understanding, not just factoid retrieval, is an open problem that FriendsQA exposes.","pith_inferences":["Because the generator and reviewers see the same script text, a model could in principle answer many questions from subtitles alone; an ablation that removes the video stream would reveal how much of the benchmark is genuinely visual, which the paper does not report.","The fact that inference questions score higher than perception questions for most models may reflect language priors in the LLM backbones rather than deep video reasoning; testing with shuffled answer orders could quantify that.","The paper's difficulty score assumes longer relevant spans make questions harder, but this is only validated indirectly; a direct human study comparing perceived difficulty with the score would strengthen the metric's use.","The same pipeline might be applied to non-English or lower-resource story videos, but that would require the character-detection and script-alignment stages to generalize beyond the sitcom's studio format."],"forward_implications":["If the benchmark is accepted, per-topic scores (e.g., P-C versus I-L) give model developers a diagnostic for exactly which story elements—characters, actions, locations, or their combinations—their system fails to model.","Cross-episode questions extend evaluation to long-range storyline reasoning beyond the typical single-scene clip, so progress on FriendsQA would signal genuine narrative understanding rather than clip-level pattern matching.","The difficulty measure separates easy, medium, and hard questions, allowing training and evaluation to target harder types of reasoning.","The automated StoryMind pipeline suggests that similar balanced, topic-tagged benchmarks can be produced for other story-rich media without the labor cost of handcrafted annotations.","The reported 44.05% accuracy of VideoChat2 (vs. 61.70% on NExT-QA) indicates that models tuned for factoid VideoQA do not transfer their performance to storyline understanding, setting a clear baseline for future work."],"supporting_citations":[{"why":"Provides the MovieQA baseline used in Table 1 for comparison of question scale and topic distribution.","marker":"(Tapaswi et al. 2016)"},{"why":"Supplies TVQA as the largest existing DVU dataset the paper compares against.","marker":"(Lei et al. 2018)"},{"why":"NExT-QA is the factoid benchmark used to demonstrate the DVU accuracy drop (61.70 to 44.05).","marker":"(Xiao et al. 2021)"},{"why":"HLVU defines the deep video understanding challenge and is a long-video baseline in the comparison.","marker":"(Curtis et al. 2020)"},{"why":"MovieChat-1K is the most recent long-video QA dataset used as comparison and a baseline model.","marker":"(Song et al. 2024)"},{"why":"Supplies the PAINS scripts with scene, dialogue, and timeline alignment that feed the question generator.","marker":"(Niu et al. 2023)"},{"why":"Provides the shot-based instance search method that detects character bounding boxes inserted into prompts.","marker":"(Li et al. 2023)"},{"why":"The earlier script-only FriendsQA dataset that this work distinguishes itself from by adding video information.","marker":"(Yang and Choi 2019)"}],"fun_headline_variants":["Video AI tops out at 47% on new story-heavy benchmark","FriendsQA benchmark: best video AI scores just 47%","New benchmark reveals video AI struggles with long storylines","Story video QA: top models hit only 47% on FriendsQA","Video AI models miss the plot in FriendsQA benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire 44.6K dataset is validated only by computer reviewers; humans checked just the first season's 4,790 questions, so the benchmark's trustworthiness rests on the assumption that the automatic filters stay just as accurate on the other 39,800+ questions.","fun_headline_variants_meta":{"raw":{"variants":["Video AI tops out at 47% on new story-heavy benchmark","FriendsQA benchmark: best video AI scores just 47%","New benchmark reveals video AI struggles with long storylines","Story video QA: top models hit only 47% on FriendsQA","Video AI models miss the plot in FriendsQA benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1662,"prompt_tokens":977,"completion_tokens":685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":600}},"tokens_in":593,"tokens_out":685,"duration_ms":6593,"temperature":1.0,"reasoning_tokens":600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:51:09.852758+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample, say, 300 questions at random from seasons 2-10, have a human verify that (a) the question is answerable from the episode and (b) exactly one choice is correct. If the human retention rate falls well below the 90-94% observed on season 1, the reported model accuracies would need to be re-estimated on a corrected subset.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MovieQA baseline used in Table 1 for comparison of question scale and topic distribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies TVQA as the largest existing DVU dataset the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HLVU defines the deep video understanding challenge and is a long-video baseline in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PAINS scripts with scene, dialogue, and timeline alignment that feed the question generator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier script-only FriendsQA dataset that this work distinguishes itself from by adding video information."}],"review_version":1}