{"id":"c6608d23-f75d-4d4e-84c0-ea438a0f25cb","arxiv_id":"2607.06618","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DA-MIVQA is a new medical video QA benchmark that labels questions by evidence type, but the paper's own results omit the crucial difficulty-split analysis.","lead":"This paper describes DA-MIVQA, a competition benchmark for question-answering on medical instructional videos that tags every question as 'simple' or 'complex' depending on whether it can be answered from subtitles alone. It aims to expose whether AI systems actually watch the video or just read the subtitles, but the reported results do not include the simple/complex breakdown that would prove this.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The simple/complex distinction is unvalidated and the reported text-only vs multimodal gap is too small to support the claim that DA-MIVQA isolates visual/procedural understanding.","rationale":"The reader identifies the weakest assumption as the validity of simple/complex labels, and I agree. The central claim—that DA-MIVQA can distinguish subtitle-matching from genuine visual/procedural understanding—depends entirely on the complex subset being visually indispensable. The manuscript itself states that 'the expected comparison between Simple Only and Complex Only results can help reveal whether a model mainly relies on subtitle matching or can truly integrate textual, visual, and procedural evidence' (Section 4.5), yet no such comparison is reported. The small text-only vs. multimodal gap across all three tracks provides direct, if preliminary, evidence against the intended distinction. This stress-test does not change the reader's CONDITIONAL verdict; it sharpens the condition under which the central claim would be accepted: the organizers must demonstrate label reliability and show that complex questions are not answerable from subtitles alone. The copy-paste participation error (Section 4.1) is secondary but reinforces the need for careful release of data and results.","tokens_in":12326,"tokens_out":2659,"duration_ms":24854,"concrete_test":"Release per-difficulty (simple-only and complex-only) results for the text-only oracle and multimodal baseline on all three tracks, and compute inter-annotator agreement (e.g., Cohen's kappa) on a held-out label set. Additionally, run a no-subtitle control: mask or remove SRT subtitles and re-run the text-only oracle and multimodal baseline on the complex subset. If the text-only oracle's complex-subset performance remains close to the multimodal baseline (within the observed 1–2 point overall gap), or if removing subtitles does not cause a large drop on complex questions, the difficulty labels do not operationalize the intended visual-evidence requirement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that DA-MIVQA distinguishes systems that rely on subtitle matching from systems that genuinely integrate visual and procedural evidence. This requires that the 'complex' subset consist of questions that cannot be answered from subtitles alone. The paper provides no inter-annotator agreement for the simple/complex labels (Section 2.3), no human validation that complex questions require visual evidence, and—despite promising per-difficulty reporting in Sections 2.2 and 4.5—no simple-only or complex-only score tables in the results (Sections 4.2–4.5). The only reported evidence relevant to the distinction undercuts it: in Track 1, the text-only oracle reaches mIoU 0.2925 vs. 0.3068 for the multimodal baseline; in Track 3, 0.2051 vs. 0.2228 Average. If the complex subset truly required visual grounding, a subtitle-only oracle should collapse on that subset relative to a multimodal system; a 1–2 point overall gap is consistent with the alternative explanation that most 'complex' questions are also answerable from subtitles, or that the baseline's gains arise from representation learning rather than from satisfying an actual visual-evidence requirement. Section 4.5 describes the expected simple/complex comparison as if it existed, but the results are absent. Thus the dataset's defining property—the validity of difficulty labels as a proxy for evidence modality—is unsupported as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is the overview of the NLPCC 2026 Shared Task 1, DA-MIVQA, a difficulty-aware multilingual and multimodal medical instructional video question answering benchmark. The task introduces a binary simple/complex difficulty label for each question, where simple questions are claimed to be answerable from subtitle-level text and complex questions are claimed to require visual grounding, procedural understanding, and cross-modal evidence integration. The benchmark contains three tracks (temporal answer grounding in a single video, video corpus retrieval, and joint retrieval+grounding). The paper describes dataset construction, evaluation metrics, a text-only oracle baseline based on Qwen3.5-9B, a multimodal encoder-decoder baseline, and the official results from participating teams. The central claim is that DA-MIVQA provides a benchmark that can distinguish systems that rely on subtitle matching from those that genuinely integrate visual and procedural evidence.","tokens_in":12625,"tokens_out":2776,"duration_ms":27948,"significance":"If the difficulty labels are valid and the evaluation is shown to separate systems on the complex subset, this benchmark would be a useful resource for the medical video QA community: it extends an established line of NLPCC shared tasks, provides multilingual data across three tasks, and ships baseline implementations. The participation and results tables also provide a snapshot of current system performance. However, the paper's central premise is currently not demonstrated. The dataset is a shared-task artifact that is likely to be reused, and the paper does make concrete falsifiable predictions (e.g., that multimodal systems should outperform text-only systems more strongly on complex questions). The strengths are the public dataset and the three-track evaluation protocol. The weakness is that the load-bearing difficulty annotation is not validated and the results that would validate it are not reported.","major_comments":[{"comment":"The simple/complex difficulty labels are the defining contribution of DA-MIVQA, yet the paper reports no inter-annotator agreement (e.g., Cohen's kappa), no annotation guideline details, and no human validation that complex questions actually require visual evidence. Without such a validation, the difficulty labels are only the authors' assertion. This is load-bearing because every downstream claim about 'difficulty-aware' evaluation rests on these labels. Please add agreement statistics and a description of the annotation protocol, or explicitly release the labels for external audit.","section":"Section 2.3, Table 2"},{"comment":"Section 2.2 states that results will be reported on simple-only, complex-only, and mixed subsets. The results section, however, reports only aggregate scores in Tables 4-6; Section 4.5 describes an 'expected comparison' between Simple Only and Complex Only results but never presents the actual numbers. The absence of per-difficulty results is a direct gap: the paper's central claim is that the benchmark reveals differences in visual grounding, and that claim cannot be evaluated without a simple/complex breakdown. Please include per-difficulty performance for all baselines and all participating teams in each track.","section":"Section 2.2 vs. Sections 4.2-4.4, Tables 4-6"},{"comment":"The evidence presented undercuts the central premise. In Track 1, the text-only oracle baseline achieves mIoU 0.2925 vs. 0.3068 for the multimodal baseline; in Track 3, Average 0.2051 vs. 0.2228. These gaps (0.014 and 0.018) are small and may be within run-to-run variation or attributable to representation learning rather than to a genuine requirement for visual evidence in 'complex' questions. If the complex subset truly required visual grounding, a subtitle-only oracle should show a large degradation on that subset relative to a multimodal system. Without the per-difficulty breakdown requested above, the reported aggregate comparison is consistent with the alternative explanation that most 'complex' questions can also be answered from subtitles. Please report the simple/complex split for these baselines and include confidence intervals or significance tests.","section":"Sections 3.1-3.2, Tables 4 and 6"},{"comment":"The participation paragraph says 'For NLPCC2025 Shared Task 4' when describing the current challenge, which is NLPCC 2026 Shared Task 1. This is more than a typo: it obscures the relationship to prior shared tasks. Please correct the wording and clarify the lineage (CMIVQA 2023, MMIVQA 2024, M4IVQA 2025, DA-MIVQA 2026).","section":"Section 4.1, lines 10-12"}],"minor_comments":[{"comment":"Figure 1 is captioned as a dataset example of the M4IVQA shared task, not DA-MIVQA. Either update the figure to DA-MIVQA examples or revise the caption to indicate that it illustrates the lineage.","section":"Figure 1"},{"comment":"The text says the dataset is 'expanded through difficulty-aware re-annotation and an increased number of question-answer pairs,' but no comparison with the sizes of CMIVQA/MMIVQA/M4IVQA is given. Add a sentence comparing dataset sizes to justify 'expanded.'","section":"Section 2.3"},{"comment":"Tracks 2 and 3 have identical sample counts. If this is intentional (same corpus, different task format), state so explicitly; otherwise the table appears suspicious.","section":"Table 2"},{"comment":"References [10] and [31] are the same paper (NLPCC 2023 shared task overview). Please merge or cite once to avoid duplication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a shared task that is clearly useful to the community, but the central scientific claim—that the difficulty labels capture whether a question requires visual grounding—is not supported by the reported evidence. The missing per-difficulty results are the key issue. This is fixable in revision if the organizers still hold the submission outputs and can recompute simple/complex splits. I would also encourage the authors to release the difficulty annotations and any agreement statistics; without them, the benchmark's defining property cannot be independently assessed. The small text-only vs. multimodal gap suggests the authors should be prepared for the possibility that the complex subset is not as visually demanding as advertised; if that is the case, the paper should be revised to make a more modest claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely new benchmark resource—difficulty-aware medical instructional video QA with three tracks and a multilingual split—and the paper gives a clear overview of how it was built and what happened in the shared task. That part is solid. The authors also deserve credit for including a text-only oracle baseline and reporting its performance alongside the multimodal baseline; that is the right way to probe whether visual grounding is needed.\n\nThe problem is the paper's central claim: that DA-MIVQA can separate systems that match subtitles from systems that actually understand the video. That claim is asserted, not shown. The difficulty labels are the authors' own annotations with no inter-annotator agreement, no label-validation experiment, and—despite promises in Sections 2.2 and 4.5—no simple-only vs complex-only results anywhere in the paper. The only relevant numbers are the overall text-only vs multimodal gaps, and they are small: 0.2925 vs 0.3068 mIoU in Track 1, 0.1418 vs 0.1779 in Track 2, 0.2051 vs 0.2228 in Track 3. If complex questions truly required visual grounding, you would expect a larger collapse for the subtitle-only baseline. The small gap is consistent with the alternative: that most 'complex' questions can be answered from subtitles too, or that the multimodal baseline gains come from representation learning rather than satisfying a real visual-evidence requirement.\n\nThere is also a copy-paste error in Section 4.1: it refers to 'NLPCC2025 Shared Task 4' and names winners (HoMaMaOvO, WuKong, Chiikawa) that do not match the tables (Amazon Inc, Team_Wukong, BIGC). That is careless in a camera-ready overview.\n\nSo: the benchmark and the shared task are worthwhile, and the paper should get a serious referee. But the evaluation framework as described does not yet establish what the conclusion claims. The authors should release the difficulty labels, report per-difficulty results, and validate the labels with annotators. If they do that, the contribution is real and useful for the medical video QA community.","headline":"New difficulty-aware medical video QA benchmark, but the difficulty labels are unvalidated and the baseline gap is too small to support the paper's main claim.","tokens_in":13105,"tokens_out":3325,"would_cite":true,"duration_ms":26123,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that labeling medical instructional video questions as simple or complex—depending on whether subtitle text suffices—lets a benchmark expose systems that match subtitles without truly understanding the video.","keywords":["difficulty-aware evaluation","medical instructional video QA","temporal answer grounding","video corpus retrieval","multimodal reasoning","multilingual medical QA","evidence types","shared task"],"falsifier":"Take a random sample of test questions, have several independent annotators re-label each as simple or complex, and measure inter-annotator agreement; then report simple-only and complex-only scores for a text-only system and a multimodal system. If annotators disagree substantially, or if complex questions are answered almost as well from subtitles alone, the difficulty-aware distinction does not hold.","tokens_in":12214,"feed_emoji":"🩺","tokens_out":7117,"duration_ms":68721,"temperature":0.7,"pith_summary":"This paper presents a shared task whose central move is to label every medical-video question as simple or complex based on the evidence needed to answer it. Simple questions can be answered from subtitle-aligned text; complex questions, the authors claim, force a system to use visual grounding, action understanding, and procedural reasoning. Three tracks test temporal grounding in a single video, retrieval from a video corpus, and the two combined, with scores reported separately for simple, complex, and mixed question sets. If the labels are valid, the benchmark offers a practical way to distinguish systems that read subtitles from systems that actually watch the medical procedure. This matters because medical instruction videos are used for first aid, nursing, rehabilitation, and emergency guidance, where text-only shortcuts can be misleading.","feed_headline":"Splits medical video QA questions by evidence needed","feed_subtitle":"Simple ones answer from subtitles; complex ones need visual grounding—so the gap reveals who truly watches.","key_machinery":"The load-bearing mechanism is the simple/complex evidence label: each question is annotated with a difficulty tag stating whether subtitle text alone can answer it. The tag drives all difficulty-aware analyses. The supporting machinery is the three-track evaluation protocol—temporal answer grounding in one video, video corpus retrieval, and retrieval with temporal grounding—scored with IoU/mIoU and R@n/MRR metrics and reported separately for simple, complex, and mixed sets. A text-only oracle baseline and a multimodal encoder–decoder baseline bracket how far subtitle-only reasoning can go and how much visual modeling adds.","core_discovery":"The paper's central claim is that difficulty-aware annotation—marking each question by the type of evidence required—turns a medical instructional video question-answering benchmark into an instrument for separating subtitle matching from genuine multimodal understanding. It defines simple questions as those where subtitle-aligned textual cues are sufficient, and complex questions as those requiring visual grounding, procedural interpretation, and cross-modal integration. The benchmark is organized into three tracks: single-video temporal answer grounding, video corpus retrieval, and corpus-level retrieval plus grounding, each evaluated on simple, complex, and mixed subsets. The reported res","pith_inferences":["The reported aggregate results show only a narrow gap between the text-only oracle and the multimodal baseline (roughly 1–2 points in the main metrics); if per-difficulty scores were broken out, many 'complex' questions might turn out to be answerable from subtitles, which would weaken the labels' intended distinction. This is an interpretation the paper does not address.","Because the paper reports no inter-annotator agreement or label-validation experiment, a cheap but decisive check on the whole framework is to re-label a sample of questions with multiple annotators and measure agreement.","If the labels are validated, they could guide data generation: writing more complex questions that genuinely resist subtitle matching would sharpen the benchmark and push models toward true visual grounding.","The simple/complex labels could also be used as a training signal, for example by upweighting complex questions or adding auxiliary objectives that require visual evidence."],"forward_implications":["If the difficulty labels hold, the simple-versus-complex gap becomes a diagnostic: a small gap signals robust multimodal understanding, while a large drop reveals dependence on subtitle cues.","Systems can be ranked not only on overall accuracy but on whether they remain accurate on questions that require visual and procedural evidence.","The corpus-level track exposes error accumulation: a system may fail by retrieving the wrong video, by grounding in the right video, or both, and the metrics separate those failure modes.","Because scores are reported in Chinese and English, the benchmark allows direct comparison of difficulty-aware behavior across languages.","The protocol is reusable: the same simple/complex partition and metric set can be applied to future multimodal reasoning benchmarks."],"fun_headline_variants":["Difficulty-aware QA splits simple vs complex medical video questions","Benchmark separates subtitle matching from real video understanding","Medical video QA: simple from subtitles, complex from visual grounding","New task grades medical video questions by evidence required","Multimodal medical video QA ranked by reasoning depth"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole evaluation rests on the assumption that annotators can reliably tell which questions truly require visual evidence; the paper asserts this distinction but gives no agreement or validation data for the labels.","fun_headline_variants_meta":{"raw":{"variants":["Difficulty-aware QA splits simple vs complex medical video questions","Benchmark separates subtitle matching from real video understanding","Medical video QA: simple from subtitles, complex from visual grounding","New task grades medical video questions by evidence required","Multimodal medical video QA ranked by reasoning depth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1002,"prompt_tokens":744,"completion_tokens":258,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":195}},"tokens_in":488,"tokens_out":258,"duration_ms":2614,"temperature":1.0,"reasoning_tokens":195,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:18:17.659256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of test questions, have several independent annotators re-label each as simple or complex, and measure inter-annotator agreement; then report simple-only and complex-only scores for a text-only system and a multimodal system. If annotators disagree substantially, or if complex questions are answered almost as well from subtitles alone, the difficulty-aware distinction does not hold.","supporting_citations":[],"review_version":2}