{"id":"34d8fc10-92ee-474d-b319-cba9618dcc91","arxiv_id":"2505.06814","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"This paper presents a new Chinese-English medical instructional video QA benchmark with three tracks and reports the leaderboard of the NLPCC 2025 shared task.","lead":"The NLPCC 2025 M4IVQA shared task organized a competition for answering bilingual medical questions about instructional videos, with three tracks testing retrieval and temporal localization. This overview paper describes the dataset, metrics, and final scores of the ten participating teams.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Track 1's primary metric mIoU(R@1) is undefined as written: Eq. 1 appears to average IoU threshold values, and the reported scores do not match the table's R@1 columns, so the leaderboard's central column is not reproducible from the paper.","rationale":"This is a shared-task overview, so its scientific value rests on dataset clarity and evaluation reproducibility. The dataset construction, bilingual annotation effort, and knowledge-graph integration are genuine assets, and the paper is generally descriptive rather than overclaimed. However, the validity of the reported rankings depends on the metric definitions, and Track 1's primary metric is defined inconsistently. This is an internal inconsistency, not merely a disagreement with an external consensus: the equation says one thing, the table says another, and the reader cannot reconstruct the headline score. The translation-equivalence and Whisper-subtitle concern raised by the reader is also real, but it is harder to falsify from the paper alone and would mainly affect cross-lingual comparability rather than the central ranking construction. The metric issue can be settled directly by checking the official evaluation script against the published numbers. I therefore recommend retaining the CONDITIONAL verdict: before the overview is used as a reference benchmark, Eq. 1 and the R@1,IoU=µ notation should be corrected, and the official evaluation code should be released or pointed to explicitly.","tokens_in":8585,"tokens_out":6701,"duration_ms":68686,"concrete_test":"Obtain the official evaluation script and raw top-1 predictions for at least Baichuan and Random Pick Method from the challenge organizers, then recompute mIoU(R@1) two ways: (a) per-sample mean IoU over all test queries and (b) average of R@1 at IoU=0.3, 0.5, and 0.7. Compare both against Table 2. If (a) reproduces the table, then Eq. 1 is a typo and the leaderboard is probably sound; if (b) reproduces it, then the column is a recall average and should be renamed, and the Track 1 ranking should be reexamined under the intended metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weak point is Section 2.2's definition of the Track 1 primary metric. Equation (1) defines mIOU as the average of three quantities labeled 'IoU = 0.3/0.5/0.7' with N=3, but IoU is a span-overlap ratio, not a threshold; the recall metrics 'R@n, IoU=µ' are never formally defined. The reported numbers are internally inconsistent with the text: averaging the three R@1 columns for Baichuan gives (0.5133+0.3612+0.2103)/3 = 0.3616, while Table 2 reports mIoU(R@1)=0.3717; for Random Pick Method the average is 0.0465 but the table reports 0.0397. If mIoU(R@1) is instead the per-sample mean IoU over all test queries, then the equation and the accompanying 'N=3' explanation are wrong. If it is the threshold average, then the column is mislabeled and the Track 1 ranking could shift. Without the official evaluation script or a corrected formula, the central claim that the leaderboard scores accurately reflect relative performance cannot be independently verified for the headline track.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an overview of the NLPCC 2025 Shared Task 4 (M4IVQA), which introduces three tracks for medical instructional video question answering: temporal answer grounding in a single video (M4TAGSV), video corpus retrieval (M4VCR), and joint temporal grounding and retrieval in a video corpus (M4TAGVC). The paper describes the dataset (1,628 videos, 7,845 QA pairs, bilingual Chinese/English questions, knowledge graphs, and Whisper-generated subtitles), defines evaluation metrics for each track, reports final leaderboard results from 10 participating teams, and briefly describes the winning systems. The central claim is that the shared task constitutes a valid benchmark for multimodal, multilingual, multi-hop medical video QA and that the reported scores reflect relative system performance.","tokens_in":8802,"tokens_out":4675,"duration_ms":41182,"significance":"If the evaluation framework is sound, the M4IVQA dataset and the three-track setup would be a useful resource for the video QA community, with the unusual addition of a multilingual (Chinese/English) dimension and a multi-hop requirement. The inclusion of a random-pick baseline in all three leaderboards is good practice. However, the paper currently has a load-bearing defect in the definition of the primary metric for Track 1, and the validity of the multilingual comparisons rests on unverified assumptions about translation equivalence and subtitle quality. The significance of the results is therefore conditional on correction of these issues; the underlying dataset and task design remain potentially valuable.","major_comments":[{"comment":"Equation (1) is internally inconsistent: it defines mIoU as the average of three quantities labeled 'IoU = 0.3/0.5/0.7' with N=3, but IoU is a per-prediction overlap ratio, not a threshold, and the recall-at-threshold metric 'R@n, IoU=µ' is never formally defined. The reported mIoU(R@1) values do not match the average of the three R@1 columns in Table 2: for Baichuan, (0.5133+0.3612+0.2103)/3 = 0.3616 versus the reported 0.3717; for Random Pick Method, (0.0571+0.0465+0.0358)/3 = 0.0465 versus the reported 0.0397. The paper must supply a correct, unambiguous definition of mIoU(R@1), and ideally the official evaluation script, so the Track 1 leaderboard can be independently reproduced.","section":"Section 2.2, Eq. (1), Table 2"},{"comment":"The multilingual evaluation rests on two unverified assumptions: that the manually crafted Chinese questions and their English translations are semantically equivalent, and that the Whisper-generated Chinese and English subtitles are accurate enough to support answer grounding. The paper asserts that English questions were translated and reviewed by native-speaking physicians and that subtitles were automatically generated, but it reports no human evaluation, inter-annotator agreement, or sample quality checks, nor does it discuss this as a limitation. Without such evidence, the cross-lingual comparisons in Tables 2-4 cannot be interpreted as measuring the same task across languages. Please add either a quality assessment of a sample or an explicit limitation paragraph.","section":"Section 2.3"},{"comment":"The 'Random Pick Method' baseline is cited as references [1] and [2], but these references are overview papers of previous shared tasks, not descriptions of a random-pick implementation. The random-pick scores in Tables 2-4 are therefore not reproducible from the cited sources. The paper should either describe the random-pick protocol explicitly (e.g., uniform sampling of segments/videos) or cite the actual implementation used.","section":"Section 3, Tables 2-4"},{"comment":"The Track 3 composite metric 'Average' depends on R@1—mIoU, R@10—mIoU, and R@50—mIoU, but the paper does not specify how these quantities are computed. In particular, it is unclear whether a query whose relevant video is not retrieved within the top-k contributes zero IoU, whether the IoU is averaged only over queries that retrieve at least one candidate, and how the predicted segment is selected within each retrieved video. This ambiguity prevents verification of the Track 3 leaderboard and should be resolved with a precise formula.","section":"Section 2.2, Eq. (4), Table 4"}],"minor_comments":[{"comment":"The section heading contains a typo: 'Dateset' should be 'Dataset'.","section":"Section 2.3 heading"},{"comment":"The text reads 'the primary ranking metric is the \"verall\" score'; this should be 'Overall'.","section":"Section 2.2, Track 2 paragraph"},{"comment":"The caption spells 'Singe Video'; it should be 'Single Video'.","section":"Figure 1 caption"},{"comment":"The dataset description mentions the 'M4TAGV task', but the corresponding track abbreviation used throughout the paper is M4TAGSV. Please standardize the terminology.","section":"Section 2.3"},{"comment":"The example temporal answer is shown as '1:541:40', which appears to be a formatting error for '1:40-1:54'.","section":"Figure 1"},{"comment":"Several references appear unrelated to the claims they support (e.g., [3] on infectious disease prediction and [15] on thyroid disease prediction in the context of medical imaging diagnosis). Please verify that each citation is relevant to the statement it accompanies.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main basis for major_revision is the undefined and internally inconsistent primary metric for Track 1 (Eq. 1 vs. Table 2). This is a correctness issue in the paper's central reporting of results and must be fixed before publication. I also recommend asking the authors to supply the missing definition or the official evaluation script, and to address the multilingual validity concerns openly. The paper shows a strong pattern of self-citation, but I do not see evidence that this affects the technical content; it may be worth reminding the authors to broaden the related-work coverage. The manuscript also needs careful copyediting before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the paper is exactly what it says: an overview of a shared task, not a research paper. The useful part is that it extends the MMIVQA line with knowledge graphs, bilingual multi-hop questions, and three tracks (single-video grounding, corpus retrieval, and the combined task). The dataset description is concrete, and the leaderboard includes ten teams plus a random-pick baseline, which is good practice. If you work in medical video QA, this is a benchmark worth knowing about.\n\nThe problem is the Track 1 metric. Equation (1) is unreadable: mIOU is defined as the average of three quantities labeled IoU = 0.3/0.5/0.7, which makes no sense—IoU is a ratio, not a threshold. And the reported mIoU(R@1) values don't match simple averaging of the three R@1 columns. For Baichuan, the average is 0.3616, but the table says 0.3717; for Random Pick, 0.0465 vs 0.0397. So the central column of the headline track is not reproducible from the paper. That's a real flaw, not a nitpick, because the leaderboard is the main result.\n\nElsewhere the metrics for tracks 2 and 3 are better specified (though \"verall\" is a typo in the text for Overall). The lack of error bars or significance tests is minor for a shared-task overview. The larger unexamined assumption is that the English translations are semantically equivalent to the Chinese questions and that Whisper subtitles are accurate enough for grounding. That limits cross-lingual interpretation, but it is a standard limitation of such benchmarks.\n\nIs this serious enough for peer review? The dataset and task design have value, and the paper is mostly honest and descriptive. But the metric definition must be fixed before anyone uses this leaderboard as a reference. If the official evaluation script was used, the numbers are probably right, but the paper should either provide the correct formula or clarify what mIoU(R@1) actually computes. With a revision, I'd be satisfied. As it stands, I would not cite it as a benchmark reference, but I would not desk-reject it either.\n\nOverall: give it a referee, but the referee should ask for a corrected metric section. The reader's conditional verdict is about right.","headline":"Useful shared-task overview with a solid dataset extension, but the Track 1 metric is undefined and the leaderboard numbers don't reproduce from the paper.","tokens_in":9378,"tokens_out":3250,"would_cite":false,"duration_ms":29701,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The M4IVQA shared task offers a valid bilingual, multimodal, multi-hop benchmark for medical video QA, and its leaderboard scores capture genuine progress.","keywords":["medical instructional video","multimodal question answering","multilingual","temporal answer grounding","video corpus retrieval","multi-hop reasoning","shared task","benchmark"],"falsifier":"Independent back-translation of the Chinese and English question pairs by a third party that reveals systematic meaning differences, or a subtitle-only system matching the top Track 1 score, would show that the reported leaderboard is not measuring the claimed multimodal, multilingual, multi-hop competence.","tokens_in":8357,"feed_emoji":"🩺","tokens_out":11116,"duration_ms":102718,"temperature":0.7,"pith_summary":"This paper presents a shared task that measures medical instructional video question answering when three difficulties are combined: questions in Chinese and English, evidence spread across video, subtitles, and knowledge graphs, and answers that require chaining evidence across those sources. The task is split into three tracks—grounding an answer span inside a single video, retrieving the right video from a corpus, and doing both together—and the paper reports the official leaderboard results for each. The central claim is that this evaluation setup is a valid and useful benchmark: the best systems clearly outscore random pick baselines, yet the absolute scores (mIoU(R@1)=0.3717, Overall=1.6059, Average=0.2314) show the problem is far from saturated. If accepted, these numbers become the reference points that future medical video QA systems will be compared against.","feed_headline":"Medical video QA gets a three-track, two-language benchmark","feed_subtitle":"Top scores stay low—0.37 on answer grounding—so the field has room to improve.","key_machinery":"The mechanism that carries the evaluation is the compound scoring protocol. For temporal grounding, the paper computes Intersection over Union between predicted and ground-truth time spans and reports the mean IoU together with R@1 at IoU thresholds 0.3, 0.5, and 0.7. For retrieval, it uses R@n with n equal to 1, 10, and 50 plus Mean Reciprocal Rank, folded into a single Overall score. For the joint track, it takes the average IoU at each retrieval depth (R@1|mIoU, R@10|mIoU, R@50|mIoU) and averages those three values. These formulas make the three tracks comparable on a common unit—how much of the correct temporal segment the system found—and they let one number rank every team within a track.","core_discovery":"The core claim is that the M4IVQA benchmark—built on an existing bilingual medical instructional video corpus, augmented with per-video knowledge graphs and machine-generated subtitles, and annotated with timestamped answers by medical professionals—is a valid instrument for measuring whether AI systems can ground answers in a single video, retrieve a relevant video from a corpus, and do both at once, in both Chinese and English. The reported results are presented as the current state of that capability: the top entries reach mIoU(R@1)=0.3717 in the single-video track, Overall=1.6059 in the retrieval track, and Average=0.2314 in the joint track, while random pick baselines sit far lower at 0.0397, 0.1674, and 0.0384 respectively. The paper treats that gap as evidence that the tasks are solvable but far from solved.","pith_inferences":["A direct test the paper leaves implicit: back-translating the Chinese and English question pairs and having a second panel of physicians re-check equivalence would show how much of the cross-lingual score depends on translation quality.","Because answer timestamps are aligned to subtitles, a system that reads only the subtitle text and ignores visual frames might score well on Tracks 1 and 3; measuring that gap would reveal whether the benchmark truly requires visual understanding.","The knowledge-graph augmentations used by the leading teams could be ablated track by track; the paper's design suggests KG facts should aid retrieval more than localization, since they enrich query and subtitle text directly.","The same protocol could be adapted to other instructional video domains, such as equipment repair or software tutorials, to test whether the measured skills are specific to medicine or general to multimodal instructional QA."],"forward_implications":["The three tracks form a graded capability ladder: single-video grounding tests fine-grained localization, corpus retrieval isolates ranking, and the joint track requires both, so a system's failure can be assigned to the specific missing skill.","Because every video carries equivalent Chinese and English questions, the benchmark makes cross-lingual gaps measurable, giving a concrete target for language-agnostic medical video understanding.","Since knowledge graphs are supplied as additional input, systems that exploit structured medical knowledge can be separated from those that rely on text and video alone.","The reported top scores and random baselines define a practical operating range for medical video QA, so future improvements can be assessed as real progress toward deployment.","Using the same test videos and knowledge graphs across all three tracks keeps the leaderboards comparable; a team's retrieval decisions can be traced from Track 2 to Track 3 without re-running the whole pipeline."],"supporting_citations":[{"why":"Prior Chinese medical instructional video QA task; this task extends its data and evaluation design.","marker":"[1]"},{"why":"Prior multilingual medical video QA task; the dataset here is built on its corpus and question pairs.","marker":"[2]"},{"why":"Establishes visual answer localization and the IoU-based evaluation used by Tracks 1 and 3.","marker":"[14]"},{"why":"Defines temporal answer localization in a video corpus and the retrieval-plus-IoU evaluation used in Tracks 2 and 3.","marker":"[22]"}],"fun_headline_variants":["Medical video QA benchmark: top scores below 40% in all tracks","Bilingual medical video QA: multi-hop, still unsolved—top mIoU 0.37","Three-track medical video QA test: models trail behind, big gap","New medical video QA benchmark: random baselines beaten, room to grow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the physician-written Chinese questions and their English translations are semantically equivalent and that the automatically generated subtitles are accurate enough to support answer grounding.","fun_headline_variants_meta":{"raw":{"variants":["Medical video QA benchmark: top scores below 40% in all tracks","Bilingual medical video QA: multi-hop, still unsolved—top mIoU 0.37","Three-track medical video QA test: models trail behind, big gap","New medical video QA benchmark: random baselines beaten, room to grow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1520,"prompt_tokens":984,"completion_tokens":536,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":451}},"tokens_in":600,"tokens_out":536,"duration_ms":6014,"temperature":1.0,"reasoning_tokens":451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:31:33.241217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independent back-translation of the Chinese and English question pairs by a third party that reveals systematic meaning differences, or a subtitle-only system matching the top Track 1 score, would show that the reported leaderboard is not measuring the claimed multimodal, multilingual, multi-hop competence.","supporting_citations":[{"cited_title":"Overview of the nlpcc 2023 shared task: Chinese medical instructional video question answering","cited_arxiv_id":null,"evidence_quote":"Prior Chinese medical instructional video QA task; this task extends its data and evaluation design."},{"cited_title":"Overview of the nlpcc 2024 shared task 7: Multi-lingual medical instructional video question answering","cited_arxiv_id":null,"evidence_quote":"Prior multilingual medical video QA task; the dataset here is built on its corpus and question pairs."},{"cited_title":"Visual answer localization with cross-modal mutual knowledge transfer","cited_arxiv_id":null,"evidence_quote":"Establishes visual answer localization and the IoU-based evaluation used by Tracks 1 and 3."},{"cited_title":"Learning to locate visual answer in video corpus using question","cited_arxiv_id":null,"evidence_quote":"Defines temporal answer localization in a video corpus and the retrieval-plus-IoU evaluation used in Tracks 2 and 3."}],"review_version":1}