{"id":"fe07435f-0ede-44db-b03e-ab660c763c72","arxiv_id":"2608.06869","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DAEP, a difficulty-aware video retrieval and temporal grounding system, ranked first among ten systems on the NLPCC 2026 medical video question answering shared task.","lead":"A team's system for answering questions about medical training videos uses the official difficulty label to change how it searches, and it reports winning a ten-team shared task. The system retrieves the right video and locates the answer-supporting span, handling simple and complex questions with different evidence strategies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Difficulty-aware planner gain is measured on the same validation split used to select its control grid, so the 0.0401 ablation gain may reflect validation selection rather than label conditioning.","rationale":"The central claim decomposes into an official-ranking claim and a causal planning claim. The official ranking is credible external evidence from the shared-task leaderboard, so the load-bearing soft spot is the attribution of validation gains to difficulty-conditioned planning. The reader identified this same weakness: the planner controls are selected on the same validation split used for the ablations, with no test-side or held-out evidence. I agree with that assessment and add a more specific technical point: the no-planner baseline in Table 4 is a single hand-picked control tuple, not the best fixed tuple, which inflates the apparent benefit of label-conditioned search. The proposed nested-split test would settle whether the planner effect survives out-of-sample comparison against a comparably tuned fixed policy. If it does, the paper's central claim stands; if not, the paper should be read as a competitive shared-task system description with a plausible but unproven planning mechanism. This does not change the reader's verdict: the paper remains CONDITIONAL, since the official result is real but the novel contribution lacks the statistical support needed for full acceptance.","tokens_in":9603,"tokens_out":6816,"duration_ms":61118,"concrete_test":"Hold out half of the released validation set (stratified by language and difficulty) for evaluation. On the selection half, re-run the same grid search for Γ and train the learned heads. On the evaluation half, compare DAEP against (a) the best single fixed tuple found on the selection half, (b) a fixed-difficulty policy using the best Simple branch for all questions, and (c) a label-permuted control. If the DAEP-vs-best-fixed-tuple Average gap on the evaluation half is not clearly positive (e.g., below about 0.005), the planner's reported 0.0401 gain is a validation-selection artifact. Repeat across at least 5 random split seeds and report mean and standard deviation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The official leaderboard result is externally supported, but the novel claim—that difficulty-conditioned evidence planning improves ranking quality (Abstract, Tables 4–5)—rests on validation comparisons where the planner's discrete lookup was itself selected. Section 4.2 says the test lookup Γ is chosen from an 80-tuple-per-difficulty-group grid on validation, and Section 5 concedes 'one validation-selected run' with no alternative splits. The 0.0401 Average drop in Table 4 and the label-conditioning gaps in Table 5 therefore conflate true label conditioning with discrete hyperparameter selection that can fit validation noise. The 'w/o DA Planner' row uses a single hand-picked tuple (K=3, τ=0.45, L=2, λ=0.15) rather than the best fixed tuple, so it is not a clean control for tuning effort. 'Fixed difficulty' in Table 5 is a better control, but it is evaluated on the same split used to choose both branches. No test-side ablations, repeated-seed variance, confidence intervals, or alternative validation splits are reported. Thus the causal claim 'difficulty-aware planning improves ranking quality, with the largest gain on complex questions' is not yet established; the official ranking establishes only the competitiveness of the complete system.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes DAEP, the BIGC submission to NLPCC 2026 Shared Task 1 Track 3 (DA-TAGVC). The system decomposes the task into video retrieval and temporal span localization, using subtitle text, visual features, and procedural context as evidence modalities. Its novel component is a difficulty-aware evidence planner: the benchmark-provided Simple/Complex label selects discrete inference controls (Top-K, boundary threshold, expansion length, and a reranking-strength schedule), while modality weights and reranking strength are learned per question. The authors report that BIGC ranked first among ten systems on the official leaderboard with an Average score of 0.2728, and they present validation ablations in which removing visual evidence, context, span reranking, or the difficulty planner reduces Average. The paper is candid about its limitations: it uses a single validation-selected run, no repeated-seed variance, no confidence intervals, and no alternative validation splits.","tokens_in":9966,"tokens_out":2529,"duration_ms":25606,"significance":"If the difficulty-planning claim is sound, the paper makes a modest but useful contribution: it provides an inspectable interface between an input difficulty label and several evidence-allocation decisions in a shared-task system. The official leaderboard result is externally grounded and demonstrates that the complete system is competitive, and the paper explicitly separates the leaderboard claim from the within-run validation analysis. The methodological strength is the clear reporting of implementation details, hyperparameter ranges, and the acknowledgement of evidential limits. However, the paper's central scientific novelty—that difficulty-conditioned planning improves ranking quality, especially on complex questions (Abstract, Tables 4–5)—is not yet established because the planner's discrete lookup is selected on the same validation split used for the ablation comparison, and the no-planner control is not tuned with comparable effort. The leaderboard establishes the competitiveness of the complete system, but not the causal contribution of the difficulty planner.","major_comments":[{"comment":"The 0.0401 Average gain attributed to the difficulty-aware planner is measured on the same validation split that was used to select the test lookup Γ from an 80-tuple grid per difficulty group (Section 4.2). The 'w/o DA Planner' row in Table 4 uses a single hand-picked tuple (K=3, τ=0.45, L=2, λ=0.15) rather than the best fixed tuple under comparable selection effort. Therefore the gain conflates label-conditioned planning with discrete hyperparameter selection that can fit validation noise. The paper's own Section 5 acknowledges 'one validation-selected run' and no alternative splits, so this is a load-bearing limitation for the abstract's causal claim. To establish the planning effect, the authors need a held-out validation split or nested cross-validation in which the lookup grid is selected on one part and evaluated on another, and the fixed-tuple control should be tuned with the same budget.","section":"Section 4.2, Tables 4–5, Section 5"},{"comment":"The 'Fixed difficulty' and 'Shuffled difficulty' rows are better controls than the no-planner row, but they are still evaluated on the same split used to select both branches of the comparison. 'Shuffled difficulty' uses one fixed permutation, so no variance or significance statement is possible, and the grid selection procedure for the shuffled branch is not described. Without repeated permutations, confidence intervals, or multiple validation folds, the larger gaps on Complex questions (0.2554 vs. 0.2288 versus Fixed, and 0.2554 vs. 0.2163 versus Shuffled) do not establish that the label itself, rather than the selected hyperparameters, carries the benefit.","section":"Table 5, Section 4.5"},{"comment":"The statement that difficulty-aware planning yields 'the largest gain on complex questions' rests on Table 3, where the Average drop without the planner is 0.0479 for Complex groups and 0.0329 for Simple groups. Because these differences are computed on a single validation split with a validation-selected planner and no repeated-seed variance or significance testing, the comparison cannot distinguish a true difficulty interaction from noise or from the different operating points of the simple and complex lookups. The claim should be softened or supported with per-group confidence intervals, ideally from leave-one-group-out or repeated subsampling of the validation split.","section":"Abstract and Table 3"},{"comment":"The hybrid planner mixes learned per-question weights (wi, λi) with discrete controls (Ki, τi, Li) selected from Γ(y) on validation. The paper states that gradients update wi and λi but not the discrete lookup (Section 3.5). This is a sensible design, but it means the discrete lookup is a validation-tuned hyperparameter, not a learned component. The manuscript should state this more prominently in the abstract and introduction, because the claimed contribution is the difficulty-conditioned discrete plan, and the current evidence for it is a validation selection result rather than a held-out evaluation.","section":"Equation (1) and Section 3.5"}],"minor_comments":[{"comment":"The order-prior and frequency-prior descriptions are too compressed; please give the exact formulas or pseudocode, and state whether these priors were tuned on validation or fixed a priori.","section":"Section 4.5, Table 5"},{"comment":"The sentence 'Repeated-seed variance and end-to-end inference latency are not reported' is useful, but it belongs in the limitations subsection of Section 5 rather than buried in implementation details.","section":"Section 4.2"},{"comment":"The phrase 'Positive-IoU candidate coverage' is introduced without definition; please define it or remove it, since the manuscript otherwise avoids undefined metrics.","section":"Section 5"},{"comment":"In Equation (2), the symbol R↓i is used both for the sorted list and (via subscripting) for its elements; using a separate notation for the ranked list, such as Li or Ri, would reduce confusion.","section":"Section 3.6"},{"comment":"The abstract says 'ranked first among ten systems on all four official metrics', which matches Table 2. Please also mention in the abstract that the leaderboard comparison includes systems with potentially different backbones and training data, since the paper itself correctly notes in Section 4.5 that these are not shared-backbone baselines.","section":"Abstract and Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"This is a shared-task system description paper. The official leaderboard ranking is externally supported and gives the paper a solid floor. The reason I am not recommending minor revision is that the abstract's central novelty claim—difficulty-aware planning improves ranking quality, with the largest gain on complex questions—rests on validation ablations whose planner lookup was selected on the same validation split. This is fixable within the manuscript's scope by adding nested or leave-one-group-out validation, tuning the fixed-planner control with the same budget, and reporting permutation variance for the shuffled-difficulty control. If the authors cannot provide such evidence, they should explicitly relabel the planner contribution as a system-design choice rather than an empirically established causal effect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a competent shared-task system paper from team BIGC, and the most important claim — first place on the official leaderboard — is externally verified by the organizer. The architecture is not new: it is the standard VCMR retrieval-localization pipeline with mBERT plus frozen CLIP. The genuinely new bit is the difficulty-conditioned planner that maps the supplied Simple/Complex label to discrete controls (K, tau, L) and learned weights/reranking strength. That is a reasonable engineering idea, and the ablations are run carefully in the sense that they control for splits, encoders, optimizer, and evaluator.\n\nThe paper does something rare in shared-task writeups: it tells you exactly what its evidence can and cannot support. Section 5 explicitly says the validation comparisons are 'diagnostic rather than unbiased test estimates' because the lookup and checkpoint were selected on the same split. That sentence defuses the stress-test concern before I raise it. The planner's 0.0401 validation gain is real only under the selected configuration; it is not established as a label-conditioning effect. The 'w/o DA Planner' row also uses a hand-picked fixed tuple rather than the best fixed tuple, so it is not a clean control for tuning effort. But the authors seem aware of this — they call them diagnostic comparisons.\n\nWhere I'd push: no repeated seeds, no confidence intervals, no test-side ablations, no code/data release. Those are addressable and would turn the planner claim from plausible to solid. The official first-place ranking, meanwhile, stands independently of all that.\n\nOverall: the paper is honest, well-scoped, and useful for anyone working on medical video QA or difficulty-conditioned retrieval. It's not a conceptual breakthrough and the absolute scores are modest, but the external leaderboard verification makes it a legitimate data point. I'd send it to review. A good referee will ask for the code and for a fair fixed-control or held-out evaluation of the planner, both of which are within reach for this team.\n\nRecommendation: accept peer review; conditional on addressing the planner's control setup and releasing code/config.","headline":"Solid shared-task system paper with an honest limitations section; the official first-place result is externally grounded, but the difficulty-planner's causal gain is only validation-level and the paper says so.","tokens_in":10401,"tokens_out":1620,"would_cite":false,"duration_ms":14142,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a task-provided Simple/Complex difficulty label can drive an inference-time evidence plan — controlling modality weights, retrieval breadth, boundary expansion, and span reranking — and that the resulting system…","keywords":["temporal answer grounding","video corpus moment retrieval","difficulty-aware evidence planning","medical instructional video question answering","multimodal retrieval","span reranking","shared task system"],"falsifier":"Run the same validation-selected checkpoint on the official hidden-test inputs with the difficulty-conditioned lookup reversed (Complex questions receive the Simple controls and vice versa) and compare Average scores; if the reversed plan does not lose to the original lookup, label-conditioned planning is not what carries the reported gain.","tokens_in":9381,"feed_emoji":"🎯","tokens_out":8401,"duration_ms":64582,"temperature":0.7,"pith_summary":"This paper tries to establish that a single binary difficulty label — the one the task already provides — can act as a plan for how a video-answer system should search, rather than an input it should merely classify. Its system, DAEP, converts Simple/Complex into an evidence plan that sets how many subtitle units feed a video score, how far boundaries expand, and how strongly whole spans are reranked. If that claim is right, temporal answer grounding gains a cheap adaptation mechanism: long answers get broad verification, compact answers get focused retrieval. The authors support the claim with a first-place finish among ten systems on the official task and with validation ablations showing the planner's removal costs the most on complex questions.","feed_headline":"Difficulty-aware evidence planning wins ten-team video grounding task","feed_subtitle":"Complex questions lose the most when the label is ignored, showing the plan's worth.","key_machinery":"The central object is the hybrid evidence plan $\\pi_i=\\{w_i,K_i,\\tau_i,L_i,\\lambda_i\\}$, produced by a planner that reads the question, the difficulty label, and initial modality match scores. The planner's learned half sets per-question modality weights $w_i$ and reranking strength $\\lambda_i$; its discrete half maps the label through a fixed lookup $\\Gamma(y)$ to the Top-$K$ aggregation count, the boundary score threshold $\\tau$, and the expansion length $L$ (in subtitle positions). That plan coordinates the three stages of the pipeline: candidate video ranking via weighted modality fusion and Top-$K$ pooling, temporal boundary decoding by thresholded anchor expansion, and whole-span consistency reranking. The same label thus changes how much evidence is collected and how far a span may grow, which is the mechanism the paper credits for the gains on complex questions.","core_discovery":"The paper's central claim is that difficulty-aware evidence planning improves video corpus temporal answer grounding. Given a question, fifty candidate videos, and a Simple/Complex label, DAEP builds a plan $\\pi_i=\\{w_i,K_i,\\tau_i,L_i,\\lambda_i\\}$: modality weights $w_i$ and reranking strength $\\lambda_i$ are learned per question, while the discrete controls — aggregation breadth $K_i$, boundary threshold $\\tau_i$, and expansion length $L_i$ — come from a difficulty-conditioned lookup fixed on validation (Simple: $(2,0.55,1)$, Complex: $(5,0.35,3)$). The system ranks videos by fusing text, visual, and procedural-context scores with adaptive Top-$K$ pooling, decodes anchored spans by expanding above a threshold, then rescales each span by a learned consistency score. In the official evaluation the complete system ranks first among ten submissions on R@1|mIoU, R@10|mIoU, R@100|mIoU, and their Average (0.2728); on validation, removing the planner lowers Average from 0.2646 to 0.2245, with the larger drop on Complex questions (0.2554 to 0.2075).","pith_inferences":["A natural extension is to replace the supplied binary label with a predicted evidence-demand score, since the paper's planner already shows that domain knowledge can be injected as discrete controls; the same framework could adapt to questions whose scope is inferred from text length or answer-duration priors.","The result suggests a transferable design principle for temporal grounding: a single coarse property of the question (here, binary difficulty) can coordinate multiple stages of a pipeline — retrieval breadth, boundary threshold, expansion, and reranking — rather than only the final scoring head.","Because the lookup and checkpoint are validation-selected and no test-side ablations are reported, an independent re-run with multiple seeds and a true held-out split is the direct way to test whether the 0.0401 planner gain is a selection artifact.","The paper's own qualitative failure modes (narration/action drift, subtitle-visual mismatch, bilingual terminology) point to cross-modal alignment as the next bottleneck; a planner that also decides when to trust subtitles versus pixels could extend the same conditional design."],"forward_implications":["The full system places first among ten submissions on all four official metrics, with an Average of 0.2728 and a 9.4% relative improvement over the runner-up.","Validation ablations within the system show every tested component contributes: removing visual evidence, context evidence, span reranking, or the planner lowers Average by 0.0170, 0.0254, 0.0097, and 0.0401, respectively.","The difficulty plan helps most where the task is hardest: without the planner, Average on Complex validation questions falls from 0.2554 to 0.2075, a larger drop than the Simple group's 0.2729 to 0.2400.","Replacing the true label with a fixed or a shuffled difficulty assignment reduces validation Average (to 0.2414 and 0.2328, respectively), so the label-conditioned lookup contributes beyond a static configuration."],"supporting_citations":[{"why":"Defines the task, its Simple/Complex labels, the four official metrics, and the leaderboard that records the first-place result.","marker":"[19]"},{"why":"Supplies the subtitle-aligned evidence-unit representation used for each candidate video.","marker":"[12]"},{"why":"Contributes the video corpus moment retrieval decomposition of retrieval and localization that DAEP follows.","marker":"[22]"},{"why":"Provides the contrastive retrieval formulation and Top-K pooling practice for video corpus moment retrieval.","marker":"[27]"},{"why":"Establishes contextual query-aware ranking, which DAEP adapts for procedural-context evidence and span reranking.","marker":"[9]"},{"why":"Supports partial-relevance handling in query-aware ranking and reranking.","marker":"[8]"},{"why":"Supplies the frozen visual-language encoder that produces the 1-fps visual evidence features.","marker":"[23]"},{"why":"Supplies the multilingual text encoder for questions, subtitles, and procedural context.","marker":"[3]"}],"fun_headline_variants":["Difficulty-aware planning tops video grounding with adaptive evidence fusion","Complex questions gain most from difficulty-aware evidence planning","Adaptive evidence plan wins video grounding, doubles down on hard queries","BIGC's DAEP first in video grounding with difficulty-aware plans","Difficulty-aware plans give largest gain on complex video questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the benchmark-supplied Simple/Complex label reliably tracks how much evidence a question needs, and that the validation split used to fix the lookup is representative of the hidden test, so the plan's gain is not just validation selection.","fun_headline_variants_meta":{"raw":{"variants":["Difficulty-aware planning tops video grounding with adaptive evidence fusion","Complex questions gain most from difficulty-aware evidence planning","Adaptive evidence plan wins video grounding, doubles down on hard queries","BIGC's DAEP first in video grounding with difficulty-aware plans","Difficulty-aware plans give largest gain on complex video questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000395,"raw_usage":{"total_tokens":2075,"prompt_tokens":955,"completion_tokens":1120,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1038}},"tokens_in":571,"tokens_out":1120,"duration_ms":7519,"temperature":1.0,"reasoning_tokens":1038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:28:17.805832+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same validation-selected checkpoint on the official hidden-test inputs with the difficulty-conditioned lookup reversed (Complex questions receive the Simple controls and vice versa) and compare Average scores; if the reversed plan does not lose to the original lookup, label-conditioned planning is not what carries the reported gain.","supporting_citations":[{"cited_title":"Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation","cited_arxiv_id":"2607.06618","evidence_quote":"Defines the task, its Simple/Complex labels, the four official metrics, and the leaderboard that records the first-place result."},{"cited_title":"In: Proceedings of ACM ICMR 2024","cited_arxiv_id":null,"evidence_quote":"Supports partial-relevance handling in query-aware ranking and reranking."},{"cited_title":"In: Proceedings of the 38th ICML","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen visual-language encoder that produces the 1-fps visual evidence features."}],"review_version":3}