{"id":"69606774-9b80-4278-b1ba-69ad872fc6f3","arxiv_id":"2412.18849","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A generative transformer framework predicts future surgical phases minute-by-minute over 20 to 30 minute horizons, reaching 41.3% F1 on AutoLaparo21 and competitive remaining-time errors on Cholec80.","lead":"This paper presents SWAG, a system that watches surgical video and generates a minute-by-minute forecast of the surgical phases that will happen over the next 20 to 30 minutes. It is a step toward giving surgical teams a live preview of the rest of an operation, rather than only recognizing the current step.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline F1 advantage on AutoLaparo21 rests on 7 test videos with no error bars; the 7.0-point gap over Naive2 may be noise.","rationale":"The reader's weakest assumption—single deterministic future trajectories—is a valid limitation but applies symmetrically to all compared methods and does not threaten the comparative claim. The more load-bearing issue is statistical reliability: the entire demonstration of superiority on complex, variable procedures comes from a 7-video test set with no error bars or repeated runs. On Cholec80, SP* is actually below Naive2, so the AutoLaparo21 gap is the sole evidence for the central contribution. This is an addressable concern, and the recommended CONDITIONAL verdict remains appropriate, with the added condition that the authors provide variance estimates and significance testing for the headline F1 comparisons.","tokens_in":13517,"tokens_out":7916,"duration_ms":72258,"concrete_test":"Run the SP*, SP, Naive2, and AR models on AutoLaparo21 with at least 5 random seeds (or bootstrap over the 7 test videos), and report mean±std of the mean F1 over the 30-minute horizon plus per-horizon curves. Compute a paired bootstrap 95% confidence interval for the SP*−Naive2 and SP*−SP differences at each horizon. If the CI includes zero or the effect is not significant (e.g., p>0.05), the superiority claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim—that the prior-knowledge single-pass model (SP*) is superior on the variable AutoLaparo21 dataset—rests on a 7-video test split (10/4/7) and is reported without error bars or repeated-seed variation. In Table 1, SP* (41.3 mean F1) beats Naive2 (34.3) by 7.0 points and SP (38.4) by 2.9 points on AutoLaparo21, but on Cholec80 the same SP* (32.1) is 7.4 points below Naive2 (39.5). The paper's 'versatility' conclusion therefore hinges entirely on the AutoLaparo21 gap. With n=7 test videos, a single atypical procedure can shift mean F1 by several points; the 4-video validation set is similarly small. No confidence intervals, bootstraps, or per-video breakdowns are provided, even though the qualitative figures show substantial variability across test cases. Because the stated contribution—prior-knowledge embeddings and generative decoding improving anticipation on complex workflows—depends on this specific gap being reliable, the headline F1 advantage is not yet established as a real effect rather than noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SWAG, a transformer-based framework for simultaneous surgical phase recognition and long-term phase anticipation. It proposes two generative decoding schemes (single-pass and auto-regressive), a prior-knowledge embedding based on class transition probabilities, and a regression-to-classification mapping. Evaluated on Cholec80 and AutoLaparo21, the single-pass model with transition-probability embeddings obtains mean F1 scores of 32.1% over 20 minutes and 41.3% over 30 minutes, respectively, and competitive remaining-time regression results.","tokens_in":13762,"tokens_out":4628,"duration_ms":38865,"significance":"If the reported results are reliable, the paper would advance surgical workflow anticipation by moving from single-event prediction to dense, long-horizon sequence generation, and by unifying recognition and anticipation in one model. The idea of conditioning future tokens on empirically estimated class transition probabilities is simple and transferable. Strengths include evaluation on two public datasets, comparison with naive baselines and prior regression methods, and clear documentation of the SegF1 metric. However, the paper does not provide code, error bars, or statistical tests, and the most important comparative claim rests on a very small test set.","major_comments":[{"comment":"The central claim that SP* outperforms the conditional-probability baseline Naive2 on AutoLaparo21 (41.3 vs. 34.3 F1) rests on a test set of only 7 videos, with no confidence intervals, bootstrap estimates, or per-video results reported. On Cholec80 the same SP* model is 7.4 points below Naive2, so the 'versatility' conclusion depends entirely on this one unquantified gap; a single atypical procedure could shift the mean by several points. Please report per-video scores and interval estimates (e.g., bootstrap CIs or a paired test) for the headline F1 comparisons.","section":"4.1, Table 1"},{"comment":"The RSD comparison is not controlled: the table reports BD-Net's results on random 4-fold splits while SWAG-SP is evaluated on consecutive 60/20 splits, as the note under the table admits. The text says 'using 4-fold cross-validation,' but the implementation differs from the comparison method's protocol. This makes the 'ranks second on MAE-5 and MAE-ALL' claim unsupported; please re-run on exactly the same split scheme or present the comparison as a qualitative reference only.","section":"4.2, Table 3"},{"comment":"The SegF1 metric relies on several hand-set parameters—IoU threshold 0.25, EOS weight 0.5, and EOS capping at 4/8 minutes—without any sensitivity analysis or demonstration that the conclusions are robust to their choice. Since the reported advantages over baselines are modest (e.g., Table 1, AutoLaparo21: SP* 41.3 vs. SP 38.4 vs. Naive2 34.3), it is important to show that the ranking is not an artifact of these thresholds.","section":"8.2, Segment-based F1 (SegF1)"},{"comment":"The paper correctly acknowledges the 'assumption of single valid future trajectories,' but this assumption also limits the evaluation metrics: cross-entropy and SegF1 compare against one annotated future sequence. If multiple futures are plausible, the reported F1 numbers systematically penalize valid predictions and the method's true predictive skill is understated. Please quantify the extent of this issue (e.g., annotator agreement on plausible futures, or a comparison with a distribution-aware metric) so the reader can judge the practical significance of the reported numbers.","section":"5, Limitations paragraph"}],"minor_comments":[{"comment":"The symbol p is used for both the key-pooling representation (p = max{...}) and the future-token probability vector p_t (Eq. 5); please disambiguate the notation.","section":"3.2"},{"comment":"The notation h0 = 0 and then hn ∈ {1,2,...,N} is slightly confusing because h0 is not part of the future index set; consider writing {0, h1, ..., hN} explicitly.","section":"3.1, Eqs. (1)-(2)"},{"comment":"The standard deviations are reported without the number of folds or test sizes used to compute them; please specify these details.","section":"Table 3"},{"comment":"The method name is typeset inconsistently: 'SW AG' appears with a space in the abstract and Section 1, while 'SWAG' appears in the text (e.g., 'SWAG-SP'); please unify the spelling.","section":"Throughout"},{"comment":"The IoU formula would be easier to follow if the union were written explicitly as the sum of lengths minus the intersection; the current compressed expression is less readable.","section":"8.1.2, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The paper is more suitable for a specialized medical-image-analysis venue than a general computer vision journal; the evaluation is thin for the strength of the claims. I encourage the editor to require the requested revisions, particularly the statistical support for the headline result and the controlled RSD comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What's actually new: SWAG treats anticipation as generative sequence modeling, predicting a minute-by-minute phase sequence 20-30 minutes ahead, conditioned on the observed video. Previous work mostly did next-event regression or short-horizon classification. The paper compares two decoding schemes (single-pass and auto-regressive), adds a transition-probability embedding, and shows a regression-to-classification mapping. The evaluation on Cholec80 and AutoLaparo21 is thorough, with ablations on context length, pooling, and model size. Credit is due for comparing against naive baselines, which many papers skip.\n\nWhere it's soft: The central comparative claim rests on a seven-video test split. SP* beats Naive2 by 7 F1 points on AutoLaparo21, but on Cholec80 Naive2 beats SP* by 7.4 points. The 'versatility' conclusion depends entirely on that one small dataset. No confidence intervals, bootstraps, or per-video numbers are given, so the 7-point gap could be noise. The RSD comparison uses mismatched splits (they disclose this in a table note, but it still weakens the comparison). The EOS capping and SegF1 thresholds are hand-set and could be tuned to favor the proposed method. The single-future-trajectory assumption is real and acknowledged, but not addressed. No code or error bars.\n\nThe paper is honest about its limitations and the ablations are useful. The core idea—dense future classification via generative decoding—is worth pursuing. But the empirical case needs more work: error bars, per-video breakdowns, and ideally code so others can reproduce.\n\nWho it's for: Surgical workflow analysis researchers, especially those working on anticipation and decision support. I'd cite it if I worked in that area, because the formulation is new. It deserves peer review, but a serious referee should ask for stronger statistical evidence.","headline":"The paper's real contribution is dense long-horizon surgical phase anticipation, but the headline advantage over a simple baseline on AutoLaparo21 is not yet established—seven test videos, no error bars.","tokens_in":14325,"tokens_out":2490,"would_cite":true,"duration_ms":21185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SWAG recasts surgical phase anticipation as generative sequence prediction and shows it beats simple priors on variable procedures.","keywords":["surgical workflow anticipation","surgical phase recognition","generative sequence modeling","remaining time regression","class transition probabilities","single-pass decoding","Cholec80","AutoLaparo21"],"falsifier":"Collect a second set of expert annotations of plausible future phase sequences for a sample of Cholec80 and AutoLaparo21 test moments, and re-score SWAG-SP* with SegF1 where a prediction is credited if it matches any expert-plausible trajectory. If scores rise substantially relative to single-ground-truth scoring, the single-future assumption, not the model, is the main cap on reported performance; if they do not rise, the model's errors are not an artifact of the metric.","tokens_in":13309,"feed_emoji":"🩺","tokens_out":6128,"duration_ms":53135,"temperature":0.7,"pith_summary":"The paper tries to establish that long-term surgical phase anticipation is best framed as generative sequence prediction, not as single-event regression. It proposes SWAG, a transformer-based framework that, from the observed video up to the current minute, generates a minute-by-minute sequence of future surgical phases over horizons up to 60 minutes. The central comparison is against two simple baselines: extending the current phase unchanged (Naive1) and sampling from training-set class transition probabilities conditioned on the recognized phase (Naive2). The claim is that the generative models match or beat these priors on frame-level F1 and, importantly, keep segment-level temporal coherence (SegF1), which the baselines lose. If true, this gives operating-room systems a way to visualize the remaining workflow and estimate completion time, not just the next event.","feed_headline":"Generative model forecasts surgical phases up to 60 minutes ahead","feed_subtitle":"On variable hysterectomy videos it beats simple priors (F1 41.3% at 30 min) and keeps segment coherence.","key_machinery":"The load-bearing mechanism is the class transition probability embedding inside the single-pass decoder. For each future minute index $h_n$, the token input is initialized with the training-set probability $P(y_{t+h_n \\cdot 60}=j \\mid y_t=i)$, the chance that class $j$ is active $h_n$ minutes after current class $i$; after a linear projection this vector is added to a learned token embedding and sinusoidal positional encoding. The decoder then uses cross-attention between these future tokens and compressed past context tokens from a windowed self-attention encoder with cumulative max-pooling, generating all $N$ future phases in one forward pass. A second mechanism, regression-to-classification (R2C), sorts the predicted remaining times to each phase into ascending order and bins them into a discrete high-resolution phase sequence, letting a regression-trained model produce the same kind of anticipation output.","core_discovery":"On the paper's own terms, the discovery is that a single-pass transformer decoder, whose future tokens are initialized with class transition probabilities extracted from training data (SP*), produces coherent minute-level forecasts of future surgical phases and outperforms both naive baselines on the more variable AutoLaparo21 dataset, with mean F1 of 41.3% over 30 minutes and SegF1 of 34.8%. On Cholec80 the same model reaches 32.1% mean F1 over 20 minutes, while Naive2 reaches 39.5% frame-level F1 but only 11.9% SegF1, which the paper reads as evidence that simple priors exploit structured workflows without temporal coherence. The same single-pass decoder, trained for remaining-time regression, achieves weighted MAEs of 0.32 and 0.48 minutes at 2- and 3-minute horizons, competitive with Bayesian and IIA-Net baselines, and the proposed regression-to-classification (R2C) mapping turns regression outputs into discrete phase sequences. The paper concludes that generative decoding unifies recognition and anticipation and becomes increasingly valuable as surgical workflow variability increases.","pith_inferences":["If multiple futures are equally plausible, the reported F1 numbers are likely conservative: re-scoring predictions against the best-matching expert-annotated future trajectory would raise both frame-level and SegF1 scores; this is an extension of the limitation the paper itself acknowledges.","The class transition probability priors are computed once from the training set; a testable extension is to update them online per procedure or per surgeon, which could sharpen the SP* embedding as more of the current surgery is observed.","The same decoder could generate instrument-usage or action tokens alongside phase tokens, since the token-level conditioning is not phase-specific; that would bring the framework closer to full workflow generation.","The strong Naive2 result on Cholec80 suggests that future surgical anticipation benchmarks should report a prior-conditioned baseline like Naive2 as a mandatory comparison point, otherwise model gains on structured datasets can be overstated."],"forward_implications":["A single model trained with maximal horizon $N$ can be evaluated at any shorter horizon without retraining, so the same checkpoint serves both short- and long-term anticipation.","Dense minute-level future phase sequences give a direct visual completion of the remaining workflow, which the paper argues supports intraoperative awareness and guidance beyond single next-event prediction.","On structured workflows like cholecystectomy, simple priors remain strong, so generative modeling's practical edge appears when phase order and durations are more variable, as in AutoLaparo21.","Because regression outputs can be converted into classification sequences via R2C, remaining-time estimation and phase anticipation cease to be separate tasks in this framework.","Segment-level F1 (SegF1) rewards correct phase boundaries and penalizes oversegmentation, so the reported advantage of generative models over naive baselines is not just frame-counting."],"supporting_citations":[{"why":"Supplies the Cholec80 dataset and phase labels used for recognition and anticipation evaluation.","marker":"[32]"},{"why":"Supplies the AutoLaparo21 dataset, the more variable procedure central to the claim that generative modeling beats simple priors.","marker":"[33]"},{"why":"Provides the Bayesian baseline and the remaining-time task formulation for next phase occurrence.","marker":"[12]"},{"why":"Provides the IIA-Net baseline and reports the prior results reproduced in Table 2.","marker":"[13]"},{"why":"Supplies the key-pooling compression and windowed attention building blocks that SWAG adapts.","marker":"[5]"},{"why":"Supplies the LoViT vision encoder whose features initialize SWAG's temporal input.","marker":"[4]"},{"why":"Supplies the GPT-2 autoregressive decoder used for the AR variant.","marker":"[14]"},{"why":"Supplies the single-pass decoding concept for generating future tokens in one forward pass.","marker":"[15]"},{"why":"Supplies the BD-Net remaining-surgery-duration estimates that SWAG-SP is compared against in Table 3.","marker":"[10]"}],"fun_headline_variants":["SWAG forecasts surgical phases with generative anticipation","Single-pass decoder predicts surgery steps up to 60 min ahead","Class transition probabilities improve surgical phase forecasting","SWAG outperforms simple priors on variable videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that for each observed moment there is one correct future phase sequence, so a prediction is graded only against the single annotated future; if several futures are equally plausible, the loss and the F1/SegF1 metrics penalize valid predictions and the reported scores understate the model's true anticipative skill.","fun_headline_variants_meta":{"raw":{"variants":["SWAG forecasts surgical phases with generative anticipation","Single-pass decoder predicts surgery steps up to 60 min ahead","Class transition probabilities improve surgical phase forecasting","SWAG outperforms simple priors on variable videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002375,"raw_usage":{"total_tokens":9209,"prompt_tokens":1077,"completion_tokens":8132,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":8072}},"tokens_in":693,"tokens_out":8132,"duration_ms":51348,"temperature":1.0,"reasoning_tokens":8072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:25:25.957765+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a second set of expert annotations of plausible future phase sequences for a sample of Cholec80 and AutoLaparo21 test moments, and re-score SWAG-SP* with SegF1 where a prediction is credited if it matches any expert-plausible trajectory. If scores rise substantially relative to single-ground-truth scoring, the single-future assumption, not the model, is the main cap on reported performance; if they do not rise, the model's errors are not an artifact of the metric.","supporting_citations":[{"cited_title":"IEEE Trans","cited_arxiv_id":null,"evidence_quote":"Supplies the Cholec80 dataset and phase labels used for recognition and anticipation evaluation."},{"cited_title":"In: International Conference on Medi- cal Image Computing and Computer-Assisted Intervention, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the AutoLaparo21 dataset, the more variable procedure central to the claim that generative modeling beats simple priors."},{"cited_title":"International Conference on Medical Image Computing and Computer-Assisted Intervention (2020)","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian baseline and the remaining-time task formulation for next phase occurrence."},{"cited_title":"Medical Image Analysis (2022)","cited_arxiv_id":null,"evidence_quote":"Provides the IIA-Net baseline and reports the prior results reproduced in Table 2."},{"cited_title":"In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the key-pooling compression and windowed attention building blocks that SWAG adapts."},{"cited_title":"Medical Image Analysis (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the LoViT vision encoder whose features initialize SWAG's temporal input."},{"cited_title":"OpenAI blog (2019)","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-2 autoregressive decoder used for the AR variant."},{"cited_title":"Computerized Medical Imaging and Graphics (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the BD-Net remaining-surgery-duration estimates that SWAG-SP is compared against in Table 3."}],"review_version":1}