{"id":"ee26924c-6684-41e9-800d-1be8dcd211e2","arxiv_id":"2501.09555","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Surg-FTDA aligns a small set of image embeddings to text embeddings and trains a text-only decoder, enabling few-shot surgical phase, triplet, and captioning tasks.","lead":"Surg-FTDA adapts a surgical vision-language foundation model to new tasks using only a few hundred image-label pairs plus text-only decoder training. The paper reports competitive results on phase recognition, triplet recognition, and image captioning compared with fully supervised training and CapDec.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The alignment MLP trained on 500 anchors lacks direct evidence of generalizing to unseen images; the central claim depends on a representativeness assumption that no quantitative test supports.","rationale":"The reader's weakest assumption identifies the representativeness of the K-selected anchors and the consequent generalization of the MLP alignment. I agree that this is the load-bearing component. The paper's headline claim is that a text-trained decoder plus a small alignment MLP can handle downstream visual tasks with few paired examples. The decoder itself is text-only and cannot correct image embeddings that are poorly mapped; all of the generalization burden falls on fMLP. Section 2.1's MSE objective only constrains outputs on the selected anchors, and Section 2.2 passes the aligned embeddings directly into the decoder. The paper's empirical support for this critical step is a qualitative t-SNE plot and aggregate task metrics, with no held-out alignment error and no variance estimates. The proposed concrete test would directly measure whether the mapping is accurate on unseen images. The broken 10% baseline and missing error bars are secondary concerns about the strength of the comparison, not about the internal validity of the method's mechanism. Since the concern is a demand for evidence rather than a demonstrated contradiction, the appropriate verdict remains CONDITIONAL, matching the reader's assessment; no adjustment is needed.","tokens_in":9515,"tokens_out":6207,"duration_ms":66528,"concrete_test":"Use the SVL-Caption training/validation split: select K=500 anchors from the training split only, train fMLP, and apply it to all 1351 validation images. For each validation image, compute the cosine distance between the aligned image embedding and the text embedding of its ground-truth caption. Compare this distribution with the mean pairwise cosine distance between distinct caption-text embeddings. If the held-out aligned distance is of the same order as the inter-caption distance, the decoder cannot reliably disambiguate captions; if it is significantly smaller, the representativeness assumption is supported. Repeat the anchor selection with 5 different random seeds to assess variance in the held-out distance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (1) trains a nonlinear MLP to minimize squared error between fMLP(image embedding) and the corresponding text embedding, but only for the K selected anchors. The text decoder is then trained purely on text embeddings of labels/captions. At inference, every test image is mapped through the same fMLP. The central claim requires that this K-point regression yields a mapping that is accurate on the entire test distribution. Nothing in the objective enforces that: in a high-dimensional embedding space, 500 points are sparse, and a two-hidden-layer MLP with 128 units can interpolate those anchors while behaving arbitrarily off-anchor. The paper provides no held-out alignment error, no nearest-neighbor accuracy of aligned test embeddings to their text labels, and no variance across anchor selections or training seeds. Figure 3 is a qualitative 2D embedding, not a quantitative measure. Absent such evidence, the reported downstream results could be explained by anchors that happen to lie near test points, especially if anchor selection is not explicitly restricted to the training split (the paper does not state this clearly). The broken 10% fully-supervised baseline and the lack of error bars further weaken the comparative evidence, but the unverified off-anchor generalization of fMLP is the load-bearing risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Surg-FTDA, a two-stage adaptation method for surgical multimodal foundation models. In the first stage, KMeans or Farthest Point Sampling selects K image embeddings from a downstream dataset; the corresponding text embeddings are retrieved, and an MLP is trained to minimize the L2 distance between image and text embeddings on those K anchor pairs. In the second stage, a GPT-2 decoder is trained with a frozen text encoder to reconstruct label texts or captions using only text strings. At inference, test image embeddings are passed through the frozen image encoder, then through the alignment MLP, and then through the decoder to generate labels or captions. The method is evaluated on phase recognition (Cholec80), triplet recognition (CholecT50), and image captioning (SVL-Caption) using both SurgVLP and CLIP, and compared with CapDec and fully supervised image-text finetuning. The central claim is that a text-trained decoder can generalize to visual tasks with only about 500 image-label pairs, approaching fully supervised performance.","tokens_in":9678,"tokens_out":5027,"duration_ms":49813,"significance":"If the central claim holds, the method offers a practical way to adapt surgical vision-language models to label-scarce tasks with a single transferable text decoder and a small alignment MLP. The paper's strengths include evaluation on two foundation models, three task types, comparison with CapDec, and a commitment to release code. It also connects to a relevant line of work on text-only training (CapDec, CLOSE). However, the empirical support has gaps that directly affect the central claim: no quantitative validation of the alignment MLP's off-anchor generalization, ambiguous anchor-split provenance, a broken 10% supervised baseline, and no error bars. The significance is therefore conditional on additional evidence that the alignment learned on a few anchors actually transfers to unseen test images.","major_comments":[{"comment":"The central inference step assumes that the MLP trained on K anchors generalizes to all test image embeddings, but no quantitative evidence is provided for this off-anchor behavior. The paper reports no held-out alignment error, no nearest-neighbor accuracy of aligned test embeddings to their corresponding label text embeddings, and no variation over anchor selections or training seeds; Fig. 3 is a qualitative 2D projection. Because a two-hidden-layer MLP with 128 units can fit 500 points while behaving arbitrarily in between them, the reported downstream gains are not yet attributable to genuine modality alignment. Please add quantitative alignment or retrieval metrics on held-out images and evaluate across seeds and anchor sets.","section":"Section 2.2, Eq. (1)"},{"comment":"It is not stated whether the KMeans/FPS anchor selection is performed only on the training split. The text refers to a 'downstream dataset with large-scale unlabeled images' and does not explicitly restrict anchors to the training set; if anchor images can come from the test split, then Eq. (1) is trained with test labels and the few-shot comparison is invalid. Please state the split explicitly, and if anchors were selected from the test split, rerun the experiments with training-only selection.","section":"Section 2.1"},{"comment":"The fully supervised model trained on 10% of the data yields all-zero metrics because it 'fails to generate phase outputs in the correct format under the same experimental settings.' This makes the comparison invalid as a measure of task performance: a formatting failure is not equivalent to low accuracy. Please tune the baseline (e.g., epochs, learning rate, decoding strategy) and report its best valid performance, or exclude it from the comparison with an explicit explanation.","section":"Section 3.3, Tables 7 and 8"},{"comment":"The decoding protocol for discriminative tasks is underspecified. The paper does not describe how free-form decoder outputs are mapped to fixed phase or triplet labels, nor whether constrained decoding is used. This is essential for reproducing the results and for diagnosing the baseline format failure mentioned in Section 3.3; please provide the exact decoding and parsing procedure.","section":"Section 2.3 and Section 3.2.1"},{"comment":"All experiments appear to be single runs with no error bars or seed variance. Given the stochasticity in the alignment MLP, GPT-2 fine-tuning, and anchor selection, comparative claims such as 'outperforms' and 'performance improves with more anchors' need confidence intervals or multiple seeds. Please add seeded repetitions and report mean and standard deviation for the main results.","section":"Tables 1-10"}],"minor_comments":[{"comment":"The notation for the aligned embedding is inconsistent (\\hat{v}^i_{image'} vs. \\hat{V}_{image'} and 'image’' with an apostrophe); please standardize the notation.","section":"Eq. (1) and surrounding text"},{"comment":"The sentence 'As shown in Tab. 4, Tab. 5, and Tab. 6, demonstrate that...' is ungrammatical; remove 'demonstrate that' or restructure the sentence.","section":"Section 3.3, first sentence"},{"comment":"The manuscript does not explain how CapDec is adapted to phase and triplet recognition, since CapDec is originally a captioning method; please clarify the setup so the comparison is meaningful.","section":"Section 3.2.1, Tables 1 and 2"},{"comment":"Figure 1(b) is referenced only in the caption and not explicitly cited in the main text; please add a callout.","section":"Figure 1"},{"comment":"The default batch size is stated as 34; if this is not a typo for 32, please comment on the choice.","section":"Section 3, model hyperparameters"},{"comment":"For SurgVLP, KMeans 100 reports accuracy 3.01 while KMeans 500 reports 27.97 and FPS 100 reports 10.28; the non-monotonic behavior and large jump deserve a brief comment.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a promising but not yet established few-shot adaptation result. The main risk is that the alignment MLP may not genuinely transfer to unseen test images; the authors should be pushed to provide retrieval or alignment metrics on held-out images and to clarify the anchor selection split. The paper fits the journal's scope if the evidence is strengthened along the lines of the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the Surg-FTDA paper. My take: this is a reasonable engineering extension of CapDec/CLOSE to surgical workflow analysis, with a few-shot anchor-selection MLP to bridge the modality gap. The core idea is not circular and the reported results are consistent with the method having some value. But the empirical support is weaker than the abstract suggests, mostly because of the handling of the low-data fully supervised baseline and the absence of any uncertainty quantification.\n\nWhat's actually new: pairing KMeans/FPS anchor selection with a two-hidden-layer MLP alignment, then training a GPT-2 decoder purely on text embeddings. The multi-task decoder with separate task-specific alignment MLPs is a nice touch. The evaluation covers three tasks and two foundation models, and the ablations compare sampling strategies and anchor counts. Releasing code and data is a plus.\n\nWhere it gets soft. First, the fully supervised 10% baseline collapses to all-zero metrics because it \"fails to generate outputs in the correct format under the same hyperparameters.\" The authors admit this, but then use that zero as evidence their method is better. That is not a fair comparison; with a few epochs of tuning the baseline would almost certainly produce nonzero numbers. Second, the stress-test concern is on point: the alignment MLP is trained on 500 anchors, and nothing in the objective guarantees it behaves sensibly off-anchor. The paper offers a qualitative 2D plot, not held-out alignment error, nearest-neighbor accuracy, or variance over anchor selections. Third, it's not clear from the text whether anchor selection is restricted to the training split; if it silently uses test images, that's a leak. Fourth, no error bars anywhere, so we can't tell if the multi-task gains (24.39 vs 23.87 F1) are noise.\n\nThe central claim—that a text-trained decoder can transfer to visual inputs with ~500 labeled pairs—is plausible and probably true in a coarse sense. But the current evidence doesn't nail it. The phase recognition result (55.33 vs 57.79 for the full-supervision model) is the strongest point in the paper's favor.\n\nFor whom? Surgical data scientists working on annotation-efficient adaptation. It deserves a serious referee—the idea is worth engaging—but the revision needs proper low-data baselines, error bars, a statement about anchor leakage, and ideally a held-out alignment metric.\n\nMy call: send to peer review, expect major revision.","headline":"Plausible text-only adaptation recipe for surgical tasks, but the empirical case is under-built: broken low-data baseline, no error bars, and the anchor-alignment generalization is asserted rather than tested.","tokens_in":10268,"tokens_out":2570,"would_cite":false,"duration_ms":27010,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes Surg-FTDA, a two-stage text-driven adaptation method that transfers pre-trained surgical vision-language foundation models to downstream tasks using only about 500 annotated image-label pairs, by training the decoder on…","keywords":["Foundation model","Surgical data science","Multi-modality learning","Surgical workflow analysis","Few-shot learning","Text-driven adaptation","Image captioning","Phase recognition"],"falsifier":"Take the same pretrained SurgVLP model and the same 500-pair budget, but replace KMeans or FPS anchor selection with random sampling of 500 image-label pairs, and retrain the MLP and decoder from scratch. If performance on phase and triplet recognition drops to near chance, the anchor-selection strategy is the load-bearing component. Alternatively, train a fully supervised classifier on the same 500 randomly selected pairs with per-task hyperparameter tuning; if it matches or beats Surg-FTDA's F1 scores, the text-driven component is not the source of the reported data efficiency.","tokens_in":9256,"feed_emoji":"🏥","tokens_out":10368,"duration_ms":82159,"temperature":0.7,"pith_summary":"The paper proposes Surg-FTDA, a two-stage adaptation method that lets a pre-trained surgical vision-language foundation model handle downstream tasks using very few paired image-label examples. The first stage selects a small, diverse set of 'anchor' images, clusters or samples them, and trains a small MLP to push their embeddings into the text-embedding space. The second stage trains a text decoder (GPT-2) using only text labels or captions, never images, so the decoder learns to produce the right output from an embedding. At inference, the aligned image embeddings are fed into that text-trained decoder. If the claim is right, surgical workflow analysis—phase recognition, action triplet recognition, and captioning—can be adapted with about 500 annotated pairs instead of thousands, with performance close to fully supervised training.","feed_headline":"Text-driven training lets 500 image pairs rival full supervision","feed_subtitle":"A decoder trained on text alone, plus a small alignment network, adapts surgical AI to phases, triplets, and captions with minimal…","key_machinery":"The load-bearing mechanism is the two-stage pipeline: (1) few-shot anchor selection via KMeans clustering or Farthest Point Sampling on frozen image embeddings, followed by an MLP trained with mean-squared error (Eq. 1) to map selected image embeddings to their corresponding text embeddings; (2) text-driven adaptation, where a GPT-2 decoder is fine-tuned to reconstruct downstream label texts or captions from embeddings produced by the frozen text encoder, with a reconstruction cross-entropy loss (Eq. 2). At inference the text encoder is swapped for the image encoder, and the MLP-aligned image embeddings are passed to the decoder. The key insight is that after alignment, the decoder cannot distinguish whether an embedding came from text or from a properly aligned image, so text-only training transfers to visual inputs.","core_discovery":"The central claim is that a decoder trained exclusively on text can act as a visual classifier and caption generator for surgical data, provided the visual embeddings are first aligned into the text embedding space by a simple MLP trained on a few selected anchors. The paper shows this by replacing the text encoder with the image encoder at inference time after alignment, and reports that Surg-FTDA approaches fully-supervised performance on Cholec80 phase recognition and CholecT50 triplet recognition, and is competitive on SVL-Caption, using only 500 image-label pairs. It further claims that a single decoder trained jointly on phase and triplet text outperforms task-specific decoders. The method is presented as a generalizable alternative to full fine-tuning when annotation is scarce.","pith_inferences":["The anchor-selection strategy is likely the main bottleneck of the pipeline: if the KMeans or FPS chosen anchors do not cover the test image distribution, the MLP will misalign out-of-distribution images, so we expect the choice of anchors to be at least as important as the alignment network's capacity.","A natural extension would be to apply the same text-driven adaptation to other medical imaging domains, such as radiology or pathology reports, where a vision-language foundation model exists and text annotations are cheaper than dense image labels.","The reported boost from a multi-task text decoder suggests that text-only training could scale toward a unified surgical language interface, with one decoder shared across phase recognition, triplet recognition, and captioning, and new tasks added by simply writing their label vocabularies.","A stronger test of the data-efficiency claim would pit Surg-FTDA against a fully supervised model whose hyperparameters are tuned specifically for the 500-pair regime; the paper's low-data fully supervised baselines were trained with the full-data hyperparameters, so they may understate the baseline's few-shot capability."],"forward_implications":["Fully supervised models trained on 10% to 50% of the surgical datasets are outperformed by Surg-FTDA using only 500 pairs, while Surg-FTDA lands only slightly below the 100%-data model.","Increasing the number of selected anchors from 100 to 500 consistently improves performance, and KMeans anchor selection generally beats Farthest Point Sampling on the reported tasks.","A single text decoder trained on mixed task text, encoding both phase and triplet labels, yields better decision boundaries than separate task-specific decoders.","The same text-driven pipeline works for both discriminative tasks, such as phase and triplet recognition, and generative tasks, such as image captioning, with the surgical-domain SurgVLP initialization giving better results than the general CLIP initialization.","Because the decoder is trained on text only, the vocabulary of possible outputs is not limited to the fixed set of image-label pairs used during adaptation, which the paper identifies as enabling generative open-vocabulary outputs like captioning."],"supporting_citations":[{"why":"Supplies the SurgVLP surgical vision-language foundation model whose frozen encoders are adapted by the proposed two-stage pipeline.","marker":"[25]"},{"why":"Provides the text-only training paradigm for image captioning that the text-driven decoder stage builds on.","marker":"[8]"},{"why":"Shows that visual tasks can be learned with only language supervision, the direct inspiration for the text-driven adaptation step.","marker":"[4]"},{"why":"CLIP is the general-purpose vision-language foundation model used as an alternative to SurgVLP to test the pipeline's generality.","marker":"[13]"},{"why":"GPT-2 is the text decoder that is fine-tuned in the text-only stage to reconstruct labels and captions.","marker":"[14]"},{"why":"CholecT50 provides the action triplet recognition task and its text labels used for decoder training and evaluation.","marker":"[9]"},{"why":"Cholec80 provides the phase recognition task and its text labels used for decoder training and evaluation.","marker":"[17]"},{"why":"KMeans clustering is one of the anchor-selection methods used to pick the few image embeddings that drive modality alignment.","marker":"[5]"},{"why":"Farthest Point Sampling is the alternative anchor-selection method compared against KMeans for choosing representative image embeddings.","marker":"[12]"}],"fun_headline_variants":["Text-only decoder trains surgical AI with just 500 pairs","Surgical AI reads text to learn from 500 image pairs","Align images to text: 500 pairs rival full supervision","Text-adapting foundation models for surgery with 500 labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the few hundred anchor images selected by KMeans or FPS are representative enough of all downstream images that the MLP trained on them will map every test image into the text embedding space closely enough for the text-only decoder to still output correct labels or captions.","fun_headline_variants_meta":{"raw":{"variants":["Text-only decoder trains surgical AI with just 500 pairs","Surgical AI reads text to learn from 500 image pairs","Align images to text: 500 pairs rival full supervision","Text-adapting foundation models for surgery with 500 labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1499,"prompt_tokens":925,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":517}},"tokens_in":541,"tokens_out":574,"duration_ms":6257,"temperature":1.0,"reasoning_tokens":517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:53:31.434643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same pretrained SurgVLP model and the same 500-pair budget, but replace KMeans or FPS anchor selection with random sampling of 500 image-label pairs, and retrain the MLP and decoder from scratch. If performance on phase and triplet recognition drops to near chance, the anchor-selection strategy is the load-bearing component. Alternatively, train a fully supervised classifier on the same 500 randomly selected pairs with per-task hyperparameter tuning; if it matches or beats Surg-FTDA's F1 scores, the text-driven component is not the source of the reported data efficiency.","supporting_citations":[{"cited_title":"I can’t believe there’s no im- ages! learning visual tasks using only language supervision, in: Pro- ceedings of the IEEE /CVF International Conference on Computer Vi- sion, pp","cited_arxiv_id":null,"evidence_quote":"Shows that visual tasks can be learned with only language supervision, the direct inspiration for the text-driven adaptation step."},{"cited_title":"Learning transferable visual models from natural language supervision, in: Inter- national conference on machine learning, PMLR","cited_arxiv_id":null,"evidence_quote":"CLIP is the general-purpose vision-language foundation model used as an alternative to SurgVLP to test the pipeline's generality."},{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"GPT-2 is the text decoder that is fine-tuned in the text-only stage to reconstruct labels and captions."},{"cited_title":"Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos","cited_arxiv_id":null,"evidence_quote":"CholecT50 provides the action triplet recognition task and its text labels used for decoder training and evaluation."},{"cited_title":"Endonet: a deep architecture for recognition tasks on laparoscopic videos","cited_arxiv_id":null,"evidence_quote":"Cholec80 provides the phase recognition task and its text labels used for decoder training and evaluation."},{"cited_title":"A k-means clustering algorithm","cited_arxiv_id":null,"evidence_quote":"KMeans clustering is one of the anchor-selection methods used to pick the few image embeddings that drive modality alignment."}],"review_version":1}