{"id":"e68583a8-39f1-423d-8f9e-9c0a19a97f30","arxiv_id":"2411.19757","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DRM fine-tunes CLIP with both true labels and soft labels derived from LLM-generated concept descriptions, improving OOD robustness on ImageNet and WILDS benchmarks.","lead":"This paper introduces DRM, a fine-tuning method that combines standard training with a worst-case-style regularizer built from text descriptions of class-defining features. The method improves out-of-distribution accuracy on ImageNet and WILDS compared to prior robust fine-tuning baselines, at about 20 percent extra training cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DRM-to-WRM bridge rests on identifying CLIP affinities to GPT-4 concept descriptions with an oracle core-feature model; this identification is validated only on 15 Hard ImageNet classes, leaving the mechanism behind the OOD gains under-supported.","rationale":"The reader's weakest_assumption identifies the same load-bearing point: the method assumes that pre-trained CLIP affinities to LLM concept descriptions estimate the oracle core-feature model pc(y|x), and this is validated only on a small Hard ImageNet study. This is the right concern because the theoretical development from IDRM to DRM is otherwise sound as a relaxation, but the replacement of max_d Rd(θ) with Rcs(θ) is not derived; it is a heuristic whose validity is exactly the empirical claim about core-feature isolation. The paper supplies useful supporting evidence: multi-seed results, ablations in Table 10 showing that concept descriptions outperform default prompts and one-hot proxies, and robustness to the choice of LLM in Table 4. Those results make it unlikely that the headline numbers are a fluke, but they do not establish that the mechanism is core-feature preservation rather than general distillation. Other potential concerns, such as the incomplete large-model comparison against concurrent baselines and the use of ID validation for selecting λ, are real but secondary: they affect the strength of the state-of-the-art claim, not the internal logic of the method. Since the reader already set CONDITIONAL, my assessment does not move the verdict; a failed version of the proposed test would justify moving toward REJECT, while a successful one would support ACCEPT after the remaining comparison gaps are filled.","tokens_in":26846,"tokens_out":4777,"duration_ms":45616,"concrete_test":"Run a controlled experiment on Hard ImageNet or a Waterbirds-style CLIP setup comparing three DRM variants: (a) faithful concept descriptions, (b) descriptions deliberately engineered to describe only non-core/background features, and (c) the un-normalized CLIP softmax from Eq. (15) as the proxy. If variant (b) retains the same OOD or worst-group improvement over FLYP as variant (a), the core-feature assumption is not load-bearing; if (b) and (c) collapse while (a) succeeds, the assumption is supported. Reporting worst-group accuracy, not only average OOD, would settle whether the proxy actually reduces non-core feature reliance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that minimizing Rcs(θ;Tcd) with soft labels p̃c(y|x) from Eqs. (8)-(9) implements worst-case risk minimization depends on the assumption that CLIP affinities to GPT-4 concept descriptions reliably isolate core visual features. The only direct supporting evidence is the 15-class Hard ImageNet foreground/background study in Table 6, and Appendix G concedes that GPT-4 produced inaccurate concept descriptions in medical imaging domains. The min-max normalization in Eq. (8) is a per-class rank transformation over training images, not a calibrated estimate of core-feature presence, and Eq. (9) can distribute probability mass to non-core classes whenever γ(x,yx)<1. Because Rcs(θ;Tcd) is the entire bridge from IDRM to DRM, a noisy or biased proxy would not refute the empirical OOD improvements over FLYP but would change the explanation for them: the gains could come from generic self-distillation or label smoothing rather than from targeted preservation of core features. This leaves the mechanism and the domain scope of the method under-supported, which is what makes the reader's CONDITIONAL verdict appropriate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes dual risk minimization (DRM) for fine-tuning zero-shot vision-language models such as CLIP. The objective adds to the standard empirical risk, computed with default prompts, a regularization term computed with LLM-generated concept descriptions; the soft targets for this second term are obtained from the pre-trained CLIP model and normalized per class. The authors argue that this regularizer estimates the worst-case risk over domains and thereby preserves core visual features. On ImageNet, WILDS-iWildCam, and WILDS-FMoW, DRM reports consistent OOD improvements over FLYP and other robust fine-tuning baselines, including with WiSE-FT, across three CLIP model sizes. The paper includes ablations on prompt choice, inference combination, normalization, LLM source, and the hyperparameter lambda.","tokens_in":27131,"tokens_out":4625,"duration_ms":44843,"significance":"If the findings hold, DRM is a practically valuable and simple addition to robust fine-tuning: it requires only an LLM-generated concept description per class and a precomputed soft-label set, and the reported gains on ImageNet variants, iWildCam, and FMoW are consistent across model sizes and seeds. The empirical evaluation is a genuine strength: five-seed runs with confidence intervals, ablations in Tables 3 and 10, sensitivity analyses for lambda and for different LLMs, and cost measurements in Appendix F.3. The central conceptual claim, however, that the concept-description soft-label risk is a faithful proxy for worst-case risk over domains, is asserted more than established; the formal derivation from IDRM to DRM has a gap. The empirical OOD improvements remain credible, but the paper's mechanistic explanation is stronger than the evidence supports.","major_comments":[{"comment":"The strong-duality argument is carried out in prediction space, not in parameter space. Lemma 1 shows convexity of the cross-entropy in the prediction function q(y|x), and the proof then applies Slater's condition to a convex program over q. However, the actual optimizations in (IDRM) and (DRM) are over theta, the parameters of a deep network, for which the objective is not convex. The conclusion that IDRM 'reduces to' min_theta [R_s(theta) + lambda* max_d R_d(theta)] is therefore not established for the parameterized problem. Since this reduction is the formal bridge from IDRM to DRM, please either prove duality under the specific parameterization, state explicit conditions under which it holds, or reframe Theorem 1 as a motivation in function space rather than a theorem about the training objective actually used.","section":"Section 3, Theorem 1 and Appendix A"},{"comment":"The identification of CLIP affinities to LLM concept descriptions with the oracle core-feature model p_c(y|x) is the load-bearing step of the method, but it is directly validated only on the 15-class Hard ImageNet study (Table 6 and Appendix D.1), and Appendix G concedes that GPT-4 produces inaccurate concept descriptions in medical imaging domains. The min-max normalization in Eq. (8) is a per-class rank transform over training images rather than a calibrated estimate of core-feature presence, and Eq. (9) distributes the residual probability mass to other classes in proportion to unnormalized affinities whenever gamma(x, y_x) < 1, so p_tilde_c can assign substantial mass on the basis of non-core similarity. The empirical OOD improvements over FLYP remain credible, but the specific claim that R_cs(theta; T_cd) is a worst-case-risk proxy is under-supported. Please provide additional evidence for the core-feature interpretation on the evaluated datasets, or soften the WRM claim and present R_cs as a regularizer motivated by core-feature preservation whose empirical value is established by the ablations.","section":"Section 4.2, Eqs. (8)-(9)"},{"comment":"The ablations convincingly show that the concept-description soft labels outperform one-hot labels and default-prompt soft labels as the regularization target, which is strong evidence against a purely generic-distillation explanation. However, the paper does not isolate the effect of the min-max normalization from the effect of the soft-label target: Row 10 of Table 3 shows that the unnormalized direct estimate fails badly, but there is no analysis of what the normalization does to the ranking of classes or to the entropy of p_tilde_c. A simple diagnostic, such as reporting the distribution of gamma(x, y_x) and the average entropy of p_tilde_c across datasets, would substantially strengthen the claim that Eqs. (8)-(9) estimate core-feature presence rather than merely rescale per-class affinities.","section":"Section 5.3, Table 3 and Table 10"}],"minor_comments":[{"comment":"The sentence 'The main challenge of applying DRM to real-world tasks is to assess of worst-case risk' contains a grammatical error; it should read 'is to assess the worst-case risk' or 'is the assessment of worst-case risk.'","section":"Section 1"},{"comment":"'It is also know that non-visual and spurious descriptions contribute significantly to CLIP's representation' should read 'It is also known that...'.","section":"Section 4.1"},{"comment":"'Quantitative study on the reliability of concept descriptions verse default prompts' should use 'versus' instead of 'verse'.","section":"Appendix D title"},{"comment":"The notation in the 'Affinity norm.' and 'w/ model' columns uses '!' and '%' symbols that are not defined in the caption; please define them or replace them with explicit labels.","section":"Table 3"},{"comment":"The rendering of Figure 1 in the submitted PDF is difficult to read: the inline text is fragmented and the diagram is not self-explanatory. Please ensure that the figure is legible and that its caption explains the relationship between ERM, WRM, and DRM without relying on the surrounding prose.","section":"Figure 1"},{"comment":"There are several typographical artifacts in the reference list, such as 'V ondrick' and 'V on Kügelgen'; these should be cleaned up before the camera-ready version.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical contribution with a clear practical payoff, and the ablation structure already addresses several natural alternative explanations. The main risk is that the theoretical framing overclaims: the strong-duality bridge is not proven for the parameterized objective, and the worst-case-risk interpretation of p_tilde_c rests on a thin validation set. I would be comfortable with acceptance after the authors either provide the missing theoretical conditions and additional proxy-validation evidence or explicitly reposition the theoretical material as motivation and the soft-label regularizer as an empirical design choice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First things first: this is a useful paper. DRM gives a simple, roughly 20% extra-cost recipe for fine-tuning CLIP while keeping OOD robustness, with consistent multi-seed gains on ImageNet variants and WILDS. The specific combination—ERM plus a regularizer whose targets are soft labels from pre-trained CLIP affinities to GPT-4 concept descriptions—is new. I believe the empirical result. The ablations are the right ones: they test single-risk variants, prompt choice, normalization, and the inference mixture, and they show that the full DRM setup is what works.\n\nThe soft spot is the theory, and it is exactly where the stress test points. The bridge from IDRM to DRM replaces max_d R_d(θ) with R_c_s(θ;T_cd), and that replacement is asserted rather than derived. The strong-duality proof is fine, but it is in prediction space; the actual optimization is over parameters of a non-convex network. And the estimate p̃_c(y|x)—min-max normalized affinities to concept descriptions—is a heuristic. The Hard ImageNet study is suggestive but covers only 15 classes. The paper's own limitation section concedes that GPT-4 descriptions fail in medical imaging. So if you read the paper as 'we minimize a worst-case-risk proxy,' the mechanism is under-supported. If you read it as 'we fine-tune with a self-distillation regularizer that helps OOD,' the evidence is solid. The authors themselves call it self-distillation at one point, so they are not hiding this.\n\nMinor issues: the exact GPT-4 descriptions are promised but not yet released, and the large-model table compares only against FLYP, not the concurrent baselines. Neither is close to fatal. The code is public, which counts.\n\nI would send this to a serious referee. The empirical claim is strong enough to deserve scrutiny, and a referee can reasonably push for either a sharper derivation of the WRM connection or a softer framing of the theoretical claim. The reader will get a practical method and an honest limitations section. The paper should be accepted after revision that re-frames the theoretical claim.","headline":"Solid empirical paper on robust fine-tuning; the worst-case-risk story is under-built, but the method works and the paper is honest about its limits.","tokens_in":27685,"tokens_out":1423,"would_cite":true,"duration_ms":14280,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning zero-shot models with dual risk minimization improves out-of-distribution robustness beyond FLYP and other baselines.","keywords":["dual risk minimization","robust fine-tuning","zero-shot models","CLIP","out-of-distribution robustness","concept descriptions","worst-case risk minimization","large language models"],"falsifier":"On a domain where LLM concept descriptions are known to be inaccurate, such as breast histology or ocular disease, compare DRM with FLYP under identical hyperparameter selection; if the OOD advantage disappears or reverses while in-distribution accuracy stays comparable, the concept-description proxy is the load-bearing part of the claim.","tokens_in":26636,"feed_emoji":"📈","tokens_out":10285,"duration_ms":82100,"temperature":0.7,"pith_summary":"This paper proposes dual risk minimization (DRM) as the training objective for fine-tuning zero-shot vision-language models, replacing the single empirical-risk objective with a sum of empirical risk and a worst-case-risk proxy. The proxy is a cross-entropy loss against soft labels computed by a pre-trained CLIP model when it is prompted with GPT-4 concept descriptions of each class's core visual features. DRM thereby preserves class-defining features while still fitting the downstream task, and the paper reports consistent out-of-distribution gains on ImageNet, WILDS-iWildCam, and WILDS-FMoW across three model sizes. A sympathetic reader would care because this is a concrete way to defend fine-tuning against distribution shifts without needing grouped domain data.","feed_headline":"DRM beats FLYP on OOD benchmarks after adding a worst-case loss","feed_subtitle":"Fine-tuning CLIP with two risks lifts ImageNet out-of-distribution accuracy to 77.1 and WILDS scores by up to 5 points.","key_machinery":"The object that carries the argument is the DRM objective (7), $$\\min_{\\$\\theta$} R_s(\\$\\theta$;$T^{{df}}$)+\\$\\lambda$ R^c_s(\\$\\theta$;$T^{{cd}}$),$$ where $R_s$ is the empirical risk with default prompts and $R^c_s$ is the core-feature risk. The core-feature risk is computed with respect to the soft labels $\\tilde{p}_c(y|x)$ defined in Eq. (9), which come from per-class min-max normalization in Eq. (8) of the pre-trained CLIP affinities between image embeddings and text embeddings of GPT-4 concept descriptions; the normalization suppresses artifact terms that would otherwise make the soft labels high-entropy. Inference uses a mixture $\\hat{p}^{dual}_\\theta=\\beta\\hat{p}^{df}_\\theta+(1-\\beta)\\hat{p}^{cd}_\\theta$. Theorem 1's strong-duality result justifies relaxing the constrained IDRM into this weighted sum, and the empirical study in Appendix D makes the case that concept-description affinities respond to foreground core features while staying stable under background changes.","core_discovery":"The central claim is that minimizing $R_s(\\theta;T^{df})+\\lambda R^c_s(\\theta;T^{cd})$—the standard ERM term supervised by one-hot labels using default prompts, plus a worst-case-risk term supervised by soft labels $\\tilde{p}_c(y|x)$ built from CLIP affinities to LLM-generated concept descriptions—preserves core features and improves robustness. The paper proves strong duality between the idealized constrained problem and the weighted dual, and then replaces the intractable worst-case risk over all domains with the core-feature risk on the training domain. On CLIP ViT-L/14@336, DRM raises OOD ImageNet accuracy from 75.9 to 77.1, WILDS-iWildCam macro F1 from 47.1 to 51.8, and WILDS-FMoW worst-region accuracy from 50.7 to 53.1 relative to FLYP plus WiSE-FT, while maintaining comparable in-distribution accuracy.","pith_inferences":["Beyond the paper's benchmarks, the method suggests a scaling rule for robust fine-tuning: any model with a text-interface zero-shot head could receive the same dual-prompt treatment, so the recipe may transfer from CLIP to other vision-language and multimodal models.","The fixed per-class min-max normalization is a natural place to look for further gains; a learned temperature or a calibrated version of $\\tilde{p}_c(y|x)$ could reduce sensitivity to $\\lambda$ on classes with weak or noisy concept affinities.","Given the paper's admitted failure of GPT-4 descriptions in medical imaging, an immediate testable extension is to generate concept descriptions with image-grounded multimodal LLMs or expert curation and check whether DRM's OOD advantage reappears in those domains.","Since DRM improves worst-case robustness without grouped domain labels, a head-to-head against domain-invariant methods on multi-domain benchmarks would separate the contribution of the core-feature proxy from the contribution of the dual objective itself."],"forward_implications":["DRM with WiSE-FT raises OOD ImageNet accuracy on CLIP ViT-L/14@336 from 75.9 to 77.1, WILDS-iWildCam macro F1 from 47.1 to 51.8, and WILDS-FMoW worst-region accuracy from 50.7 to 53.1.","On CLIP ViT-B/16, DRM's relative OOD gains over FLYP are about 5.0% on ImageNet, 12.4% on iWildCam, and 11.1% on FMoW; with WiSE-FT the gains are 1.9%, 11.6%, and 9.8%.","The ablation shows that using concept descriptions for both the ERM and the WRM term is worse than separating the prompts, so the dual-prompt design is load-bearing for the result.","DRM is not tied to one generation model: concept descriptions from GPT-3.5, GPT-4, and three Llama-3 sizes all improve OOD performance over FLYP on iWildCam.","Compared with FLYP, DRM adds roughly 20% to training and inference cost, from the extra text-encoder pass."],"supporting_citations":[{"why":"Supplies the CLIP zero-shot model and the affinity-based classifier that DRM fine-tunes and uses to compute soft labels.","marker":"Radford et al., 2021"},{"why":"FLYP, the main baseline and the ERM backbone DRM extends; contributes the fine-tuning recipe and the contrastive loss used for $R_s$.","marker":"Goyal et al., 2023"},{"why":"WiSE-FT, the weight-space averaging baseline whose combination with DRM yields the reported state-of-the-art OOD numbers.","marker":"Wortsman et al., 2022"},{"why":"LP-FT evidence that fine-tuning distorts pre-trained features, motivating the core-feature-preserving dual objective.","marker":"Kumar et al., 2022"},{"why":"GPT-4, the LLM that generates the concept descriptions used to build the worst-case-risk proxy.","marker":"Achiam et al., 2023"},{"why":"Earlier work on LLM-generated image descriptions that DRM adapts by explicitly asking for core visual features rather than generic image descriptions.","marker":"Pratt et al., 2023"},{"why":"WILDS, the benchmark suite that supplies iWildCam and FMoW evaluation sets used in the main experiments.","marker":"Koh et al., 2021"},{"why":"The worst-case risk minimization principle from which DRM's WRM term and the IDRM formulation are derived.","marker":"Wald, 1945"},{"why":"GroupDRO, a representative worst-case risk method contrasted with DRM because it needs grouped domain data.","marker":"Sagawa et al., 2020"},{"why":"Hard ImageNet provides the segmentations used in Appendix D to quantify that concept-description affinities respond to core features and ignore backgrounds.","marker":"Moayeri et al., 2022"}],"fun_headline_variants":["Two risks beat one: DRM boosts CLIP OOD accuracy","DRM fine-tuning: dual risk for stronger OOD robustness","LLM core-feature proxies enable worst-case risk in DRM","Dual risk minimization lifts CLIP robustness on real benchmarks","DRM: standard plus worst-case loss outperforms on OOD shifts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that CLIP's similarity scores for GPT-4 concept descriptions reliably track the class-defining visual attributes and ignore background and context; the soft labels built from those scores are only a trustworthy stand-in for worst-case risk if this holds.","fun_headline_variants_meta":{"raw":{"variants":["Two risks beat one: DRM boosts CLIP OOD accuracy","DRM fine-tuning: dual risk for stronger OOD robustness","LLM core-feature proxies enable worst-case risk in DRM","Dual risk minimization lifts CLIP robustness on real benchmarks","DRM: standard plus worst-case loss outperforms on OOD shifts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1650,"prompt_tokens":973,"completion_tokens":677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":588}},"tokens_in":589,"tokens_out":677,"duration_ms":5701,"temperature":1.0,"reasoning_tokens":588,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:50:59.553516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a domain where LLM concept descriptions are known to be inaccurate, such as breast histology or ocular disease, compare DRM with FLYP under identical hyperparameter selection; if the OOD advantage disappears or reverses while in-distribution accuracy stays comparable, the concept-description proxy is the load-bearing part of the claim.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP zero-shot model and the affinity-based classifier that DRM fine-tunes and uses to compute soft labels."},{"cited_title":"Fine-tuning can distort pretrained features and underperform out-of-distribution","cited_arxiv_id":null,"evidence_quote":"LP-FT evidence that fine-tuning distorts pre-trained features, motivating the core-feature-preserving dual objective."},{"cited_title":"Statistical decision functions which minimize the maximum risk","cited_arxiv_id":null,"evidence_quote":"The worst-case risk minimization principle from which DRM's WRM term and the IDRM formulation are derived."},{"cited_title":"Hashimoto, and Percy Liang","cited_arxiv_id":null,"evidence_quote":"GroupDRO, a representative worst-case risk method contrasted with DRM because it needs grouped domain data."},{"cited_title":"Hard imagenet: Segmentations for objects with strong spurious cues","cited_arxiv_id":null,"evidence_quote":"Hard ImageNet provides the segmentations used in Appendix D to quantify that concept-description affinities respond to core features and ignore backgrounds."}],"review_version":1}