{"id":"1caacf24-736a-4c14-86a6-19cd3d7841ed","arxiv_id":"2412.12504","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DaRL improves LLM relevance ranking by augmenting training data with OOD-detected samples, applying multi-stage fine-tuning, and calibrating overconfident predictions.","lead":"DaRL is a fine-tuning framework for LLM-based search relevance that combines out-of-distribution sample selection, multi-stage fine-tuning, and a confidence-calibrating loss. The authors report offline gains and online deployment in Alipay insurance search, but the OOD test set is built using the same selection module being evaluated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD generalization is measured on a test set built by the same DASA selection rule used to construct the augmented training set, so the claimed OOD gains may be selection-matched rather than general.","rationale":"Reader verdict is CONDITIONAL, and I agree with that disposition, but for a slightly different reason. The reader's weakest_assumption flags the Gaussian model in Eq. 2; my reading is that the more load-bearing issue is the construction of T_ood by the same selection rule used to create D_aug. Even a perfectly Gaussian representation would not fix the fact that the OOD benchmark is distributionally aligned with the augmented training set. The alpha selection on T_ood compounds the problem, since the peak OOD F1 in Fig. 4 may reflect test-set tuning. The ID results (Table 3 Exp1-Exp4) and the DASA-vs-random curve (Fig. 3) provide real support for the recipe as an in-distribution and data-selection technique, and the online A/B test is genuine deployment evidence, though it compares DaRL-0.3B to BERT-325M, not to a fine-tuned LLM, so it does not isolate the OOD contribution. Therefore I would not reject; I would keep the conditional acceptance, with the condition that an independent OOD benchmark confirm the generalization claim.","tokens_in":13470,"tokens_out":4761,"duration_ms":44407,"concrete_test":"Construct an independent OOD test set from a later time window or from held-out product categories in production logs, without applying Eq. 4; select alpha and the DASA thresholds using only ID validation data; rerun the Table 2 comparison between DaRL-0.3B and ProMoT-0.3B. If the DaRL advantage on this independent set is not significant at p<0.05, the OOD claim rests on selection-matched evaluation; if the advantage persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1.1 defines T_ood as 'constructed according to the DASA module,' while D_aug is built by DASA from the same unlabeled pool U (Eq. 4, Eq. 5). The OOD test set is therefore not an independent distribution shift: it is the distribution induced by the same Mahalanobis-plus-kNN selection criterion, with thresholds derived from the same training representation. A model trained on D_aug is evaluated on a distribution it was explicitly trained to cover, which can inflate T_ood scores without demonstrating robustness to other real-world shifts (temporal drift, new query intents, unseen item categories). In addition, the LPFT interpolation coefficient alpha is selected by maximizing F1 on T_ood (Fig. 4), so the reported OOD result embeds test-set hyperparameter tuning. This does not invalidate the ID-side results, the DASA-vs-random comparison in Fig. 3, or the online A/B evidence, but it means the central OOD generalization claim is not certified by the current benchmark. The reader's Gaussian assumption is a secondary risk: even if the representation were exactly Gaussian, the evaluation would still be selection-matched.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DaRL, a framework for fine-tuning LLMs for graded relevance modeling in e-commerce search (Alipay insurance search). DaRL has three components: (1) DASA, which uses Mahalanobis and kNN cosine distances to select out-of-distribution samples from an unlabeled pool for annotation and augmentation; (2) a multi-stage fine-tuning strategy LPFT that first trains a classification head with LoRA, then fine-tunes all parameters, and finally interpolates the two sets of weights; and (3) an over-confidence calibration (OCC) auxiliary loss that uses KL divergence to reshape the predicted score distribution toward a hand-defined prior per relevance grade. The authors report offline experiments on industrial data showing improvements in F1 and accuracy on in-distribution, out-of-distribution, and production test sets, plus online A/B testing results with improved PV-CTR, PV-CVR, and manual relevance metrics, and state that DaRL-0.3B has been deployed.","tokens_in":13736,"tokens_out":2501,"duration_ms":23309,"significance":"If the claimed OOD generalization improvements hold, the paper would make a useful practical contribution: it addresses a real problem (LLM fine-tuning degrading under distribution shift in search relevance) and combines data augmentation, multi-stage training, and calibration into one deployable system. The paper's strengths include real industrial data, a deployed system with online A/B evidence, and an ablation study that isolates the contribution of each component. However, the central OOD generalization claim is not certified by the current benchmark: the OOD test set is constructed by the very same DASA selection rule used to build the augmented training set, and the LPFT interpolation coefficient is tuned on that same OOD test set. This is a load-bearing issue because the paper's headline contribution is robustness to distribution shift. The ID-side results, the DASA-versus-random comparison, and the online A/B evidence are less affected, but the quantitative OOD improvements in Table 2 and Table 3 should be interpreted with caution.","major_comments":[{"comment":"The OOD test set T_ood is not an independent distribution shift: §5.1.1 states that T_ood is 'constructed according to the DASA module,' and Eq. (4) shows that DASA selects samples for D_aug using the same Mahalanobis and kNN cosine thresholds derived from the training representation. A model trained on D_aug is therefore evaluated on a distribution that was explicitly selected to match the same criterion used for augmentation. This selection-matched evaluation can inflate T_ood gains without demonstrating robustness to other real-world shifts such as temporal drift, new query intents, or unseen item categories. To certify the OOD generalization claim, the authors should evaluate on an OOD test set constructed independently of the DASA selection rule, e.g., time-based splits, query-intent splits, or item-category splits.","section":"§5.1.1, Eq. (4), Eq. (5)"},{"comment":"The interpolation coefficient alpha in Eq. (8) is selected by maximizing F1 on the OOD test T_ood (Fig. 4 reports F1 on T_ood as a function of alpha, with the optimal value at alpha = 0.6). This means the reported T_ood numbers embed test-set hyperparameter tuning, which is a form of leakage and further weakens the claim that the method improves OOD generalization. The authors should select alpha (and any other hyperparameters) using a validation split that is disjoint from T_ood, or report T_ood results for a pre-specified alpha.","section":"§5.2.4, Fig. 4, Eq. (8)"},{"comment":"The target distribution q(x) in Eq. (9) is not fully specified. The paper says q(x) is 'over the labels (0, 1)', but then presents intervals such as [1−ρ, ρ] for IR and [2ρ, 1−2ρ] for WR. For the default ρ = 0.1, these intervals are [0.9, 0.1] and [0.2, 0.8]; the first is written in descending order and appears to be a typo (likely should be [ρ, 1−ρ] if the label for IR is 0). More importantly, the KL divergence L_kl = KL(p(x), q(x)) in Eq. (10) is asymmetric, and it is unclear whether q(x) is a proper probability distribution over the binary label or over the score interval. If q is intended as a target distribution over the scalar score, the KL direction and the normalization need clarification. The authors should specify the exact form of p(x), the intended label convention, and why this particular set of target intervals is appropriate for graded relevance.","section":"§4.3, Eq. (9)"},{"comment":"The Gaussian assumption P_id = N(x̄, Σ) is a strong modeling assumption for the representation distribution. If the training representation is multimodal or non-Gaussian, Mahalanobis distance may not be a valid OOD score, and DASA may select uninformative or even adversarial samples. The paper does not provide any diagnostic evidence for Gaussianity, nor does it compare against non-parametric OOD scores beyond the kNN term already used. This is secondary to the selection-matched evaluation issue, but it affects the interpretation of the DASA ablation (Exp3 and Fig. 3); a robustness check on the choice of OOD score would strengthen the paper.","section":"§4.1, Eq. (2)"}],"minor_comments":[{"comment":"Table 2 reports no variance or confidence intervals for any metric except the significance asterisks. Since the paper claims statistically significant improvements (t-test, p < 0.05), the reader cannot assess the stability of the results; reporting mean and standard deviation over multiple runs would be more informative, especially for the small OOD improvements (e.g., 0.6678 vs 0.6431 F1 on T_ood).","section":"Table 2"},{"comment":"The online A/B test compares DaRL against a BERT-325M baseline rather than a fine-tuned LLM baseline such as AntGLM or ProMoT. This is a weaker comparison for the paper's central claim that DaRL helps LLM-based relevance modeling; the offline comparison against ProMoT is more informative, and the online section should acknowledge this discrepancy or provide an online comparison against an LLM baseline if available.","section":"§5.3"},{"comment":"The description of T_prod is vague: 'comprising samples from T_id and T_ood, along with long-tail samples' makes it unclear how the production test set relates to the other two test sets and whether it overlaps with the augmented training set. Please clarify the construction and overlap of T_prod.","section":"§5.1.1, Table 1"},{"comment":"The symbol phi is reused: Eq. (8) defines phi as the interpolated weights, but Eq. (6) and Eq. (7) use phi_lp and phi_ft. This is not an error, but the notation could be made more distinct to avoid confusion, especially because Eq. (8) is central to the LPFT module.","section":"§4.2, Eq. (8)"},{"comment":"The threshold d1 and d2 are said to be computed using 'the best ID F1 scores on the validation set', but the validation set is not described in Section 5.1. Please specify the validation set construction and how exactly the thresholds are derived from it.","section":"§4.1, Eq. (4)"},{"comment":"Equation (1) defines the objective as a sum of ID and OOD expected losses, but it is not used subsequently in the method section (L_ce and L_kl appear in Eq. (10) without reference to the two-distribution objective). The connection between the formulation and the actual training loss could be made explicit.","section":"§3, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is from an industrial group with deployment evidence, which is valuable. The main concern I would raise with the editor is that the headline OOD generalization claim is supported by a circular evaluation design (same selection rule for both training augmentation and test-set construction) and by test-set hyperparameter tuning (alpha). This is fixable with additional experiments on an independent OOD split, but as it stands the central claim is not established. The ID-side results and the ablation are plausible and internally consistent. The paper would be a reasonable fit for the conference if the authors can provide a non-circular OOD evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: DaRL is a sensible industrial recipe and the in-distribution results look solid, but the OOD generalization claim is weaker than it appears because the OOD test set is selected by the same DASA module the paper is evaluating.\n\nWhat's actually new: the specific combination of grade-dependent KL target distributions (Eq. 9), distribution-aware sample augmentation via Mahalanobis + kNN distances, and a three-stage LPFT with weight interpolation. Each piece is borrowed, but the packaging for fine-grained relevance is new, and the authors are upfront about the Gaussian assumption behind the Mahalanobis score.\n\nWhat the paper does well: The ablation study is clean and incremental; Exp1 to Exp4 isolate each component and the DASA-vs-random comparison in Fig 3 is a nice demonstration that selective augmentation beats random sampling at the same labeling budget. The online A/B testing, even against a BERT baseline, gives some evidence the pipeline works in production, and the manual evaluation details are transparent.\n\nThe soft spots: The main one is the construction of T_ood. Section 5.1.1 says it is 'constructed according to the DASA module,' meaning it is drawn from the same distribution as the D_ood samples that were added to the training set. So the OOD test is not a blind generalization test; it's a selection-matched test. The model is evaluated on the exact kind of samples it was trained to cover. The 6-point OOD gain from DASA in Exp3 is likely inflated. Related but separate: the interpolation coefficient alpha is chosen by maximizing F1 on T_ood (Fig 4), which is test-set tuning. That should have been done on a validation split or shown to be robust across a range. Also, Table 2 reports no variance, and the online baseline is BERT, not a fine-tuned LLM, so the online claim doesn't prove superiority over the actual previous best model. These issues don't invalidate the ID-side results or the relative ranking of methods on the same test sets, but they undercut the central claim of OOD robustness.\n\nFor whom: this paper is useful for practitioners building LLM-based relevance models in industrial search, and for anyone thinking about how to evaluate OOD robustness in a data augmentation context. The circularity lesson is valuable.\n\nRecommendation: worth a serious referee, but I would ask for an independent OOD test set (e.g., temporally shifted data or manually curated novel intents) and a pre-registered alpha selection. If that is out of scope, the authors should moderate the OOD claim and frame T_ood as a difficulty-matched subset rather than an out-of-distribution generalization benchmark.","headline":"DaRL is a sensible industrial recipe with solid ID results, but the OOD test set is selected by the same DASA module being evaluated, so the OOD generalization claim is not yet proven.","tokens_in":14251,"tokens_out":5667,"would_cite":false,"duration_ms":45555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DaRL, a distribution-aware robust learning framework, improves LLM-based relevance modeling on both in-distribution and out-of-distribution data, and its deployed version lifts Alipay insurance search's valid PV-CTR by 1.2% and PV-CVR by…","keywords":["Search Relevance","Relevance Modeling","Large Language Model","Out-of-distribution Generalization","Distribution-Aware Sample Augmentation","Fine-tuning","E-commerce Search","Mahalanobis Distance"],"falsifier":"Run DASA (Eq. 4) on a public query-item relevance dataset whose training representation distribution is visibly multi-modal (e.g., estimated by clustering or a normality test), then compare OOD F1 against training on the same number of randomly selected labeled samples; if the random selection matches or beats DASA, the Gaussian assumption is load-bearing for the result. A second check: replace the Mahalanobis term with a non-parametric density estimate and see whether the OOD F1 gain on the Alipay-style data persists.","tokens_in":13278,"feed_emoji":"🔍","tokens_out":10425,"duration_ms":79835,"temperature":0.7,"pith_summary":"This paper claims that the two known weaknesses of fine-tuned LLMs in search relevance modeling—overconfidence on fine-grained relevance grades and performance collapse under distribution shift—can be addressed together by a distribution-aware robust learning framework called DaRL. DaRL combines three mechanisms: DASA selects unlabeled samples that the original training distribution poorly covers and adds them to the fine-tuning set; LPFT first probes a classification head, then full-fine-tunes, then interpolates the two weight sets; OCC regularizes the score distribution toward a smoothed prior. On Alipay insurance product search data, DaRL-0.3B improved out-of-distribution F1 from 0.6431 to 0.6678 and in-distribution F1 from 0.9252 to 0.9349 over the strongest baseline, with larger models gaining more; online A/B testing showed a 1.2% PV-CTR and 3.04% PV-CVR lift. The paper thus establishes that data-level augmentation guided by OOD detection plus robust fine-tuning is a practical path to closing the ID–OOD gap without scaling up model size.","feed_headline":"LLM relevance F1 up 2.5 points on out-of-distribution data","feed_subtitle":"Distribution-aware data selection plus multi-stage fine-tuning closes the out-of-distribution gap in Alipay's live search.","key_machinery":"The load-bearing machinery is the interaction of three named mechanisms. DASA uses Mahalanobis distance under a Gaussian assumption on the training representations together with a cosine-distance nearest-neighbor score to flag out-of-distribution unlabeled samples, which are then labeled and merged into the training set. LPFT freezes the backbone to train a classification head, fine-tunes all parameters, and interpolates the two resulting weight vectors to avoid catastrophic forgetting. OCC adds a KL-divergence auxiliary loss that regularizes predicted relevance scores toward a hand-set smoothed prior distribution for the three relevance grades. Together they convert fine-tuned LLMs into models that keep in-distribution accuracy while gaining out-of-distribution robustness.","core_discovery":"DaRL is built from three components, each targeting a different failure mode of vanilla LLM fine-tuning. DASA (Distribution-Aware Sample Augmentation) assumes the training representations form a Gaussian and scores each unlabeled search-log sample by Mahalanobis distance plus cosine distance to its nearest training neighbor; samples exceeding both thresholds are labeled and added to the training set, giving the model exposure to the long tail of the query-item distribution. LPFT (Linear-Probing then Fine-Tuning, adapted from prior work) freezes the backbone and trains only a classification head, then fine-tunes all parameters, then linearly interpolates the two weight vectors, which reduces feature distortion and catastrophic forgetting. OCC (Over-Confidence Calibration) adds a KL-divergence loss pulling the predicted score distribution toward a smoothed prior for strong, weak, and irrelevant labels, preventing the model from collapsing all weak-relevance predictions to a single extreme. The paper shows on Alipay insurance search data that each module contributes to the gain, with DASA contributing the largest OOD improvement (about 6 F1 points alone), and that the deployed DaRL-0.3B model improved valid PV-CTR by 1.2% and PV-CVR by 3.04% relative to the BERT baseline in online A/B testing.","pith_inferences":["Because DASA scores samples with the pre-fine-tuning model's representations, the selected 'OOD' set is relative to the initial model; an iterative variant that re-runs DASA after each fine-tuning round (or after LPFT's interpolation step) could yield further OOD gains, but the paper does not explore this.","The Gaussian assumption in Eq. 2 is the framework's most fragile premise; on representation spaces that are naturally multi-modal (mixed item categories, domains, or languages), a non-parametric OOD score or a Gaussian-mixture alternative might change which samples DASA picks, and could either improve or degrade the final model.","The KL prior in Eq. 9 is defined for three relevance grades with a smoothing factor rho tuned by grid search; extending the same calibration idea to four-point or five-point grading scales would require redefining the prior and probably re-tuning rho, which is a simple but untested generalization.","The online gains (CTR, CVR) also reflect Alipay-specific traffic and labeling; transferring DaRL to other e-commerce verticals or to web search would test whether the ID–OOD gap behavior replicates."],"forward_implications":["DaRL-0.3B is the deployed model in Alipay insurance search; in online A/B tests it lifted valid PV-CTR by 1.2% and PV-CVR by 3.04% relative to the BERT baseline.","The DASA module lets a labeling budget go further: distribution-aware sample selection yields larger OOD F1 gains than the same number of randomly selected labeled samples (Fig. 3).","LPFT's weight interpolation at alpha = 0.6 yields better OOD F1 than either linear probing alone or full fine-tuning alone (Fig. 4), indicating a sweet spot between feature preservation and task adaptation.","OCC reduces score overlap between weak and strong relevance predictions (Fig. 2), which is what enables a downstream trained-to-rank layer to use the scores as graded signals.","Model scale still matters: DaRL-2B and DaRL-10B beat DaRL-0.3B on all metrics, but latency and compute made the 0.3B model the industrial choice."],"supporting_citations":[{"why":"Provides the Mahalanobis-distance OOD scoring method and the Gaussian assumption on which DASA's sample selection rests.","marker":"[19]"},{"why":"Supplies the deep nearest-neighbor distance used as DASA's local OOD score in Eq. (3).","marker":"[36]"},{"why":"Contributes the original LPFT strategy (linear probing then fine-tuning) that DaRL adapts into its multi-stage training.","marker":"[46]"},{"why":"Establishes that fine-tuning can distort pretrained features and hurt OOD performance, the motivation for LPFT.","marker":"[18]"},{"why":"LoRA is used in the first LPFT stage to train a good classifier head efficiently.","marker":"[14]"},{"why":"The PET verbalizer approach is both the baseline model and the source of the overconfidence problem that OCC targets.","marker":"[34]"},{"why":"ProMoT is the strongest baseline (two-stage prompt-then-fine-tune) that DaRL-0.3B must outperform on ID and OOD F1.","marker":"[39]"}],"fun_headline_variants":["Alipay search: robust LLM learning lifts OOD F1 by 2.5 pts","Distribution-aware robust fine-tuning boosts LLM relevance on OOD","DaRL: distribution-aware data selection plus multi-stage tuning wins","Alipay insurance search: DaRL lifts PV-CVR by 3.04%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The DASA module assumes the representation space of the training data is a single Gaussian, so a sample's Mahalanobis distance is a reliable score for how out-of-distribution it is; if the true representation manifold is multi-modal or non-Gaussian, DASA may select samples that are not actually informative and the reported OOD gains may shrink.","fun_headline_variants_meta":{"raw":{"variants":["Alipay search: robust LLM learning lifts OOD F1 by 2.5 pts","Distribution-aware robust fine-tuning boosts LLM relevance on OOD","DaRL: distribution-aware data selection plus multi-stage tuning wins","Alipay insurance search: DaRL lifts PV-CVR by 3.04%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000521,"raw_usage":{"total_tokens":2604,"prompt_tokens":1110,"completion_tokens":1494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":1409}},"tokens_in":726,"tokens_out":1494,"duration_ms":11917,"temperature":1.0,"reasoning_tokens":1409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:00:41.757279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DASA (Eq. 4) on a public query-item relevance dataset whose training representation distribution is visibly multi-modal (e.g., estimated by clustering or a normality test), then compare OOD F1 against training on the same number of randomly selected labeled samples; if the random selection matches or beats DASA, the Gaussian assumption is load-bearing for the result. A second check: replace the Mahalanobis term with a non-parametric density estimate and see whether the OOD F1 gain on the Alipay-style data persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Mahalanobis-distance OOD scoring method and the Gaussian assumption on which DASA's sample selection rests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the deep nearest-neighbor distance used as DASA's local OOD score in Eq. (3)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the original LPFT strategy (linear probing then fine-tuning) that DaRL adapts into its multi-stage training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that fine-tuning can distort pretrained features and hurt OOD performance, the motivation for LPFT."},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"LoRA is used in the first LPFT stage to train a good classifier head efficiently."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The PET verbalizer approach is both the baseline model and the source of the overconfidence problem that OCC targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ProMoT is the strongest baseline (two-stage prompt-then-fine-tune) that DaRL-0.3B must outperform on ID and OOD F1."}],"review_version":1}