{"id":"eb797fcf-17e6-4063-8bb1-9bc06c3068ad","arxiv_id":"2608.05857","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Embedding models rank query-chunk pairs similarly but score them on systematically different scales; isotonic regression learned on synthetic query-chunk pairs can partly align those scales for threshold transfer.","lead":"This paper introduces Synthetic Query Probing, a method that uses an LLM to generate queries from document chunks, and uses it to compare similarity scores across embedding models. It finds that models agree on ranking but use very different score scales, and learned mappings can partly transfer thresholds across models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of improved threshold portability is never directly tested: mappings are fit and evaluated on the full dataset, with no held-out transfer of a converted threshold.","rationale":"Reading the paper in good faith, SQP is a reasonable methodology for obtaining paired query-chunk scores across models, and the distributional analysis is interesting. The claim that thresholds can be transferred, however, is the load-bearing part of the abstract, and it is not directly supported. No experiment moves a threshold from one model to another; conversion functions are fit and evaluated in-sample. This is a correctness risk that is independent of the synthetic-label validity concern: even if the labels are perfect, the mapping's calibration and its effect on threshold transfer at a fixed operating point are unmeasured. The paper explicitly flags the train-test limitation in Section 5, so the issue is acknowledged, but acknowledging it does not make the headline claim true. The reader's identified weakest assumption (synthetic queries as a proxy) is a legitimate secondary concern, but the more load-bearing gap is the absent end-to-end threshold-transfer evaluation. The proposed check—a held-out transfer of a converted threshold—would settle the question. The paper also lacks error bars, code/data release, and uses only 100 chunks per corpus, but these are secondary. Given the explicit limitation, the reader's CONDITIONAL verdict is appropriate; our concern does not change the verdict but sharpens the condition: the central claim should be reworded or the threshold-transfer experiment performed.","tokens_in":11850,"tokens_out":3857,"duration_ms":33014,"concrete_test":"Run a threshold-transfer experiment on the existing SQP data. Split the query-chunk pairs by chunk (e.g., 70% of chunks for training, 30% held out) so that no pair sharing a chunk appears in both sets. On the training set, fit each mapping f from model A (e.g., Titan 1024) to model B (e.g., Ada-002) and calibrate a threshold τ_A on A that achieves precision ≥ 0.95. Convert to τ_B = f(τ_A). On the held-out set, measure precision, recall, and F1 for B using τ_B, and compare against B's own directly calibrated threshold and against no conversion (naive reuse of τ_A). Report the degradation. If the converted threshold does not approximately match the directly calibrated operating point, the threshold-portability claim fails. This test directly targets the central claim and can be run on the authors' existing data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that learned mappings 'improve threshold portability.' For this to hold, one must show that a threshold calibrated on model A, after conversion through a learned mapping, yields acceptable retrieval performance on model B's score scale, ideally on data not used to fit the mapping. Section 4.5 does not do this: it fits linear, isotonic, and quantile mappings on the full 3,000 query-chunk pairs per corpus and reports MAE/R^2 on the same data. Section 4.3 computes per-model thresholds (Table 2) but never converts one model's threshold to another or measures the resulting precision/recall. The strongest observed evidence is that score distributions are related with high R^2 (e.g., 0.945 for 1024→Ada on SciFact). High R^2 on the training data does not imply that a converted threshold, applied at a specific operating point, preserves precision at the desired level; small systematic errors in the mapping can shift the effective threshold substantially in a steep region of the precision-recall curve. The paper's own Section 5 concedes: 'conversion functions are fitted on the full dataset and should be extended with proper train-test protocols for reusable calibration.' This explicit limitation directly undermines the headline claim as stated. The claim should be weakened to 'mappings can be fit with high accuracy on SQP data' until a threshold-transfer experiment is performed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Synthetic Query Probing (SQP), a reference-free method that uses an LLM to generate PARAPHRASE, RELEVANT, and IRRELEVANT queries for sampled chunks, yielding 3,000 labeled query–chunk pairs per corpus. Using these pairs, the authors compute cosine scores for four embedding configurations (Amazon Titan Text Embeddings V2 at 256/512/1024 dimensions and OpenAI text-embedding-ada-002) on SciFact and a proprietary Pegasystems corpus. They analyze per-class score distributions, derive precision-based thresholds for each configuration, and fit linear, isotonic, and quantile mappings between all 12 directional model pairs, reporting R^2 and MAE. The central claim is that learned mappings partially align similarity spaces and improve threshold portability, with isotonic regression performing best.","tokens_in":12123,"tokens_out":6931,"duration_ms":57605,"significance":"The problem of cross-model score comparability is real and practically important for RAG migrations, and SQP is a simple, scalable, annotation-free way to collect paired scores across models. The distributional results (e.g., Ada's compressed score range, the Titan dimensionality gradient, and preserved class ordering) are informative and could be useful to practitioners. However, the headline claim of improved threshold portability is not directly tested: all conversion functions are fitted and evaluated on the same full dataset, and no threshold-transfer experiment is performed. The paper's contribution is therefore a promising methodology and a set of descriptive findings, not yet a validated solution for threshold reuse.","major_comments":[{"comment":"The conversion functions in Section 4.5 are fitted on the full 3,000 query–chunk pairs per corpus and evaluated on the same data; Table 3 therefore reports in-sample R^2 and MAE, not predictive accuracy. The abstract's claim that mappings 'improve threshold portability' is never tested directly: no experiment converts a threshold calibrated on one model through a learned mapping and measures precision/recall on held-out data. Section 5 explicitly concedes this ('conversion functions are fitted on the full dataset and should be extended with proper train-test protocols for reusable calibration'). The authors should add a held-out or cross-validated threshold-transfer experiment, or weaken the claim to 'mappings can be fit accurately on SQP data.'","section":"§4.5, Table 3, §5"},{"comment":"The precision operating thresholds in Table 2 are selected by sweeping Equation (1) over the full SQP set, and the reported precision/recall/F1 are computed on the same data. This makes the thresholds in-sample by construction. If the goal is to demonstrate portability, the threshold calibration and evaluation should be separated (e.g., fit thresholds on a training split, evaluate on a test split, and report variability). As it stands, the argument that 'naively reusing a 1024-d threshold at lower dimensionalities would lead to a noticeable drop in precision' is based on thresholds and precisions both measured on the same set.","section":"§4.3, Table 2"},{"comment":"The ground-truth relevance labels are generated automatically by Claude Sonnet 4.6 with no human verification, and all subsequent analyses (distributions, thresholds, mappings) treat these labels as ground truth. The paper cites [2] for the predictive validity of synthetic benchmarks, but a small human-checked subset or a comparison against a standard retrieval benchmark would reduce the risk that the observed score relationships are specific to the query generator rather than to real RAG traffic. I raise this as a correctness-risk concern, not as a claim that the method is circular.","section":"§4.1, §5"}],"minor_comments":[{"comment":"The quantile mapping columns report only MAE, not R^2, although the text claims isotonic regression achieves the highest R^2 across all methods. Please include R^2 for quantile mapping or explain the omission.","section":"Table 3"},{"comment":"The set-builder notation for the threshold sweep is garbled; please correct the LaTeX so that T_mi is clearly defined.","section":"Equation (1)"},{"comment":"The statement that R^2 is 'identical in both directions' for the 1024 ↔ Ada pair holds for linear regression (0.841) but not for isotonic regression (0.945 vs. 0.889 in Table 3a). Please clarify that the symmetry claim applies only to the linear model.","section":"§4.5"},{"comment":"A data/code availability statement is missing; providing the SQP dataset or a reproducible pipeline would strengthen the paper.","section":"Overall"}],"recommendation":"major_revision","confidential_remarks":"The central problem is that the abstract and conclusion claim threshold portability, while Section 5 admits the conversion functions are in-sample. A held-out transfer experiment is the single most important revision. The synthetic-query validity concern is secondary but worth a quick human check. If the authors add the experiment, I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a genuinely useful empirical study of how cosine similarity scores differ across embedding models, and it introduces a practical methodology for studying that. But the headline claim—that learned mappings \"improve threshold portability\"—is not directly supported by the experiments as run. The paper itself concedes this in Section 5, which is honest but means the abstract overstates what was shown.\n\nWhat's new and good: Synthetic Query Probing is a simple, reference-free way to generate controlled query-chunk pairs with paraphrase/relevant/irrelevant labels, and the paper uses it to characterize score distributions across four embedding configurations on two corpora. The distributional findings are solid and clearly presented: Titan dimensionalities produce a consistent upward shift as dimensions decrease, Ada compresses scores into a narrow high band, ranking structure is preserved across models, and class separability drives conversion quality. The cross-model S-shaped nonlinearity and the directional asymmetry in error (Ada to Titan has higher MAE than the reverse) are real observations, and the comparison of linear, isotonic, and quantile mappings is reasonable. Within the Titan family, the near-lossless conversions are a useful practical result.\n\nThe soft spots are concentrated in the evaluation protocol. The conversion functions are fitted and evaluated on the full dataset, so the reported R^2 and MAE are in-sample goodness of fit, not predictive accuracy. There is no experiment that takes a threshold calibrated on one model, converts it to another model's scale via the learned mapping, and measures precision/recall on held-out data. That is the load-bearing experiment for the portability claim, and it is missing. The paper's own Section 5 admits this: \"conversion functions are fitted on the full dataset and should be extended with proper train-test protocols.\" This is an explicit limitation that directly weakens the abstract's claim. I therefore agree with the stress-test note that the claim should be toned down to \"mappings can be fit with high accuracy on SQP data\" until the transfer experiment is done.\n\nOther concerns are real but secondary: synthetic queries are a proxy for real user traffic (the paper cites prior work suggesting they are predictive, but it is still a proxy), the sample is small (100 chunks per corpus, 3,000 pairs total), there are no error bars, and no code or data are released. None of these are fatal; they are addressable with additional experiments.\n\nWho is this for? RAG practitioners and anyone doing model migration who wants a lightweight way to reason about score scales and thresholds. It will not settle the field, but it provides a useful framework and initial evidence. It deserves serious refereeing, because the problem is real and the methodology is new, but it needs major revision: add a held-out threshold-transfer experiment, report variability, and ideally release the SQP data or code. I would not cite the portability claim as established, but I would cite the distributional characterization and the SQP methodology.","headline":"Useful empirical study of cross-model similarity score distributions, but its headline claim about threshold portability is not actually tested: the mappings are fit and evaluated on the same full dataset, with no held-out transfer experiment.","tokens_in":12647,"tokens_out":1654,"would_cite":true,"duration_ms":18359,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cosine similarity scores from different embedding models cannot be directly compared, but learned monotone mappings fitted on synthetic query–chunk pairs can partially transfer retrieval thresholds across models, with isotonic regression…","keywords":["embedding models","cosine similarity","threshold calibration","isotonic regression","synthetic query probing","retrieval-augmented generation","score distribution mapping","quantile mapping"],"falsifier":"Fit an SQP conversion on a corpus that also has human relevance judgments or logged real user queries, apply the converted threshold, and measure precision at the target operating point on the real queries; if precision falls systematically below the synthetic performance, the reference-free assumption is falsified.","tokens_in":11608,"feed_emoji":"🔁","tokens_out":2130,"duration_ms":21374,"temperature":0.7,"pith_summary":"The paper claims that cosine similarity scores from different embedding models are not directly comparable, and that naive threshold reuse across models fails. It introduces Synthetic Query Probing (SQP), a reference-free method that generates controlled query–chunk pairs labeled by semantic relatedness to analyze cross-model score distributions. The authors show that learned score conversion functions—especially isotonic regression—partially align these distributions and improve threshold portability. If correct, this provides a practical calibration step for migrating retrieval-augmented generation systems between embedding models and corpora.","feed_headline":"Score mapping transfers retrieval thresholds across embedding models","feed_subtitle":"Isotonic regression fitted on synthetic queries best aligns cosine similarity spaces, enabling threshold reuse in RAG migration.","key_machinery":"Synthetic Query Probing (SQP) is the central mechanism: from a chunk of a corpus, an LLM generates queries labeled as PARAPHRASE, RELEVANT, or IRRELEVANT, producing paired similarity scores across embedding models. The cross-model conversion functions are fitted on these paired scores using linear regression, isotonic regression, and quantile (CDF) mapping, and threshold analysis sweeps a binary classifier over cosine similarity to find operating points at target precision levels.","core_discovery":"The paper's central claim is that while embedding models largely agree on the relative ranking of query–chunk pairs, their absolute similarity scores exhibit systematic, model-dependent and corpus-dependent distortions. These distortions can be captured by monotone mapping functions learned from synthetic query probing data: linear regression works for near-linear within-family pairs, while isotonic regression best models the S-shaped non-linearity between different model families. The resulting mappings partially align similarity spaces and make precision-targeted thresholds more portable, though conversion accuracy degrades on heterogeneous corpora and is directionally asymmetric.","pith_inferences":["SQP could serve as a continuous drift-monitoring tool: periodically regenerated synthetic probes on a live corpus could flag when an embedding model's score distribution has shifted enough to require recalibration, an application the paper lists as future work.","The S-shaped Titan–Ada relationship likely reflects differences in how the models normalize embedding magnitudes; testing other model families beyond these two vendors would reveal whether such non-linearity is generic or idiosyncratic.","The label-free nature of SQP means it could be extended beyond cosine similarity to other distance metrics, and the method's dependence on LLM-generated labels suggests a testable extension: compare SQP-fitted thresholds against thresholds derived from real user query logs on the same corpus to quantify synthetic-to-real transfer."],"forward_implications":["Within a model family, dimensionality changes are nearly lossless for score conversion, with $R^2 \\geq 0.97$ on both corpora, so lightweight linear mappings suffice for such migrations.","Cross-model threshold reuse without calibration can lead to large precision drops; the paper's mappings reduce this gap, with isotonic regression achieving the highest $R^2$ and lowest MAE across all twelve directional pairs.","Thresholds are primarily corpus-dependent: changing the corpus induces larger threshold shifts than changing the embedding model within a family, implying per-corpus calibration remains necessary even when the model is fixed.","Conversion reliability is predictable from class separability: when relevant and irrelevant queries are cleanly separated, SQP-derived mappings are tighter, suggesting SQP statistics can serve as a prior indicator of transfer quality."],"supporting_citations":[{"why":"Supplies the SciFact corpus, one of the two evaluation datasets used to measure score distributions and threshold behavior.","marker":"[15]"},{"why":"Justifies the use of synthetic queries as a proxy for real retrieval behavior, directly supporting the paper's reference-free assumption.","marker":"[2]"},{"why":"Provides the closest prior work: learning monotone calibrations of similarity scores with linear and isotonic regression, which the paper extends to cross-model mappings.","marker":"[13]"},{"why":"Motivates the study by showing that benchmark scores do not imply comparable retrieval behavior across models, grounding the need for score-space analysis.","marker":"[1]"},{"why":"Establishes cosine similarity as the dominant choice in production embedding retrieval, justifying the paper's focus on cosine scores.","marker":"[12]"}],"fun_headline_variants":["Synthetic probing aligns embedding scores for threshold reuse","Isotonic mapping best transfers retrieval thresholds across models","Synthetic queries reveal score distortions and improve thresholds","Cross-model score calibration via synthetic query probing","Score mapping makes retrieval thresholds portable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire threshold and mapping analysis treats the LLM-generated synthetic queries and their automatic labels (PARAPHRASE, RELEVANT, IRRELEVANT) as a faithful substitute for real user queries and true relevance on the target corpus.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic probing aligns embedding scores for threshold reuse","Isotonic mapping best transfers retrieval thresholds across models","Synthetic queries reveal score distortions and improve thresholds","Cross-model score calibration via synthetic query probing","Score mapping makes retrieval thresholds portable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2770,"prompt_tokens":801,"completion_tokens":1969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":417,"completion_tokens_details":{"reasoning_tokens":1901}},"tokens_in":417,"tokens_out":1969,"duration_ms":13882,"temperature":1.0,"reasoning_tokens":1901,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:13:48.212398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit an SQP conversion on a corpus that also has human relevance judgments or logged real user queries, apply the converted threshold, and measure precision at the target operating point on the real queries; if precision falls systematically below the synthetic performance, the reference-free assumption is falsified.","supporting_citations":[{"cited_title":"(eds.) Machine Learning and Principles and Practice of Knowledge Discovery in Databases","cited_arxiv_id":null,"evidence_quote":"Justifies the use of synthetic queries as a proxy for real retrieval behavior, directly supporting the paper's reference-free assumption."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the closest prior work: learning monotone calibrations of similarity scores with linear and isotonic regression, which the paper extends to cross-model mappings."},{"cited_title":"In: Petroni, F., Siciliano, F., Silvestri, F., Trappolini, G","cited_arxiv_id":null,"evidence_quote":"Motivates the study by showing that benchmark scores do not imply comparable retrieval behavior across models, grounding the need for score-space analysis."}],"review_version":1}