{"id":"73222a3c-a49a-40dc-b2ff-96491952a7c2","arxiv_id":"2505.07345","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A fine-tuned ensemble of a generative small language model and an embedding model outperformed zero-shot LLMs on Korean search relevance labeling, with reported Cohen's kappa of 0.646 versus 0.387 and 60x lower latency.","lead":"The authors combined a text-generating small language model with an embedding-based small language model to judge how relevant a search result is to a query. Their system, called QUPID, is claimed to be more accurate and about 60 times faster than large language model baselines on Korean search relevance, and it improved real search ranking by a small amount.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline SLM-vs-LLM comparison is confounded: QUPID is fine-tuned on ~1M labeled pairs, while all LLM baselines are zero-shot.","rationale":"I considered leakage as an alternative, but it is speculative: the paper does not describe deduplication, yet the test sets could still be independently sampled. The fine-tuning confound, by contrast, is explicit in the experimental design and is logically prior: even with zero leakage, Table 1 cannot support the abstract's strong claim. The system-level contributions (62 ms latency, single-token generation, live nDCG gain) are plausible and not attacked here. The reader's conditional verdict is appropriate; my concern strengthens the condition by making 'fine-tuned' a required qualifier in the headline conclusion.","tokens_in":13092,"tokens_out":8688,"duration_ms":83730,"concrete_test":"Fine-tune a representative strong LLM (e.g., Qwen-2.5-72B-instruct or LLaMA-3.3-70B-instruct) on the same training corpus and label scheme as QUPID, using the same evaluation protocol (same test sets, same Cohen's Kappa/AUC calculation, matching token-probability scoring where applicable). If this fine-tuned LLM achieves kappa >= 0.646, the claimed SLM advantage is a fine-tuning effect, not an architecture effect. A minimal variant is to fine-tune a 7B/8B LLM on the same data to bound the effect of scale.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 1, where QUPID's 0.646 kappa is compared with 0.387 for ChatGPT-4o. Section 4.2.1 states the LLM baselines are evaluated zero-shot, while QUPID is fine-tuned on roughly one million human-labeled query-document pairs (Section 3.1) drawn from the same three document types as the test sets (Section 4.1). The comparison therefore measures fine-tuning plus architecture, not architecture alone. A fine-tuned LLM on the same data could plausibly match or exceed 0.646; without that baseline, the abstract's claim that combining two SLMs can outperform LLMs is not established. Table 2's heterogeneous-vs-homogeneous ablation is the right internal control, but all rows are fine-tuned with the same data and no error bars, so it supports only the weaker claim that adding an embedding-based scoring head to a fine-tuned generative model helps.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QUPID, a relevance-assessment method that combines two small language models of different architectures: a generative SLM that scores relevance through token probabilities and an embedding-based SLM that scores through a pooled representation with a linear head. Both components are fine-tuned on roughly one million Korean query-document pairs, including synthetic hard negatives, and combined by a weighted average. The authors report that QUPID achieves a Cohen's kappa of 0.646 versus 0.387 for the best zero-shot LLM baseline, reduces inference latency to 62 ms versus seconds, and improves production ranking nDCG@5 by 1.9%. They also present use cases for filtering low-quality pairs, evaluating query refinement, and assessing snippet quality.","tokens_in":13302,"tokens_out":4163,"duration_ms":38268,"significance":"If the empirical claims are supported, the paper would make a useful practical contribution: a fine-tuned heterogeneous SLM ensemble could serve as a cheaper, faster alternative to zero-shot LLM relevance assessors in a deployed Korean search engine. The paper's strengths include its real-world production setting, three document-type test sets, and an internal ablation comparing homogeneous and heterogeneous ensembling. However, the central headline claim is weakened by a confounded experimental design and a lack of uncertainty quantification. The paper does not release code or data, which limits reproducibility, but the methodology is straightforward and the internal ablation is informative.","major_comments":[{"comment":"The central comparison is confounded: QUPID is fine-tuned on roughly one million labeled query-document pairs (Section 3.1), while all LLM baselines in Table 1 are evaluated zero-shot with a single prompting strategy (Section 4.2.1). The abstract's claim that 'combining two distinct SLMs with different architectures can outperform LLMs' is not established by this design because the comparison measures the combined effect of fine-tuning and architecture, not architecture alone. A fine-tuned LLM on the same data could plausibly match or exceed the reported 0.646 kappa. Please add fine-tuned LLM baselines, or at least a zero-shot variant of QUPID, and qualify the language in the abstract and contributions accordingly.","section":"§4.2.1 vs. §3.1"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any kappa, AUC, or nDCG value. The headline difference (0.646 vs. 0.387) and the production nDCG@5 improvement (+1.9% on 719 queries) cannot be assessed for statistical reliability. Please report bootstrap confidence intervals or paired significance tests (e.g., signed tests on per-query nDCG) for the key comparisons.","section":"Table 1 and Table 5"},{"comment":"Training and test data are drawn from the same three document types (snippets, UGC, web documents), but the paper does not describe any deduplication or near-duplicate removal between the roughly one million training pairs and the test sets. If near-duplicate query-document pairs occur in both, kappa and AUC values would be inflated. Please report an overlap analysis (e.g., exact-match and embedding-similarity duplicate rates) and, if leakage is found, evaluate on a disjoint held-out set.","section":"§3.1 vs. §4.1"},{"comment":"The ensemble weights w_gen and w_emb are tuned on a held-out validation set, and the label weights y_k are manually fixed. This is acceptable in principle, but the paper should report the selected weight values and a sensitivity analysis to show that the reported gains are not an artifact of overfitting the validation set. Also state explicitly whether the validation set is disjoint from the three test sets used in Tables 1 and 2.","section":"§3.2 and §3.4"}],"minor_comments":[{"comment":"The abstract states '60x faster inference times,' but Table 3 shows 62 ms vs. 3258 ms, which is approximately 52.5x; please recompute or reword.","section":"Abstract and Table 3"},{"comment":"The column header 'AUC (Relevant / Irrelevant)' is ambiguous; please clarify that these are class-wise AUC values and describe how they are computed.","section":"Table 1"},{"comment":"The description of the JudgeBlender baseline should clarify whether its constituent models were fine-tuned on Korean data or used zero-shot, and exactly how majority voting and average ensembling were applied.","section":"§4.2.2"},{"comment":"The caption says the rows are trained 'with different hyperparameters,' but the specific hyperparameters varied are not listed; please provide this information or refer to the appendix.","section":"Table 2 caption"},{"comment":"Table 4 lists a row 'T: inference temperature' but no temperature value appears in the table; the text mentions temperature 3.0 only later. Please make the table consistent with the prose.","section":"Appendix A.3"},{"comment":"The reference to Aho and Ullman (1972) in Section 3.1.2 does not appear related to hard-negative generation; please verify and replace if mis-cited. Also, de Souza P. Moreira et al. (2024) and (2025) appear to be the same preprint; consolidate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industry report with no code or data release, and the LLM baselines include a proprietary model (ChatGPT-4o), so exact replication is not possible. The main experimental claim would be much stronger if the authors added fine-tuned LLM baselines and statistical uncertainty measures. The contribution label 'first' in the introduction should be verified against prior work on heterogeneous ensembles of generative and embedding models. The production-ranking section would benefit from a more detailed description of how QUPID was integrated and how the baseline ranking model was configured."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead QUPID. Bottom line: if you run Korean search at scale and want a cheap relevance judge, this is a believable engineering result. But the abstract's headline claim—that combining two SLMs can outperform LLMs—is not established by the experiment as run.\n\nWhat's actually new: the heterogeneous ensemble pairing a generative SLM (token-probability scoring) with an embedding SLM (mean-pooled classifier) is a sensible combination that I don't think JudgeBlender covers. The internal ablation in Table 2 is the strongest part: heterogeneous ensemble beats either single model and beats homogeneous ensembles of the same architecture. The 60x speedup is plausible because the system generates one special token with a 10-token system prompt. The production nDCG@5 gain of 1.9% on 719 queries is a useful real-world signal, and the limitations section is honest.\n\nSoft spots, in order of severity. First, the headline SLM-vs-LLM comparison is confounded: QUPID is fine-tuned on roughly one million human-labeled pairs from the same three document types as the test sets, while ChatGPT-4o, LLaMA, Mistral, and Qwen run zero-shot with a single prompting strategy. The stress-test note is right. This measures fine-tuning plus architecture, not architecture alone. A fine-tuned LLM on the same data could plausibly beat 0.646; without that baseline the abstract overclaims. Second, no error bars or significance tests anywhere; the kappa differences between QUPIDGEN and QUPIDEMB are small (0.558 vs 0.590) and the ensemble gain could be noise. Third, possible train/test leakage: training and test sets both draw from the same three sources (snippets, UGC, web docs), and the paper gives no deduplication or overlap removal. That could inflate AUC and kappa. Fourth, the label weights y_k and ensemble weights w_gen/w_emb are tuned on validation; minor, since the test sets are separate, but worth disclosing.\n\nThe central engineering claim—a fine-tuned SLM ensemble can serve as a production relevance assessor in Korean—holds up reasonably well. The scientific claim about architectural diversity beating LLMs needs a fair baseline (fine-tuned LLM) and error bars. The paper deserves a serious referee: it is a real industrial system with concrete results, not a toy. I'd send it out, but the authors should be asked to add a fine-tuned LLM baseline or soften the headline, report confidence intervals, and clarify train/test separation.\n\nRegards.","headline":"Believable engineering win for Korean relevance labeling, but the SLM-vs-LLM headline is confounded by fine-tuning; the internal ablation is the paper's strength.","tokens_in":13824,"tokens_out":2670,"would_cite":false,"duration_ms":23824,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two small, differently built language models beat large ones at judging search relevance.","keywords":["relevance assessment","small language models","heterogeneous ensemble","query-document relevance","Korean search","Cohen's kappa","search ranking","token-probability scoring"],"falsifier":"Recompute QUPID's Cohen's kappa and AUC after removing from the three test sets every pair that is an exact or near-duplicate of a training pair (using, say, MinHash on normalized text), and compare with the LLM baselines on the same filtered sets; if the gap from 0.646 to 0.387 shrinks substantially, the claim that heterogeneous SLMs beat LLMs on genuinely unseen data fails. A complementary check is to hold out a fresh batch of query-document pairs collected after the training corpus was frozen and re-run the same evaluation.","tokens_in":12905,"feed_emoji":"🔎","tokens_out":8901,"duration_ms":75514,"temperature":0.7,"pith_summary":"QUPID claims that the best practical judge of whether a search result matches a query is not a large language model but a fine-tuned pair of small models with different architectures: a generative model that scores relevance from token probabilities and an embedding model that scores it from pooled hidden states, combined by weighted averaging. On Korean query-document data across snippets, user-generated content, and general web documents, the ensemble reaches Cohen's kappa 0.646 against human labels, where the strongest zero-shot LLM baseline reaches 0.387, and it does so at 62 milliseconds per judgment versus about 3.3 seconds. The paper also reports that the same relevance scores, plugged into a live search pipeline, improve ranking quality by 1.9% in nDCG@5. A sympathetic reader takes the claim to be that architectural diversity among small models can substitute for raw scale in this task.","feed_headline":"Small, differently built models beat LLMs at relevance labeling","feed_subtitle":"Generative + embedding SLM pair scores kappa 0.646 on Korean search data, 60x faster than large models.","key_machinery":"The load-bearing object is the heterogeneous ensemble itself: a weighted average $s_{\\text{final}} = w_{\\text{gen}} s_{\\text{gen}} + w_{\\text{emb}} s_{\\text{emb}}$ of two fine-tuned SLM scores. The generative score follows the token-probability method: the model is fine-tuned to output exactly one of three label tokens, and the score is $\\sum_k p_{i,k} y_k$, the probability-weighted expectation over labels. The embedding score mean-pools the hidden states of the query-document pair and maps the pooled vector through a learned linear layer and softmax. Both models are fine-tuned on roughly one million Korean query-document pairs, including synthetic hard negatives, and the final weights are tuned on a held-out validation set. An operational detail that carries much of the efficiency gain is prompt compression: after fine-tuning, the model needs only a ten-token system prompt plus one special task token and emits a single token, which is why inference takes tens of milliseconds rather than seconds.","core_discovery":"The paper's central discovery is that a heterogeneous ensemble of two fine-tuned small language models, one generative and one embedding-based, can label query-document relevance more accurately than leading zero-shot LLMs while running about 60 times faster. The generative arm, QUPID_GEN, is fine-tuned to emit one of three special label tokens and converts their log probabilities into a softmax distribution, with the relevance score taken as the probability-weighted expectation over label values; the embedding arm, QUPID_EMB, mean-pools the token hidden states of a decoder-only model and passes the pooled vector through a linear layer with softmax. Their scores are combined as a weighted average with weights chosen on a validation set. Across the three test collections the ensemble attains an average kappa of 0.646 and relevant/irrelevant AUC of 0.945/0.871, beating every LLM and every same-architecture ensemble tested, and the paper attributes this to complementary strengths of generative reasoning and similarity-based representation.","pith_inferences":["Editorial: the reported comparison mixes two variables, architecture diversity and fine-tuning on roughly one million labeled pairs, so the paper does not isolate which one drives the kappa gain; a fair test would fine-tune a single SLM of comparable size on the same data and compare.","Editorial: if the mechanism is genuinely architectural complementarity, the same generative-plus-embedding recipe should transfer to other languages and to tasks like answer relevance in retrieval-augmented generation; the paper only demonstrates Korean at scale, with one English anecdote.","Editorial: because the test sets and training corpus share the same three document sources and no deduplication is reported, the practical gap over LLMs might shrink on genuinely fresh data; a temporal split would settle this.","Editorial: the 60x speedup partly reflects the short prompt and single-token output, not just model size; an LLM fine-tuned with the same compressed prompt might narrow the latency gap, which would change the cost-benefit comparison."],"forward_implications":["If QUPID's claimed advantage holds, production search engines can afford to run relevance assessment on every query-document pair, not just samples, because per-judgment cost drops to 62 ms.","Heterogeneous SLM ensembles, mixing a generative and an embedding-based model, become a viable alternative to zero-shot LLM assessors, especially for languages where large models underperform.","The same relevance scores can serve multiple pipeline roles: filtering low-quality pairs at precision above 0.9, auditing query rewriting and auto-completion, flagging misleading snippets, and feeding the ranker directly.","Fine-tuning on task data can compress long natural-language prompts into a short task prefix, cutting both latency and serving cost.","Homogeneous ensembles of the same architecture do not capture the benefit; Table 2 shows that ensembling three or five copies of QUPID_GEN or QUPID_EMB yields smaller gains than combining the two different architectures."],"supporting_citations":[{"why":"Supplies the token-probability method for generative relevance scoring that QUPID_GEN is built on.","marker":"Zhuang et al., 2024"},{"why":"Provides the HCX-S backbone model that both QUPID arms are fine-tuned from.","marker":"Yoo et al., 2024"},{"why":"Defines JudgeBlender, the same-architecture SLM ensemble baseline that QUPID must beat.","marker":"Rahmani et al., 2024a"},{"why":"Provides the LLMJudge benchmark whose best prompting strategy is used for the LLM baselines.","marker":"Rahmani et al., 2024b"},{"why":"Supplies the criteria-based LLM relevance-evaluation prompt used for the representative LLM baselines.","marker":"Farzi and Dietz, 2024"},{"why":"Motivates the synthetic hard-negative generation used to augment the training corpus.","marker":"de Souza P. Moreira et al., 2024"},{"why":"Supports the choice of mean pooling for the embedding arm's sentence representation.","marker":"Lee et al., 2025"}],"fun_headline_variants":["Small SLM duo outperforms LLMs for search relevance","Generative + embedding SLMs beat LLMs at relevance, 60x faster","Heterogeneous SLM pair beats LLMs on Korean search relevance","Two small models outclass LLMs for relevance labeling","SLM combo: better relevance judgments, 60x faster than LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The test sets of 3,000 snippet, 20,000 user-generated, and 9,000 web-document pairs are drawn from the same three document sources as the roughly one million training pairs, and the paper reports no deduplication or near-duplicate overlap removal; if those test pairs overlap with training data, the reported kappa and AUC are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Small SLM duo outperforms LLMs for search relevance","Generative + embedding SLMs beat LLMs at relevance, 60x faster","Heterogeneous SLM pair beats LLMs on Korean search relevance","Two small models outclass LLMs for relevance labeling","SLM combo: better relevance judgments, 60x faster than LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1211,"prompt_tokens":923,"completion_tokens":288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":197}},"tokens_in":539,"tokens_out":288,"duration_ms":2667,"temperature":1.0,"reasoning_tokens":197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:18:31.580964+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute QUPID's Cohen's kappa and AUC after removing from the three test sets every pair that is an exact or near-duplicate of a training pair (using, say, MinHash on normalized text), and compare with the LLM baselines on the same filtered sets; if the gap from 0.646 to 0.387 shrinks substantially, the claim that heterogeneous SLMs beat LLMs on genuinely unseen data fails. A complementary check is to hold out a fresh batch of query-document pairs collected after the training corpus was frozen and re-run the same evaluation.","supporting_citations":[],"review_version":1}