{"id":"eb018313-a227-4923-8d4b-1d183ee560e4","arxiv_id":"2508.20559","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 0.1B-parameter model, trained by distillation from a 10B teacher plus SFT, DPO, and inference optimizations, matches a 10B model on query-focused webpage summarization and runs at about 50,000 queries per second.","lead":"This paper describes a system that trains a small 0.1B-parameter model to generate query-focused webpage summaries in Baidu search, replacing a multi-stage extractive pipeline. It reports that the small model matches a 10B model's summary quality while running at about 50,000 queries per second on clusters of L20 GPUs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generative outputs are not constrained to satisfy the formal extractive definition (S subset of D in Sec.","rationale":"The reader's weakest_assumption was annotation bias in the GSB and ROUGE evaluations; that is a real threat to the magnitude of the reported gain and is one reason the verdict should stay CONDITIONAL. My review identifies a more direct internal inconsistency: the paper's task definition and motivation commit to extractive, faithful output, while the method is an unconstrained generative language model with no faithfulness checks. The ROUGE and GSB tests cannot settle faithfulness because they compare against human references and overall preference rather than verifying that generated text is a subset of the source. This is not a disagreement with current consensus; it is a mismatch between the formalization in Section 3.1 and the implementation in Sections 3.3-3.5. The proposed test would decide whether the mismatch is material: if hallucination rates are low and exact-span extraction is high, the concern is resolved and the deployment claim is reasonable; if not, the paper should either add an extraction or verification mechanism or explicitly reframe itself as abstractive summarization. The reader already assigned CONDITIONAL, so my concern does not change the verdict; it sharpens the missing evidence that should be required before acceptance.","tokens_in":13028,"tokens_out":10441,"duration_ms":103597,"concrete_test":"On the 1,021 test triples, compute for each generated summary (a) the fraction of generated sentences that are exact contiguous spans of the source document, and (b) the fraction of summary tokens that appear verbatim in the document; in addition, ask two annotators to flag sentences containing content not supported by the document. If the exact-span extraction rate is not close to 100% or hallucination flagging is non-negligible, the model is abstractive or paraphrase-prone, and the formalization in Section 3.1 plus the faithfulness claim in Section 1 is violated; the paper would need constrained decoding (e.g., sentence selection with re-ranking) or a revised formulation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central value proposition is that a 0.1B generative model can replace Baidu's multi-stage extractive QDTS pipeline while producing summaries that 'faithfully reflect the content of the original document' (Section 1) and satisfy the formal definition S={s_o1,...,s_ok} subset of D (Section 3.1). However, no component of the method enforces this constraint. The student is a GPT-2 autoregressive language model trained with MLE on teacher-generated text (Eq. 3), then DPO (Eq. 5); it is never trained or verified to select a subset of source sentences, and no inference-time copying or constrained-decoding mechanism is described. The paper also reports no faithfulness or hallucination metric, even though Section 1 explicitly rejects 'entirely rephrased or hallucinated content.' Table 1's ROUGE scores (e.g., ROUGE-2 51.33) cannot distinguish extractive selection from fluent paraphrase, and the human GSB evaluation (Table 2) only measures overall quality and relevance, not whether the summary is a faithful subset of the document. Thus the core assertion that this is an extractive QDTS system suitable for production rests on an unverified property that the training objective does not positively enforce.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents QDGenSumRT, a four-stage framework for query-driven text summarization in Baidu Search: distillation from a 10B-parameter ERNIE-Lite-8K teacher into a 0.1B-parameter GPT-2-style student, full-parameter SFT on 6,160 human-annotated examples, DPO on click-based preference data, and deployment with FP8 quantization, TensorRT-LLM, and lookahead decoding. Offline evaluation reports ROUGE on a 1,021-pair held-out set, human evaluation using the ΔGSB metric against two systems, online A/B metrics, efficiency measurements, and ablations. The central claim is that this 0.1B model outperforms the production extractive baseline, matches a 10B nearline model, and achieves a new state of the art at much lower latency.","tokens_in":13272,"tokens_out":6603,"duration_ms":59027,"significance":"The production-scale evidence is the main strength. The +20.68% ΔGSB gain over Base (p=1.3e-6), the uniformly positive online A/B metrics, and the 149.7 QPS with 55 ms average inference time on an L20 GPU make a credible case that a small generative model can replace a multi-stage extractive pipeline in a real-time search setting. The ablation isolating distillation, SFT, and DPO is clear and informative. However, the paper's 'new state of the art' claim is not supported by its own Table 2, and the formal extractive definition is not enforced by any component of the method. These issues are fixable by reframing the claims and adding a faithfulness or extraction-rate evaluation.","major_comments":[{"comment":"The paper defines QDTS as selecting a subset S of D and motivates in Section 1 that summaries should be extractive so that users are not shown 'entirely rephrased or hallucinated content.' However, the student is an autoregressive GPT-2-style model trained with the MLE objective in Eq. (3) and the DPO objective in Eq. (5); no mechanism such as constrained decoding, sentence copying, or a selection loss enforces S ⊆ D, and no faithfulness or hallucination metric is reported. The ROUGE and ΔGSB metrics cannot distinguish extractive sentence selection from fluent abstractive generation, so the production-safety property asserted in Section 1 is unverified. Please either add a constraint or enforcement mechanism and report an extraction-rate or faithfulness metric, or explicitly reframe the system as abstractive with an accompanying hallucination and factuality evaluation.","section":"Section 3.1, Sections 3.2–3.4, Eqs. (1)–(5)"},{"comment":"The claim that the model 'achieves a new state of the art' is contradicted by the paper's own comparison against NGS: ΔGSB is -0.12% with p=0.88, and ROUGE-1 and ROUGE-L are lower than NGS (59.21 vs. 60.48 and 57.12 vs. 58.31, respectively). Only ROUGE-2 is numerically higher (51.33 vs. 50.25). The claim should be weakened to 'competitive with the 10B nearline model' unless a statistically significant advantage over NGS is demonstrated.","section":"Abstract and Section 4.5, Table 2"},{"comment":"Both the SFT training set (6,160 triplets) and the test set (1,021 triplets) are produced by Baidu's in-house annotation platform under the same stated criteria, which creates a distributional-circularity risk for the reported gains. The paper does not report inter-annotator agreement, and it does not specify how 'factual consistency, relevance, and centrality' were operationalized. Please add agreement statistics and, if possible, an independently annotated evaluation set that was not produced under the same conventions as the SFT data.","section":"Sections 4.1–4.2"},{"comment":"The production baseline Base is the anchor for both the headline ΔGSB gain and the online A/B results, but its design is concealed with the statement 'Specific details are omitted due to confidentiality constraints.' The online metrics are reported only as relative percentages with a blanket 'p-values for these experimental metrics are less than 0.05'; exact p-values and confidence intervals are missing, and with four correlated metrics the multiple-comparison issue is not addressed. Please provide a high-level description of Base, exact p-values and confidence intervals for Table 3, and state how many comparisons were tested.","section":"Sections 4.4 and 4.5, Table 3"}],"minor_comments":[{"comment":"The heading 'Evaluation Metircs' contains a typo; it should be 'Evaluation Metrics.'","section":"Section 4.2 heading"},{"comment":"The ROUGE-L formula is not the standard F-measure definition and does not define L(X, Y) or state how recall is combined with precision; please use the standard ROUGE-L F-score and define all symbols.","section":"Section 4.2, Eq. (7)"},{"comment":"The statement that 14 million samples yield 130B tokens implies roughly 9,300 tokens per sample; please clarify whether this count includes the source document content and whether near-duplicate or low-quality teacher generations were filtered before training.","section":"Section 3.2"},{"comment":"Click-based preference collection at the same display position is a reasonable weak signal, but the paper should specify how display order was randomized and how position bias or novelty effects were controlled when constructing D_dpo.","section":"Section 3.4"},{"comment":"The column heading 'Train Time (per epoch)' is inconsistent with the text, which compares total training time ('reduces training time by more than 35x'); please clarify whether the reported values are per-epoch or total and define what 'E2E Latency' includes.","section":"Table 4"},{"comment":"The abstract's figure of 334 NVIDIA L20 GPUs for ~50,000 queries per second is never derived in the body; Table 5 reports 149.7 QPS per GPU, which implies about 334 GPUs, but the calculation should be stated explicitly in the experiments.","section":"Abstract and Section 4.5"},{"comment":"The phrase 'significantly outperforms most existing methods' overstates the comparison, since DeepExtract and BART are offline, out-of-domain baselines and the only statistically significant comparison in the paper is against Base.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"This is an industrially grounded paper with a credible production evaluation, but the advertised 'state of the art' claim is not supported by the reported comparison with NGS, and the extractive framing is not backed by any enforcement or faithfulness measurement. I would not require the authors to disclose proprietary details of Base, but they should clearly state the limitations of a confidential baseline and temper the abstract and conclusion accordingly. A major revision with these corrections would make the paper sound and publishable in an applied venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid industrial systems paper with one overclaimed headline. The genuinely useful piece is the recipe — distilling a 10B ERNIE into a 0.1B GPT-2 student, then SFT on 6k human pairs, DPO on click-derived preferences, and FP8 + lookahead decoding for ~50k QPS at 55ms on 334 L20s. That's a reproducible engineering stack, and the online A/B results (+0.81% CTR, +0.33% SE, +0.62% SCR, +0.54% DR, all p<0.05) plus the human GSB win over Base (+20.68%, p=1.3e-6) give real evidence the system works in production.\n\nNow the soft spots, in proportion. The 'new state of the art' claim does not survive their own numbers: against NGS, ΔGSB is -0.12% with p=0.88, and ROUGE-1 and ROUGE-L are actually lower. The honest claim is 'comparable quality to a 10B model at a fraction of the serving cost,' which is plenty strong. The abstract should say that.\n\nThe bigger conceptual issue is the extractive framing. Section 3.1 defines QDTS as a subset-of-sentences selection problem, and Section 1 argues extractive output is preferred to avoid hallucination. But the generative model is trained with MLE on teacher text plus DPO; nothing enforces a copy constraint, and there is no faithfulness metric in the evaluation. ROUGE cannot distinguish a fluent paraphrase from a true extraction. The teacher might well be producing extractive summaries, and the student might be copying in practice, but the paper doesn't show it. This is the weakest link: either add a constraint, add a faithfulness eval, or drop the 'extractive' claim and talk about 'faithful generation'.\n\nTwo smaller points. SFT training data and test labels come from the same in-house annotation platform, so there is a distributional-bias risk, though the online A/B partially mitigates it. And the Base system is confidential, so the +20.68% cannot be independently benchmarked. Neither is disqualifying, but they should be acknowledged.\n\nBottom line: this deserves a serious referee. It's a valuable systems contribution with strong deployment evidence. The authors should temper the SOTA claim and add a section on faithfulness. I'd send it out and expect a major revision.","headline":"A deployable distil-and-align recipe for low-latency query summarization with real online gains, but the SOTA claim is a statistical tie and the extractive guarantee is asserted, not enforced.","tokens_in":13873,"tokens_out":2480,"would_cite":true,"duration_ms":24408,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 0.1-billion-parameter generative model can replace a multi-stage extractive summarization pipeline in real-time web search.","keywords":["generative summarization","query-driven text summarization","model distillation","direct preference optimization","lookahead decoding","FP8 quantization","low-latency inference","web search"],"falsifier":"Conduct a blind head-to-head evaluation on the 1,021 query-document test pairs in which summaries are produced by both systems but judged by a completely independent annotation team that has never seen the training data conventions; if the +20.68% $\\Delta$GSB advantage over the production baseline does not survive independent judging, or the ROUGE-2 lead over the 10B nearline generator reverses, the central claim of state-of-the-art quality is not supportable.","tokens_in":12802,"feed_emoji":"⚡","tokens_out":16000,"duration_ms":123102,"temperature":0.7,"pith_summary":"The paper claims that a 0.1-billion-parameter generative model, trained by distilling a 10-billion-parameter teacher and then refined with human annotations and online preference signals, can replace the multi-stage extractive summary pipeline of a large-scale web-search engine. On a production test set of 1,021 query-document pairs, the small model reports the highest ROUGE-2 score (51.33), edges out a 10-billion-parameter nearline generator on that metric, and is judged by human annotators to be substantially better than the deployed extractive baseline ($\\Delta$GSB +20.68%). It also meets real-time serving constraints: with FP8 quantization and lookahead decoding, about 334 GPUs handle roughly 50,000 queries per second at an average latency of 55 ms. If true, this gives search engines a cheaper, lower-latency path to query-relevant summaries without the cumulative information loss of funnel-style extractive pipelines.","feed_headline":"0.1B generative model beats extractive search summarizers","feed_subtitle":"Distilled from a 10B teacher and tuned on real clicks, it serves 50,000 queries per second at 55 ms.","key_machinery":"The load-bearing object is QDGenSumRT, a four-stage training-and-serving pipeline wrapped around the 0.1B-parameter student model, a 12-layer, 768-hidden-dimension decoder-only transformer. The machinery that carries the argument is the combination of (1) distillation from a 10B teacher, which transfers the ability to select query-relevant sentences; (2) full-parameter supervised fine-tuning on 6,160 curated human summaries, which imposes the roughly 80-token, three-to-five-point structured format and filters out ambiguous supervision; (3) direct preference optimization on click-derived preferences, which nudges output structure toward what users actually choose; and (4) FP8 quantization plus lookahead speculative decoding, which removes the latency that ordinarily makes 80-token generative output too slow for live search. The prompt is deliberately kept minimal to match the student's weaker instruction-following, and the output constraint $|S|\\leq 80$ tokens is what ties the generative model to the display budget of a search result page.","core_discovery":"On its own terms, the paper's discovery is that a task-specialized 0.1B decoder-only transformer, trained in four stages, reaches or exceeds the summary quality of a 10B generative model while running in real time. Stage one distills the teacher into a lightweight student on a synthetic corpus of 14 million query-title-content-summary samples (about 130B tokens). Stage two fine-tunes the student on 6,160 human-annotated examples that enforce a consistent three-to-five-point, roughly 80-token output format. Stage three applies direct preference optimization using summaries ranked by real user click behavior, interleaved with periodic SFT refreshes. Stage four deploys the model with FP8 weight-activation quantization and a lookahead speculative-decoding strategy. In offline tests the model scores 59.21 ROUGE-1, 51.33 ROUGE-2, and 57.12 ROUGE-L; human judges prefer it to the production extractive baseline by +20.68% $\\Delta$GSB (p = 1.3e-6) and find it statistically indistinguishable from the 10B nearline generator ($\\Delta$GSB -0.12%, p = 0.88); and a live A/B test shows gains of +0.81% CTR, +0.33% search engagement, +0.62% satisfactory consumption, and +0.54% distribution ratio.","pith_inferences":["This suggests the same distillation-SFT-DPO recipe could transfer to other constrained-output generation tasks in search, such as query-biased title rewriting, answer snippets, or structured product highlights, where extractive pipelines currently dominate.","A testable extension would be to measure whether the click-derived preference signal alone, without the SFT anchor, is stable over time; the paper interleaves DPO with periodic SFT refreshes, implying that pure preference drift may degrade format consistency.","One could also probe generalization: since the student is distilled on a 10B teacher and 14M queries, holding out an entire query category would reveal whether the apparent state of the art comes from task specialization or from memorizing the teacher's preferences.","The reported $\\Delta$GSB near-parity with the 10B nearline generator suggests that, at equal quality, serving cost becomes the deciding factor; the paper's own numbers imply the 0.1B model is the cheaper deployment choice by a large margin."],"forward_implications":["A 0.1B generative model can match a 10B teacher on a narrow, well-specified summarization task, so task-specific distillation plus preference alignment can substitute for raw parameter count.","The funnel-style extractive pipeline can be replaced by one end-to-end model, removing the cumulative information loss and the weakest-stage bottleneck.","FP8 quantization and lookahead decoding make generative 80-token output viable under industrial latency budgets: about 334 GPUs sustain roughly 50,000 queries per second at 55 ms average latency.","Online engagement metrics (CTR, search engagement, satisfactory consumption, distribution ratio) improve with the generative summaries at p < 0.05, implying users find the new summaries more useful, not just higher-scoring on offline metrics.","Because the summaries stay extractive, users can still click through to find the exact source sentence, so the generative approach does not compromise verifyability against the landing page."],"supporting_citations":[{"why":"Supplies the lookahead speculative decoding algorithm that accelerates autoregressive generation in the serving stack.","marker":"[8]"},{"why":"Provides the direct preference optimization objective used to align summaries with user click preferences.","marker":"[32]"},{"why":"Defines the decoder-only transformer architecture on which the 0.1B student model is built.","marker":"[31]"},{"why":"Defines the FP8 numeric format used for weight-activation quantization in deployment.","marker":"[21]"},{"why":"Defines the ROUGE metrics used for offline evaluation of summary quality.","marker":"[18]"},{"why":"Supplies the BART baseline that the system must outperform on the ROUGE evaluation.","marker":"[16]"},{"why":"Supplies the DeepExtract baseline representing a recent LLM-based extractive approach.","marker":"[27]"},{"why":"Provides the inference-serving framework used to deploy the quantized model at low latency.","marker":"[26]"},{"why":"Supplies the scaling-law analysis that motivates why a well-tuned lightweight model can rival a larger one on a narrow task.","marker":"[15]"},{"why":"Motivates the criticism that multi-stage extractive pipelines accumulate information loss.","marker":"[37]"}],"fun_headline_variants":["Distilled 0.1B model outperforms extractive baselines at 50k QPS","Real-time QDTS: 0.1B model beats extractive baselines","0.1B distilled from 10B, beats extractive summarizers","0.1B model beats extractive search summarizers at 50k QPS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human-annotated summaries and preference judgments used for both training and evaluation measure query relevance and summary quality without systematic bias; if those annotations share stylistic or content biases with the trained model, the reported gains over the production baseline could shrink or vanish in genuinely independent evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Distilled 0.1B model outperforms extractive baselines at 50k QPS","Real-time QDTS: 0.1B model beats extractive baselines","0.1B distilled from 10B, beats extractive summarizers","0.1B model beats extractive search summarizers at 50k QPS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001107,"raw_usage":{"total_tokens":4687,"prompt_tokens":1090,"completion_tokens":3597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":3505}},"tokens_in":706,"tokens_out":3597,"duration_ms":22595,"temperature":1.0,"reasoning_tokens":3505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:43:31.287843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Conduct a blind head-to-head evaluation on the 1,021 query-document test pairs in which summaries are produced by both systems but judged by a completely independent annotation team that has never seen the training data conventions; if the +20.68% $\\Delta$GSB advantage over the production baseline does not survive independent judging, or the ROUGE-2 lead over the 10B nearline generator reverses, the central claim of state-of-the-art quality is not supportable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DeepExtract baseline representing a recent LLM-based extractive approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the inference-serving framework used to deploy the quantized model at low latency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the criticism that multi-stage extractive pipelines accumulate information loss."}],"review_version":2}