Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

Scheduling LLM requests by learned pairwise rankings of expected response length, not by raw length prediction, brings latency down by up to 15.7x under burst loads.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A prompt-ranking scheduler that predicts relative output lengths with pairwise learning-to-rank and runs shorter LLM requests first reduces serving latency by up to 7.7x in burst tests.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Solid pairwise-ranking scheduler for LLM serving, but missing overhead measurement and an inconsistent 15.7x claim need addressing. the 4 major comments →

arxiv 2510.03243 v3 pith:C52SEG36 submitted 2025-09-25 cs.LG cs.AIcs.DCcs.PF

Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

Yiheng Tao , Yihe Zhang , Matthew Dearing , Xin Wang , Yuping Fan , Michael E. Papka , Zhiling Lan This is my paper
classification cs.LG cs.AIcs.DCcs.PF
keywords LLM inference servinglearning to rankshortest-job-first schedulinglatency reductionhead-of-line blockingmargin ranking losscross-model transferprompt-aware scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PARS aims to make shortest-job-first scheduling practical for LLM inference by replacing the hard task of predicting a request's output length with the easier task of deciding, for any pair of queued prompts, which one will generate the longer response. A lightweight BERT-based scoring model is trained with margin ranking loss on filtered prompt pairs—only pairs whose expected lengths differ by at least a calibrated threshold—so that noisy, near-tie comparisons do not corrupt the learned order. The resulting scores rank the waiting queue, and the scheduler then admits requests to the running batch in shortest-first order, in iteration-level continuous batching. Measured on real workloads and models, including reasoning models with long chain-of-thought outputs, PARS reports significantly lower average and p90 per-token latency than FCFS and prior pointwise/listwise SJF approximations, and it transfers across LLMs without per-model retraining. The pith is that pairwise preference learning with filtered training pairs is a more robust and cheaper proxy for SJF than direct length estimation.

Core claim

The paper claims that pairwise ranking is the right learning formulation for LLM task scheduling. Instead of regressing a length value or optimizing a full-list ranking, the predictor maximizes pairwise comparison accuracy under a margin, after discarding prompt pairs whose ground-truth length gap falls within the LLM's natural run-to-run variance (min_length_difference >= delta). The predictor is a BERT-base-uncased encoder followed by a linear scalar head producing a score; scheduling is SJF ordered by that score, with a starvation-prevention priority boost after two minutes. In the reported experiments the pairwise predictor reaches Kendall's tau as high as 0.96 (GPT-4/Alpaca) versus 0.70

What carries the argument

Margin ranking loss with filtered pairs: for each pair, the loss max(0, -y*(s_A - s_B) + margin) enforces that the predicted score of the longer-response prompt exceeds the other by margin=1.0. Before training, pairs with min_length_difference = |L_A - L_B| / max(L_A, L_B) below delta are discarded, where delta is calibrated on 30 prompts (0.2 for Llama 3.1/GPT-4, 0.25 for DeepSeek-R1) so that the model learns only comparisons that are stable against stochastic generation. The scoring model is BERT-base-uncased taking the [CLS] pooler embedding into a linear scalar. The same score ranks the waiting queue and drives iteration-level admittance to the running batch in vLLM, with a 2-minute star

Load-bearing premise

The load-bearing assumption is that a single relative-variance threshold measured on 30 prompts (delta = 0.2 for Llama 3.1/GPT-4, 0.25 for DeepSeek-R1, with temperature 0.7/top-p 0.9) captures the output-length variability of the full real-world workload; if actual request traffic exhibits heavier or more workload-dependent output variability, the filtered training pairs no longer isolate meaningful orderings and the scheduling gains degrade.

What would settle it

Measure output-length variability across a production request stream with diverse prompts and decoding settings; then train and deploy PARS with its fixed delta. If a large fraction of request pairs have length ratios within the natural variability band and PARS' ordering of those pairs is no better than chance, its average and p90 latency should converge toward FCFS under heavy load. A direct check: on a synthetic workload of near-uniform-length requests, the pairwise filter should remove most training signal and PARS should show no latency advantage over a random-order scheduler.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the ranking is accurate enough, near-optimal SJF can be achieved with a lightweight encoder and no knowledge of the exact output lengths, so latency-critical serving systems can avoid caching or simulating generation to estimate job durations.
  • For reasoning LLMs, including full chain-of-thought traces in the length label makes the pairwise order meaningful; otherwise the ranking would ignore the dominant cost component.
  • Since predictors trained on one LLM (GPT-4) rank requests for other LLMs (Llama 3.1, DeepSeek-R1) with modest degradation, a single shared predictor can be deployed across a heterogeneous pool of serving models, avoiding per-model retraining.
  • The filtering of near-tie pairs is a measurable ingredient: removing it lowers Kendall's tau by a consistent margin across datasets (e.g., 0.93 to 0.96 on GPT-4/Alpaca), so the benefit of the method depends on the stability of the length signal.
  • Under burst loads, the method keeps tail latency close to the oracle SJF bound, meaning it directly attacks head-of-line blocking rather than just improving average throughput.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper leaves implicit: as sampling temperature and top-p increase, run-to-run length variance grows, so the fixed delta calibration should be re-measured; if a deployment uses more stochastic decoding, the filter may retain too many noisy pairs and the ranking benefit should shrink—an empirical prediction one could test by sweeping delta.
  • The pairwise formulation suggests a connection to preference-based reinforcement learning: the same margin-ranking objective that orders documents by relevance is being repurposed to order jobs by resource cost, which could be extended to multi-objective scheduling (latency, fairness, energy) by making the pair label depend on a cost vector.
  • If the transfer across LLMs holds because 'task complexity' is the underlying signal, then the BERT predictor might also rank unknown future models zero-shot; a testable extension is fine-tuning on a small set of pairs from a new model and measuring how few labels restore full accuracy.
  • The starvation-prevention threshold (2 minutes) is a crude fairness knob; in production one might expect an adaptive threshold that trades tail latency against fairness, which could be tuned as a second learned layer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PARS, a prompt-aware scheduler for LLM serving that approximates shortest-job-first (SJF) by learning a pairwise ranking over prompts. A BERT-base encoder with a linear scoring head is trained with margin ranking loss on pairs of prompts filtered by a minimum relative length difference. At serving time, PARS scores each request in the waiting queue, sorts by predicted length, and feeds the vLLM scheduler with continuous batching, plus a starvation-prevention timeout. Experiments on Alpaca and LMSYS-Chat-1M with GPT-4, Llama 3.1, and DeepSeek-R1 report that PARS achieves higher Kendall tau than pointwise and listwise ranking baselines, improves average and p90 per-token latency over FCFS in burst scenarios, and transfers across models without retraining.

Significance. If the reported numbers are reliable, PARS is a useful practical contribution: it avoids exact length prediction, which is noisy and expensive, and instead learns a relative ordering that is robust to output-length variability. The pairwise filtering idea is well motivated and the consistent ranking improvements over pointwise/listwise baselines are a genuine strength. Cross-model generalization, if supported by the data, is valuable for deployment. However, the empirical claims are not yet verifiable: there is no measurement of predictor overhead, no confidence intervals or repeated-run statistics, no code or data release, and a discrepancy between the abstract's headline speedup and the body's reported numbers. These issues are load-bearing for the central claims of low-latency serving with minimal overhead.

major comments (4)
  1. [Abstract vs. §IV-D] The abstract claims latency reduction "up to 15.7x" compared to vLLM's default scheduler, but §IV-D reports "up to 7.7x" average speedup and "up to 8x" p90 speedup, and no 15.7x figure appears anywhere in the body. This is a factual inconsistency in the headline quantitative claim. Please reconcile the abstract with the reported experiments, or provide the specific configuration that yields 15.7x.
  2. [§III-B, §IV-D] The paper never measures predictor inference time or scheduling overhead. In the burst experiment with 2000 simultaneous requests, §III-B says the predictor computes a ranking score for each request in the waiting queue. With BERT-base-uncased (110M parameters), scoring a queue of 2000 prompts could be far from negligible. The "minimal overhead" claim and the end-to-end latency comparisons are unsubstantiated unless this cost is measured and shown to be included in the reported per-token latencies. Please provide an overhead breakdown and clarify whether scheduling time is counted.
  3. [§IV-D, Tables II-IV] All quantitative claims are point estimates without confidence intervals, seeds, or repeated runs. Given the high stochasticity of LLM output lengths, differences of tens to hundreds of ms/token between schedulers could be within run-to-run noise. The paper should report error bars or variance across runs for latency results, and ideally multiple training seeds for Kendall tau values, to support the claimed advantages over FCFS, pointwise, and listwise baselines.
  4. [§III-A] The training-pair filter threshold δ is calibrated on 30 prompts run 10 times on Llama 3.1 and DeepSeek-R1. No sensitivity analysis is given; the paper only compares "without filtering" to one δ value. If real workloads exhibit heavier or more variable output lengths, the chosen δ may not cleanly separate informative from noisy pairs, directly affecting ranking quality and scheduling benefit. Please vary δ (e.g., 0.1, 0.2, 0.3 for Llama; 0.15, 0.25, 0.35 for R1) and report ranking and scheduling results, or justify the transfer from the small calibration set.
minor comments (6)
  1. [§IV-D] The text describes "average per-token latency across varying arrival rates" and "burst scenarios" but the corresponding plots/figures are not present in the manuscript text. Please include the actual latency curves and burst results, not just prose summaries.
  2. [§IV] The Kendall’s Tau formula is garbled in the text ("τ b = nc−nd√ (n0−n1)(n0−n2)"). The mathematical notation should be typeset correctly.
  3. [References] Reference [6] (Fu et al., "Efficient LLM Scheduling by Learning to Rank") is incomplete: no venue, year, or arXiv identifier is given. Also "Alpaca [42], [49]" cites two separate references for the same dataset; please clean up the citation.
  4. [Throughout] Minor typos: "PARS(Prompt-" is missing a space; "PARSframework" appears in the introduction; "Alpaca [42], [49]" is redundant. These should be fixed.
  5. [§IV] Please clarify how GPT-4 is served on the described A100 testbed: is it accessed via API or a locally deployed model? This matters for interpreting latency comparisons and for the reproducibility of the cross-model experiments.
  6. [§V] The conclusion says the software and datasets "will be released" but no code or data repository is provided with the manuscript. For verification of the quantitative claims, please make the artifacts available or state clearly how to obtain them.

Circularity Check

0 steps flagged

No significant circularity: the predictor is trained and evaluated on disjoint data, and no load-bearing claim reduces to its inputs.

full rationale

PARS trains a BERT-base pairwise ranker on output-length labels from a training split and evaluates both ranking accuracy (Kendall's tau) and end-to-end scheduling latencies on a held-out test split. The training labels and the evaluated scheduling outcomes are not the same objects: the predictor never sees test-prompt ground-truth lengths during training, so the reported latency improvements are measured, not constructed. The only data-dependent hyperparameter, the min_length_difference threshold δ, is set from a small auxiliary variance experiment (Section III-A) and then ablated in Section IV-C; choosing a threshold from a pilot sample is parameter tuning, not fitting the reported result. Cross-model PARS is trained on GPT-4 labels and applied to Llama 3.1 and DeepSeek-R1, so the generalization claim is not tautological. No load-bearing self-citations appear; references to prior LTR and SJF work are external. Separately, the paper omits a direct measurement of predictor inference overhead in the scheduling loop (Section III-B vs Section IV), and the abstract's 'up to 15.7x' does not match the body's reported maximum speedups (7.7x average, 8x p90 in Section IV-D). These are missing-measurement and consistency concerns, not circular reductions, and therefore do not raise the circularity score.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on several domain assumptions about LLM generation variability, the proxy quality of BERT embeddings, and the transferability of output-length rankings across models. None are formally proven, and the delta calibration uses a very small sample. No new physical or theoretical entities are introduced.

free parameters (4)
  • min_length_difference threshold delta = 0.2 for Llama 3.1/GPT-4, 0.25 for DeepSeek-R1
    Set from measured relative output length variance of 30 prompts x 10 runs (Section III-A, Figure 2); controls which prompt pairs enter training.
  • margin = 1.0
    Fixed margin in the margin ranking loss for all experiments; hand chosen, not derived.
  • starvation timeout = 2 minutes (default)
    Hand-chosen fairness threshold in the scheduler to boost long-waiting requests; not derived (Section III-B).
  • training hyperparameters = 5 epochs, batch size 128, learning rate 2e-5
    Fixed for all predictor trainings; no sensitivity analysis is reported, though these values affect the trained predictor.
axioms (5)
  • domain assumption Output token count is a valid proxy for LLM execution time because decoding dominates latency.
    Section II: 'the inference time is primarily determined by the response output length [6]-[8]'; PARS schedules by predicted length.
  • domain assumption A fixed delta threshold separates informative from noisy prompt pairs across the full workload.
    Section III-A: threshold calibrated on 30 prompts x 10 runs; assumes similar variability on the full Alpaca and LMSYS workloads.
  • domain assumption BERT [CLS] embedding contains enough semantic signal to rank response lengths, including across different LLMs.
    Section III-A and IV-B: empirically validated on Alpaca and LMSYS, but no theoretical guarantee is provided.
  • domain assumption Pairwise preference accuracy yields a reliable global ranking.
    Section III-A: 'making accurate pairwise decisions leads to a reliable global ranking'; this justifies the pairwise objective.
  • domain assumption SJF scheduling approximated by predicted ranking improves latency under continuous batching without severe starvation.
    Section III-B: starvation prevention mechanism is assumed sufficient; no formal fairness or throughput analysis is given.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank." pith.science (2026). https://pith.science/paper/C52SEG36

@misc{pith2026251003243,
  author       = {Pith},
  title        = {Pith review of: Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C52SEG36}},
  note         = {Machine review of arXiv:2510.03243}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable. Traditional strategies like First Come, First-Serve (FCFS) often suffer from Head-of-Line (HOL) blocking, where long-running tasks delay shorter ones queued behind them. In this paper, we introduce PARS, a prompt-aware LLM task scheduler that mitigates HOL blocking by approximating shortest-job-first (SJF) scheduling through pairwise ranking with a margin ranking loss. PARS effectively predicts response-length-based task ordering directly from prompts, thereby optimizing scheduling decisions with minimal overhead. In addition, it integrates seamlessly with vLLM, a state-of-the-art LLM serving system, for the research community. Extensive experiments across multiple LLM models and real-world inference use cases, including chat, math, and code generation, demonstrate that PARS significantly reduces latency by up to 15.7x compared to the vLLM default scheduler. Cross-model evaluations demonstrate that our design generalizes effectively, allowing effective scheduling across diverse LLMs without requiring model-specific retraining.

Figures

Figures reproduced from arXiv: 2510.03243 by Matthew Dearing, Michael E. Papka, Xin Wang, Yiheng Tao, Yihe Zhang, Yuping Fan, Zhiling Lan.

Figure 1
Figure 1. Figure 1: PARS Workflow Specifically, the training dataset consists of paired prompts (A, B), with each pair annotated by a binary label y indicating which prompt is expected to generate a longer response by the LLM. The labeling scheme is defined as: (i) y = 1 → Prompt A is expected to yield a longer response than Prompt B; or (ii) y = −1 → Prompt A is expected to yield a shorter response than Prompt B. Due to the … view at source ↗
Figure 2
Figure 2. Figure 2: Relative variance from ten inference runs of 30 prompts [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

    cs.AI 2026-06 unverdicted novelty 7.0

    SVF scheduling algorithm achieves a competitive ratio of 3 for LLM serving and integrates into vLLM to reduce average and tail latency.

  2. Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference

    cs.DC 2026-05 unverdicted novelty 7.0

    Kairos improves SLO attainment and throughput in LLM serving by adapting to request length imbalance with priority scheduling and adaptive batching.

  3. Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving

    cs.AR 2026-07 conditional novelty 6.0

    NeuScale routes LLM inference requests to the most energy/cost-efficient configuration of heterogeneous NPU chips using roofline allocation and runtime auto-scaling.

  4. Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference

    cs.DC 2026-05 unverdicted novelty 4.0

    Kairos applies urgency-based priority scheduling on prefill and slack-guided adaptive batching on decode to raise TTFT, TPOT, and end-to-end SLO attainment by up to 33.8% and decode throughput by up to 19.3% versus baselines.

Reference graph

Works this paper leans on

53 extracted references · 2 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  2. [2]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024

    2024

  3. [3]

    Melody Y . Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, and Amelia Glaese. Deliberative alignment: Reasoning enables safer language models, 2025

    2025

  4. [4]

    Phi-4-reasoning technical report, 2025

    Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachan- dran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio C´esar Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Pa- pailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, Vibhav Vineet, Yue Wu, Safo...

    2025

  5. [5]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  6. [6]

    Efficient LLM Scheduling by Learning to Rank

    Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. Efficient LLM Scheduling by Learning to Rank

  7. [7]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI 24), pages 193–210, Santa Clara, CA, July 2024. USENIX Association

    2024

  8. [8]

    Response length perception and sequence scheduling: An llm-empowered llm inference pipeline, 2023

    Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. Response length perception and sequence scheduling: An llm-empowered llm inference pipeline, 2023

    2023

  9. [9]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023

    2023

  10. [10]

    Orca: A distributed serving system for Transformer- Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer- Based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carls- bad, CA, July 2022. USENIX Association

    2022

  11. [11]

    Zygos: Achieving low tail latency for microsecond-scale networked tasks

    George Prekas, Marios Kogias, and Edouard Bugnion. Zygos: Achieving low tail latency for microsecond-scale networked tasks. InProceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, page 325–341, New York, NY , USA, 2017. Association for Computing Machinery

    2017

  12. [12]

    Fast distributed inference serving for large language models, 2024

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models, 2024

    2024

  13. [13]

    Kalbarczyk, Tamer Bas ¸ar, and Ravishankar K

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T. Kalbarczyk, Tamer Bas ¸ar, and Ravishankar K. Iyer. Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

  14. [14]

    Power- aware Deep Learning Model Serving withµ-Serve

    Haoran Qiu, Weichao Mao, Archit Patke, and Shengkun Cui. Power- aware Deep Learning Model Serving withµ-Serve

  15. [15]

    GPT-4 Technical Report

    OpenAI, Josh Achiam, Steven Adler, and Agarwaland others. GPT-4 Technical Report

  16. [16]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    A Survey on Efficient Inference for Large Language Models, July 2024

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. A Survey on Efficient Inference for Large Language Models, July 2024

    2024

  18. [18]

    LLM Inference Serving: Survey of Recent Advances and Opportunities

    Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. LLM Inference Serving: Survey of Recent Advances and Opportunities

  19. [19]

    LLM Inference Unveiled: Survey and Roofline Model Insights

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. LLM Inference Unveiled: Survey and Roofline Model Insights

  20. [20]

    Assessing inference time in large language models

    Bartosz Walkowiak and Tomasz Walkowiak. Assessing inference time in large language models. In Wojciech Zamojski, Jacek Mazurkiewicz, Jarosław Sugier, Tomasz Walkowiak, and Janusz Kacprzyk, editors, System Dependability - Theory and Applications, pages 296–305, Cham,

  21. [21]

    Learning to rank for information retrieval

    Tie-Yan Liu. Learning to rank for information retrieval. InProceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’10, page 904, New York, NY , USA, 2010. Association for Computing Machinery

    2010

  22. [22]

    Learning to rank for recommender systems

    Alexandros Karatzoglou, Linas Baltrunas, and Yue Shi. Learning to rank for recommender systems. InProceedings of the 7th ACM Conference on Recommender Systems, RecSys ’13, page 493–494, New York, NY , USA, 2013. Association for Computing Machinery

    2013

  23. [23]

    Learning to optimize tensor programs, 2019

    Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Learning to optimize tensor programs, 2019

    2019

  24. [24]

    Lero: applying learning-to-rank in query optimizer.The VLDB Journal, 33(5):1307–1331, April 2024

    Xingguang Chen, Rong Zhu, Bolin Ding, Sibo Wang, and Jingren Zhou. Lero: applying learning-to-rank in query optimizer.The VLDB Journal, 33(5):1307–1331, April 2024

    2024

  25. [25]

    Subset ranking using regression

    David Cossock and Tong Zhang. Subset ranking using regression. InLearning Theory - 19th Annual Conference on Learning Theory, COLT 2006, Proceedings, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pages 605–619, Germany, 2006. Springer. 19th Annual Conference on Learnin...

    2006

  26. [26]

    Burges, and Qiang Wu

    Ping Li, Christopher J.C. Burges, and Qiang Wu. Mcrank: Learning to rank using multiple classification and gradient boosting. InAdvances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference, Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference, 2009. 21st Annual Conference on Neural Informat...

    2007

  27. [27]

    Ranking relevance in yahoo search

    Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, Chikashi Nobata, Jean-Marc Langlois, and Yi Chang. Ranking relevance in yahoo search. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 323–332, New York, NY , USA, 2016. Asso...

    2016

  28. [28]

    Schapire, and Yoram Singer

    Yoav Freund, Raj Iyer, Robert E. Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences.J. Mach. Learn. Res., 4(null):933–969, December 2003

    2003

  29. [29]

    Learning to rank using gradient descent

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. InProceedings of the 22nd international conference on Machine learning, pages 89–96, 2005

    2005

  30. [30]

    A regression framework for learning ranking functions using relative relevance judgments

    Zhaohui Zheng, Keke Chen, Gordon Sun, and Hongyuan Zha. A regression framework for learning ranking functions using relative relevance judgments. InProceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 287–294, 2007

    2007

  31. [31]

    Learning to rank with nonsmooth cost functions.Advances in neural information processing systems, 19, 2006

    Christopher Burges, Robert Ragno, and Quoc Le. Learning to rank with nonsmooth cost functions.Advances in neural information processing systems, 19, 2006

    2006

  32. [32]

    Adapting boosting for information retrieval measures.Information Retrieval, 13:254–270, 2010

    Qiang Wu, Christopher JC Burges, Krysta M Svore, and Jianfeng Gao. Adapting boosting for information retrieval measures.Information Retrieval, 13:254–270, 2010

    2010

  33. [33]

    From ranknet to lambdarank to lambdamart: An overview.Learning, 11(23-581):81, 2010

    Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview.Learning, 11(23-581):81, 2010

    2010

  34. [34]

    Adarank: a boosting algorithm for information retrieval

    Jun Xu and Hang Li. Adarank: a boosting algorithm for information retrieval. InProceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 391–398, 2007

    2007

  35. [35]

    Softrank: optimizing non-smooth rank metrics

    Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. Softrank: optimizing non-smooth rank metrics. InProceedings of the 2008 International Conference on Web Search and Data Mining, pages 77–86, 2008

    2008

  36. [36]

    Learning to rank: from pairwise approach to listwise approach

    Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. InProceedings of the 24th international conference on Machine learning, pages 129–136, 2007

    2007

  37. [37]

    Listwise approach to learning to rank: theory and algorithm

    Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pages 1192–1199, 2008

    2008

  38. [38]

    Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting

    Przemysław Pobrotyn and Radosław Białobrzeski. Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting. arXiv preprint arXiv:2102.07831, 2021

    Pith/arXiv arXiv 2021

  39. [39]

    S$ˆ{3}$: Increasing GPU Utilization during Generative Inference for Higher Throughput, June 2023

    Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. S$ˆ{3}$: Increasing GPU Utilization during Generative Inference for Higher Throughput, June 2023

    2023

  40. [40]

    Inference without Interference: Dis- aggregate LLM Inference for Mixed Downstream Workloads, January 2024

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without Interference: Dis- aggregate LLM Inference for Mixed Downstream Workloads, January 2024

    2024

  41. [41]

    DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency, August 2024

    Jovan Stojkovic, Chaojie Zhang, ´I˜nigo Goiri, Josep Torrellas, and Esha Choukse. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency, August 2024

    2024

  42. [42]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stan- ford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford alpaca, 2023

    2023

  43. [43]

    PAIR: Prompt-aware margIn ranking for counselor reflection scoring in motivational interviewing

    Do June Min, Ver ´onica P ´erez-Rosas, Kenneth Resnicow, and Rada Mihalcea. PAIR: Prompt-aware margIn ranking for counselor reflection scoring in motivational interviewing. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 148– 158, Abu Dhabi, United Ar...

    2022

  44. [44]

    Ranking with large margin principle: Two approaches

    Amnon Shashua and Anat Levin. Ranking with large margin principle: Two approaches. In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems, volume 15. MIT Press, 2002

    2002

  45. [45]

    Multi- task deep model with margin ranking loss for lung nodule analysis.IEEE Transactions on Medical Imaging, 39(3):718–728, 2020

    Lihao Liu, Qi Dou, Hao Chen, Jing Qin, and Pheng-Ann Heng. Multi- task deep model with margin ranking loss for lung nodule analysis.IEEE Transactions on Medical Imaging, 39(3):718–728, 2020

    2020

  46. [46]

    Bert: Pre-training of deep bidirectional transformers for language un- derstanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language un- derstanding, 2019

    2019

  47. [47]

    P Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023

    2023

  48. [48]

    M. G. Kendall. A new measure of rank correlation.Biometrika, 30(1/2):81–93, 1938

    1938

  49. [49]

    Instruction tuning with gpt-4, 2023

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023

    2023

  50. [50]

    A comprehensive survey on transfer learning, 2020

    Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning, 2020

    2020

  51. [51]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022

    2022

  52. [52]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023

    2023

  53. [2024]

    Springer Nature Switzerland

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.