REVIEW 4 major objections 7 minor 21 references
Quantile Regression with Large Language Models for Price Prediction
T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Fine-tuned LLM with quantile head beats price baselines
desk verdict Useful empirical recipe for text-to-distribution pricing, but the LLM-based data cleaning needs a harder look before the headline numbers are trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is a quantile regression head attached to the LLM's last hidden layer and fine-tuned end-to-end with a smoothed pinball loss $L_\tau(\hat{q}_\tau,y)=\tau(y-\hat{q}_\tau)+\alpha\log(1+e^{(\hat{q}_\tau-y)/\alpha})$, averaged over $K=200$ quantile levels. Summing over many $\tau$ values makes the loss a discrete approximation of the CRPS, a strictly proper scoring rule, so minimizing it pushes the model toward the true conditional distribution rather than only its mean. The paper also describes a delta-encoding variant that predicts the first quantile plus non-negative gaps, guaranteeing monotone quantiles by construction, and linear interpolation for evaluating arbitrary $\tau$. For point forecasts the median quantile is used; the paper argues this is more robust to outliers than squared-error regression, with non-median quantiles acting as regularizers.
What would settle it
Re-run the evaluation on a large independently human-validated test set (thousands of items per domain) drawn from the original, uncleaned listings, and compare Mistral-7B-Quantile against the best embedding baseline on both raw and cleaned subsets. If the MAPE advantage shrinks to statistical overlap, or if the model performs worse on items the LLM cleaner rejected, the cleaned benchmark rather than the modeling approach explains the reported gains.
Extended reading notes
Core claim
The central claim is that fine-tuning an LLM with a multi-quantile regression head, trained end-to-end with a smoothed pinball loss, produces better calibrated price distributions and better point forecasts than the alternatives tested. The model takes the last hidden state of the tokenized text, passes it through a quantile head that predicts $K=200$ equally spaced quantile levels, and uses the $\tau=0.5$ quantile as the point estimate. On retail product listings it reaches 16.86% MAPE versus 42.68% for the strongest embedding-based baseline; on used cars it reaches 6.30% versus best baselines above 58%; on used boats it reaches 21.20%. Distributionally it achieves calibration errors of 0.04–0.08 and CRPSS of 0.73–0.92, and its predicted densities show unimodal, bimodal, and skewed shapes that track price uncertainty. The paper also claims the quantile-trained median outperforms the same model trained with squared-error loss, and that few-shot in-context learning underperforms fine-tuning by large margins.
Load-bearing premise
The load-bearing premise is that the LLM-guided removal of roughly 111,000 'erroneous' prices did not preferentially delete hard or atypical examples, so the cleaned test sets measure genuine modeling ability rather than an artificially easier target distribution; the human validation covered only a few hundred samples.
Editorial extensions
If this is right
- Price predictions can come with confidence intervals: the model outputs a full distribution per item, so a seller or buyer can see not just a point estimate but a range, with tight intervals for standardized goods and wide ones for luxury or custom items.
- Fine-tuning an LLM with a quantile head is a practical recipe: it beats embedding-plus-ML pipelines and few-shot prompting without any domain-specific components, so the same head should transfer to other text-to-number tasks.
- Model scale and training data matter: the paper finds consistent gains from larger models (7B beats 3B and 500M decoders) and from more training examples, with MAPE roughly halving as training data grows from 1K to 100K samples.
- Data quality is part of the method: removing 'unreasonable' prices by LLM judgment before training contributes to the headline accuracy, so benchmark construction and model choice jointly determine the reported gains.
- Few-shot in-context regression has a non-monotonic shot curve: beyond an optimal number of examples, performance degrades, so practitioners should tune the shot count rather than assume more examples help.
Reading between the lines
- Editorial extension: the paper's own human-validated numbers leave room for a cleaning-bias story. On the retail product data, the quantile model's MAPE on the human-validated subset is 43.76% with a 95% CI of [14.9%, 87.5%], so a larger human-labeled test set could narrow or erase the gap to the embedding baselines.
- Testable extension the paper leaves open: run the same quantile head on listings created after the base LLM's pre-training cutoff and compare against the cleaned benchmark; a large drop would signal data contamination rather than genuine price understanding.
- The method's logic is not price-specific: because the multi-quantile pinball loss approximates a strictly proper scoring rule, the same head could be pointed at any text-to-number distribution task, such as financial forecasting or readability scoring, though the paper does not evaluate these.
- A direct way to isolate the cleaning contribution: train the same Mistral-7B-Quantile pipeline on the raw, uncleaned data and compare against the cleaned-data run, attributing the accuracy delta to the curation step rather than the model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies probabilistic regression with LLMs for price prediction. The authors attach a K-quantile regression head (K=200) to decoder-only and encoder-only language models and fine-tune with a smoothed pinball loss, comparing against point-regression heads, embedding-based regressors (Ridge, XGBoost, quantile MLP, kNN variants), and few-shot in-context LLMs on Amazon products, Craigslist used cars, and boats. The main reported finding is that a fine-tuned Mistral-7B with a quantile head outperforms all baselines on point metrics (MAPE, WAPE, MPE) and distributional metrics (CE, CRPSS, RCIW), and that LLM-assisted data cleaning of erroneous prices is unbiased. The paper also reports scaling experiments, category breakdowns, and qualitative analyses of predicted distributions.
Significance. The contribution is a useful empirical benchmark: the systematic comparison across architectures, loss functions, training scales, and few-shot strategies is more comprehensive than prior work on LLM text regression, and the released cleaned datasets with standardized splits are a practical asset. The paper also ships bootstrap confidence intervals for all main metrics, which is a strength. If the cleaning claim held, the result that a 7B decoder with a quantile head beats embedding-based distributional baselines would be a solid practical finding. However, the 'no systematic bias' claim is load-bearing and currently rests on a small human-validation study with a large unexplained performance gap on the Amazon subset; resolving this, along with clarifying the log-transform and monotonicity implementations, is necessary before the headline numbers can be fully trusted.
major comments (4)
- [Section 3 / Appendix A.1] The evidence that LLM-based cleaning is unbiased is not sufficient. Human validation covers only 341 Amazon and 153 car rows and none for boats (Table 6), yet the appendix reports a 2.7x gap on Amazon between Mistral-7B-Quantile MAPE on LLM-filtered data (16.3%) and on the human-validated subset (43.76%, 95% CI [14.9, 87.5]). The Fisher test p=0.198 only shows that the small sample cannot reject the null; it does not establish equivalence. Because roughly 16% of Amazon rows and 3% of car rows were removed before training, a cleaning rule that preferentially removes atypical-but-valid prices would make the benchmark easier for all methods and could change the relative conclusions, so the abstract's 'without systematic bias' claim needs stronger support, such as a larger human-validated test set or a sensitivity analysis on retained versus removed rows.
- [Appendix A.1] The composition of the 'human-validated subset' used for the 43.76% MAPE is unspecified. If it includes LLM-rejected rows, it is not a clean test set; if it is restricted to human-accepted rows, the validation rests on a very small number of examples (at most 325 agreements). The paper should state exactly which rows are in this subset and report performance on the restricted human-accepted set separately, together with the subset size.
- [Section 4.2 / Appendix B.1] The main architecture in Eq. (4) does not enforce quantile monotonicity, while the evaluation metrics in C.2.1 assume sorted quantiles. Appendix B.1 describes delta encoding and interpolation using 'can implement' and 'we can adjust', but it is never stated whether these structural refinements were actually used in the reported experiments. Please specify whether monotonicity was enforced and interpolation was applied; if they were not used, describe how non-monotone predictions were handled in computing CE, CRPSS, and RCIW.
- [Section 5 / Appendix C.1] The paper states that all models are trained on log-transformed targets but does not state how predictions are mapped back to the price scale before computing MAPE, MPE, WAPE, CE, CRPSS, and RCIW. This mapping is essential for interpreting every reported number and for reproducibility. Please specify explicitly whether exponentiation was applied to point predictions and to each predicted quantile before metric computation.
minor comments (7)
- [Table 2] In the Amazon Products row for Qwen-7B-Emb+Ridge, the MPE 95% CI is written as [32.02, -29.02], which has the bounds in the wrong order and is inconsistent with the point value 30.36; please correct this entry.
- [Section 6.1] The sentence claiming that traditional approaches 'systematically underestimate prices, with negative biases ranging from -24% to -135%' is contradicted by the positive MPE values for Qwen-7B-Emb+Ridge on Amazon (30.36) and for the two few-shot models on Amazon (14.32 and 19.12); please reconcile the statement with the table.
- [Appendix A.1] The appendix reports Mistral-7B-Quantile Amazon MAPE as 16.3% on LLM-filtered data, while Table 2 reports 16.86% for the same model and dataset; please clarify which test subset each number refers to.
- [Section 7] There is a typo in 'insights into they make pricing decisions'; this should be 'insights into how they make pricing decisions'.
- [Table 6] The column headers 'LLM Acc., Human Rej.' and 'LLM Rej., Human Acc.' are ambiguous; please reword them to make explicit which rows are counted in each column (for example, 'LLM accepted, human rejected').
- [References] Vacareanu et al. (2024a) and Vacareanu et al. (2024b) appear to be the same paper with identical titles; please consolidate the duplicate reference.
- [Section 6.1 / Section 4] The comparison between Mistral-7B-Quantile and Mistral-7B-Point conflates the multi-quantile head with the loss function, since the point model is trained with squared error while the quantile model uses pinball loss; an ablation with a point head trained on the median pinball loss would help isolate the benefit of multi-quantile training.
Circularity Check
No significant circularity: the central quantile-versus-point comparison is empirical, evaluated on held-out test splits, and the theoretical justification is imported from external references.
full rationale
The paper's central claims are empirical benchmark results rather than derivations. The headline comparison (Mistral-7B-Quantile versus Mistral-7B-Point, Table 2) pits a 200-quantile head trained with smoothed pinball loss against a point head trained with squared error, both evaluated on the same held-out test splits, so the outcome is not forced by construction. Section 6.3's theoretical discussion invokes external, independently checkable results (Koenker and Bassett 1978 for quantile consistency; Gneiting and Raftery 2007 for CRPS as a strictly proper scoring rule) and does not present them as novel predictions. No fitted parameter is renamed as a prediction: hyperparameters are selected via 5-fold cross-validation, and the Figure 4 data-scaling curves are training-subset experiments, not test-set evaluations. The two mildly self-referential elements (Claude-3.5-Sonnet both cleaning the data in Section 3 and serving as a few-shot baseline, and Amazon's Nova Pro as a baseline) are not load-bearing: the decisive comparisons among fine-tuned architectures do not involve Claude, and the cleaning is independently validated against human judgments in Appendix A.1 with 95.3% and 94.1% agreement on 341 and 153 samples. The Appendix's disclosed gap between 16.3% MAPE on the LLM-filtered Amazon test set and 43.76% on the human-validated subset is a benchmark-validity concern about cleaning bias, not a circular reduction; under the review rules it belongs under correctness risk rather than circularity.
Assumptions & free parameters
free parameters (2)
- Number of quantile levels K =
200
- Pinball smoothing alpha =
1e-2
assumptions (4)
- standard math Pinball loss minimization yields consistent conditional quantile estimates.
- standard math CRPS is a strictly proper scoring rule, and minimizing integrated pinball loss approximates CRPS.
- domain assumption Claude-3.5-Sonnet zero-shot judgments about whether a price is 'reasonable' are reliable enough to serve as ground-truth label cleaning.
- domain assumption The three datasets (Amazon products, Craigslist used cars, used boats) are representative of text-to-price prediction tasks.
Cite this review
Pith. "Pith review of Quantile Regression with Large Language Models for Price Prediction." pith.science (2026). https://pith.science/paper/GV4T4QPA
@misc{pith2026250606657,
author = {Pith},
title = {Pith review of: Quantile Regression with Large Language Models for Price Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/GV4T4QPA}},
note = {Machine review of arXiv:2506.06657}
}
read the original abstract
Large Language Models (LLMs) have shown promise in structured prediction tasks, including regression, but existing approaches primarily focus on point estimates and lack systematic comparison across different methods. We investigate probabilistic regression using LLMs for unstructured inputs, addressing challenging text-to-distribution prediction tasks such as price estimation where both nuanced text understanding and uncertainty quantification are critical. We propose a novel quantile regression approach that enables LLMs to produce full predictive distributions, improving upon traditional point estimates. Through extensive experiments across three diverse price prediction datasets, we demonstrate that a Mistral-7B model fine-tuned with quantile heads significantly outperforms traditional approaches for both point and distributional estimations, as measured by three established metrics each for prediction accuracy and distributional calibration. Our systematic comparison of LLM approaches, model architectures, training approaches, and data scaling reveals that Mistral-7B consistently outperforms encoder architectures, embedding-based methods, and few-shot learning methods. Our experiments also reveal the effectiveness of LLM-assisted label correction in achieving human-level accuracy without systematic bias. Our curated datasets are made available at https://github.com/vnik18/llm-price-quantile-reg/ to support future research.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
while maintaining monotonicity, as linear interpolation preserves order relationships. B.2 Few-shot Learning Few-shot learning enables models to make predictions with limited training examples, a capability that has proven particularly effective with LLMs (Wang et al., 2020). Recent theoretical work has demonstrated that this ability, also known as in-con...
work page 2020
-
[4]
arXiv e-prints, arXiv:2303.12712
Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv e-prints, arXiv:2303.12712. Tingting Chen and Shijing Si
-
[5]
Predicting rental price of lane houses in Shanghai with machine learn- ing methods and large language models. arXiv preprint arXiv:2405.17505. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al
-
[6]
arXiv preprint arXiv:2407.10759
Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov
-
[7]
In 2024 8th Interna- tional Conference on Cloud and Big Data Comput- ing
Predicting stock prices with FinBERT-LSTM: Inte- grating news sentiment analysis. In 2024 8th Interna- tional Conference on Cloud and Big Data Comput- ing. Ragıp Gürlek, Francis de Véricourt, and Donald K.K. Lee
work page 2024
-
[9]
Regression aware inference with LLMs. In Find- ings of the Association for Computational Linguistics: EMNLP 2024, pages 13667–13678, Miami, Florida, USA. Association for Computational Linguistics. Mistral
work page 2024
-
[11]
arXiv e-prints, arXiv:2303.08774
GPT-4 Technical Report. arXiv e-prints, arXiv:2303.08774. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe
-
[13]
arXiv preprint arXiv:2411.00852
EF-LLM: Energy forecasting LLM with AI-assisted automation, enhanced sparse prediction, hallucination detection. arXiv preprint arXiv:2411.00852. Xingyou Song, Oscar Li, Chansoo Lee, Bangding Yang, Daiyi Peng, Sagi Perel, and Yutian Chen
Show all 21 references
-
[14]
CoRR, abs/2402.14547
Om- nipred: Language models as universal regressors. CoRR, abs/2402.14547. Eric Tang, Bangding Yang, and Xingyou Song
-
[15]
CoRR, abs/2411.14708
Understanding LLM embeddings for regression. CoRR, abs/2411.14708. Robert Vacareanu, Victor-Andrei Negru, Vlad Suciu, and Mihai Surdeanu. 2024a. From words to num- bers: Your large language model is secretly a capable regressor when given in-context examples. arXiv preprint ar...
-
[16]
A.1 Validating LLM-based Data Filtering To address potential concerns about whether the LLM filtering removed hard examples or created unintended biases, we made two key observations. First, we noted that Claude, the model performing the cleanup, shows poor performance in zero...
2014
-
[19]
Few-shot SOTA LLMs: We evaluate the zero-shot and few-shot performance of two state-of-the-art LLMs, Claude-3.5-Sonnet and Nova Pro (Anthropic, 2024; Amazon, 2024)
Fine-tuned LLM with Regression Head: We fine-tune Mistral-7B, the largest LLM in our set, with a regression head to study the impact of quantile prediction versus point estimation. Few-shot SOTA LLMs: We evaluate the zero-shot and few-shot performance of two state-of-the-art L...
2024
-
[20]
that are limited to open-source models. For the Amazon Products and Boats dataset, even with the best-performing category-based sampling strategy and optimal shot count (256), both Claude and Nova-pro achieve MAPEs more than 35%, lagging significantly behind fine-tuned Mistral...
2007
-
[21]
Unimodal Distributions: Products such as the Merritt tumbler, Toyota Corolla, Ford Mustang, and wedding guest book exhibit single-peaked distributions
D.2 Distributional Patterns in Price Predictions The probability distributions shown in Figure 3 and Figure 8 show different patterns that reflect the underlying market dynamics of different product categories. Unimodal Distributions: Products such as the Merritt tumbler, Toyo...
2020
-
[200]
We use models that produce a distribution both for generating probabilistic outputs and for point predictions
We therefore used this number for all our experiments involving a trained model with a quantile regression head. We use models that produce a distribution both for generating probabilistic outputs and for point predictions. In the latter case, we take the predicted quantile at...
2024
-
[2019]
Jus- tifying recommendations using distantly-labeled re- views and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), pages 1...
2019
-
[2020]
In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901
Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi L...
1901
-
[2021]
arXiv preprint arXiv:2005.11401
Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401. Ilya Loshchilov and Frank Hutter
2005 arXiv
-
[2022]
arXiv preprint arXiv:2203.02155
Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. Zihang Qiu, Chaojie Li, Zhongyang Wang, Renyou Xie, Borui Zhang, Huadong Mo, Guo Chen, and Zhaoyang Dong
-
[2023]
Manufacturing & Service Operations Management, 25(4):1489–1508
Probabilistic forecasting of patient waiting times in an emergency department. Manufacturing & Service Operations Management, 25(4):1489–1508. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, and Kai et al. Dang. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609. Yu Bai...
-
[2024]
Preprint, arXiv:2404.14219
Phi-3 technical report: A highly capable lan- guage model locally on your phone. Preprint, arXiv:2404.14219. Amazon
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.