Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Do LLM Recommenders Know When They're Hallucinating? Auditing Confidence Calibration in Catalog Faithfulness

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM recommenders are systematically under-confident about catalog faithfulness, so verbalized-confidence abstention cannot separate correct items from hallucinations.

desk verdict A genuinely useful audit with a real under-confidence result across four vendors, but the fuzzy-match label oracle needs a sensitivity analysis before the headline numbers harden. read the letter →

arxiv 2608.10008 v1 pith:2K7OSKIT submitted 2026-08-07 cs.IR cs.CLcs.LG

Srijith Ravikumar This is my paper · ORCID
classification cs.IRcs.CLcs.LG
keywords LLMrecommendershallucinationconfidencecalibrationcatalogfaithfulnessabstentionconformalpredictionOOD@10verbalized
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the confidence LLM recommenders attach to their top-K suggestions is calibrated against generic recommendation quality, not against whether the suggested item actually exists in the target catalog. It audits four zero-shot LLMs from different vendors across three catalogs and finds that all of them are systematically under-confident: they verbalize mean confidence of 67–86 on items that are in-catalog 92–100% of the time. Because the confidence channel cannot tell a real catalog entry from a hallucinated one, a conformal abstention threshold reduces hallucination by at most 0.7 percentage points while sacrificing 4–21 points of coverage. The paper argues that audits of LLM recommenders should report calibration alongside hallucination rate, and that confidence elicitation should be anchored to catalog membership rather than left generic.

What carries the argument

The central object is the joint distribution of the catalog-membership indicator (whether a generated title exists in the catalog, coded by exact match plus a fuzzy token_set_ratio >= 90 fallback) and the verbalized confidence elicited by the "Just Ask" template, measured with ECE, Brier, and reliability diagrams. A split-conformal quantile over the nonconformity score $s = 1 - c$ sets the abstention threshold. The mismatch between what the prompt asks for (recommendation quality) and what the audit measures (catalog existence) is the mechanism that clusters nearly all scores in the band $[0.05, 0.55]$, so quantile thresholds remove correct items faster than they remove hallucinations.

What would settle it

Re-run the audit on MovieLens with exact-match-only coding and with fuzzy-match thresholds at 85 and 95; if OOD@10 rises above 0.2% or the conformal sweep reduces hallucination by more than 0.7 percentage points, then the lenient matching threshold is material to the abstention-ceiling claim.

Watch

Extended reading notes

Core claim

Across all four LLMs, all three catalogs, and both popularity strata, verbalized confidence is systematically under-confident with respect to catalog faithfulness: models report mean confidence in the 67–86 range while their actual in-catalog accuracy is 92–100%. This is the opposite direction from the over-confidence usually reported in LLM-hallucination work, and it holds even on MovieLens where the hallucination rate is near zero. The under-confidence is best explained as an elicitation mismatch: the "Just Ask" prompt elicits a generic recommendation-quality rating, not a probability that the title exists in the catalog. As a result, a split-conformal abstention threshold built on the nonconformity score $s = 1 - c$ fails to separate correct items from hallucinations, reducing hallucination by at most 0.7 percentage points across the alpha sweep at 4–21 points of coverage cost.

Load-bearing premise

The reported hallucination rates rest on a hand-set fuzzy-matching rule that counts near-miss titles as in-catalog once token_set_ratio reaches 90; if that rule is too lenient, hallucination is undercounted and the abstention ceiling is measured against the wrong baseline.

Editorial extensions

If this is right

  • Model selection changes when calibration is reported alongside OOD: GPT-OSS-120B is worst on hallucination rate on long-tail catalogs but best-calibrated on Yelp, so choosing on one axis alone trades off the other.
  • Conformal abstention composed on verbalized confidence is not a free safety net for zero-shot LLM recommenders; the finite-sample guarantee holds on score coverage but not on hallucination rate.
  • Catalog-anchored confidence prompts can materially move calibration, but in opposite directions by vendor, so prompt design is part of the calibration problem, not a neutral measurement choice.
  • Audits of grounded, fine-tuned, or retrieval-constrained recommenders should measure the same joint distribution, since the zero-shot results may not characterize those production configurations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The near-zero MovieLens hallucination rate may be inflated by the lenient fuzzy-match rule; re-coding with exact-match-only or a stricter threshold could reveal a nonzero OOD rate, which would weaken the "calibration matters even when hallucination is zero" framing.
  • If the elicitation mismatch is the cause, then training-time calibration objectives that target catalog membership directly (e.g., rewarding doubt on out-of-catalog items) might restore a usable abstention signal, but the paper does not test this.
  • The logprob check on Llama times Yelp is also under-confident, so the finding is not purely a verbalization artifact; this suggests the under-confidence is a property of the model's internal probability estimates for this task, not just of the prompt.
  • The paper's twelve-cell design supports a strong cross-vendor claim, but a larger sample of models and catalogs would tell whether under-confidence is a general zero-shot phenomenon or specific to this prompt-and-catalog configuration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript audits the joint distribution of catalog faithfulness and verbalized confidence for four zero-shot LLM recommenders (Mistral Large, Llama-3.3-70B, GPT-OSS-120B, Claude Sonnet 4.6) across three catalogs (MovieLens-25M, Amazon Toys, Yelp), with popularity-stratified calibration metrics. The main claims are (i) OOD@10 is catalog-dependent (0-0.2% on MovieLens, 4.5-8.3% on Amazon, 2.2-8.4% on Yelp); (ii) all four models are systematically under-confident across all twelve model-catalog cells, verbalizing mean confidence 67-86 on items that are correct 92-100% of the time; and (iii) a split-conformal abstention threshold over verbalized confidence reduces hallucination by at most 0.7 pp at 4-21 pp coverage cost, showing that the confidence channel, not the conformal procedure, is the binding constraint. The paper also runs prompt-perturbation ablations and a logprob check, and it explicitly disclaims that the conformal score-quantile guarantee transfers to hallucination-rate control.

Significance. The manuscript addresses a genuine gap: prior generative-recommender evaluations report binary OOD rates without asking whether the model's confidence separates correct items from hallucinations. The cross-vendor replication (four independent model families), the exact prompt templates, the model identifiers and access dates, and the explicit statement of the conformal guarantee's limits are strengths. If the under-confidence and 0.7 pp ceiling findings survive a robustness audit of the label oracle, they have clear operational consequences for abstention policies. The main weakness is that every headline quantity passes through a single hand-set fuzzy-matching threshold, and calibration metrics are reported without uncertainty intervals; both issues are addressable in revision.

major comments (4)
  1. [§3.3, Tables 1 and 3] The in-catalog label oracle is a single hand-set rule: after exact match fails, a title is counted as in-catalog if token_set_ratio >= 90 on lowercased, punctuation-stripped strings. The manuscript reports no counts of fuzzy-only matches per catalog, no manual error-rate estimate for that rule, no sensitivity analysis around the threshold, and no exact definition of the fuzzy metric. Because both accuracy (Figure 1, Table 2) and OOD@10 (Table 1) are computed from this binary label, a lenient threshold would silently reclassify near-miss hallucinations as catalog hits, inflating accuracy and deflating OOD; this could exaggerate or reverse the under-confidence gap and would directly contaminate the 0.7 pp abstention ceiling, since a filter that removes hallucinations would be recorded as removing correct items. Please add exact-only versus fuzzy-only OOD rates, a threshold sweep (for example 80, 85, 90, 95, and exact-only), and a hand-audited sample of fuzzy-only matches, with particular attention to the near-zero MovieLens result where the fuzzy rule may be doing most of the matching.
  2. [Table 2 and §4.3] Table 2 reports ECE and Brier to three decimals without any uncertainty intervals or significance tests, in contrast to Table 1, which has bootstrap CIs. The cross-model and head-tail comparisons in Section 4.3 (for example, "Claude has the highest ECE in every (dataset, stratum) cell" and "ECE-tail exceeds ECE-head on Yelp for every LLM") are therefore not accompanied by evidence about sampling variation. Given the 300 users per audit set and bin counts as small as 21 items in Figure 1, many of these differences may be within noise. Please provide cluster-bootstrap or percentile intervals for ECE/Brier and for the head-tail differences, and state the effective per-cell sample sizes.
  3. [§4.3 and Table 2] The text says that "MovieLens tail samples (7-26 items per cell) are too thin for stratified inference," yet Table 2 still reports ECE-tail and Brier-tail for MovieLens for all four models. Metrics computed on 7-26 items cannot support the quantitative comparisons made in Section 4.2, and the reported three-decimal values are misleading. Please either remove the MovieLens tail columns or flag them clearly as not interpretable, and exclude them from any cross-cell summary claims.
  4. [§3.5 and Table 3] The conformal calibration description is ambiguous about the calibration set composition: it says the empirical quantile is computed "over in-catalog calibration items." If the quantile is estimated only from in-catalog items, the exchangeability assumption with test items that include hallucinations does not hold, and the stated marginal score-quantile guarantee does not follow. Moreover, Table 3 reports only hallucination rates and no actual coverage values, so the text's claim that coverage tracks 1-alpha within ±0.07 cannot be verified from the table. Please clarify whether calibration scores are computed on all calibration items or only in-catalog ones, and report the coverage column in Table 3.
minor comments (5)
  1. [Abstract and Table 2] The abstract states that models verbalize a mean confidence of 67-86 on items with 92-100% accuracy, but these numbers do not appear in any table; Table 2 shows ECE and Brier only. Please add a table or text block reporting per-cell mean confidence and accuracy.
  2. [§3.1] The paper says per-cell audit counts are 285-300 users after dropping errored responses, but it does not give exact counts per cell; please add a footnote or small table with final N per model-dataset cell.
  3. [§4.2] The two prompt-perturbation ablations (Llama Yelp ECE 0.224 -> 0.141; Claude Yelp -> 0.434) are reported without sample sizes or uncertainty information. Please state the per-ablation N and whether the same audit users are used.
  4. [Figure 1] The caption says marker size is proportional to bin count, but the legend does not give the scale; adding a size legend or numeric bin-count annotations would improve interpretability.
  5. [§5] The text says "the median reduction is below 0.1 pp," but no median or distribution of reductions is shown in Table 3; please add a summary statistic or a small table of all 32 reductions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the audit is empirical and self-contained, with all headline comparisons resting on independently measured catalog labels and verbalized confidence.

full rationale

The paper makes no derivation that reduces to its own inputs. The three central claims—catalog-dependent OOD@10, systematic under-confidence, and the 0.7 pp abstention ceiling—are all empirical measurements on externally sourced catalogs (MovieLens-25M, Amazon Toys, Yelp) and externally hosted LLMs. The under-confidence finding is defined by comparing verbalized confidence to catalog-hit accuracy, which are measured independently from the same responses but not constructed to be equal: the paper explicitly reports reliability curves sitting above the diagonal and ECE values, which are computed from raw per-item data rather than fitted to match the claim. The conformal abstention threshold is an empirical score quantile, and the paper explicitly states that the score-quantile guarantee does not transfer to hallucination rate unless scores are calibrated indicators, which the data show they are not; it therefore does not smuggle the conclusion into the method. There are no self-citations: the author does not cite prior work by the same author, and the one cited elicitation template (Tian et al., 'Just Ask') is external prior work used as a standard prompt, with the paper's own perturbation ablation showing that the template's phrasing materially changes the channel. The fuzzy-matching threshold raised by the skeptic is a measurement-validity concern, not a circularity: it is a fixed, pre-specified label rule with no parameter fitted to the outcome, and the paper's claims could be wrong if the rule is miscalibrated, which is exactly what a non-circular empirical claim should allow. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors, and no known result is repackaged under new coordinates. The paper is self-contained against external benchmarks and its limitation statements (marginal guarantee, exchangeability failure, zero-shot regime only) further confirm that the derivation chain is open rather than closed.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The audit relies on standard calibration metrics and conformal prediction, with no fitted parameters tuned to produce the reported results. The main hand-set choices are the fuzzy-match threshold and context length, both of which affect the headline numbers in ways the paper does not quantify.

free parameters (2)
  • fuzzy_match_threshold = token_set_ratio >= 90
    Hand-set threshold in Section 3.3 for coding generated titles as in-catalog when exact match fails. OOD@10 rates and the abstention-ceiling analysis depend on this choice, with no sensitivity analysis.
  • interaction_history_length = 30 most recent items
    Fixed context window for all users and models (Section 3.2, Appendix B). This design choice shapes the conditioning information and is not varied or justified against other lengths.
assumptions (4)
  • standard math Split-conformal exchangeability between calibration and test items
    Section 3.5 invokes split-conformal calibration [1]; the finite-sample coverage guarantee holds under exchangeability, which the paper notes fails under temporal or popularity drift.
  • domain assumption Public benchmark catalogs are adequate proxies for production catalog canonicity
    Section 3.1 selects MovieLens, Amazon Toys, and Yelp to represent different canonicity regimes; the limitations section acknowledges production catalogs may differ in churn, inventory drift, and staleness.
  • domain assumption Exact plus fuzzy matching correctly identifies catalog membership
    Section 3.3 assumes token_set_ratio >= 90 after normalization absorbs formatting variants without misclassifying true hallucinations as hits; this is load-bearing for all OOD rates.
  • domain assumption The 'Just Ask' prompt elicits a meaningful cross-vendor confidence channel
    Section 3.4 uses the Tian et al. 'Just Ask' template uniformly; Section 4.2's own ablations show the channel is prompt- and vendor-sensitive, so the main results rest on this specific elicitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do LLM Recommenders Know When They're Hallucinating? Auditing Confidence Calibration in Catalog Faithfulness." pith.science (2026). https://pith.science/paper/2K7OSKIT

@misc{pith2026260810008,
  author       = {Pith},
  title        = {Pith review of: Do LLM Recommenders Know When They're Hallucinating? Auditing Confidence Calibration in Catalog Faithfulness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2K7OSKIT}},
  note         = {Machine review of arXiv:2608.10008}
}
abstract

LLM recommenders for top-$K$ item suggestion regularly emit titles outside the target catalog. Prior audits measure this as a binary out-of-domain rate; none ask whether the model knew it was hallucinating. We jointly audit hallucination rate (OOD@10) and verbalized-confidence calibration (ECE, Brier, reliability) for four zero-shot LLM recommenders from four independent vendors (Mistral Large, Llama-3.3-70B, GPT-OSS-120B, Claude Sonnet 4.6), not grounded or fine-tuned systems, across three catalogs (MovieLens-25M, Amazon Reviews 2023 Toys, Yelp Open Dataset), stratified by item popularity. Hallucination is catalog-dependent (0--0.2\% on MovieLens, 4.5--8.3\% on Amazon, 2.2--8.4\% on Yelp), but verbalized confidence is materially miscalibrated even when hallucination is zero (ECE up to 0.223 on MovieLens despite 0\% OOD). All four LLMs are systematically \emph{under}-confident across all twelve cells, verbalizing a mean of 67--86 on items they recommend with 92--100\% accuracy. This is the opposite of the over-confidence usually emphasized in LLM-hallucination work. The under-confidence is best read as an \emph{elicitation mismatch}: ``Just Ask'' elicits a generic recommendation-quality rating, not a catalog-membership probability. A conformal abstention threshold over verbalized confidence reduces hallucination by at most 0.7\,pp across $\alpha \in \{.05, .10, .15, .20\}$, at 4--21\,pp of coverage cost: the under-confident channel cannot separate correct items from hallucinations, so the threshold mostly removes correct items. We recommend that audits of LLM recommenders report calibration alongside OOD, and use catalog-anchored elicitation rather than generic confidence prompts.

Figures

Figures reproduced from arXiv: 2608.10008 by the authors.

Figure 1
Figure 1. Reliability per catalog. Mistral (blue), Claude (pur [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 12 canonical work pages

  1. [1]

    Yasin Abbasi-Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. 2024. Mitigating LLM Hallucinations via Conformal Abstention. InAdvances in Neural Information Processing Systems 37 (NeurIPS). arXiv:2405.01563

    arXiv 2024

  2. [2]

    David Bani-Harouni, Chantal Pellegrini, Paul Stangel, Ege Özsoy, Kamilia Zaripova, Nassir Navab, and Matthias Keicher. 2025. Rewarding Doubt: A Re- inforcement Learning Approach to Calibrated Confidence Expression of Large Language Models.arXiv preprint arXiv:2503.02623(2025)

    arXiv 2025

  3. [3]

    Keqin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang, Yancheng Luo, Chong Chen, Fuli Feng, and Qi Tian. 2025. A Bi-Step Grounding Paradigm for Large Language Models in Recommendation Systems.ACM Transactions on Recommender Systems(2025). doi:10.1145/3716393 arXiv:2308.08434

    arXiv 2025

  4. [4]

    Yashar Deldjoo, Nikhil Mehta, Maheswaran Sathiamoorthy, Shuai Zhang, Pablo Castells, and Julian McAuley. 2025. Toward Holistic Evaluation of Recommender Systems Powered by Generative Models. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25). doi:10.1145/3726302.3730354

    arXiv 2025

  5. [5]

    Jianfeng Deng, Qingfeng Chen, Debo Cheng, Jiuyong Li, and Lin Liu. 2025. Logit Space Constrained Fine-Tuning for Mitigating Hallucinations in LLM- Based Recommender Systems. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 29311–29324. https://aclanthology.org/2025...

    work page 2025

  6. [6]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting Hallucinations in Large Language Models Using Semantic Entropy.Nature630 (2024), 625–630. doi:10.1038/s41586-024-07421-0

    doi:10.1038/s41586-024-07421-0 2024

  7. [7]

    Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A Survey of Confidence Estimation and Calibration in Large Language Models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 6577–6595. https://aclanthology.org/2024.naacl-long.366/

    work page 2024

  8. [8]

    Maxwell Harper and Joseph A

    F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context.ACM Transactions on Interactive Intelligent Systems5, 4, Article 19 (2015). doi:10.1145/2827872 MovieLens-25M release used: https: //grouplens.org/datasets/movielens/25m/

    doi:10.1145/2827872 2015

Show all 29 references
  1. [9]

    Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley

  2. [10]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran- Johnson, et al. 2022. Language Models (Mostly) Know What They Know.arXiv preprint arXiv:2207.05221(2022)

    2022 arXiv

  3. [11]

    Vempala, and Edwin Zhang

    Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. 2025. Why Language Models Hallucinate.arXiv preprint arXiv:2509.04664(2025)

    2025 arXiv

  4. [12]

    Adam Tauman Kalai and Santosh S. Vempala. 2024. Calibrated Language Models Must Hallucinate. InProceedings of the 56th Annual ACM Symposium on Theory of Computing (STOC). arXiv:2311.14648

    2024 arXiv

  5. [13]

    Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective Question Answering under Domain Shift. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). 5684–5696. https://aclanthology.org/2020. acl-main.503/

    2020

  6. [14]

    Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. 2025. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. InAdvances in Neural Information Processing Systems (NeurIPS). arXiv:2506.09038

    2025 arXiv

  7. [15]

    Wonbin Kweon, Sanghwan Jang, SeongKu Kang, and Hwanjo Yu. 2025. Uncer- tainty Quantification and Decomposition for LLM-based Recommendation. In Proceedings of the ACM Web Conference 2025 (WWW ’25). doi:10.1145/3696410. 3714601 arXiv:2501.17630

    2025 arXiv

  8. [16]

    Enze Liu, Bowen Zheng, Cheng Ling, Lantao Hu, Han Li, and Wayne Xin Zhao

  9. [17]

    Wenyu Liu, Yupeng Hou, An Yan, Junwei Lin, and Julian McAuley. 2025. Elimi- nating Out-of-Domain Recommendations in LLM-based Recommender Systems: A Unified View.arXiv preprint arXiv:2505.03336(2025). https://arxiv.org/abs/ 2505.03336

    2025 arXiv

  10. [18]

    Tran, Jonah Samost, Maciej Kula, Ed H

    Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Mahesh Sathiamoorthy. 2023. Recommender Systems with Generative Retrieval. InAdvances in Neural Inform...

    2023

  11. [19]

    Yedan Shen, Kaixin Wu, Yuechen Ding, Jingyuan Wen, Hong Liu, Mingjie Zhong, Zhouhan Lin, Jia Xu, and Linjian Mo. 2025. Alleviating LLM-based Generative Retrieval Hallucination in Alipay Search. InProceedings of the 48th International ACM SIGIR Conference on Research and Develo...

    2025

  12. [20]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. InProceed...

    2023

  13. [21]

    Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. 2025. Know Your Limits: A Survey of Abstention in Large Language Models.Transactions of the Association for Computational Linguistics (2025). https://aclanthology.org/2025.tacl-1.26/

    2025

  14. [22]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. InThe Twelfth International Conference on Learning Representations (ICLR). arXiv:2306.13063

    2024 arXiv

  15. [23]

    Yelp. 2024. Yelp Open Dataset. https://www.yelp.com/dataset

    2024

  16. [24]

    Just Ask

    Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024. R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’. InProceedings of the 2024 Conference of the North American Chapter of the Association for C...

    2023 arXiv

  17. [28]

    <item title> | <confidence> ... {k}. <item title> | <confidence> Do not output any preamble, explanations, or commentary. D Template 2: catalog-anchored perturbation prompt Used for the prompt-perturbation ablation in Section 4.2, applied to Llama-3.3-70B and Claude Sonnet 4.6...

  18. [29]

    <item title> | <confidence>

  19. [30]

    <item title> | <confidence> ... {k}. <item title> | <confidence> Do not output any preamble, explanations, or commentary. Do LLM Recommenders Know When They’re Hallucinating? Auditing Confidence Calibration in Catalog Faithfulness E Models, endpoints, and access dates All reco...

    2026

  20. [2024]

    Bridging Language and Items for Retrieval and Recommendation: Bench- marking LLMs as Semantic Encoders.arXiv preprint arXiv:2403.03952(2024)

    2024 arXiv

  21. [2025]

    InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25)

    Generative Recommender with End-to-End Learnable Item Tokenization. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25). doi:10.1145/3726302.3729989

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.