{"id":"47093012-8b53-4137-b265-20dc074cf04c","arxiv_id":"2412.12148","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A recipe that maps stakeholder risk tolerance to a confidence level and then uses statistical methods, with conformal prediction performing best, to set thresholds on LLM evaluation metrics.","lead":"This paper proposes a step-by-step recipe for choosing a cutoff on LLM evaluation metrics like Faithfulness, borrowing model risk management practice from finance. It demonstrates the recipe on a public hallucination dataset and compares statistical methods for setting the cutoff.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main empirical claim is not supported because conformal prediction confidence is reported as prediction-set coverage, while the recipe's risk tolerance concerns Type I/II error rates of a single threshold; Table 1's thresholds lack a defined mapping from Eqs. 4-6.","rationale":"I considered the reader's weakest assumption about risk-tolerance translation (Sec 2.1.3) and agree it is a real limitation; it is explicitly acknowledged as out of scope, so it is less decisive for the statistical contribution. The output-side gap is more direct: the paper's confidence levels are attached to prediction-set coverage, but the recipe's output is a single threshold whose error rates were never evaluated. This is falsifiable from the paper's own tables: Table 1 coverage and Table 2 recall are not comparable, so the superiority of conformal prediction is asserted rather than demonstrated. The paper has strengths: public data, reproducible implementations of three faithfulness metrics, clear step-by-step structure, and honest appendix reporting that Z-scores fail. The issue is addressable by re-analysis, so I keep the conditional verdict.","tokens_in":14265,"tokens_out":10421,"duration_ms":111203,"concrete_test":"Ask the authors to release the code that maps conformity scores to the single thresholds in Table 1, rerun the stratified 5-fold CV, and for every method and confidence level compute the held-out false positive and false negative rates of the resulting threshold classifier (not prediction-set coverage). If, for example, the 95% conformal threshold yields a false-positive rate far from 5%, or if GAM/poly widths are not significantly smaller than standard LR across folds by a paired test, the paper's promised threshold semantics and discriminative-power claim fail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Sec 2.3.5, conformal prediction is described as producing prediction sets (Eq. 5), which for binary classification can be {FAIL}, {PASS}, or {FAIL, PASS}. The text says thresholds are 'mapped back to feature space' but no mapping is given, and Table 1 reports a single faithfulness threshold per confidence level. This is a gap: a single cutoff cannot represent the set-valued predictor's ambiguous region, so the reported threshold is not determined by Eqs. 4-6. The paper validates conformal methods by coverage rate and width (Sec 4.5, Table 1), but coverage is a property of the prediction sets, not of the threshold classifier that practitioners would deploy. Step 1 (Sec 2.1.3) defines risk tolerance in terms of Type I/II error percentages ('only 5% hallucination accepted'), so the quantity that matters for the recipe is the error rate of the threshold classifier, not set coverage. Moreover, the comparison with KDE, empirical recall, and ROC-based thresholds uses recall (Table 2), so the methods are evaluated on different quantities; the paper itself notes (Sec 4.5) that the thresholds are not directly comparable. Thus the headline claim that conformal prediction has 'better discriminative power' is not established. The concern is load-bearing because the central practical contribution is a defensible single threshold at a specified confidence level.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a step-by-step recipe for choosing a threshold on a continuous LLM evaluation metric. The recipe begins with identifying application risks and stakeholder risk tolerance, translates that tolerance into a statistical confidence level, prepares ground-truth data, and then derives a threshold using one of several statistical methods: Z-scores, kernel density estimation, empirical recall, AUC-ROC, or split conformal prediction. The methods are demonstrated on the HaluBench dataset using three faithfulness implementations (RAGAS, DeepEval, UpTrain) with gpt-4o-mini as the judge. The paper reports thresholds at confidence levels from 80% to 99% and claims that conformal prediction, especially when based on GAM or polynomial logistic regression, has better discriminative power than the other methods.","tokens_in":14590,"tokens_out":4357,"duration_ms":47818,"significance":"If the proposed methodology were fully validated, it would give practitioners a defensible, risk-aware procedure for threshold selection in LLM evaluation, which is currently an underexplored part of LLM deployment. The paper's main strengths are its clear problem framing, its use of publicly available data and libraries, and its explicit discussion of stakeholder risk as a starting point. These strengths make the paper potentially useful as an applied recipe. However, the contribution is not supported by a new statistical derivation, and the empirical comparison has several load-bearing gaps, so the practical claims should be treated as provisional until the evaluation criteria are aligned across methods.","major_comments":[{"comment":"The central claim that conformal prediction yields better thresholds is not established because the mapping from conformal prediction sets to a single faithfulness threshold is never defined. Equations (4)-(6) produce set-valued predictions that can be {FAIL}, {PASS}, or {FAIL, PASS}, and the text only says that thresholds are 'mapped back to feature space' without giving a rule. Table 1 then reports a single threshold per confidence level, but no derivation is provided from those equations. Since the paper itself notes in Section 4.5 that the thresholds in Tables 1 and 2 are not directly comparable, the comparison of conformal methods with KDE, empirical recall, and ROC-based methods is not valid evidence for the claimed 'better discriminative power.' The evaluation should define the operational threshold classifier (e.g., using one boundary of the ambiguous region or a decision rule for the two-label set), and then report the same error metrics for all methods.","section":"Section 2.3.5, Table 1"},{"comment":"The foundational step of the recipe is asserted rather than operationalized. The paper states that risk preferences 'should be translated into a corresponding statistical confidence level' and gives the example 'only 5% hallucination is accepted ... required confidence level is 95%,' but it provides no procedure for this translation. The experiments simply choose confidence levels from 80% to 99% without any stakeholder-elicitation method. Because this translation is the input to every downstream method, the recipe cannot be used by practitioners as written. The authors should either supply a concrete elicitation and mapping procedure or reformulate the recipe so that risk tolerance directly specifies quantities such as acceptable false-positive and false-negative rates.","section":"Section 2.1.3"},{"comment":"The non-conformal methods are evaluated only by recall, which is insufficient for a risk-tolerance framework whose stated quantities are Type I and Type II error rates. Table 2 reports recall values of 100% at high confidence levels, but without precision or false-positive rates it is impossible to tell whether those thresholds are sensible or simply classify almost everything as passing. The paper should report a full confusion-matrix-based comparison for every method, including the conformal classifier after a threshold is extracted, so that methods are compared on the same operational quantities.","section":"Section 3.3, Table 2"},{"comment":"The data preprocessing is not justified with sensitivity analysis. The original HaluBench dataset is described as containing around 15K samples, but filtering by answer length and removing NaN scores reduces it to 7,703 samples. The paper does not characterize what is lost by this filtering, nor does it check whether the thresholds are stable under different filtering choices. In addition, all faithfulness scores come from a single judge (gpt-4o-mini) with default hyperparameters and single draws, so no estimate of judge or sampling variability is provided. A robustness analysis should be added to support the generality of the reported thresholds.","section":"Section 3.1, Section 3.2"},{"comment":"The sentence 'Overall, all methods achieve valid coverage aligned with the pre-specified confidence levels' is not supported by the tables. Table 2 reports recall, not coverage, and Table 1 shows coverage values that are often substantially higher than the nominal level (e.g., 90% nominal coverage reported as 98.35% for Uptrain). The claim should be restricted to conformal prediction, and even there the exact coverage should be reported rather than described as 'aligned.'","section":"Section 4.5"}],"minor_comments":[{"comment":"The notation in Equations (4) and (5) is ambiguous: the subscript on the calibrated probability in Equation (5) appears to be missing, and the text following Equation (4) does not clearly define whether the expression is the probability of the observed label or of each candidate label.","section":"Section 2.3.5, Equations (4)-(5)"},{"comment":"Equation (1) is a confidence interval for the population mean, not a threshold for an individual score. The paper notes in Appendix C that the resulting bounds are outside [0,1], but the conceptual mismatch should be stated explicitly in the main text so that readers do not interpret the method as a quantile estimator.","section":"Section 2.3.1, Equation (1)"},{"comment":"There is a typo in the first sentence: 'The results od KDE' should read 'The results of KDE.'","section":"Section 4.3.1"},{"comment":"The split conformal workflow is described only informally. The paper should specify how the training set I1, the calibration set I2, and the test set are created inside the stratified 5-fold cross-validation, and how the calibrated probabilities are obtained for each fold.","section":"Section 2.3.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of an applied recipe than a new statistical method, which may be appropriate for an applied ML journal but is a weak fit for a theory-oriented journal. The main empirical comparison is currently missing a well-defined threshold mapping for conformal prediction, and the evaluation metrics differ across methods. These issues are fixable, but they require nontrivial re-analysis rather than simple editing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper gives regulated-industry practitioners a sensible workflow for setting thresholds on LLM evaluation metrics, and it includes a genuinely useful empirical comparison of three faithfulness implementations on the public HaluBench dataset. The main weakness is that the conformal prediction section evaluates coverage while the recipe's risk-tolerance input is about Type I/II error of a deployed threshold, so the headline claim about 'better discriminative power' is not supported.\n\nWhat's new: not the statistical components—Z-score, KDE, ROC, conformal prediction are all standard—but the synthesis. The paper starts from stakeholder risk tolerance, translates it to a confidence level, and then compares five threshold-selection methods on RAGAS, UpTrain, and DeepEval faithfulness scores. The authors are honest about limitations: the Z-score fails on bimodal data, KDE is bandwidth-sensitive, empirical recall defaults to zero at high confidence. Appendices are detailed and reproducible, and the data and libraries are public.\n\nThe load-bearing problem: Step 1 defines risk tolerance in terms of accepted Type I/II error percentages, e.g., 'only 5% hallucination accepted.' Conformal prediction, as implemented, produces prediction sets with marginal coverage guarantees. A single faithfulness threshold is not a prediction set—it cannot represent the ambiguous region where the set includes both labels. The paper says thresholds are 'mapped back to feature space' but no mapping is given. Table 1 reports a single threshold per confidence level, but Eqs. 4–6 define a set-valued predictor, so the threshold is not determined by those equations. Coverage and width are properties of the sets, not of the threshold classifier a practitioner would deploy. The paper itself notes in Sec 4.5 that Tables 1 and 2 are not directly comparable, which undercuts the discriminative-power claim. This is a major gap, not a nitpick.\n\nOther issues, in proportion: the risk-tolerance-to-confidence translation is asserted but not operationalized; experiments simply pick 80–99%. Data filtering from ~15K to 7,703 samples lacks sensitivity analysis. Only recall is reported for non-conformal methods, and LLM-judge scores are single draws with no variance. All fixable.\n\nWho this is for: practitioners in finance, healthcare, or other regulated sectors who need a defensible rationale for threshold choices, and researchers working on LLM evaluation. It deserves a serious referee—the empirical comparison is valuable and the recipe is reasonable—but the conformal prediction section needs substantive revision, either by supplying the missing mapping or evaluating the threshold classifier's actual error rates. I would send it to review with that expectation.","headline":"Practical recipe for LLM metric thresholds, but the conformal prediction claim overreaches: coverage is not the same as the Type I/II error rate the recipe needs.","tokens_in":15107,"tokens_out":4232,"would_cite":false,"duration_ms":37200,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A recipe for picking LLM evaluation thresholds from risk tolerance using conformal prediction.","keywords":["threshold selection","LLM evaluation metrics","conformal prediction","model risk management","faithfulness metric","hallucination detection","generalized additive models"],"falsifier":"Run the same threshold recipe on an LLM evaluation metric whose scores are roughly uniformly distributed instead of bimodal; if GAM-based conformal prediction no longer produces narrower prediction sets than standard logistic regression, the claimed advantage is an artifact of the bimodal faithfulness scores used here.","tokens_in":14098,"feed_emoji":"🎯","tokens_out":4020,"duration_ms":36853,"temperature":0.7,"pith_summary":"This paper proposes a step-by-step methodology for choosing a threshold on an LLM evaluation metric, borrowing model risk management practices from regulated finance. The method starts with identifying the risks of the specific LLM application and the risk tolerance of its stakeholders, then translates that tolerance into a statistical confidence level, and finally computes a threshold from ground-truth data. As a demonstration, the authors apply the recipe to the Faithfulness metric for RAG hallucinations, using three public implementations and the HaluBench dataset. Their central empirical claim is that conformal prediction—especially with GAM or polynomial logistic regression as the underlying calibrator—identifies thresholds with better discriminative power than KDE, empirical recall, or standard logistic regression, while maintaining guaranteed coverage.","feed_headline":"Conformal prediction wins at picking LLM metric thresholds","feed_subtitle":"A finance-inspired recipe maps risk appetite to confidence levels and beats simple baselines on faithfulness scores.","key_machinery":"The load-bearing machinery is split conformal prediction combined with Platt-style probability calibration. Faithfulness scores are first mapped to calibrated probabilities of the label using a classifier; the paper compares standard logistic regression, polynomial logistic regression, and Generalized Additive Models (GAMs). These calibrated probabilities are turned into conformity scores $s_i = 1 - \\hat{\\mu}(X_i)_{Y_i}$ on a hold-out set, and the $(1-\\alpha)$-quantile of those scores defines a prediction set $\\hat{C}_\\alpha(X_{test}) = \\{y : \\hat{\\mu}(X_{test}) \\geq 1 - Q_{1-\\alpha}\\}$ that is guaranteed to contain the true label with confidence $1-\\alpha$. The threshold on the original faithfulness score is obtained by mapping this quantile back to the feature space, and narrower prediction sets indicate more informative thresholds.","core_discovery":"On the paper's own terms, the central claim is that threshold selection for LLM evaluation metrics should not be a heuristic choice but a disciplined procedure that starts from stakeholder risk appetite and ends with a statistically grounded cutoff. The authors establish this by translating financial model risk management guidance into a concrete recipe, and they argue that split conformal prediction with flexibly calibrated probabilities—via GAMs or polynomial logistic regression—gives the best thresholds: it achieves the claimed coverage at each confidence level while producing narrower, more informative prediction sets than KDE, empirical recall, or standard logistic regression. The demonstration on Faithfulness scores from RAGAS, DeepEval, and UpTrain against human labels in HaluBench shows that the simpler baselines collapse to zero thresholds at high confidence levels, whereas conformal prediction continues to produce non-trivial cutoffs.","pith_inferences":["The missing risk-tolerance-to-confidence translation could be made operational by fitting Prospect Theory utility functions to stakeholder choices and deriving the implied acceptable error rates, a step the paper flags but leaves to future work.","Because conformal prediction only requires exchangeability, the recipe could be adapted to streaming or drift-monitoring settings by re-calibrating thresholds on rolling windows.","A natural next experiment is to test whether GAM-based conformal thresholds remain superior when the ground-truth labels are noisy or when the LLM judge used in the metric changes."],"forward_implications":["Practitioners get a defensible route from stakeholder risk appetite to a specific cutoff on an LLM evaluation metric, instead of an ad hoc choice.","Conformal prediction with flexible calibration supplies thresholds that come with a guaranteed coverage rate, so the stated confidence level is an actual bound.","KDE and empirical recall thresholds degrade to zero at high confidence levels, so they are not suitable for high-stakes settings.","The recipe is metric-agnostic and intended to carry over from Faithfulness to other LLM and GenAI evaluation metrics.","The paper positions the procedure as a foundation for systematic threshold selection in any generative AI application, not just LLMs."],"supporting_citations":[{"why":"Provides the broader model risk management approach for RAG systems that this paper positions itself against and complements.","marker":"[17]"},{"why":"Supplies the HaluBench dataset with human-annotated hallucination labels used in all experiments.","marker":"[14]"},{"why":"Introduces the conformal prediction framework and the gentle-introduction formulation used to build prediction sets.","marker":"[2]"},{"why":"Defines the least ambiguous set-valued classifier construction that underlies the conformity-score prediction sets.","marker":"[16]"},{"why":"Provides the Generalized Additive Models used as the flexible probability-calibration method.","marker":"[8]"},{"why":"Gives the standard logistic regression and ROC methodology that serve as baselines and the core statistical tools.","marker":"[7]"},{"why":"Explains why AUC favors extreme score distributions, used to interpret the differing performance across the three faithfulness libraries.","marker":"[12]"}],"fun_headline_variants":["Conformal prediction nails LLM metric thresholds","Risk-first recipe for LLM evaluation cutoffs","Finance rules guide LLM metric threshold choice","Pick LLM thresholds with conformal confidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole recipe rests on the assumption that a stakeholder's risk tolerance can be translated into a specific statistical confidence level; the paper states this translation as a requirement but gives no procedure for doing it, and the experiments simply pick confidence levels from 80% to 99%.","fun_headline_variants_meta":{"raw":{"variants":["Conformal prediction nails LLM metric thresholds","Risk-first recipe for LLM evaluation cutoffs","Finance rules guide LLM metric threshold choice","Pick LLM thresholds with conformal confidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1412,"prompt_tokens":893,"completion_tokens":519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":509,"tokens_out":519,"duration_ms":5075,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:23:16.595952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same threshold recipe on an LLM evaluation metric whose scores are roughly uniformly distributed instead of bimodal; if GAM-based conformal prediction no longer produces narrower prediction sets than standard logistic regression, the claimed advantage is an artifact of the bimodal faithfulness scores used here.","supporting_citations":[{"cited_title":"Sudjianto and S","cited_arxiv_id":null,"evidence_quote":"Provides the broader model risk management approach for RAG systems that this paper positions itself against and complements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the conformal prediction framework and the gentle-introduction formulation used to build prediction sets."},{"cited_title":"Sadinle, J","cited_arxiv_id":null,"evidence_quote":"Defines the least ambiguous set-valued classifier construction that underlies the conformity-score prediction sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Generalized Additive Models used as the flexible probability-calibration method."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the standard logistic regression and ROC methodology that serve as baselines and the core statistical tools."},{"cited_title":"Schroedinger's Threshold: When the AUC doesn't predict Accuracy","cited_arxiv_id":"2404.03344","evidence_quote":"Explains why AUC favors extreme score distributions, used to interpret the differing performance across the three faithfulness libraries."}],"review_version":1}