Pith. sign in

REVIEW 3 major objections 6 minor 73 references

Conformal Language Model Reasoning with Coherent Factuality

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper guarantees 'coherent factuality' of language-model reasoning chains at any user-set error rate.

desk verdict A genuinely new conformal framing for reasoning coherence, with a clean lower bound and a credible but currently under-supported upper bound; worth refereeing. read the letter →

arxiv 2505.17126 v1 pith:MOE777ML submitted 2025-05-21 cs.CL cs.LGstat.ML

Maxon Rubin-Toles , Maya Gambhir , Keshav Ramji , Aaron Roth , Surbhi Goel This is my paper · ORCID
classification cs.CLcs.LGstat.ML
keywords coherentfactualityconformalpredictiondeducibilitygraphlanguagemodelreasoningclaimfilteringmathematicalhallucinationmitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language-model reasoning is only trustworthy if each step follows from what came before, not merely true in isolation. This paper defines "coherent factuality" to mean exactly that—every claim in an output ordering must be deducible from its prefix together with the question and a ground truth—and gives an algorithm that filters model outputs so the retained chain is coherently factual with probability at least $1-\alpha$ for a user-chosen error rate $\alpha$. The filter represents each generation as a directed deducibility graph and runs split conformal prediction over its subgraphs rather than over independent claims. On MATH problems the method is calibrated within the theoretical bounds, reaching 90% coherent factuality while keeping close to 80% of claims, whereas filtering claims independently leaves chains incoherent. The payoff is a checkable, per-output bound on logical integrity, not just on isolated truth.

What carries the argument

The load-bearing object is the approximate deducibility graph (Definition 4): a DAG whose vertices are the claims of a generation and whose edges record that a claim is sufficiently substantiated by its ancestors. The definition requires that any ancestor-connected subgraph is coherently orderable exactly when it admits at least one coherently factual ordering, and that enlarging such a subgraph never turns an incoherent set into a coherent one. On this graph, Algorithm 1 generates at most $|S(Y)| + 1$ candidate subgraphs by thresholding per-claim risk scores and removing any node whose ancestors were removed; each subgraph is paired with its risk threshold. The nonconformity score $r(X, Y, \mathcal{U}_T)$ is the largest threshold at which every considered subgraph is coherently factual, and a standard split-conformal quantile turns this into a calibrated cutoff $\hat{q}_\alpha$ for test outputs. The graph is what makes the proof go through: a risk score above the cutoff is then equivalent to the output failing coherent factuality.

What would settle it

Take a fresh sample of MATH outputs, manually construct ideal deducibility graphs, and check each GPT-4o proxy against Definition 4; if even one proxy admits an ancestor-connected subgraph whose topological ordering is incoherent while its risk score sits above the calibrated threshold, the conformal upper bound is violated. A direct test is to remove a single required edge from an otherwise-good graph and run the pipeline: the selected subgraph can then be incoherent at a score above the cutoff, which is the silent failure mode the definition is meant to prevent.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 1: for exchangeable calibration and test data and any target error rate $\alpha$, the output produced by subgraph filtering is coherently factual with probability at least $1-\alpha$; if, in addition, every deducibility graph is an approximate deducibility graph in the sense of Definition 4 and every output contains at least one hallucination (so the nonconformity score is finite), the probability is at most $1-\alpha + 1/(n+1)$. The empirical companion claim is that GPT-generated deducibility graphs satisfy Definition 4 often enough that calibration curves fall inside the bounds, and that graph-aware filtering outperforms both the independent-claim conformal baseline and a simple linear-order graph on MATH and FELM. In short, the paper claims that coherence—the dependency structure of a reasoning chain—is the quantity that should be calibrated, and that a DAG of claim dependencies is the right carrier for the conformal argument.

Load-bearing premise

The upper-bound guarantee relies on every language-model-generated deducibility graph being an approximate deducibility graph under Definition 4, a property the paper verified on only the first ten GPT-4o graphs (all satisfied) and the first ten Llama graphs (four satisfied), with the remaining graphs used unverified; the human annotations that fix coherent factuality are also acknowledged to be subjective.

Editorial extensions

If this is right

  • A user who sets error rate $\alpha$ gets a chain-level coverage guarantee on new outputs under exchangeability, with the lower bound $1-\alpha$ holding even when the deducibility graphs are imperfect.
  • Calibrating over the subgraph family is necessary: both the independent-claim baseline and a naive linear graph fail to achieve calibrated coherent factuality on MATH, while subgraph filtering lands inside the bounds.
  • The guarantee is not bought by deleting everything: on MATH the method attains 90% coherent factuality while retaining close to 80% of claims, and 85% factuality while retaining nearly 90%.
  • Feeding the filtered coherent chain back to the model as starter work reduces downstream error more than feeding an independently filtered chain, because coherent partial solutions are easier to continue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One editorial extension: because the lower bound is graph-agnostic, a natural stress test is to corrupt calibration graphs deliberately and confirm that the cost is over-filtering rather than silent incoherence; that would make the guarantee robust even to adversarial graph errors.
  • The method's practical upper bound becomes fully assumption-free only if deducibility graphs come from a verifiable extractor rather than from prompting a proprietary model, and the paper's own 40% pass rate on Llama graphs suggests such extractors need validation before deployment.
  • The same subgraph-filtering protocol transfers to code generation, where a compiler or proof checker supplies a formal substantiation relation; conversely, outputs without directed dependency structure, such as biography-style generations, are outside the method's reach, a boundary the paper observes when graph generation fails.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes a graph-aware conformal prediction method for filtering LLM reasoning outputs. It defines coherent factuality (Definition 3), in which a claim is acceptable only if it is deducible from the input, the ground truth, and the preceding claims in the output. The method relies on a 'deducibility graph' (Definition 4) that approximates the logical dependency structure among claims, generates a family of ancestor-connected subgraphs by thresholding claim risk scores (Algorithm 1), and applies split conformal prediction over a nonconformity score defined as the largest risk threshold below which all candidate subgraphs are coherent (Definition 5). Theorem 1 gives a lower bound on the probability that the filtered output is coherently factual and, under the additional assumption that every graph is an approximate deducibility graph, an upper bound. Experiments on MATH and FELM use GPT-4o-generated graphs and a Llama-3.1-70B variant, reporting that subgraph filtering achieves high coherent factuality with moderate claim retention, outperforming the independent-factuality baseline.

Significance. The contribution is a meaningful step toward conformal guarantees for structured reasoning outputs: the lower-bound argument in Appendix C is a clean split-conformal application that does not depend on graph quality, the paper includes gold human annotations on MATH, and the code is released. The notion of coherent factuality is a sensible and clearly-motivated refinement of independent factuality for multi-step reasoning. However, the distinctive two-sided calibration claim rests on properties of the model-generated graphs that are verified on only 10 examples per model, with the open-source graphs conceded to often violate the required definition. As a result, the upper-bound coverage claim is not currently supported, and the practical lower-bound guarantee depends on the reliability of silver annotations that inherit the graphs' assumptions. These issues are fixable, but they affect the paper's central quantitative claims.

major comments (3)
  1. [§5, Appendix F, Theorem 1] The upper-bound statement of Theorem 1 requires every graph Gi (including the test graph) to be an approximate deducibility graph under Definition 4. The paper verifies this only on the first 10 GPT-4o proxies (100% satisfied) and the first 10 Llama-3.1-70B proxies (40% satisfied), and Appendix E states that Llama graphs 'were further from ideal and less often satisfied Definition 4.' All remaining calibration and test graphs are used without checking Definition 4. This matters because the upper-bound proof in Appendix C invokes property (2) of Definition 4 on the test graph: it must be impossible to extend an incoherent ancestor-connected subgraph to a coherent superset. When that property fails, the event {r(X_{n+1}) < 1 - q_alpha} no longer implies that the filtered output is incoherent, and the claimed upper bound P[Y_filtered coherent] ≤ 1 - α + 1/(n+1) is not guaranteed. The two-sided 'calibrated' claim is therefore unsupported for Llama and only spot-checked for GPT-4o; the authors should verify Definition 4 on all graphs used or explicitly restrict the theorem and experiments to verified graphs.
  2. [§5 'Annotation', Definition 5] The conformal calibration in practice uses silver subset annotations that inherit the graph's assumptions: 'Subset factuality was measured by checking (1) ancestor connectedness and (2) whether any claim in the subset had an individual annotation of "No," assuming the graph proxies are reliable.' If a graph is not an approximate deducibility graph, a subgraph may be labeled coherent by the silver rule even though it is not coherent under Definition 3, inflating the nonconformity score r in Definition 5 relative to the oracle. In that case the empirical lower-bound coverage can fall below 1 - α even where the mathematical lower bound would hold with oracle labels. Gold annotations on MATH provide a partial check, but FELM has only silver annotations and the paper itself notes miscalibration in Appendix D, so the FELM results should be presented as exploratory rather than as validated conformal guarantees.
  3. [Appendix F] The paper reports that the first 10 GPT-4o proxies are approximate deducibility graphs and have mean edit distance 1.8 to manually-constructed ideal graphs, but it does not describe how Definition 4 was verified. Property (2) of Definition 4 requires checking every ancestor-connected subgraph of the graph, so a claim of 100% satisfaction without stating the checking procedure is not sufficient evidence. Since the upper bound in Theorem 1 depends on property (2), the authors should provide the verification protocol (e.g., how many subgraphs were checked, whether human coherence labels were used) or alternatively weaken the statements to the lower bound only where graphs are unverified.
minor comments (6)
  1. [Table 1 and §5 text] The table and its discussion contain a 'Table ??' placeholder (Table 1 in the pdf), which should be replaced with the actual table number.
  2. [Appendix J] The sentence 'we present our results in Appendix .' has an empty cross-reference; it should point to Appendix E.
  3. [§4 'Descendant weighting'] The median descendant-weighting score is selected with β = 0.5 based on robustness, but no sensitivity analysis is reported; a small sweep plot or a sentence quantifying the robustness would support the choice.
  4. [§3.1 and §5] The 'dependency' property of GPT-generated graphs is referred to as an empirical property but is never formally defined; since it is used to explain the utility of the graphs, a precise definition would help.
  5. [Definition 5] The notation UT ⊆ U × T uses U both for the set of candidate subgraphs and for a generic subgraph; renaming the set (e.g., S) would avoid ambiguity.
  6. [Appendix D] The text says 'see Appendix 5a' where it presumably means Figure 5a; please fix the cross-reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: conformal coverage is standard, and the upper-bound assumption (Definition 4) is an explicit empirical hypothesis rather than a derivation from the target claim.

full rationale

The derivation chain is self-contained. Theorem 1's lower bound is the standard split-conformal guarantee applied to the nonconformity score r (Definition 5); r is deliberately defined so that r >= 1 - qhat iff the filtered output is coherent, and calibrating such a score on labeled examples is the normal conformal construction, not circular. The upper bound is proved as a conditional statement: assuming each graph is an approximate deducibility graph (Definition 4) and r < infinity, property (2) of Definition 4 supplies the monotonicity that makes the contrapositive go through; this is an explicit modeling assumption, not the conclusion smuggled in. Observation 2 gives an independent constructive existence proof (ideal graph minus vtrue) rather than citing the target. The silver annotations used for calibration assume proxy graphs are reliable, but the headline results are validated on gold annotations that do not assume the graph, so the empirical evaluation is not circular. The beta=0.5 descendant-weighting hyperparameter is tuned on the same datasets, a mild selection-risk concern, but it does not constitute a fitted parameter renamed as a prediction. The main limitation, that Definition 4 is checked on only the first 10 GPT-4o proxies (100% satisfied) and first 10 Llama proxies (40% satisfied), Appendix F, undermines the empirical support for the upper-bound hypothesis, especially for Llama (Appendix E concedes these graphs 'less often satisfied Definition 4'), but that is a correctness and evidence gap, not circular reasoning. No load-bearing self-citation was found: the only self-related citations (Detommaso et al. 2024 and Jung et al. 2023, both involving co-author Aaron Roth) appear in related-work context and do not carry the proof.

Assumptions & free parameters 2 free parameters · 6 assumptions · 3 invented entities

The central theory rests on the standard conformal exchangeability assumption plus domain-level commitments: the meaning of 'deducible', the reliability of claim splitting, and the quality of LLM-generated deducibility graphs. The graph-quality assumption is the most fragile: it is asserted for all data but verified on only 10 instances per model. The method itself introduces no new physical entities; the new formal objects (coherent factuality, approximate deducibility graphs, dependency) are defined in the paper and have no externally falsifiable handle beyond the paper's own annotations.

free parameters (2)
  • beta (descendant weighting) = 0.5
    Swept over [0,1] and chosen for good claim-retention performance on the MATH dataset (Section 4, footnote 6); the conformal guarantee is valid for any fixed score, but the reported retention numbers depend on this choice.
  • k (number of alternate generations for self-consistency scoring) = 5
    The frequency-based claim scores use 5 GPT-4 generations (Section 4, Scoring functions); this affects the score distribution and therefore the empirical calibration, though the conformal guarantee holds for any fixed score function.
assumptions (6)
  • standard math Exchangeability of calibration and test draws (X_i, Y_i) ~ D
    Invoked before Theorem 1 (Section 4) for the split conformal guarantee.
  • domain assumption Assumption 1 (Superstring deducibility): adding good information preserves deducibility
    Used to justify that adding edges to an ideal graph yields approximate deducibility graphs (Section 3.1).
  • domain assumption Existence of a claim splitter S: Y -> 2^C
    Assumed in Section 2 without formalization; sentence boundaries of atomic claims are left informal.
  • domain assumption Ground truth C_true exists and annotators agree on it
    Section 2 and Section 6; correctness is 'only consistent with the annotator's subjective notion of truth'.
  • ad hoc to paper GPT-4o-generated proxies satisfy Definition 4 (approximate deducibility graph)
    Section 5 asserts 'Model-generated proxies ensure the conformal upper bound under Definition 4'; verified only on first 10 examples in Appendix F.
  • domain assumption r(X,Y,·) < ∞ for each output (each output contains a hallucination)
    Footnote 8, Section 5; needed for the upper bound in Theorem 1.
invented entities (3)
  • Coherent factuality (Definition 3)
    purpose: Formal target notion: every claim deducible from the prefix, X, and ground truth.
    A subjective definition evaluated by human annotators, not an independently falsifiable object.
  • Approximate deducibility graph (Definition 4)
    purpose: Graph proxy for claim dependency used to define subgraph filtering and conformal scoring.
    The paper shows existence via an ideal graph, but the practical proxies are LLM-generated; only 10 examples were checked for the defining property.
  • Dependency (empirical property of LLM graphs)
    purpose: Heuristic edge structure between erroneous claims that improves subgraph selection.
    Reported as an empirical observation (50% of graphs with errors have edges between erroneous nodes), not a formal construct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conformal Language Model Reasoning with Coherent Factuality." pith.science (2026). https://pith.science/paper/MOE777ML

@misc{pith2026250517126,
  author       = {Pith},
  title        = {Pith review of: Conformal Language Model Reasoning with Coherent Factuality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOE777ML}},
  note         = {Machine review of arXiv:2505.17126}
}
read the original abstract

Language models are increasingly being used in important decision pipelines, so ensuring the correctness of their outputs is crucial. Recent work has proposed evaluating the "factuality" of claims decomposed from a language model generation and applying conformal prediction techniques to filter out those claims that are not factual. This can be effective for tasks such as information retrieval, where constituent claims may be evaluated in isolation for factuality, but is not appropriate for reasoning tasks, as steps of a logical argument can be evaluated for correctness only within the context of the claims that precede them. To capture this, we define "coherent factuality" and develop a conformal-prediction-based method to guarantee coherent factuality for language model outputs. Our approach applies split conformal prediction to subgraphs within a "deducibility" graph" that represents the steps of a reasoning problem. We evaluate our method on mathematical reasoning problems from the MATH and FELM datasets and find that our algorithm consistently produces correct and substantiated orderings of claims, achieving coherent factuality across target coverage levels. Moreover, we achieve 90% factuality on our stricter definition while retaining 80% or more of the original claims, highlighting the utility of our deducibility-graph-guided approach.

Figures

Figures reproduced from arXiv: 2505.17126 by the authors.

Figure 1
Figure 1. Here, the previous method (Output 1) removes the erroneous claims outlined in red, but [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The nodes above correspond to the subclaims enumerated in Figure [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Even though Claim 6 is technically true, it detracts from the coherent solution as it [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: We evaluate our post-hoc (green) and subgraph filtering algorithms (using descendant [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Results on the FELM dataset using GPT-4 for responses and GPT-4o for graphs. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Results on the MATH dataset solely using Llama-3.1-70B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Performance of GPT-generated vs. human-constructed graphs for [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: These figures depict the performance of independent factuality calibration validated [PITH_FULL_IMAGE:figures/full_fig_p033_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 69 canonical work pages

  1. [1]

    A temperature of 1.0 was used to generate alternate responses for frequency scoring; a tem- perature of 0.0 was used for all other API calls

    work page

  2. [2]

    GPT-4 was used for the generation of outputs for the MATH questions

    work page

  3. [3]

    GPT-4 was used for self-consistency scoring, described in Section 4

    work page

  4. [4]

    commentary

    GPT-4o was used for graph generation. K.1 Dependency Graph Generation Prompt (MATH/FELM) GPT-4o Our prompt for graph generation includes in-context exemplars annotated with ratio- nales (“commentary”) for guided decomposition of the model-generated output into claims and their relation to one another. I’m going to give you a question and a series of claim...

    work page

  5. [5]

    We conclude that the sum of the zeroes of this function is −3 + 5 = 2. Desired Output: [[0,0,0,0,0],[1,0,0,0,0],[0,1,0,0,0],[0,0,1,0,0],[0,0,0,1,0]] Now provide your adjacency list for the following question and subclaims: K.2 Self-consistency (frequency) scoring prompt You will get a list of claims and piece of text. For each claim, score whether the tex...

    work page

  6. [10]

    a” depends on subclaim “ b

    Graph Description: - Represent the dependency relationships between subclaims as a directed graph. - Each subclaim is a vertex in the graph. - An edge ( b → a) exists if subclaim “ a” depends on subclaim “ b.” - Subclaims that are “a priori” (e.g., assumptions or definitions) should not have any ancestors

    work page

  7. [11]

    - Each entry in the adjacency list is a list of n integers: - A value of 1 at position i in row j indicates that subclaim j depends on subclaim i

    Output Format: - Provide your graph as an adjacency list of size NUM × NUM, where NUM is the number of subclaims (this will be given at the beginning of the prompt). - Each entry in the adjacency list is a list of n integers: - A value of 1 at position i in row j indicates that subclaim j depends on subclaim i. - A value of 0 indicates no dependency. - En...

    work page

  8. [12]

    - Each row and column must be exactly n integers

    Rules: - The adjacency list must be square, with n rows and n columns, where n is the exact number of subclaims provided. - Each row and column must be exactly n integers. Do not include extra rows, columns, or misaligned entries. - The output must consist solely of the adjacency list (e.g., [[0, 1, 0], [0, 0, 1], [0, 0, 0]]); do not include explanations,...

    work page

Show all 73 references
  1. [13]

    For example, if subclaim j implicitly relies on subclaim i (even if not stated directly), include the edge ( i → j) in the graph

    Dependencies: - Consider explicit and implicit dependencies between subclaims. For example, if subclaim j implicitly relies on subclaim i (even if not stated directly), include the edge ( i → j) in the graph. - Always represent dependencies, even if the subclaims are incorrect...

  2. [14]

    A function has vertical asymptotes exactly where its denominator equals zero

  3. [15]

    For all real values of x, x2 + 1 > 0

    To solve for the vertical asymptotes of the function y = x/(x2 + 1), we therefore must solve x2 + 1 = 0.3. For all real values of x, x2 + 1 > 0

  4. [16]

    Desired Output: [[0,0,0,0],[1,0,0,0],[0,1,0,0],[0,1,1,0]] - Input: Question: Consider the function y = x2 + 2x + 15

    Thus, we conclude that the function y = x/(x2 + 1) has no vertical asymptotes. Desired Output: [[0,0,0,0],[1,0,0,0],[0,1,0,0],[0,1,1,0]] - Input: Question: Consider the function y = x2 + 2x + 15. What is the sum of the zeroes of this function? NUM = 5 Subclaims:

  5. [17]

    The zeroes of a function are the x-values of its x-intercepts. 27

  6. [18]

    To find the zeroes of y = x2 + 2x + 15, we set the right-hand side equal to 0, writing 0 = x2 + 2x + 15

  7. [19]

    To solve 0 = x2 + 2x + 15, we factor it as 0 = ( x + 3)(x − 5)

  8. [20]

    This means that the zeroes of y = x2 + 2x + 15arex = −3, 5

  9. [23]

    Coherent F actuality:

    Converting 1 1 6 to an improper fraction gives 7 6 = 14w. Coherent F actuality:

  10. [24]

    Multiplying both sides by w gives 1 1 6 = w · 42 3

  11. [25]

    Simplifying the right side gives 11 6 = 14w

  12. [26]

    28 Example 2 Question: Mr

    Converting 1 1 6 to an improper fraction gives 7 6 = 14w. 28 Example 2 Question: Mr. Madoff invests 1000 dollars in a fund that compounds annually at a constant interest rate. After three years, his investment has grown to 1225 dollars. What is the annual interest rate, as a p...

  13. [37]

    Coherent F actuality:

    Rounded to the nearest integer, the an- nual interest rate is 7%. Coherent F actuality:

  14. [38]

    The formula for compound interest is A = P (1 + r/n)nt

  15. [39]

    In the formula, A is the amount of money accumulated after n years, including in- terest

  16. [40]

    In the formula, P is the principal amount (the initial amount of money)

  17. [41]

    In the formula, r is the annual interest rate (in decimal)

  18. [42]

    In the formula, n is the number of times that interest is compounded per year

  19. [43]

    In this case, A = 1225, P = 1000, n = 1, and t = 3 years

  20. [44]

    Substituting the given values into the for- mula, we get: 1225 = 1000(1 + r/1)1∗3

  21. [45]

    Simplifying the equation, we get: 1225 = 1000(1 + r)3

  22. [46]

    Dividing both sides by 1000, we get: 1.225 = (1 + r)3

  23. [47]

    Taking the cube root of both sides, we get: 1 .07 = 1 + r

  24. [48]

    Subtracting 1 from both sides, we get: r = 0.07

  25. [49]

    The annual interest rate is 0 .07, or 7% when expressed as a percentage

  26. [50]

    29 Example 3 Question: What is the smallest real number x in the domain of the function g(x) = p (x − 3)2 − (x − 8)2 ? Independent F actuality:

    Rounded to the nearest integer, the an- nual interest rate is 7%. 29 Example 3 Question: What is the smallest real number x in the domain of the function g(x) = p (x − 3)2 − (x − 8)2 ? Independent F actuality:

  27. [54]

    Coherent F actuality:

    The smallest such x is 11 2 . Coherent F actuality:

  28. [55]

    The expression under the square root must be nonnegative

  29. [56]

    We must have ( x − 3)2 − (x − 8)2 ≥ 0

  30. [57]

    This simplifies to ( x − 3 + x − 8)(x − 3 − (x − 8)) ≥ 0

  31. [58]

    Example 4 Question: If (2x + 5)(x − 3) = 14, find the sum of the possible values of x

    This further simplifies to (2 x − 11)(5) ≥ 0. Example 4 Question: If (2x + 5)(x − 3) = 14, find the sum of the possible values of x. Independent F actuality:

  32. [61]

    The sum of the roots of the equation is 1 2 Coherent F actuality:

  33. [62]

    The left side of the equation is expanded to 2x2 − 6x + 5x − 15 = 14

  34. [63]

    The expanded equation simplifies to 2x2 − x − 29 = 0

  35. [64]

    Vieta’s formulas are used to find the sum of the roots of the equation

  36. [65]

    The point (13 , q) lies on the same line

    The sum of the roots of the equation is 1 2 30 Example 5 Question: The points (x, y) represented in this table lie on a straight line. The point (13 , q) lies on the same line. What is the value of p + q? Express your answer as a decimal to the nearest tenth. x y 2 −5 p −14 p ...

  37. [68]

    The slope of the line is also calculated as −14−(−5) p−2 = −9 p−2

  38. [69]

    Setting this slope equation equal to −3 2 gives −3 2 = q+5 11

  39. [70]

    Coherent F actuality:

    Setting these two slope equations equal gives −3 2 = −9 p−2 . Coherent F actuality:

  40. [71]

    The slope of the line is calculated as −17−(−14) p+2−p = −3 2

  41. [72]

    Setting these two slope equations equal gives −3 2 = −9 p−2

  42. [73]

    The slope of the line is also calculated as q−(−5) 13−2 = q+5 11

  43. [74]

    L.2 FELM Dataset Outputs Example 1 Question: Jessica makes $2,000.00 a month

    Setting this slope equation equal to −3 2 gives −3 2 = q+5 11 . L.2 FELM Dataset Outputs Example 1 Question: Jessica makes $2,000.00 a month. She sets 25% of her paycheck aside to put towards fancy shoes. Each pair of shoes she buys costs $1,000.00. How many shoes can she buy ...

  44. [76]

    Coherent F actuality:

    So Jessica can buy 6 pairs of shoes in a year with the money she sets aside from her paycheck. Coherent F actuality:

  45. [77]

    Jessica sets aside 25% of her paycheck, which is: $2,000.00 x 0.25 = $500.00

  46. [78]

    31 Example 2 Question: The ratio of men to women on a bus is 5:9

    To figure out how many pairs of shoes she can buy in a year, we need to multi- ply the number of pairs she can buy in a month by 12 (the number of months in a year):$500.00 x 12 = $6,000.00. 31 Example 2 Question: The ratio of men to women on a bus is 5:9. If the total number ...

  47. [81]

    Answer: 34

    If 20 women alight from the bus, then the number of women remaining on the bus is 54 − 20 = 34 women. Answer: 34 . Coherent F actuality:

  48. [82]

    The total ratio of men and women on the bus is 5 + 9 = 14

  49. [83]

    If the total number of passengers on the bus is 84, then each ratio unit represents 84 14 = 6 passengers

  50. [84]

    The number of women on the bus is 9 units × 6 passengers/unit = 54 passengers

  51. [85]

    Answer: 34

    If 20 women alight from the bus, then the number of women remaining on the bus is 54 − 20 = 34 women. Answer: 34 . Example 3 Question: John injured his back and needs to go to physical therapy. He went to physical therapy for 6 weeks. Each week he went twice for 2 hours at a t...

  52. [88]

    Coherent F actuality:

    Answer: 3000 . Coherent F actuality:

  53. [89]

    He went to therapy 6 × 2 = ≪ 6 × 2 = 12 ≫ 12 times

  54. [90]

    Each session cost 2 ×125 = $ ≪ 2×125 = 250 ≫ 250

  55. [91]

    So he paid 250 × 12 = $ ≪ 250 × 12 = 3000 ≫ 3000

  56. [92]

    false positive

    Answer: 3000 . 32 M Independent Factuality Baselines We also report the baseline results of independent factuality as in (Mohri & Hashimoto, 2024) for the problems we analyze; these plots are analogous to those we report in Section 6. (a) Calibration plot (b) Fraction of claim...

    2024

  57. [93]

    GPT-4 outputs, GPT-4o as judge Table 2: Results of legibility experiment with LLM-as-a-judge with our method. Outcome Proportion True Positive 0.22 True Negative 0.59 False Positive 0.17 False Negative 0.02 33 Table 3: Results of legibility experiment with LLM-as-a-judge with ...

  58. [94]

    GPT-4 outputs, Llama-3.1-70B-Instruct as judge Table 4: Results of legibility experiment with LLM-as-a-judge with our method. Outcome Proportion True Positive 0.15 True Negative 0.61 False Positive 0.15 False Negative 0.10 Table 5: Results of legibility experiment with LLM-as-...

  59. [95]

    Outcome Proportion True Positive 0.08 True Negative 0.64 False Positive 0.26 False Negative 0.03

    Llama-3.1-70B-Instruct outputs, GPT-4o as judge Table 6: Results of legibility experiment with LLM-as-a-judge with our method. Outcome Proportion True Positive 0.08 True Negative 0.64 False Positive 0.26 False Negative 0.03

  60. [96]

    Outcome Proportion True Positive 0.06 True Negative 0.53 False Positive 0.36 False Negative 0.05 Table 8: Results of legibility experiment with LLM-as-a-judge with our method

    Llama-3.1-70B-Instruct outputs, Llama-3.1-70B-Instruct as judge 34 Table 7: Results of legibility experiment with LLM-as-a-judge with the baseline, independent factuality method. Outcome Proportion True Positive 0.06 True Negative 0.53 False Positive 0.36 False Negative 0.05 T...

  61. [2019]

    Subgraph Generator,

    URL https://arxiv.org/abs/1905.03222. Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. J. Mach. Learn. Res. , 9: 371–421, jun 2008. ISSN 1532-4435. Vaishnavi Shrivastava, Percy Liang, and Ananya Kumar. Llamas know what gpts don’t show: Surrogate models for c...

    1905 arXiv

  62. [2021]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu

    URL https://arxiv.org/abs/2103.03874. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and o...

    2023 arXiv

  63. [2024]

    suggest that LMs will always hallucinate while there exists unknown knowledge, while others such as (Ahdritz et al., 2024) seek to identify uncertainty due to lack of knowledge via linear probes. At the same time, (Zhang et al., 2023) demonstrate LLMs can independently identif...

    2024

  64. [3203]

    URL http://dx.doi.org/10.1016/j.patcog.2021

    doi: 10.1016/j.patcog.2021.108496. URL http://dx.doi.org/10.1016/j.patcog.2021. 108496. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset,

    2021

  65. [5399]

    URL http://dx.doi.org/10.1609/aaai.v38i16.29720

    doi: 10.1609/aaai.v38i16.29720. URL http://dx.doi.org/10.1609/aaai.v38i16.29720. Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness, 2023. URL https://arxiv.org/abs/2308.16175. Shiqi Chen, Yiran Zhao, J...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.