Pith. sign in

REVIEW 3 major objections 6 minor 48 references

Argumentatively Coherent Judgmental Forecasting

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper defines argumentative coherence (a forecaster's prediction must match their own argumentative reasoning) and claims that filtering out incoherent forecasters consistently improves forecasting accuracy for humans and LLM-based…

desk verdict Solid formal contribution, but the 'consistent improvement' claim rides on in-sample and underpowered evaluations; the coherence notion deserves a serious referee, not a desk reject. read the letter →

arxiv 2507.23163 v2 pith:LWN7PSPR submitted 2025-07-30 cs.AI

Deniz Gorur , Antonio Rago , Francesca Toni This is my paper · ORCID
classification cs.AI
keywords judgmentalforecastingargumentationframeworksargumentativecoherenceQBAFDF-QuADlargelanguagemodelsforecastaggregationArguCast
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a forecaster's numerical prediction should be consistent with their own stated reasoning about the supporting and attacking arguments in a forecasting debate, and it formalises this requirement as argumentative coherence. It claims that filtering out forecasters who violate this coherence improves aggregate forecasting accuracy, and shows this in an in-person study with humans and in experiments with four LLM-based forecasters, where restricting to coherent predictions improved or preserved accuracy across the board. A crowd-sourced study adds that ordinary people do not naturally reason in this coherent way, which the paper takes as a reason for forecasting systems to enforce the filter automatically rather than relying on users' intuition. If the claim holds, any debate-style forecasting platform can screen forecasters with nothing more than the internal consistency of their own reasoning, without waiting for historical outcomes.

What carries the argument

The carrying mechanism is the forecaster QBAF, a Quantitative Bipolar Argumentation Framework built from a single user's votes, where disagreeing with an argument flips the direction of its attack or support edges, evaluated with the DF-QuAD gradual semantics (a function that combines an argument's base score with the strengths of its attackers and supporters into a final strength $\sigma(f)$). The coherence check then compares $\sigma(f)$ against a reasoning-midpoint threshold $\xi_1$ and the prediction against an expected-probability threshold $\xi_2$, declaring the forecaster prediction coherent only if the two comparisons agree in direction. This machinery does the work because the entire filter is defined on it, and because the neutrally set base scores endow it with the neutral-forecasting and bias-based-forecasting properties that earlier frameworks lacked, ensuring that abstention leaves strength at 0.5 and that one-sided agreement visibly moves strength away from 0.5.

What would settle it

Take the LLM experiments and filter the same number of predictions at random, matching the coherence filter's retention rate for each model and breadth, while keeping everything else fixed; if random subsampling reproduces the same accuracy gains, then coherence is not the active ingredient and the central claim is refuted. A cleaner variant would repeat the in-person study with the coherence threshold $\xi_2$ fixed from an independent source (for example a neutral 0.5) while evaluating against the original baseline, since gains that vanish whenever the filter is not tuned to the evaluation target would also undermine the claim that coherence itself improves accuracy.

Watch

Extended reading notes

Core claim

The central discovery is a formal, parameterised definition of argumentative coherence plus an empirical regularity that follows from it: forecasters whose predictions contradict their own argumentative reasoning are measurably more likely to be wrong. For each forecaster the framework builds a forecaster QBAF, a graph of supporting and attacking arguments in which a 'disagree' vote flips the stance of the relevant edge, and the DF-QuAD gradual semantics turns that graph into a strength $\sigma(f)$ for the forecasting argument. The forecaster is prediction coherent exactly when $\sigma(f)$ crossing the reasoning midpoint $\xi_1$ implies the prediction $P(u,f)$ crossing the expected-probability threshold $\xi_2$ in the same direction, with equality handled by a small tolerance. In the in-person ArguCast study applying this filter moved average predictions closer to the Metaculus and Betfair baselines, and in the LLM experiments it lifted accuracy consistently (for example GPT-4o from 69% to 80% at breadth 1,1), while the crowd-sourced study showed users diverge from the notion, aligning with it more as debate complexity increases.

Load-bearing premise

Closeness to a single external baseline (Metaculus or Betfair) is assumed to measure forecasting accuracy, and in the in-person study the same baseline value is used both as the coherence threshold and as the evaluation target, so if those baselines are noisy or the filter is tuned to the target, the accuracy gains are not independent evidence that coherence itself helps.

Editorial extensions

If this is right

  • Argumentation-based forecasting tools such as ArguCast should enforce a coherence filter before aggregating group forecasts, because users do not spontaneously satisfy the property.
  • The filter transfers to LLM-based forecasting without retraining: across Mistral, Mixtral, Llama 3, and GPT-4o, restricting to coherent predictions improved or preserved accuracy, with GPT-4o rising from 69% to 80% at breadth 1,1.
  • The coherence parameters matter: a domain-tuned threshold $\xi_2$ (e.g., derived from prediction-market probabilities) outperforms a neutral 0.5 threshold, so the notion should be parameterised per forecasting task.
  • Coherence provides a historical-data-free quality signal for weighting or filtering forecasters, complementing accuracy-based methods that require outcome data.
  • Making the notion graded (a degree of coherence rather than a binary outcome) would let systems weight predictions by coherence, which the paper identifies as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A matched random-subsample control is the natural next test: because coherence filtering removes many samples (as few as 44% of samples retained for Llama 3 at breadth 1,1), part of the accuracy gain could be a sample-size selection effect rather than coherence itself, and the paper does not run this control.
  • In the in-person study the same baseline value $\gamma$ plays two roles: it sets the coherence threshold $\xi_2$ and it is the target used to judge improvement; a cleaner test would fix $\xi_2$ from an independent source and evaluate against $\gamma$ only afterwards.
  • If the effect is real, coherence filtering is a free quality screen: it needs no historical outcomes, only the forecaster's own argument structure, so it could be applied to debate-style platforms for questions that have no prior baseline data at all.
  • The complexity result suggests the user mismatch on simple debates may be partly presentational; reformulating double-negative votes (disagreeing with an attacker) in more intuitive language could raise both user alignment and achievable accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes a formal notion of argumentative coherence for judgmental forecasting: a forecaster is prediction coherent if, for every forecasting argument, the sign of the difference between their prediction and a threshold xi2 matches the sign of the difference between the argument's dialectical strength and a threshold xi1 (with an epsilon neighbourhood for equality). The notion is instantiated in a variant of the ArguCast system with QBAFs and the DF-QuAD semantics, and two sanity properties (neutral and bias-based forecasting) are proved. The authors report three evaluations: an in-person study at a festival comparing raw average predictions with averages after filtering by coherence against Metaculus/Betfair baselines; an LLM experiment in which ArgLLMs generate supporting/attacking arguments and the coherent subset's accuracy is compared with raw accuracy on 2,923 predictions; and a Prolific study measuring user agreement with fictitious forecasters' predictions. The paper claims that filtering by coherence improves forecasting accuracy consistently and that users do not naturally align with the notion, motivating automatic filtering.

Significance. The formal definition is a useful step: it is parameterised, clearly operationalised through QBAFs, and comes with two proved sanity properties. If the empirical claims were solid, the coherence filter would be an inexpensive and broadly applicable quality filter for human and LLM forecasting. The authors also make their data available and use an appropriate McNemar test and sample-size planning in the crowdsourced study. However, the load-bearing empirical claim, that filtering by coherence consistently improves accuracy, is not established by the present experiments. The in-person evaluation is a small in-sample comparison with no statistical testing and uses the evaluation target as a filter threshold; the LLM evaluation lacks a random-filtering null and shows no improvement for one model; and the crowdsourced study measures agreement with predictions rather than perceived coherence. These shortcomings make the contribution's claimed practical value uncertain and require either additional analyses or a substantial softening of the claims.

major comments (3)
  1. [Section 5.4, Table 1] The in-person study as reported does not provide independent evidence that coherence improves accuracy. In the column labelled xi2 = gamma, the baseline gamma used as the evaluation target is also the coherence threshold in Definition 2. The filtering rule keeps predictions that are on the 'correct' side of gamma relative to the argument's strength, so the movement of the filtered average toward gamma is partly an artifact of the selection rule. In addition, footnote 8 states that no statistical testing was performed, while the text concludes 'we can reject the null hypothesis for RQ1'; with retained sample sizes of N=6 for Labour and Nvidia, this conclusion is unsupported by the data. The raw-versus-coherent comparison is in-sample and cannot be used to infer a general forecasting improvement.
  2. [Section 6.3, Table 2] The LLM experiment does not establish a causal role for coherence. Since the coherent subsets retain between 44% and 78% of the original 2,923 predictions, a random subset of the same size would already vary in accuracy around the raw mean. No bootstrap confidence intervals, error bars, or random-filtering control are reported, so the observed accuracy differences (e.g., 75% to 78% for Mixtral at breadth 1,1) cannot be separated from sampling variation. Moreover, the claim of 'consistently' improving accuracy is not literally supported by Table 2: Mixtral at breadth n,k shows 75% accuracy before and after filtering. The conclusion that RQ2 is 'answered affirmatively' is therefore premature.
  3. [Sections 7.2 and 7.4] The crowd-sourced study's dependent variable is whether the user agrees with Alex's prediction, not whether the user judges the underlying reasoning to be coherent. Figure 4 and the design description in Section 7.2 show that users are asked to agree or disagree with a fictitious forecaster's prediction, and Table 3 labels this as 'User Coherent?'. Agreement with a prediction is a plausible behavioural proxy, but it is not a direct elicitation of perceived coherence; the conclusion that users do not 'align with this coherence property' is thus an interpretation of the proxy rather than a measurement of it. The authors should either elicit coherence judgments directly or explicitly frame the result as concerning agreement behaviour and temper the conceptual conclusions.
minor comments (6)
  1. [Section 6.3] The sentence 'at breadth n,k generally retaining more samples compared to breadth n,k' appears to contain a typo; it should compare breadth n,k with breadth 1,1.
  2. [Section 5.4] Footnote 8 states that no statistical testing was performed, while the main text says the null hypothesis is rejected; please remove or qualify the rejection claim.
  3. [Table 2 caption] The caption says 'For raw LLMs, we show accuracy as (correct/total)', but the table displays percentages; please make the notation consistent.
  4. [Section 7.1, Definition 4] The formal condition for vote complexity uses equalities between votes without specifying whether the votes are '+' or '−', making it hard to parse; consider rewriting the condition with explicit signs.
  5. [Section 8] The sentence 'These results from the in-person and LLM experiments support the practical need for integrating argumentative coherence' is stronger than the evidence warrants; suggest 'provide preliminary evidence for'.
  6. [Section 4, after Definition 2] The phrase 'with arguments' base scores set up as in Definition 2' is confusing because base scores are defined in Definition 1, not Definition 2; please correct the cross-reference.

Circularity Check

1 steps flagged · score 4.0 of 10

The in-person study's xi2=gamma arm uses the same baseline value both as the coherence filter threshold and as the accuracy target, so part of the reported improvement is induced by the filter; the LLM and crowd studies are not circular, making the overall circularity partial.

  1. fitted input called prediction [Section 5.2 (Experimental Design), Section 5.4 (Results, Table 1) with Definition 2 (Section 4)]
    "we applied our notion of coherence with xi1(f) = 0.5 and two options for xi2: xi2(f) = 0.5, which is a neutral assignment, and xi2(f) = gamma, where gamma is the initial prediction, which we obtained from prediction markets and Metaculus. ... Due to the small sample size and exploratory design, we did not perform statistical testing. ... Overall, we can reject the null hypothesis for RQ1, since applying coherence (for appropriate choices of the parameters) improves forecasting accuracy."

    In the xi2=gamma arm, the same value gamma serves both as the coherence threshold in Definition 2 (if sigma(f)<0.5 then P<gamma; if sigma(f)>0.5 then P>gamma) and as the evaluation target against which the filtered average is scored as 'improved'. The filter is thus defined using the answer key: it discards predictions lying on the wrong side of gamma given the forecaster's computed argument strength, so the retained average is nudged towards gamma before any comparison is made.

full rationale

The coherence definition (Definition 2) is self-contained, and the crowd-sourced user study (RQ3/RQ4) is independent of the accuracy claim, using McNemar's test and one-sided t-tests with stated sample-size calculations. The LLM study is also not circular: xi2 is set to 0.5 and accuracy is scored against real GJOpen outcomes, so the filter does not use the evaluation target; its weakness is the absence of a matched random-filtering control and null distribution, and Mixtral at breadth n,k shows no gain (75% vs 75%), which undercuts 'consistently' but is a statistical, not circular, issue. The one genuine circular step is the in-person xi2=gamma arm (Sections 5.2 and 5.4): the same gamma is both the coherence threshold and the accuracy target, no statistical testing was performed (stated in the paper), and retained samples are as small as 6, so part of the observed movement toward the baseline is induced by the filter's construction. Self-citations (DF-QuAD, Rago et al. 2016; ArguCast, Gorur et al. 2023; ArgLLMs, Freedman et al. 2025; the ECAI version, Gorur et al. 2025) are load-bearing in assembling the pipeline but not circular, because the empirical comparisons are against external baselines (Metaculus, Betfair, GJOpen ground truth) and the coherence property itself is newly defined here. Net result: one partially-circular evaluation arm while the central claim retains independent content, so the circularity score is 4.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The core definition of coherence is self-contained, but its empirical support rests on domain assumptions about how votes map to argument strength, about the validity of external baselines as ground truth, and about the interpretation of user agreement as perceived coherence. There are no invented physical entities.

free parameters (5)
  • xi1(f): argument-strength threshold = Set to 0.5 in all experiments.
    Threshold separating predominantly attacked from predominantly supported forecasting arguments; the paper calls it task-specific and says it needs optimisation, but fixes it at 0.5 (end of Section 4).
  • xi2(f): prediction threshold = Set to 0.5 or to gamma, the Metaculus/Betfair initial probability.
    Expected probability used to decide whether a prediction is coherent with argument strength; in the in-person study xi2=gamma is the same baseline used to measure accuracy (Sections 5.2 and 5.4).
  • epsilon in Definition 2 = Unspecified, described only as some small epsilon.
    Width of the coherence band when argument strength exactly equals xi1; never instantiated in the experiments.
  • Base score tau(f) for forecasting arguments = Set to 0.5 in the LLM experiments; generally allowed in [0,1].
    Confidence in the forecasting argument before debate; the paper notes it can be varied to control weight but fixes it to 0.5 (Section 6.2).
  • Base-score mapping for non-forecasting arguments = Set to 0.5 for positive and negative votes, 0 for neutral and undefined votes.
    Chosen in Definition 1 so neutral votes have no effect and agree and disagree votes start from the same weight; not empirically calibrated.
assumptions (5)
  • domain assumption DF-QuAD gradual semantics correctly computes the dialectical strength of a forecasting argument from a forecaster QBAF.
    The coherence definition in Definition 2 and all experiments rely on sigma(f) from DF-QuAD (Rago et al. [2016]) as the representation of a forecaster's reasoning.
  • ad hoc to paper A forecaster's votes can be converted into flipped attack and support edges with base scores 0.5 or 0, and the resulting QBAF represents their internal reasoning.
    Definition 1 constructs forecaster QBAFs this way; no independent evidence is given that this mapping captures human reasoning.
  • domain assumption Metaculus and Betfair initial predictions are accurate ground truth for the three in-person questions.
    Section 5.2 treats these baselines as the target and Section 5.4 evaluates ArguCast by closeness to them.
  • ad hoc to paper User agreement with a fictitious forecaster's prediction measures alignment with the proposed coherence notion.
    Section 7.2 asks whether users agree with Alex's prediction and Section 7.4 interprets the results as evidence about perceived coherence; the equivalence is not validated.
  • standard math The background QBAF and ArguCast formalisms from Baroni et al. [2019] and Gorur et al. [2023] are assumed.
    The framework builds on these cited definitions; no new foundational axioms are introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Argumentatively Coherent Judgmental Forecasting." pith.science (2026). https://pith.science/paper/LWN7PSPR

@misc{pith2026250723163,
  author       = {Pith},
  title        = {Pith review of: Argumentatively Coherent Judgmental Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LWN7PSPR}},
  note         = {Machine review of arXiv:2507.23163}
}
read the original abstract

Judgmental forecasting employs human opinions to make predictions about future events, rather than exclusively historical data as in quantitative forecasting. When these opinions form an argumentative structure around forecasts, it is useful to study the properties of the forecasts from an argumentative perspective. In this paper, we advocate and formally define a property of argumentative coherence, which, in essence, requires that a forecaster's reasoning is coherent with their forecast. We then conduct three evaluations with our notion of coherence. First, we assess the impact of enforcing coherence on human forecasters as well as on Large Language Model (LLM)-based forecasters, given that they have recently shown to be competitive with human forecasters. In both cases, we show that filtering out incoherent predictions improves forecasting accuracy consistently, supporting the practical value of coherence in both human and LLM-based forecasting. Then, via crowd-sourced user experiments, we show that, despite its apparent intuitiveness and usefulness, users do not generally align with this coherence property. This points to the need to integrate, within argumentation-based judgmental forecasting, mechanisms to filter out incoherent opinions before obtaining group forecasting predictions.

Figures

Figures reproduced from arXiv: 2507.23163 by the authors.

Figure 1
Figure 1. The QBAF in Example 1 as a graph, with arguments repre [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. An example of ArgLLM (with Mistral-7B-Instruct-v0.3 as [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Example of a debate shown to users. This includes a [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Supporting and attacking arguments generated for the [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 7
Figure 7. Figure 7: Supporting and attacking arguments generated for the [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Full debate for the forecasting question ‘Will X win their next tennis match against Y?’. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Full debate for the forecasting question ‘Will Party A win the next election against Party B?’. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Abstract structure of the simple debate variant. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 15
Figure 15. Figure 15: Abstract structure of the vote/depth complex debate [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 12
Figure 12. Figure 12: Abstract structure of the breadth complex debate variant. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Abstract structure of the depth complex debate variant. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 17
Figure 17. Figure 17: Abstract structure of the vote/depth/breadth complex [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Experimental design for our user experiments where subfigure a) shows the experimental design for the Simple-Complex group [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 40 canonical work pages

  1. [1]

    Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AI

    MAhdi Abolghasemi, Odkhishig Ganbold, and Kristian Rotaru. Humans vs large language models: Judgmental forecasting in an era of advanced AI . CoRR , 2023. URL https://doi.org/10.48550 https://doi.org/10.48550/arXiv.2312.06941

    work page Pith review arXiv doi:10.48550/arxiv.2312.06941 2023

  2. [2]

    Evaluation of arguments in weighted bipolar graphs

    Leila Amgoud and Jonathan Ben-Naim. Evaluation of arguments in weighted bipolar graphs. Int. J. of Approximate Reasoning , 99:39--55, 2018. URL https://www.sciencedirect.com https://www.sciencedirect.com/science/article/pii/S0888613X1730590X

    work page 2018

  3. [3]

    Hal R. Arkes. Overconfidence in Judgmental Forecasting , pages 495--515. Springer US, Boston, MA, 2001

    work page 2001

  4. [4]

    Towards artificial argumentation

    Katie Atkinson, Pietro Baroni, Massimiliano Giacomin, Anthony Hunter, Henry Prakken, Chris Reed, Guillermo Ricardo Simari, Matthias Thimm, and Serena Villata. Towards artificial argumentation. AI Magazine , 38(3):25--36, 2017. URL https://doi.org/10.1609 https://doi.org/10.1609/aimag.v38i3.2704

    doi:10.1609/aimag.v38i3.2704 2017

  5. [5]

    Handbook of Formal Argumentation

    Pietro Baroni, Dov Gabbay, Massimiliano Giacomin, and Leendert van der Torre, editors. Handbook of Formal Argumentation . College Publications, 2018

    work page 2018

  6. [6]

    From fine-grained properties to broad principles for gradual argumentation

    Pietro Baroni, Antonio Rago, and Francesca Toni. From fine-grained properties to broad principles for gradual argumentation. IJAR , 105:252--286, February 2019. URL https://linkinghub.elsevier.com https://linkinghub.elsevier.com/retrieve/pii/S0888613X18304651

    work page 2019

  7. [7]

    Benjamin, Fred Morstatter, Ali E

    Daniel M. Benjamin, Fred Morstatter, Ali E. Abbas, Andr \' e s Abeliuk, Pavel Atanasov, Stephen Bennett, Andreas Beger, Saurabh Birari, David V. Budescu, Michele Catasta, Emilio Ferrara, Lucas Haravitch, Mark Himmelstein, K. S. M. Tozammel Hossain, Yuzhong Huang, Woojeong Jin, Regina Joseph, Jure Leskovec, Akira Matsui, Mehrnoosh Mirtaheri, Xiang Ren, Gle...

    work page 2023

  8. [8]

    Developing expert political judgment

    Welton Chang, Eva Chen, Barbara Mellers, and Philip Tetlock. Developing expert political judgment. Judgment and Decision Making , 11(5):509--526, September 2016. URL https://www.cambridge.org https://www.cambridge.org/core/product/identifier/S1930297500004599/type/journal_article

    work page 2016

Show all 48 references
  1. [9]

    The Llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. CoRR , 2024. URL https://doi.org/10.48550 https://doi.org/10.48550/arXiv.2407.21783

    2024 doi arXiv

  2. [10]

    Budescu, David Mandel, and Mark Himmelstein

    Yuyu Fan, David V. Budescu, David Mandel, and Mark Himmelstein. Improving accuracy by coherence weighting of direct and ratio probability judgments. Decis. Anal. , 16:197--217, 2019. URL https://doi.org/10.1287/deca.2018.0388 https://doi.org/10.1287/deca.2018.0388

    2019

  3. [11]

    Argumentative large language models for explainable and contestable claim verification

    Gabriel Freedman, Adam Dejl, Deniz Gorur, Xiang Yin, Antonio Rago, and Francesca Toni. Argumentative large language models for explainable and contestable claim verification. In AAAI , 2025. URL https://doi.org/10.1609/aaai.v39i14 https://doi.org/10.1609/aaai.v39i14.33637

    2025 doi

  4. [12]

    Moran, Nicholas G

    Graham Casey Gibson, Kelly R. Moran, Nicholas G. Reich, and Dave Osthus. Improving probabilistic infectious disease forecasting through coherence. PLoS Comput. Biol. , 17:1--20, 2021. URL https://doi.org/10.1371/journal.pcbi https://doi.org/10.1371/journal.pcbi.1007623

    2021 doi

  5. [13]

    FORECAST2023: A forecast and reasoning corpus of argumentation structures

    Kamila Gorska, John Lawrence, and Chris Reed. FORECAST2023: A forecast and reasoning corpus of argumentation structures. In LREC/COLING , 2024. URL https://aclanthology.org/2024.lrec-main https://aclanthology.org/2024.lrec-main.652

    2024

  6. [14]

    Argucast: A system for online multi-forecasting with gradual argumentation

    Deniz Gorur, Antonio Rago, and Francesca Toni. Argucast: A system for online multi-forecasting with gradual argumentation. In Arg&App@KR , 2023

    2023

  7. [15]

    Argumentatively coherent judgmental forecasting

    Deniz Gorur, Antonio Rago, and Francesca Toni. Argumentatively coherent judgmental forecasting. In ECAI , 2025

    2025

  8. [16]

    Approaching human-level forecasting with language models

    Danny Halawi, Fred Zhang, Chen Yueh - Han, and Jacob Steinhardt. Approaching human-level forecasting with language models. In NeurIPS , 2024. URL http://papers.nips.cc http://papers.nips.cc/paper\_files/paper/2024/hash/5a5acfd0876c940d81619c1dc60e7748-Abstract-Conference.html

    2024

  9. [17]

    Ho, David V

    Emily H. Ho, David V. Budescu, and Mark Himmelstein. Measuring probabilistic coherence to identify superior forecasters. Int. J. of Forecasting , 41:596--612, 2024. URL https://www.sciencedirect.com https://www.sciencedirect.com/science/article/pii/S0169207024000104

    2024

  10. [18]

    Forecasting argumentation frameworks

    Benjamin Irwin, Antonio Rago, and Francesca Toni. Forecasting argumentation frameworks. In KR , 2022

    2022

  11. [19]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. CoRR , 2023. URL https://doi.org/10.48550/arXiv.2310 https://doi.org/10.48...

    2023 doi arXiv

  12. [20]

    Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. CoRR , 2024. URL https://doi.org/10.48550/arXiv https://doi.org/10.4855...

    2024 doi arXiv

  13. [21]

    Zhang, Xiaoming Shi, Pin - Yu Chen, Yuxuan Liang, Yuan - Fang Li, Shirui Pan, and Qingsong Wen

    Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin - Yu Chen, Yuxuan Liang, Yuan - Fang Li, Shirui Pan, and Qingsong Wen. Time- LLM : Time series forecasting by reprogramming large language models. In ICLR , 2024

    2024

  14. [22]

    Thinking, Fast and Slow

    Daniel Kahneman. Thinking, Fast and Slow . Farrar, Straus and Giroux, New York, 2011

    2011

  15. [23]

    Karvetski, Kenneth C

    Christopher W. Karvetski, Kenneth C. Olson, David R. Mandel, and Charles R. Twardy. Probabilistic coherence weighting for optimizing expert forecasts. Decis. Anal. , 10:305--326, 2013. URL https://doi.org/10.1287/deca.2013 https://doi.org/10.1287/deca.2013.0279

    2013

  16. [24]

    Karvetski, Carolyn Meinel, Daniel T

    Christopher W. Karvetski, Carolyn Meinel, Daniel T. Maxwell, Yunzi Lu, Barbara A. Mellers, and Philip E. Tetlock. What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters? Int. J. of Forecasting , 38:688--704, 2022. URL https://www.scienced...

    2022

  17. [25]

    Judgmental forecasting: A review of progress over the last 25 years

    Michael Lawrence, Paul Goodwin, Marcus O'Connor, and Dilek Önkal. Judgmental forecasting: A review of progress over the last 25 years. Int. J. of Forecasting , 22:493--518, 2006. URL https://www.sciencedirect.com https://www.sciencedirect.com/science/article/pii/S0169207006000501

    2006

  18. [26]

    Argumentation mining: State of the art and emerging trends

    Marco Lippi and Paolo Torroni. Argumentation mining: State of the art and emerging trends. ACM Transactions on Internet Technology , 16, 2016

    2016

  19. [27]

    Taming pre-trained LLMs for generalised time series forecasting via cross-modal knowledge distillation

    Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, and Shu - Tao Xia. Taming pre-trained LLMs for generalised time series forecasting via cross-modal knowledge distillation. CoRR , 2024

    2024

  20. [28]

    Note on the sampling error of the difference between correlated proportions or percentages

    Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika , 12:153--157, June 1947

    1947

  21. [29]

    Olivares, Geoffrey N \' e giar, Ruijun Ma, O

    Kin G. Olivares, Geoffrey N \' e giar, Ruijun Ma, O. Nangba Meetei, Mengfei Cao, and Michael W. Mahoney. Probabilistic forecasting with coherent aggregation. CoRR , 2023

    2023

  22. [30]

    Gpt-4o mini: advancing cost-efficient intelligence, 2024

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2024. URL https://openai.com https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  23. [31]

    Anastasios Panagiotelis, Puwasala Gamakumara, George Athanasopoulos, and Rob J. Hyndman. Probabilistic forecast reconciliation: Properties, evaluation and score optimisation. Eur. J. Oper. Res. , 306:693--706, 2023

    2023

  24. [32]

    Rothschild, Andrew Gordon, Zak Evernden, and Ekaterina Damer

    Eyal P \'e er, David M. Rothschild, Andrew Gordon, Zak Evernden, and Ekaterina Damer. Data quality of platforms and panels for online behavioral research. Behavior Research Methods , 54:1643--1662, 2021

    2021

  25. [33]

    Interpreting neural networks as quantitative argumentation frameworks

    Nico Potyka. Interpreting neural networks as quantitative argumentation frameworks. In AAAI , 2021. URL https://doi.org/10.1609/aaai.v35i7 https://doi.org/10.1609/aaai.v35i7.16801

    2021 doi

  26. [34]

    Discontinuity-free decision support with quantitative argumentation debates

    Antonio Rago, Francesca Toni, Marco Aurisicchio, and Pietro Baroni. Discontinuity-free decision support with quantitative argumentation debates. In KR . AAAI Press, 2016. URL http://www.aaai.org http://www.aaai.org/ocs/index.php/KR/KR16/paper/view/12874

    2016

  27. [35]

    Interactive explanations by conflict resolution via argumentative exchanges

    Antonio Rago, Hengzhi Li, and Francesca Toni. Interactive explanations by conflict resolution via argumentative exchanges. In KR , 2023

    2023

  28. [36]

    Werner, Konstantinos Benidis, Pedro Mercado, Jan Gasthaus, and Tim Januschowski

    Syama Sundar Rangapuram, Lucien D. Werner, Konstantinos Benidis, Pedro Mercado, Jan Gasthaus, and Tim Januschowski. End-to-end learning of coherent probabilistic forecasts for hierarchical time series. In ICML , 2021

    2021

  29. [37]

    Coherent probabilistic forecasting of temporal hierarchies

    Syama Sundar Rangapuram, Shubham Kapoor, Rajbir - Singh Nirwan, Pedro Mercado, Tim Januschowski, Yuyang Wang, and Michael Bohlke - Schneider. Coherent probabilistic forecasting of temporal hierarchies. In AISTATS , 2023

    2023

  30. [38]

    Philipp Schoenegger and Peter S. Park. Large language model prediction capabilities: Evidence from a real-world forecasting tournament. CoRR , 2023. URL https://doi.org/10.48550 https://doi.org/10.48550/arXiv.2310.13014

    2023 doi arXiv

  31. [39]

    Park, Ezra Karger, and Philip E

    Philipp Schoenegger, Peter S. Park, Ezra Karger, and Philip E. Tetlock. AI -augmented predictions: LLM assistants improve human forecasting accuracy. CoRR , 2024. URL https://doi.org/10.48550 https://doi.org/10.48550/arXiv.2402.07862

    2024 doi arXiv

  32. [40]

    Time series forecasting with LLMs : Understanding and enhancing model capabilities

    Hua Tang, Chong Zhang, Mingyu Jin, Qinkai Yu, Zhenting Wang, Xiaobo Jin, Yongfeng Zhang, and Mengnan Du. Time series forecasting with LLMs : Understanding and enhancing model capabilities. SIGKDD Explor. , 26(2):109--118, 2024

    2024

  33. [41]

    Tetlock and Dan Gardner

    Philip E. Tetlock and Dan Gardner. Superforecasting . Random House, 2016

    2016

  34. [42]

    Tetlock, Barbara A

    Philip E. Tetlock, Barbara A. Mellers, Nick Rohrbaugh, and Eva Chen. Forecasting tournaments: Tools for increasing transparency and improving the quality of debate. Current Directions in Psychological Science , 23, 2014

    2014

  35. [43]

    Thomson, Andrew C

    Mary E. Thomson, Andrew C. Pollock, Dilek Önkal, and M. Sinan Gönül. Combining forecasts: Performance and coherence. International Journal of Forecasting , 35:474--484, 2019. URL https://www.sciencedirect.com https://www.sciencedirect.com/science/article/pii/S0169207018301869

    2019

  36. [44]

    Forecasting with jury-based probabilistic argumentation

    Francesca Toni, Antonio Rago, and Kristijonas Cyras. Forecasting with jury-based probabilistic argumentation. J. Appl. Non Class. Logics , 33:224--243, 2023

    2023

  37. [45]

    Wallsten and David V

    Thomas S. Wallsten and David V. Budescu. A review of human linguistic probability processing: General principles and empirical evidence. Knowl. Eng. Rev. , 10(1):43--62, 1995. URL https://doi.org/10.1017 https://doi.org/10.1017/S0269888900007256

    1995 doi

  38. [46]

    Abbas, David V

    Maximilian Zellner, Ali E. Abbas, David V. Budescu, and Aram Galstyan. A survey of human judgement and quantitative forecasting methods. Royal Society Open Science , 8:201187, 2021

    2021

  39. [47]

    Retrieval-augmented generation for ai-generated content: A survey

    Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. CoRR , 2024

    2024

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.