Consistency in Language Models: Current Landscape, Challenges, and Future Directions

Borhane Blili-Hamelin; Carol Anderson; Domenic Rosati; Jekaterina Novikova; Subhabrata Majumdar

REVIEW 3 major objections 5 minor 2 cited by

Consistency in Language Models: Current Landscape, Challenges, and Future Directions

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This review argues that large language models are frequently inconsistent and that the field lacks any standard definition, metric, or benchmark for consistency, making reliability hard to measure.

desk verdict A clear, useful taxonomy of consistency research, but its field-level percentages rest on an undocumented survey corpus and should not be taken as measured fact. read the letter →

arxiv 2505.00268 v2 pith:CHOIHC43 submitted 2025-05-01 cs.CL cs.AI

Jekaterina Novikova , Carol Anderson , Borhane Blili-Hamelin , Domenic Rosati , Subhabrata Majumdar This is my paper · ORCID

classification cs.CLcs.AI

keywords consistencylargelanguagemodelsevaluationbenchmarkslogicalsemanticcross-lingualself-consistencyreliability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This review argues that large language models frequently give different answers to equivalent questions, and that researchers cannot yet reliably measure this because there is no shared definition, metric, or benchmark for consistency. The paper surveys consistency research from 2019 to 2025, organizes the field into logical/formal and nonlogical/informal types of consistency, and finds that a slim majority of studies use standard NLP tasks while most evaluations focus on English, generative transformer models. It also finds that very few methods actually improve consistency, and that most existing approaches address symptoms rather than root causes. If the paper is right, progress on model reliability is currently hard to measure, compare, or verify.

What carries the argument

The organizing device is a behavioral-consistency taxonomy that splits logical/formal consistency from nonlogical/informal consistency, anchored in the negational, symmetric, transitive, and additive classification and in the semantic equivalence property that equivalent inputs should produce equivalent outputs. The measurement machinery is pairwise evaluation: input-based sampling creates paraphrases or perturbed prompts, output-based sampling generates multiple decodings from identical inputs, and base metrics such as BERTScore, ROUGE, and entailment or contradiction scores are aggregated over pairs, sometimes through semantic entropy. This taxonomy and sampling-and-metric pipeline is what lets the paper compare otherwise disparate studies and identify the gaps in the landscape.

What would settle it

A reader could settle the landscape claim by running a preregistered systematic search over the same 2019–2025 window with explicit inclusion criteria; if it surfaced many peer-reviewed studies on encoder-only models, consistency-oriented pretraining, or adversarial consistency attacks that the review omitted, the claim that those directions are underexplored would need revision. The central claim that no standard evaluation exists would be falsified by the emergence of a benchmark that the field widely adopts as the default consistency test.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that consistency research is fragmented: authors define consistency differently or not at all, evaluation relies on ad hoc pairwise similarity metrics applied to paraphrased or repeated prompts, and no standard benchmarks have emerged. The authors organize existing work into logical/formal consistency, which includes negational, symmetric, transitive, additive, and semantic consistency, and nonlogical/informal consistency, which includes moral, norm, and informational/factual consistency. They find that most studies use standard NLP tasks in English with decoder-only or encoder-decoder transformer models, and that current improvement methods, such as fine-tuning on consistent input-output pairs or encouraging self-consistency in chain-of-thought reasoning, mainly address symptoms. The paper concludes with a call for standardized definitions, comprehensive multilingual benchmarks, combined automatic and human evaluation, and research into the structural basis of consistency.

Load-bearing premise

The paper's landscape description rests on the assumption that the peer-reviewed publications and influential preprints it selected from 2019 to 2025 are representative of consistency research as a whole.

Editorial extensions

If this is right

Until definitions and metrics are standardized, consistency scores reported across studies are not comparable, and the performance of state-of-the-art models can be overestimated.
Evaluation protocols that use high-temperature output sampling can inflate apparent inconsistency, so consistency results need to be reported with their sampling setup.
Current improvement methods, such as fine-tuning on consistent input-output pairs and self-consistency decoding, mainly treat symptoms; lasting progress likely requires consistency-oriented pretraining or architectural changes.
Cross-lingual deployment is riskier than monolingual deployment because safety guardrails, factual associations, and even political positions can shift with the input language.
Consistency should be evaluated together with factuality, safety, and helpfulness, since a model can be internally consistent yet factually wrong, or it may sacrifice consistency to maintain safety.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the taxonomy were adopted widely, a natural next step is a consistency report card that scores a model separately on logical, semantic, factual, and moral consistency rather than reporting one aggregate number.
The review's emphasis on root causes suggests a testable hypothesis: if inconsistency is not merely a decoding artifact, then meaning-preserving interpolations in a model's internal representation space should produce consistent outputs, and checking this would localize the source of inconsistency.
A low-cost extension would be to build a multilingual paraphrase set from the existing benchmarks the paper reviews, which would directly quantify the cross-lingual consistency gap the paper identifies.
The review's observation that adversarial consistency attacks are underexplored implies that subtle prompt perturbations could serve as a practical stress test for consistency before deployment.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a read if you work on consistency or evaluation of LLMs. The paper is a workshop review that organizes the scattered literature into a sensible two-part taxonomy: logical/formal consistency against nonlogical/informal, with the Jang et al. categories and the newer moral/norm/factual variants mapping cleanly into that frame. It also gives a digestible summary of metrics (pairwise similarity, semantic entropy, etc.) and tasks (QA, summarization, NLI, reasoning). The writing is clear, and the call for standardized definitions and holistic benchmarks is reasonable. As a synthesis, it earns its place.

The load-bearing weakness is the survey methodology, or rather the lack of one. The paper reports proportions—a slim majority of studies use standard NLP tasks, about a third use custom tasks, more than two-thirds use generative transformers, etc.—but never defines the denominator. There is no search strategy, no inclusion/exclusion criteria, no screening process, and no table of included studies. The reference list compounds the problem: Asai & Hajishirzi appears twice (2020a and 2020b), Liu et al. twice (2023 and 2024b), and Raj et al. twice (2022 and 2025), so an independent reader cannot even count the corpus reliably. This matters because the paper's core empirical claim—that the field is fragmented and lacks standard approaches—rests partly on these proportions. If the sample is skewed, the diagnosis could be an artifact of the selection. I would push back on any suggestion that this flaw is minor; it is the difference between a measured landscape and an informed opinion. The author's own citations (Raj et al.) appear as part of the landscape, not as premises for the diagnosis, so I do not see a circularity problem.

Minor points: the multimodal appendix is thin, but the paper explicitly scopes itself to text-only, so that is not a real flaw. Deeper treatment of the trade-off between consistency and creativity is promised but only sketched; again, fine for a workshop position paper.

Verdict: conditional acceptance. A serious referee should ask for the screening protocol and a table of surveyed papers so the percentages are auditable. Without that, the quantitative claims should be softened. The taxonomy and recommendations are worth keeping.

Referee Report

3 major / 5 minor

Summary. This paper is a narrative review of research on consistency in text-only language models, covering literature from 2019 to 2025. It proposes a taxonomy that separates logical/formal consistency (negational, symmetric, transitive, additive, semantic) from nonlogical/informal consistency (moral, norm, informational/factual), and it surveys which tasks, datasets, models, and evaluation metrics are used in the field. The paper's central claims are that most advanced LLMs exhibit inconsistent behavior, that there is no standard terminology or benchmark for consistency, and that the field needs standardized definitions, comprehensive multilingual benchmarks, and evaluation protocols combining automatic and human assessment. It closes with a call to action and a short appendix on multimodal consistency.

Significance. If established, the paper's central diagnosis would be valuable: it identifies a core reliability property of LMs that currently lacks shared definitions, metrics, and benchmarks, and it connects this to downstream risks in high-stakes and multilingual settings. The taxonomy of consistency types is a useful organizing device, and the discussion of challenges such as decoding-time bias, adversarial consistency attacks, and multilingual gaps is sensible. The paper also gives concrete recommendations that a research community could act on. However, the quantitative landscape claims—the proportions of studies using certain tasks, models, and evaluation styles—are presented without a documented corpus, so the field-level diagnosis is not yet reproducible. The qualitative examples and taxonomy are useful, but they alone do not establish the strength of the headline claims.

major comments (3)

[Abstract and §2 (Challenges)] The quantitative statements in §2 are the backbone of the claim that consistency research lacks standard approaches: the paper reports that 'a slim majority of studies' use standard NLP tasks, 'approximately a third' use custom tasks, 'more than two-thirds' use transformer-based generative LMs, 'slightly more than half' test proprietary models, and 'about a quarter' focus on encoder-only models. Yet no denominator is defined. There is no search strategy, no inclusion or exclusion criteria, no screening protocol, no coding scheme, and no table of the included studies. The reference list mixes surveyed primary studies with background citations and contains duplicate entries (Asai & Hajishirzi 2020a and 2020b are the same paper; Liu et al. 2023 and Liu et al. 2024b are versions of the same work), so an independent reader cannot reconstruct the sample or verify any of the percentages. This undermines the reproducibility of the central field-level diagnosis. Please add a documented search and screening protocol, a table of included studies with coding, and counts for each reported proportion.
[Abstract and §2 (Challenges)] The abstract states that 'most advanced large language models (LLMs) struggle with consistency and frequently demonstrate inconsistent behavior,' and the introduction repeats this as a field-level fact. The evidence cited is Elazar et al. (2021) and Raj et al. (2025), which demonstrate inconsistency on specific tasks and model families, not a systematic survey of all advanced LLMs. As written, this overstates the strength of the evidence. Please either support the claim with a systematic synthesis of the reviewed corpus or soften it to a claim such as 'many state-of-the-art LLMs have been shown to exhibit inconsistent behavior in a range of tasks and settings.' This distinction matters because the paper's call to action rests on the severity and generality of the inconsistency problem.
[§2 (Challenges) and §3 (Improving Consistency)] Several other quantitative or comparative claims are asserted without supporting counts. In §2, the paper says 'very few studies explore how inconspicuous or subtle manipulation of prompts can lead to inconsistent LLM responses,' and in §3 it says 'There are surprisingly few approaches that actually increase the consistency of LMs.' These are the kinds of landscape claims that require a denominator and an explicit search. Without a documented corpus, these statements are not auditable. Please provide the number of studies reviewed and the number classified into each category, or rephrase as qualitative observations with representative examples.

minor comments (5)

[§2 (Analyzed Tasks)] The percentages for task categories are presented without stating whether the categories are mutually exclusive; a study using both a standard task and a custom task would need a coding rule. Please clarify the coding procedure or acknowledge overlaps.
[§2 (Terminology)] The sentence 'Definitions of factual consistency are often not clearly specified, and instead are replaced with human annotations' is unclear: human annotations can define ground truth, but they do not replace a definition. Please rephrase to clarify the intended contrast.
[§3 (Improving Consistency)] The list of approaches that improve consistency is confusingly phrased: 'Elazar et al. (2021) used a custom loss function, Raj et al. (2025) used knowledge distillation from more consistent teacher models, and Raj et al. (2025); Zhao et al. (2024) used synthetic datasets...' The repetition of Raj et al. (2025) and the semicolon are likely formatting errors. Please revise for clarity and ensure each cited work is credited for the correct contribution.
[Appendix A] The opening sentence of Appendix A, 'Until 2022, every consistency study was analyzing robustness of LMs to various text perturbations or to semantically equivalent texts only,' is a universal negative claim over all prior studies. This is not supported by the review's documented corpus and could be easily falsified. Please qualify the claim with respect to the reviewed literature.
[Throughout] There are minor typographical and formatting errors, for example 'parahrasing' in §2 (Dataset Size and Availability), 'Enteprise' in the affiliation block, and inconsistent use of 'LM' versus 'LLM' in places. A careful proofreading pass would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; self-citations are incidental and the survey's landscape claims rest on external literature, with an undocumented corpus as a transparency weakness.

full rationale

This paper is a survey and position paper rather than a derivation: it does not fit parameters, define a target quantity in terms of its own output, or invoke a uniqueness theorem. The central claims—that LLMs are inconsistent and that the field lacks standard evaluation approaches—are supported by citations to external works (e.g., Elazar et al. 2021; Jang et al. 2022) and by the survey's own qualitative taxonomy of consistency types. The two self-citations (Raj et al. 2022 and Raj et al. 2025) appear only as examples of semantic-consistency measurement and of a knowledge-distillation improvement method; they are descriptive entries in the landscape, not load-bearing premises, so they do not create circularity. The most significant weakness is that the corpus behind quantitative statements such as 'A slim majority of studies' and 'Approximately a third of existing studies' is not documented with a search strategy, inclusion criteria, or a study table, making those proportions unauditable; however, an undocumented sample is a transparency and reproducibility concern, not a case where a prediction reduces by construction to its inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities. The review's load-bearing premises are the representativeness of the surveyed literature and the chosen framing of behavioral, text-only consistency.

assumptions (3)

domain assumption The surveyed literature from 2019-2025, described as peer-reviewed publications and influential preprints, is representative of LM consistency research.
All landscape-level claims in Section 2 about task distribution, model types, and evaluation approaches rest on this selection.
domain assumption Behavioral consistency is the appropriate frame for studying consistency in LMs.
Section 2 states that the paper limits 'the otherwise broader concept of consistency to behavioral consistency'; this framing determines which papers are included.
domain assumption Text-only LMs are the relevant scope; multimodal consistency is excluded from the main analysis.
Section 1 states that the paper restricts attention to text-only LMs, with multimodal consistency covered only briefly in Appendix A.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Consistency in Language Models: Current Landscape, Challenges, and Future Directions." pith.science (2026). https://pith.science/paper/CHOIHC43

@misc{pith2026250500268,
  author       = {Pith},
  title        = {Pith review of: Consistency in Language Models: Current Landscape, Challenges, and Future Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CHOIHC43}},
  note         = {Machine review of arXiv:2505.00268}
}

read the original abstract

The hallmark of effective language use lies in consistency: expressing similar meanings in similar contexts and avoiding contradictions. While human communication naturally demonstrates this principle, state-of-the-art language models (LMs) struggle to maintain reliable consistency across task- and domain-specific applications. Here we examine the landscape of consistency research in LMs, analyze current approaches to measure aspects of consistency, and identify critical research gaps. Our findings point to an urgent need for quality benchmarks to measure and interdisciplinary approaches to ensure consistency while preserving utility.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Measuring and Improving Behavioral Consistency in Large Language Models through Fact-Heuristic-Emotion State Enforcement
cs.CL 2026-06 conditional novelty 6.0 of 10

A prompt that forces LLMs to separate facts, inferences, and emotions reduces repeated-answer variability (+0.016 to +0.021 SI on a ~0.95 baseline) and, under injected state persistence, cuts decision-flip rate by 82%...
Graph RAG as Human Choice Model: Building a Data-Driven Mobility Agent with Preference Chain
cs.AI 2025-08 unverdicted novelty 5.0 of 10

A graph-retrieval-augmented LLM agent, the Preference Chain, matched real-world transport mode choices on the Replica dataset better than a standard LLM.

Reference graph

Works this paper leans on

13 extracted references · 2 canonical work pages · cited by 2 Pith papers

[5]

org/CorpusID:263609159

URL https://api.semanticscholar. org/CorpusID:263609159. Lin, W., Gerchanovsky, A., Akgul, O., Bauer, L., Fredrik- son, M., and Wang, Z. Llm whisperer: An incon- spicuous attack to bias llm responses. arXiv preprint arXiv:2406.04755, 2024a. Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y ., and Zhang, H. Towards trustworthy llms: a review on debiasing and ...

arXiv
[8]

org/CorpusID:264145744

URL https://api.semanticscholar. org/CorpusID:264145744. Qin, L., Xie, T., Huang, S., Chen, Q., Xu, X., and Che, W. Don’t be contradicted with anything! CI-ToD: To- wards benchmarking consistency for task-oriented dia- logue system. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Con- ference on Empirical Methods ...

2021
[10]

Tan, Z., Yang, X., Ye, Z., Wang, Q., Yan, Y ., Nguyen, A., and Huang, K

Association for Computational Linguistics. Tan, Z., Yang, X., Ye, Z., Wang, Q., Yan, Y ., Nguyen, A., and Huang, K. Ssd: Towards better text-image consis- tency metric in text-to-image generation. arXiv preprint arXiv:2210.15235, 2022. ¨Ust¨un, A., Aryabumi, V ., Yong, Z.-X., Ko, W.-Y ., D’souza, D., Onilude, G., Bhandari, N., Singh, S., Ooi, H.-L., Kayid...

arXiv
[11]

Wang, A., Cho, K., and Lewis, M

doi: 10.1145/3613904.3642135. Wang, A., Cho, K., and Lewis, M. Asking and answer- ing questions to evaluate the factual consistency of sum- maries. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 5008–5020, Online, July 2020. Association for C...

arXiv
[13]

org/CorpusID:270870062

URL https://api.semanticscholar. org/CorpusID:270870062. Yang, L., Ma, Y ., and Zhang, Y . Measuring consistency in text-based financial forecasting models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13751–13765, Toronto, Can...

arXiv 2022
[134]

acl-long.134

URL https://aclanthology.org/2021. acl-long.134. Ohmer, X., Bruni, E., and Hupkes, D. From form (s) to meaning: Probing the semantic depths of language mod- els using multisense consistency. Computational Linguis- tics, pp. 1–51, 2024. Paleka, D., Hadjikyriacou, A., Daneshjou, R., Gleave, A., and Steinhardt, J. Consistency checks for language model foreca...

arXiv 2021
[173]

acl-main.173/

URL https://aclanthology.org/2020. acl-main.173/. Mitchell, E., Noh, J. J., Li, S., Armstrong, W. S., Agar- wal, A., Liu, P., Finn, C., and Manning, C. D. Enhanc- ing self-consistency and performance of pre-trained lan- guage models through natural language inference. arXiv preprint arXiv:2211.11875, 2022. M¨undler, N., He, J., Jenko, S., and Vechev, M. S...

arXiv 2020
[182]

emnlp-main.182

URL https://aclanthology.org/2021. emnlp-main.182. Rabinovich, E., Ackerman, S., Raz, O., Farchi, E., and Anaby Tavor, A. Predicting question-answering per- formance of large language models through semantic consistency. In Gehrmann, S., Wang, A., Sedoc, J., Clark, E., Dhole, K., Chandu, K. R., Santus, E., and Sedghamiz, H. (eds.), Proceedings of the Thir...

arXiv 2021

Show all 13 references

[382]

acl-main.382

URL https://aclanthology.org/2020. acl-main.382. Chen, A., Phang, J., Parrish, A., Padmakumar, V ., Zhao, C., Bowman, S. R., and Cho, K. Two failures of self- consistency in the multi-step reasoning of LLMs. Trans- actions on Machine Learning Research , 2024. ISSN 2835-8856. U...

2020
[450]

what it can create, it may not understand

URL https://aclanthology.org/2020. acl-main.450. Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y ., Gao, J., Awadallah, A. H., and Li, B. Adversarial glue: A multi-task benchmark for robustness evaluation of lan- guage models, 2022. URL https://arxiv.org/ abs/2111.02840. Wang, X...

2020 arXiv
[991]

emnlp-main.991

URL https://aclanthology.org/2023. emnlp-main.991. Jang, M., Kwon, D. S., and Lukasiewicz, T. Becel: Bench- mark for consistency evaluation of language models. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 3680–3696, 2022. Jin, Y ., Chan...

2023
[2023]

org/CorpusID:264405758

URL https://api.semanticscholar. org/CorpusID:264405758. Kim, S. S. Y ., Vaughan, J. W., Liao, Q. V ., Lombrozo, T., and Russakovsky, O. Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies. In CHI Conference on Human ...

2025
[2024]

Cui, W., Zhang, J., Li, Z., Damien, L., Das, K., Malin, B., and Kumar, S

doi: 10.1145/3613904.3641904. Cui, W., Zhang, J., Li, Z., Damien, L., Das, K., Malin, B., and Kumar, S. Dcr-consistency: Divide-conquer- reasoning for consistency evaluation and improvement of large language models, 2024. URL https://arxiv. org/abs/2401.02132. Dediu, D. and Cy...

2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [5]

org/CorpusID:263609159

URL https://api.semanticscholar. org/CorpusID:263609159. Lin, W., Gerchanovsky, A., Akgul, O., Bauer, L., Fredrik- son, M., and Wang, Z. Llm whisperer: An incon- spicuous attack to bias llm responses. arXiv preprint arXiv:2406.04755, 2024a. Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y ., and Zhang, H. Towards trustworthy llms: a review on debiasing and ...

arXiv

[2] [8]

org/CorpusID:264145744

URL https://api.semanticscholar. org/CorpusID:264145744. Qin, L., Xie, T., Huang, S., Chen, Q., Xu, X., and Che, W. Don’t be contradicted with anything! CI-ToD: To- wards benchmarking consistency for task-oriented dia- logue system. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Con- ference on Empirical Methods ...

2021

[3] [10]

Tan, Z., Yang, X., Ye, Z., Wang, Q., Yan, Y ., Nguyen, A., and Huang, K

Association for Computational Linguistics. Tan, Z., Yang, X., Ye, Z., Wang, Q., Yan, Y ., Nguyen, A., and Huang, K. Ssd: Towards better text-image consis- tency metric in text-to-image generation. arXiv preprint arXiv:2210.15235, 2022. ¨Ust¨un, A., Aryabumi, V ., Yong, Z.-X., Ko, W.-Y ., D’souza, D., Onilude, G., Bhandari, N., Singh, S., Ooi, H.-L., Kayid...

arXiv

[4] [11]

Wang, A., Cho, K., and Lewis, M

doi: 10.1145/3613904.3642135. Wang, A., Cho, K., and Lewis, M. Asking and answer- ing questions to evaluate the factual consistency of sum- maries. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 5008–5020, Online, July 2020. Association for C...

arXiv

[5] [13]

org/CorpusID:270870062

URL https://api.semanticscholar. org/CorpusID:270870062. Yang, L., Ma, Y ., and Zhang, Y . Measuring consistency in text-based financial forecasting models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13751–13765, Toronto, Can...

arXiv 2022

[6] [134]

acl-long.134

URL https://aclanthology.org/2021. acl-long.134. Ohmer, X., Bruni, E., and Hupkes, D. From form (s) to meaning: Probing the semantic depths of language mod- els using multisense consistency. Computational Linguis- tics, pp. 1–51, 2024. Paleka, D., Hadjikyriacou, A., Daneshjou, R., Gleave, A., and Steinhardt, J. Consistency checks for language model foreca...

arXiv 2021

[7] [173]

acl-main.173/

URL https://aclanthology.org/2020. acl-main.173/. Mitchell, E., Noh, J. J., Li, S., Armstrong, W. S., Agar- wal, A., Liu, P., Finn, C., and Manning, C. D. Enhanc- ing self-consistency and performance of pre-trained lan- guage models through natural language inference. arXiv preprint arXiv:2211.11875, 2022. M¨undler, N., He, J., Jenko, S., and Vechev, M. S...

arXiv 2020

[8] [182]

emnlp-main.182

URL https://aclanthology.org/2021. emnlp-main.182. Rabinovich, E., Ackerman, S., Raz, O., Farchi, E., and Anaby Tavor, A. Predicting question-answering per- formance of large language models through semantic consistency. In Gehrmann, S., Wang, A., Sedoc, J., Clark, E., Dhole, K., Chandu, K. R., Santus, E., and Sedghamiz, H. (eds.), Proceedings of the Thir...

arXiv 2021