Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles

Shu-Kai Hsieh; Yu-Fei Shih; Zheng-Lin Lin

REVIEW 4 major objections 6 minor 1 cited by

Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that on Rosetta Stone linguistic puzzles, plain Input-Output prompting makes GPT-4 more accurate than Chain-of-Thought or Solo Performance Prompting, and that explicit reasoning traces are frequently unfaithful to the…

desk verdict Useful empirical comparison of prompting methods on linguistic puzzles, but a CharacTER sign error contradicts the paper's central claim and needs to be fixed before the results can be trusted. read the letter →

arxiv 2502.00817 v1 pith:2K55R7UK submitted 2025-02-02 cs.CL

Zheng-Lin Lin , Yu-Fei Shih , Shu-Kai Hsieh This is my paper · ORCID

classification cs.CL

keywords MachineTranslationReasoningLargeLanguageModelsExplainabilityLinguisticOlympiadTextAnalyticsChain-of-ThoughtPromptingRosettaStoneproblem

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether prompting a large language model to reason aloud helps it solve Rosetta Stone linguistic puzzles, in which the solver must infer a mini-grammar and vocabulary from a few translation pairs and then translate unseen sentences. The paper reports that on both the Puzzling Machine competition set and an adapted Linguistics Olympiad set, GPT-4 0603 scored higher with plain Input-Output prompting than with Chain-of-Thought prompting or Solo Performance Prompting on every metric, including exact match and multiple translation-quality scores. It also documents failures in the explicit reasoning traces: baseless rule and dictionary assumptions, English-biased grammar, rare genuine disagreement among multi-expert personas, and contradictions between a stated rule or dictionary and the final answer. The result matters because it suggests that for low-resource translation puzzles, asking for an explicit reasoning path can add noise rather than accuracy, and that the traces should not be read as faithful explanations of the model's decisions.

What carries the argument

The central object is the Rosetta Stone problem format, a puzzle with a meta section naming the foreign language, a train set of translation pairs, and a test set with one side missing, so that solving requires inducing a small grammar and dictionary. The argument is carried by a controlled comparison of three prompting conditions: IO, which supplies the task and answer format only; CoT, which adds step-by-step reasoning directives inside a two-phase extract-rules-then-apply strategy; and SPP, which simulates discussion among expert personas under a facilitator. The paper evaluates all conditions with exact match, BLEU-2, chrF, characTER, and cosine similarity of Sentence-BERT embeddings, and it classifies reasoning failures into dictionary contradictions and rule contradictions to explain why explicit reasoning underperforms.

What would settle it

Re-run all three prompting conditions on the same 86 Puzzling Machine problems and 28 LO problems with nonzero sampling temperature across, say, ten seeds, and compare confidence intervals on EM and the translation metrics; if the IO-over-CoT intervals overlap broadly, the paper's central ordering is not established. Alternatively, a single held-out set of comparable Rosetta Stone puzzles on which CoT matches or exceeds IO would falsify the claimed universality of the effect.

Watch

Extended reading notes

Core claim

The central claim is that the simplest prompt wins. Across the 86-problem Puzzling Machine set and the 28-problem Linguistics Olympiad set, IO prompting outperformed both CoT and SPP on every reported metric: exact match, BLEU-2, chrF, characTER, and cosine similarity of sentence embeddings. The paper attributes the gap to the quality of the reasoning traces themselves: CoT repeatedly commits to unsupported rules and dictionary pairings without retrospective correction, SPP expert personas rarely challenge one another, and GPT-4's own stated rules or dictionaries are sometimes contradicted by its answers. The paper's explanation is that an incomplete or wrong intermediate grammar becomes noise that hurts translation, whereas IO answers directly from the example pairs.

Load-bearing premise

The load-bearing premise of the comparison is that one run of each prompting method adequately represents that method's performance; the paper reports no sampling variance, temperature, or significance test, so the small observed margins could change or reverse on another run.

Editorial extensions

If this is right

For Rosetta Stone-style translation puzzles, IO is the empirically stronger baseline, and explicit-reasoning prompts need to justify their cost.
CoT's inferred rules and dictionaries should not be treated as reliable; without retrospective correction they can propagate initial errors.
SPP's expert dialogue added little critical disagreement in this domain, so the value of multi-persona prompting is not automatic.
The dictionary-vs-rule contradiction taxonomy provides a concrete diagnostic for checking whether a model's stated reasoning matches its output.
If the paper's noise explanation is right, then methods that use the reasoning trace only to verify or revise an answer, rather than to produce it, may combine the best of both approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because each condition appears to have been run once, with no reported temperature, seed, or variance, the small IO-over-CoT margins (e.g., EM 0.217 vs 0.183 on the LO set) could reverse under sampling; re-running with multiple seeds is a direct, testable check.
A natural extension the paper does not pursue is to compare IO against CoT with self-consistency or a verification stage, which would test whether the problem is reasoning itself or the lack of correction.
The contradiction taxonomy could be automated and applied to other LLM reasoning tasks as a faithfulness measure, independently of whether the final answer is correct.
The observed language-familiarity tiers suggest that prompting effects may differ by whether GPT-4 had substantial exposure to the target language; a stratified analysis by tier would sharpen the comparison.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's genuine contribution is a first systematic comparison of Input-Output, Chain-of-Thought, and Solo Performance Prompting on Rosetta Stone puzzles, plus a newly adapted LO dataset and a qualitative look at reasoning contradictions. The pattern that IO beats CoT and SPP on EM, BLEU, chrF, and CosSim is consistent across two datasets and is a legitimate empirical observation worth reporting. The contradiction analysis is interesting and honestly presented.

The soft spot is more serious than run-to-run stochasticity. The paper defines CharacTER via Wang et al. (2016), which is an edit rate where lower is better. Yet every ranking treats higher CharacTER as better. Table 2 shows IO at 0.642, CoT 0.618, SPP 0.516, ZeroEx 0.317; under the standard definition, IO is the worst and ZeroEx is the best, directly contradicting the text that says IO surpasses all methods and ZeroEx lags behind. Table 3 and the CTER columns in Table 1 show the same inversion. This is not a minor footnote; it invalidates the claim that IO wins on every metric. It also affects Section 5.3, where ZeroEx CharacTER is used to assign proficiency tiers, because the text elsewhere dismisses ZeroEx as uniformly worst. The paper needs to correct the sign interpretation or explain why they are using a nonstandard direction, and re-run the analysis accordingly.

The reader's weaker assumption about missing error bars is also valid but secondary. Each condition is run once, no variance, no significance tests; the margins on the LO dataset are thin, so the ordering could be noise. The LO dataset is small, adapted without release, and the exact prompts are not included, so the study is not reproducible as written. The language-tier thresholds are arbitrary. Single model, single version.

Who is this for? People working on prompting for reasoning and constrained translation tasks will find the comparison useful once the metric issue is resolved. As it stands, the paper needs major revision. I would send it to review, but only after the authors fix the CharacTER error, run repeated trials, and release the data and prompts. Without those changes the central claim cannot be taken seriously.

Referee Report

4 major / 6 minor

Summary. The paper compares four prompting strategies (Input-Output, Chain-of-Thought, Solo Performance Prompting, and a Zero-Example variant) applied to GPT-4 on two collections of Rosetta Stone linguistic puzzles: 86 problems from the Puzzling Machine competition and 28 problems obtained or adapted from UKLO/NACLO. It evaluates outputs with EM, BLEU, chrF, CharacTER, and cosine similarity, and reports that plain IO is the best prompting method on essentially all metrics while CoT and SPP produce explicit reasoning paths that are frequently incomplete or self-contradictory. The paper also proposes a three-tier categorization of GPT-4's language proficiency based on ZeroEx CharacTER scores and discusses qualitative examples of rule and dictionary contradictions.

Significance. If the main finding held, it would be a meaningful empirical counterpoint to the common assumption that eliciting explicit reasoning paths improves LLM performance on constrained translation and reasoning tasks. The evaluation uses external benchmarks and standard metrics, the dataset construction is mostly transparent, and the qualitative analysis of CoT/SPP failure modes is a genuine strength; there is no fitted parameter or circular derivation that would make the result an artifact of the authors' assumptions. However, the impact is currently limited by the small LO dataset, the absence of repeated runs or significance tests, and a serious metric-direction error involving CharacTER, which invalidates the claim that IO wins on 'every' reported metric.

major comments (4)

[§4, §5.1, Tables 2–3] Section 4 defines CharacTER by citing Wang et al. (2016), i.e., a character-level translation edit rate for which lower values are better, but Section 5.1 and Tables 2–3 consistently treat larger values as better. In Table 2, ZeroEx's CharacTER of 0.317 is actually the best score, contrary to the sentence 'ZeroEx lags behind in all metrics'; IO's 0.642 is the worst on that metric, and CoT (0.618) and SPP (0.516) both beat IO. The same inversion appears in Table 3, and Table 1's FE_CTER and EF_CTER columns presumably suffer from the same issue if they are error rates. As a result, the central claim that IO outperforms CoT and SPP on every reported metric is false as stated, and the proficiency-tier analysis in Section 5.3, which builds on ZeroEx CharacTER values in Figure 4, needs to be re-derived under the correct direction of the metric.
[§3.2, Tables 1–3] The comparative claim rests on a single set of outputs per condition, with no report of sampling temperature, number of runs, variance, confidence intervals, or significance tests. The margins separating IO from CoT and SPP are small (e.g., EM 33.79 vs 31.37 in Table 1 and 0.217 vs 0.183 in Table 2), so under GPT-4's stochastic decoding the observed ranking could plausibly reverse on a second run. The authors should either add repeated runs with error bars or significance tests, or explicitly reframe the findings as single-run exploratory observations without claiming stable superiority.
[§3.1, §3.2] The adapted LO dataset is not released, the adaptation procedure is not specified in enough detail to reproduce, and the exact prompt templates for IO, ZeroEx, CoT, and SPP are not provided. Without the adapted problems, the full prompt texts, and a clear statement of which conditions used the two-phase reasoning strategy, the evaluation cannot be independently checked or re-run by other researchers.
[§5.3] The assignment of languages to Limited, Moderate, and High Proficiency tiers in Section 5.3 is presented without any stated threshold or statistical criterion, and Figure 4 is described only as a plot of ZeroEx CharacTER scores without specifying axis labels or a legend. Please state the quantitative rule used to separate the tiers and confirm whether the tier assignments survive after correcting the CharacTER direction issue.

minor comments (6)

[§5.1] In Section 5.1, 'Information Ordering (IO)' and 'Self-Paced Learning (SPP)' are incorrect expansions; these should be Input-Output Prompting and Solo Performance Prompting.
[Tables 1–3] The metric name is written inconsistently as 'CharacTER', 'characTER', and 'ChF-3'; please use 'characTER' and 'chrF' consistently throughout.
[References] The entries Wei et al. 2023a and 2023b appear to be the same paper, and Wu et al. 2023a and 2023b likewise duplicate; please distinguish or merge them.
[Conclusion] The phrase 'our results(see figure)' in the conclusion does not name a figure; it should refer to Figure 4 or be removed.
[Abstract] The abstract contains formatting errors such as 'explorespecificpromptingtechniques' and 'LLMs’ability'; please proofread the text.
[§3.2] The description of ZeroEx says it 'excludes example pairs,' but it is unclear whether ZeroEx still receives the Meta section and test prompt; including one full example of each prompt type would remove this ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical benchmark comparisons, not derivations from fitted inputs or self-citation chains.

full rationale

This paper reports an empirical evaluation of three prompting methods (IO, CoT, SPP) plus a ZeroEx variant on two external benchmark datasets (Puzzling Machine Competition and Linguistics Olympiad problems). The central claim, that IO outperforms CoT and SPP on the reported metrics, is an observed result, not a quantity derived from definitions or from fitted parameters. The evaluation metrics (EM, BLEU, chrF, characTER, CosSim) are standard external measures with stated references, and the benchmark data are external; no metric score is defined in terms of the paper's own conclusions. The only self-citation in the reference list, Lin et al. (2023), is not cited in the body and is not used to justify any load-bearing assumption, uniqueness claim, or ansatz. The paper's reasoning-quality discussion is qualitative and does not feed back into the scores. One notable internal issue is that characTER is an error rate (lower is better) while the text ranks higher values as better, and this affects the accuracy of some narrative claims; however, this is a correctness or interpretation problem, not a circularity problem, because the scores themselves were computed independently and are not constructed to force IO's superiority. There is no equation, fitted parameter, or self-citation chain that makes the observed outcome equivalent to the paper's inputs, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard measurement assumptions and dataset-construction choices, none of which are validated in the paper. No free parameters are fitted, and no new entities are postulated.

assumptions (3)

domain assumption The automatic metrics (BLEU-2, characTER, chrF, CosSim) are valid proxies for translation quality on Rosetta Stone problems.
Section 4 adopts these metrics without task-specific validation or human evaluation, so the reported scores are assumed to reflect actual translation accuracy.
domain assumption The adapted LO problems preserve the difficulty and linguistic validity of the original competition puzzles.
Section 3.1 says the problems were 'modified to match the format used in the Puzzling Machine Challenge', but no analysis is given of how this changes their properties.
domain assumption Single-run generation is representative of each prompting method's performance.
The paper reports one set of scores per method (Tables 1-3) with no repeated runs or configuration details, implying determinism or negligible variance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles." pith.science (2026). https://pith.science/paper/2K55R7UK

@misc{pith2026250200817,
  author       = {Pith},
  title        = {Pith review of: Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2K55R7UK}},
  note         = {Machine review of arXiv:2502.00817}
}

read the original abstract

This paper investigates the utilization of Large Language Models (LLMs) for solving complex linguistic puzzles, a domain requiring advanced reasoning and adept translation capabilities akin to human cognitive processes. We explore specific prompting techniques designed to enhance ability of LLMs to reason and elucidate their decision-making pathways, with a focus on Input-Output Prompting (IO), Chain-of-Thought Prompting (CoT), and Solo Performance Prompting (SPP). Utilizing datasets from the Puzzling Machine Competition and various Linguistics Olympiads, we employ a comprehensive set of metrics to assess the performance of GPT-4 0603, a prominent LLM, across these prompting methods. Our findings illuminate the potential of LLMs in linguistic reasoning and complex translation tasks, highlighting their capabilities and identifying limitations in the context of linguistic puzzles. This research contributes significantly to the broader field of Natural Language Processing (NLP) by providing insights into the optimization of LLM applications for improved reasoning and translation accuracy, thereby enriching the ongoing dialogue in NLP advancements.

Figures

Figures reproduced from arXiv: 2502.00817 by the authors.

**Figure 2.** An example of dictionary contradiction within GPT-4’s reasoning process using CoT Prompting [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Problem format example of northern Algeria language, Kabyle, collected and refined from UKLO. 4. Evaluation This section outlines the methodology adopted to evaluate GPT-4’s proficiency in solving Rosetta Stone Problems, utilizing two distinct datasets: the Puzzling Machine Competition data and a dataset compiled from various Linguistic Olympiads (LO). Central to our investigation are two primary objectives: first… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: characTER score of zero example identified contradictions in the second phase don’t necessarily correlate with output quality. There were instances where GPT-4 disregarded incorrect pre-established rules, resulting in the correct answer, as well as cases where it over…

**Figure 5.** Figure 5: The figure shows the baseless assumption occurring in the CoT discussion on Kiche language. Linguistic anthropologists first propose baseless vocabulary pairs and lexicographers reaffirm the opinion. We conducted a detailed analysis of GPT-4’s reasoning process prim…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Think Through a Bottleneck: Hourglass Reasoning for Rigorous Induction
cs.AI 2026-07 conditional novelty 6.0 of 10

Strict stage isolation that passes only a compressed symbolic schema and rule between LLM calls improves few-shot inductive reasoning more than self-refinement or explicit verbalization alone.

Reference graph

Works this paper leans on

25 extracted references · 5 canonical work pages · cited by 1 Pith paper

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked islrn pid label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprintur...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Bozhidar Bozhanov and Ivan Derzhanski. 2013. Rosetta stone linguistic problems. pages 1--8

work page 2013
[4]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv 2020
[5]

Kahneman Daniel. 2017. Thinking, fast and slow

2017
[6]

Ivan Derzhanski and Thomas Payne. 2010. The linguistics olympiads: Academic competitions in linguistics for secondary school students. Linguistics at school: language awareness in primary and secondary education, pages 213--26

work page 2010
[7]

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. http://arxiv.org/abs/2301.00234 A survey on in-context learning

arXiv 2023
[8]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. http://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the math dataset

arXiv 2021

Show all 25 references

[9]

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. https://doi.org/10.1145/3571730 Survey of hallucination in natural language generation . ACM Computing Surveys , 55(12):1--38

2023 doi
[10]

Chuanyang Jin, Songyang Zhang, Tianmin Shu, and Zhihan Cui. 2023. http://arxiv.org/abs/2308.14242 The cultural psychology of large language models: Is chatgpt a holistic or analytic thinker?

2023 arXiv
[11]

Zheng-Lin Lin, Chiao-Han Yen, Jia-Cheng Xu, Deborah Watty, and Shu-Kai Hsieh. 2023. https://aclanthology.org/2023.rocling-1.33 Solving linguistic olympiad problems with tree-of-thought prompting . In Proceedings of the 35th Conference on Computational Linguistics and Speech Pr...

2023
[12]

OpenAI. 2023. http://arxiv.org/abs/2303.08774 Gpt-4 technical report

2023 arXiv
[13]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

2002
[14]

Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever. 2022. http://arxiv.org/abs/2202.01344 Formal mathematics statement curriculum learning

2022 arXiv
[15]

Stanislas Polu and Ilya Sutskever. 2020. http://arxiv.org/abs/2009.03393 Generative language modeling for automated theorem proving

2020 arXiv
[16]

Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics

2015 doi
[17]

Nils Reimers and Iryna Gurevych. 2019. http://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks

2019 arXiv
[18]

Jannis Vamvas. 2022. https://vamvas.ch/translation-puzzles-are-in-context-learning-tasks Translation puzzles are in‑context learning tasks

2022
[19]

Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. https://doi.org/10.18653/v1/W16-2342 C harac T er: Translation edit rate on character level . In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 505--510,...

2016 doi
[20]

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 a . http://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models

2023 arXiv
[21]

Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023 b . http://arxiv.org/abs/2307.05300 Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration

2023 arXiv
[23]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023 b . http://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models

2023 arXiv
[25]

Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. 2023 b . http://arxiv.org/abs/2306.01337 An empirical study on challenging math problem solving with gpt-4

2023 arXiv
[26]

Griffiths, Yuan Cao, and Karthik Narasimhan

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. http://arxiv.org/abs/2305.10601 Tree of thoughts: Deliberate problem solving with large language models

2023 arXiv
[27]

Gözde İşgüder, Yova Kementchedjhieva, Phillip Rust, and Iryna Gurevych. 2020. https://doi.org/10.18653/v1/2020.acl-main.115 Puzzling machines: A challenge on learning from small data . pages 1241--1254

2020 doi

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked islrn pid label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprintur...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Bozhidar Bozhanov and Ivan Derzhanski. 2013. Rosetta stone linguistic problems. pages 1--8

work page 2013

[4] [4]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv 2020

[5] [5]

Kahneman Daniel. 2017. Thinking, fast and slow

2017

[6] [6]

Ivan Derzhanski and Thomas Payne. 2010. The linguistics olympiads: Academic competitions in linguistics for secondary school students. Linguistics at school: language awareness in primary and secondary education, pages 213--26

work page 2010

[7] [7]

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. http://arxiv.org/abs/2301.00234 A survey on in-context learning

arXiv 2023

[8] [8]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. http://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the math dataset

arXiv 2021