Pith. sign in

REVIEW 10 cited by

KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.06526 v3 pith:NXPBA6PO submitted 2024-10-09 cs.DB

Kaijing Ma , Xinrun Du , Yunran Wang , Haoran Zhang , Zhoufutu Wen , Xingwei Qu , Jian Yang , Jiaheng Liu
show 4 more authors
Minghao Liu Xiang Yue Wenhao Huang Ge Zhang
This is my paper · ORCID
classification cs.DB
keywords kor-benchreasoningknowledge-orthogonalmodelstaskbenchmarkcipherconcept
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we introduce Knowledge-Orthogonal Reasoning (KOR), a concept aimed at minimizing reliance on domain-specific knowledge, enabling more accurate evaluation of models' reasoning abilities in out-of-distribution settings. Based on this concept, we propose the Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench), encompassing five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench emphasizes models' effectiveness in applying new rule descriptions to solve novel rule-driven questions. O1-Preview and O1-Mini achieve accuracies of 72.88% and 70.16%, surpassing Claude-3.5-Sonnet and GPT-4o (58.96% and 58.00%), highlighting the effectiveness of KOR-Bench. We perform detailed analyses, identifying bottlenecks in the Cipher task with Stepwise Prompting, where two rounds of Self-Correction yield optimal results. We evaluate performance across three integrated tasks, explore the impact of Tricks on the Puzzle task, and visualize rule-focused attention. Additionally, we conduct an ablation study on dataset size, benchmark correlations, and zero-shot and three-shot "only questions" experiments. KOR-Bench aims to enhance reasoning evaluation and support further research in this area.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    CreditDecoding accelerates parallel decoding in diffusion LLMs by fusing accumulated Trace Credit with current logits to accept early-correct tokens sooner, yielding up to 5.48x speedup and accuracy gains.

  2. Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Enigmata's synthetic puzzles with verifiable rewards lift a 32B model to 32.8% on ARC-AGI, above o3-mini-high and o1, and give small apparent gains on math and STEM when added to Seed1.5-Thinking.

  3. SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond

    cs.AI 2025-05 conditional novelty 7.0 of 10

    A synthetic logic dataset with verifiable answers, trained via reinforcement learning, improves LLM logical reasoning and generalizes to math and coding.

  4. Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions

    cs.LG 2025-08 conditional novelty 6.0 of 10

    Counterintuitive RL phenomena in LLMs (one-shot, spurious reward, negative-only) appear only under strong model-task alignment measured by pass@k, not because of data contamination.

  5. Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code

    cs.CL 2025-07 conditional novelty 6.0 of 10

    TeaR uses GRPO reinforcement learning on test-case output prediction for algorithmic problems, with no code shown, and reports broad reasoning gains across 17 benchmarks.

  6. CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Training on 3.5M code input-output prediction tasks with natural-language chain-of-thought improves LLM performance on math, logic, symbolic, scientific, and commonsense reasoning benchmarks.

  7. Reasoning Can Hurt the Inductive Abilities of Large Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Chain-of-thought reasoning can hurt LLMs' ability to infer hidden rules from gameplay transcripts, and structured interventions recover the lost accuracy.

  8. The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants

    cs.CL 2025-05 reject novelty 5.0 of 10

    Clustering-based routing plus self-consistency voting among ten 7B open models reportedly outranks GPT-4.1 and GPT-4.5 on average over 15 diverse benchmarks.

  9. KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    KORGym introduces a 51-game, text and visual, multi-turn benchmark with a normalized scoring scheme, and uses it to compare 19 LLMs and 8 VLMs on six reasoning dimensions.

  10. Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.

Pith tools