Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimate

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By inducing an abstract rule from the query before retrieval and then re-scoring candidate rules for usefulness, retrieval and reasoning accuracy improve across datasets, rule formats, retrievers, and model sizes.

desk verdict A competent, well-scoped retrofit of generation-augmented retrieval and LLM reranking to rule retrieval, but the abstract's 'consistently' claim is contradicted by the paper's own CLUTRR/7B numbers, and no code or error bars are released. read the letter →

arxiv 2505.10870 v1 pith:X4K4HVKO submitted 2025-05-16 cs.CL cs.AIcs.IR

Ziyang Huang , Wangtao Sun , Jun Zhao , Kang Liu This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords ruleretrievalretrieve-then-reasonself-inductionrelevancere-estimationsemanticmisalignmentLLMpromptingBenchqueryexpansion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rule retrieval sits at the bottleneck of retrieve-then-reason pipelines: queries state concrete facts ('Alice moved to California, where a new law mandates recycling') while library rules are abstract templates ('If PersonX moves to RegionZ and LegislationY applies to RegionZ, then PersonX must obey LegislationY'). Vanilla sparse and dense retrievers fail on this mismatch, and feeding their noisy top results into an LLM often lowers reasoning accuracy below not using rules at all. This paper claims that two prompting-based additions fix the problem without any training: SIAR has the LLM abstract the query into a self-induced rule and uses that rule (alone or with the query) for retrieval, and R3 has the LLM re-rank the retrieved rules by whether their abstract knowledge can be instantiated back into the query's facts and actually help reasoning. On CLUTRR, ULogic, and CAIL2018, SIAR and R3 improve Recall@k and downstream Match scores over vanilla retrieval across natural and formal rule formats, sparse and dense retrievers, and 7B/72B open models plus GPT-4o. If the claim holds, rule-based reasoning becomes reliably grounded in libraries of abstract knowledge rather than being derailed by retrieval noise.

What carries the argument

The load-bearing object is the self-induced rule (SI): an LLM-generated rewrite of the query that replaces concrete entities with typed variables and hypothesizes an inferential relationship, e.g., 'If PersonX moves to RegionY and RegionY has implemented LegislationZ, then PersonX needs to follow LegislationZ.' Its role is to project the query into the abstract 'rule subspace' so that sparse or dense retrievers can match it against library rules sharing the same underlying logic. The second mechanism is the R3 listwise re-ranking prompt, modeled on RankGPT-style prompting, which asks the LLM to order retrieved rules by two criteria: whether the rule's abstract knowledge can be instantiated into the query's facts, and whether the rule would help answer the query. This re-estimation corrects retrievers' reliance on surface similarity and adds a second chance to fix imperfect inductions. The two mechanisms are inserted before and after the retrieval stage, respectively, so the whole retrieval pipeline remains training-free and only requires LLM prompting.

What would settle it

Compare SIAR to an oracle variant in which the query is replaced by the library's own golden rule (variable names intact). If the oracle retrieval Recall@1 is not substantially higher than vanilla retrieval, the semantic-gap explanation collapses; if SIAR approaches the oracle, the self-induction step is doing the hypothesised projection. A second, cheaper check: on CLUTRR with Qwen2.5-7B, where the paper reports SIAR (w/ SI) Recall@1 dropping from 6.67 to 2.29, show whether the drop persists across query splits.

Watch

Extended reading notes

Core claim

The central discovery is that the retrieval bottleneck in rule-based reasoning is a representational one: queries live in an instantiated, concrete subspace while rules live in an abstract, variable-based subspace, and standard lexical or dense similarity cannot bridge the two. The paper's solution is a two-stage projection. First, Self-Induction Augmented Retrieval (SIAR) prompts the LLM to summarize and abstract the facts in the query into a hypothesized inferential rule—a 'self-induced rule'—that is used as the retrieval query; this projects the query toward the rule subspace and lifts Recall@1, with gains that grow with model scale. Second, Rule Relevance ReEstimate (R3) prompts the LLM to rank the top-n retrieved rules by whether the abstract knowledge in each rule can be instantiated into the query's facts and whether applying the rule would help reasoning, yielding a reranked list that is more useful for downstream reasoning. The combination consistently beats vanilla retrieval on three datasets and across rule formats, retrievers, and LLMs, and the paper reports that R3 alone applied to vanilla retrieval also helps, confirming that relevance re-estimation and query-side induction are complementary.

Load-bearing premise

The load-bearing premise is that an LLM can, from the query alone, generate a self-induced rule that is semantically close enough to the correct library rule to move retrieval in the right direction; if induction fails, SIAR not only stops helping but can hurt, as the paper itself finds on CLUTRR with Qwen2.5-7B.

Editorial extensions

If this is right

  • When the golden rule is supplied, reasoning accuracy jumps by roughly 24–32 points on average, so improving rule retrieval is a direct lever on reasoning quality.
  • SIAR consistently improves Recall@1 over vanilla retrieval across natural and formal rule libraries, with the largest gains (up to ~67 points on CAIL2018 with dense retrieval) coming when the self-induced rule alone is used as the dense-retrieval query.
  • R3 further boosts both retrieval and reasoning on ULogic and CAIL2018 across nearly all setups, and even 'vanilla retrieval + R3' (without self-induction) improves over vanilla, showing that relevance re-estimation is independently valuable.
  • Larger models benefit more from both components, indicating that rule-induction and relevance-estimation capabilities scale with model size.
  • Sparse retrieval with the concatenated query (SI+input) tends to win, while dense retrieval prefers the pure SI query—so the optimal query form depends on the retriever, not on the rule format.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the self-induction projection is genuinely the cause of the gains, then the same two-stage recipe should transfer to other retrieval-about-abstractions tasks, such as retrieving legal statutes from crime narratives, medical guideline rules from symptom descriptions, or policy clauses from incident reports.
  • The paper's own failure case (CLUTRR + Qwen2.5-7B, where SIAR w/ SI Recall@1 drops from 6.67 to 2.29) suggests a testable boundary: when the LLM's induction is weaker than the retriever's lexical overlap, self-induction can hurt; a guard that falls back to the original query when the induced rule is low-confidence could make the method robust.
  • Because both components are prompt-only and training-free, they can be layered on top of any retriever or reasoning LLM; a natural next experiment is to measure how the gains scale with rule-library size beyond the 166–1,048 rules used here, where retrieval noise becomes the dominant error source.
  • The paper treats the query-subspace/rule-subspace split as nearly non-overlapping; if that picture is right, then contrastive training of a dense retriever on (query, induced-rule) pairs should outperform fine-tuning on (query, golden-rule) pairs, because the induced rules are the actual bridge.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses rule retrieval in retrieve-then-reason pipelines, where instantiations in a query are hard to match against abstract rules. It proposes Self-Induction Augmented Retrieval (SIAR), which uses an LLM to rewrite the query into an abstract 'self-induced' rule, optionally concatenated with the original query, before sparse or dense retrieval; and Rule Relevance ReEstimate (R3), an LLM-based reranker that re-orders retrieved rules by estimated applicability to the query. Experiments on CLUTRR, ULogic, and CAIL2018 from RuleBench, with BM25 and bge retrievers and several LLMs, report Recall@1/5/10 and a Match reasoning metric. The paper claims that SIAR and R3 consistently improve retrieval and reasoning across settings.

Significance. The paper identifies a real and understudied problem: the semantic gap between instantiated queries and abstract rules in rule retrieval. The proposed methods are training-free, require no fitted parameters, and can be layered on top of existing retrievers, which is a practical strength. In many configurations the reported gains are large (e.g., CAIL2018 Recall@1 rising from 25.30 with vanilla sparse retrieval to 83.73 with SIAR-R3 (w/ SI+input) for GPT-4o), and the ablations across retrievers, model families, and rule-base size provide useful evidence of broad applicability. However, the headline claim of consistency is contradicted by the paper's own tables, and the evaluation protocol involves post-hoc selection of the query format on the same test sets, which weakens the strength of the generalizability claims.

major comments (3)
  1. [Section 4.1, Tables 1 and 4] The claim that 'SIAR can consistently improve performance compared to vanilla retrieval' is directly contradicted by the reported numbers. On CLUTRR with Qwen2.5-7B-Instruct and sparse BM25 over the natural-language rule base, vanilla retrieval has Recall@1 = 6.67, while SIAR (w/ SI) drops to 2.29, SIAR-R3 (w/ SI) drops to 2.00, and even SIAR-R3 (w/ SI+input) reaches only 4.58; Recall@5 and Recall@10 also drop for SIAR (w/ SI) (16.60/8.30 and 24.52/11.93). The formal-language table (Table 4) shows the same pattern (Recall@1 of 2.39 vs. 6.58 for SIAR w/ SI). CLUTRR is one of only two synthetic benchmarks and Qwen2.5-7B is one of three model families tested, so this is not a negligible edge case. The 'consistently' wording in the abstract, introduction, and Section 4.1 should be replaced with a quantified per-configuration summary and an attempt to explain the failure mode on CLUTRR with smaller models.
  2. [Section 4.2 and Appendix D] The query-format choice (w/ SI vs. w/ SI+input) is made per retriever after observing the retrieval results on the same test sets. Section 4.2 states: 'Based on the conclusion from the previous section, for sparse retrieval, we use SI+input as the query, while for dense retrieval, we use SI as the query for retrieval.' No validation split or pre-registered protocol is described. The reasoning improvements in Table 2 are therefore not unbiased estimates of the expected performance of the pipeline; they are selected after peeking at the test data. The authors should fix the choice a priori using a held-out split, report both query formats for reasoning results, or use a nested evaluation design.
  3. [Section 4 and Appendix B] No seeds, repeated runs, standard deviations, or significance tests are reported, and no code, prompts, or constructed rule libraries are released. Several differences that underlie the claims are very small in absolute terms (e.g., Table 4, CLUTRR sparse, 72B: vanilla Recall@1 = 6.58 and SIAR w/ SI = 6.58), while the CLUTRR 7B results show large drops that could be noise. Without variance estimates or multiple runs, the reader cannot judge whether the reported improvements are stable. The authors should report at least 3 seeds with mean and standard deviation, and release the code, prompts, and rule libraries for reproducibility.
minor comments (5)
  1. [Appendix C] The heading contains a typo: 'Talbe 4' should be 'Table 4'; Figure 2 also contains 'Seif-Induction' instead of 'Self-Induction'.
  2. [Section 5.3] Section 5.3 refers to 'Table 3' when describing the counterfactual-rule robustness experiment; the actual table appears to be Table 6. Section 5.2 also uses lowercase 'table 5'.
  3. [Table 6] In Table 6, the second block is labeled 'w/ sparse retrieval' again; it should be 'w/ dense retrieval'.
  4. [Appendix C, Section 4.1] The gpt-4o model is omitted from all formal-language experiments for cost reasons, yet Section 4.1 claims that 'Open-source models have comparable performance with closed-source models' across settings. This limitation should be stated in the main experimental design, not only in the appendix, because it restricts the scope of the comparability claim.
  5. [Section 4 and Related Work] The paper uses RuleBench as the evaluation suite but does not clearly state in the main text that RuleBench is the authors' own resource (Sun et al., 2024b). This is not itself a flaw, but it should be disclosed to the reader for transparency.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: SIAR and R3 are untrained query-side augmentations evaluated on external benchmarks; the only self-citation (RuleBench) is not load-bearing.

full rationale

I walked the claimed derivation chain. SIAR uses an LLM to induce an abstract rule from the query, then uses that induced rule (alone or concatenated with the original query) as the retrieval query against a fixed rule library. R3 reranks the retrieved rules by prompting the LLM to assess relevance to the original query, and reasoning uses the top-1 rule. None of these stages trains parameters, fits benchmark labels, or defines its output in terms of the other method's output. The self-induced rule is not inserted into the rule library, so the retrieval target remains the externally defined golden rule. The paper's choice of w/SI versus w/SI+input per retriever is a post hoc design selection made after inspecting retrieval tables, which raises a generalizability/cherry-picking concern but is not a circular reduction: the reported Recall and Match numbers are still measured against external labels rather than being forced by construction. The claimed 'consistent improvement' is contradicted by the CLUTRR/Qwen2.5-7B rows in Tables 1 and 4, but an internal empirical contradiction is a correctness issue, not circularity. The only self-reference is the use of RuleBench (Sun et al., 2024b), which includes the present authors; however, RuleBench wraps external datasets (CLUTRR, ULogic, CAIL2018) and no load-bearing argument or uniqueness theorem is imported from that citation. The limitation paragraph acknowledging small rule libraries further weakens external generality but does not create an equation-level identity between inputs and predictions. No specific circular step can be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on LLM capabilities and benchmark construction rather than on free parameters. Only two top-k thresholds and a post-hoc query-format choice are hand-set, and no new physical or formal entities are introduced.

free parameters (3)
  • top_k = 10
    The paper retrieves top-10 rules for retrieval evaluation and uses top-1 for reasoning; the value is fixed without sensitivity analysis or validation.
  • top_n = 20
    SIAR-R3 feeds top-20 SIAR results into R3 and then takes top-10; hand-set threshold with no sensitivity analysis.
  • query_format_selection = dense: SI alone; sparse: SI + original input
    Chosen per retriever from test-set observations in Appendix D, then applied in the Table 2 reasoning experiments; no held-out validation, so it is effectively a post-hoc selection.
assumptions (4)
  • domain assumption Queries and rules lie in nearly non-overlapping semantic subspaces, and self-induction projects a query into the rule subspace.
    Stated in Section 3.1 as the theoretical motivation for SIAR; if false, the query augmentation has no basis.
  • domain assumption The LLM has sufficient inductive capability to write an abstract rule from the query that resembles the correct library rule.
    Section 3.1 says self-induction highly relies on the inductive capability of the LLM; the paper's CLUTRR and 7B-model results show this capability can fail.
  • domain assumption The LLM can judge whether a rule can be instantiated to align with query facts and would help reasoning.
    Section 3.2 defines R3's relevance criterion; the entire reranking stage depends on this capability.
  • domain assumption The rule library contains the golden rule for each query and is small enough for sparse and dense retrievers to be meaningful.
    RuleBench libraries contain 1,048, 830, and 166 rules respectively (Limitations); the benchmark assumes one useful rule exists per query.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimate." pith.science (2026). https://pith.science/paper/X4K4HVKO

@misc{pith2026250510870,
  author       = {Pith},
  title        = {Pith review of: Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X4K4HVKO}},
  note         = {Machine review of arXiv:2505.10870}
}
abstract

This paper systematically addresses the challenges of rule retrieval, a crucial yet underexplored area. Vanilla retrieval methods using sparse or dense retrievers to directly search for relevant rules to support downstream reasoning, often suffer from low accuracy. This is primarily due to a significant semantic gap between the instantiated facts in the queries and the abstract representations of the rules. Such misalignment results in suboptimal retrieval quality, which in turn negatively impacts reasoning performance. To overcome these challenges, we propose Self-Induction Augmented Retrieval (SIAR), a novel approach that utilizes Large Language Models (LLMs) to induce potential inferential rules that might offer benefits for reasoning by abstracting the underlying knowledge and logical structure in queries. These induced rules are then used for query augmentation to improve retrieval effectiveness. Additionally, we introduce Rule Relevance ReEstimate (R$^3$), a method that re-estimates the relevance of retrieved rules by assessing whether the abstract knowledge they contain can be instantiated to align with the facts in the queries and the helpfulness for reasoning. Extensive experiments across various settings demonstrate the effectiveness and versatility of our proposed methods.

Figures

Figures reproduced from arXiv: 2505.10870 by the authors.

Figure 1
Figure 1. (a) and (b) show the different characteris [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The workflow of retrieve-then-reason augmented with our method is shown in the middle of the Figure. To address the semantic misalignment issues, self-induction is first utilized to generate the hypothesized rule for query augmentation. Then, the new query is used for rule retrieval. And the retrieved rules are concatenated with the original query for reasoning. Building on this, we can reestimate the relevance of t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 4 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  2. [2]

    01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu,...

    arXiv 2025

  3. [3]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268

    arXiv 2016

  4. [4]

    Chen Bowen, Rune S tre, and Yusuke Miyao. 2024. https://aclanthology.org/2024.findings-eacl.22 A comprehensive evaluation of inductive reasoning capabilities and problem solving in large language models . In Findings of the Association for Computational Linguistics: EACL 2024, pages 323--339, St. Julian ' s, Malta. Association for Computational Linguistics

    work page 2024

  5. [5]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024 a . https://arxiv.org/abs/2402.03216 Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . Preprint, arXiv:2402.03216

    arXiv 2024

  6. [6]

    Xinran Chen, Xuanang Chen, Ben He, Tengfei Wen, and Le Sun. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.708 Analyze, generate and refine: Query expansion with LLM s for zero-shot open-domain QA . In Findings of the Association for Computational Linguistics ACL 2024, pages 11908--11922, Bangkok, Thailand and virtual meeting. Association for Comp...

    doi:10.18653/v1/2024.findings-acl.708 2024

  7. [7]

    Kewei Cheng, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Binxuan Huang, Ruirui Li, Shiyang Li, Zheng Li, Yifan Gao, Xian Li, et al. 2024. Inductive or deductive? rethinking the fundamental reasoning abilities of llms. arXiv preprint arXiv:2408.00114

    arXiv 2024

  8. [8]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. https://arxiv.org/abs/2301.00234 A survey on in-context learning . Preprint, arXiv:2301.00234

    arXiv 2024

Show all 46 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  2. [10]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empiric...

    2020 doi

  3. [11]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  4. [12]

    Haochen Li, Xin Zhou, and Zhiqi Shen. 2024. https://doi.org/10.18653/v1/2024.acl-long.75 Rewriting the code: A simple method for large language model augmented code search . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    2024 doi

  5. [13]

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations. arXiv preprint arXiv:2102.10073

    2021 arXiv

  6. [14]

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.322 Query rewriting in retrieval-augmented large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 53...

    2023 doi

  7. [15]

    Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. https://doi.org/10.18653/v1/2021.acl-long.316 Generation-augmented retrieval for open-domain question answering . In Proceedings of the 59th Annual Meeting of the Association ...

    2021 doi

  8. [16]

    OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/

    2024

  9. [17]

    Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. 2024. https://openreview.net/forum?id=bNt7oajl2a Phenomenal yet puzzling: Testing inductive reasoning capabilities of langua...

    2024

  10. [18]

    David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Vassilina Nikoulina, and Stéphane Clinchant. 2024. https://arxiv.org/abs/2407.01102 Bergen: A benchmarking library for retrieval-augmented generation . Preprint, arXiv:2407.01102

    2024 arXiv

  11. [19]

    Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. 2023. https://arxiv.org/abs/2305.15269 Testing the general deductive reasoning capacity of large language models using ood examples . Preprint, arXiv:2305.15269

    2023 arXiv

  12. [20]

    Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay S...

    2024 arXiv

  13. [21]

    Hamilton

    Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. https://doi.org/10.18653/v1/D19-1458 CLUTRR : A diagnostic benchmark for inductive reasoning from text . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Proces...

    2019 doi

  14. [22]

    Shangeetha Sivasothy, Scott Barnett, Rena Logothetis, Mohamed Abdelrazek, Zafaryab Rasool, Srikanth Thudumu, and Zac Brannelly. 2024. https://arxiv.org/abs/2406.06835 Large language models for generating rules, yay or nay? Preprint, arXiv:2406.06835

    2024 arXiv

  15. [23]

    Wangtao Sun, Haotian Xu, Xuanqing Yu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.150 I t D : Large language models can teach themselves induction through deduction . In Proceedings of the 62nd Annual Meeting of the Associati...

    2024 doi

  16. [24]

    Wangtao Sun, Xuanqing Yu, Shizhu He, Jun Zhao, and Kang Liu. 2023 a . https://openreview.net/forum?id=1Xht3SKAoY Expnote: Black-box large language models are better task solvers with experience notebook . In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  17. [25]

    Wangtao Sun, Chenxiang Zhang, Xueyou Zhang, Ziyang Huang, Haotian Xu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. 2024 b . Beyond instruction following: Evaluating rule following of large language models. arXiv preprint arXiv:2407.08440

    2024 arXiv

  18. [26]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.923 Is C hat GPT good at search? investigating large language models as re-ranking agents . In Proceedings of the 20...

    2023 doi

  19. [27]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  20. [28]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024 a . https://doi.org/10.1007/s11704-024-40231-1 A survey on large language model based autonomous agents . Fron...

    2024 doi

  21. [29]

    Liang Wang, Nan Yang, and Furu Wei. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.585 Query2doc: Query expansion with large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9414--9423, Singapore. Association...

    2023 doi

  22. [30]

    Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah Goodman. 2024 b . https://openreview.net/forum?id=G7UtIGQmjm Hypothesis search: Inductive reasoning with language models . In The Twelfth International Conference on Learning Representations

    2024

  23. [31]

    Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.406 Can LLM s reason with rules? logic scaffolding for stress-testing and improving LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Li...

    2024 doi

  24. [32]

    Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. 2024 d . https://arxiv.org/abs/2408.13654 Symbolic working memory enhances language models for complex rule application . Preprint, arXiv:2408.13654

    2024 arXiv

  25. [33]

    Xiaohan Wang, Xiaoyan Yang, Yuqi Zhu, Yue Shen, Jian Wang, Peng Wei, Lei Liang, Jinjie Gu, Huajun Chen, and Ningyu Zhang. 2024 e . Rulealign: Making large language models better physicians with diagnostic rule alignment. arXiv preprint arXiv:2408.12579

    2024 arXiv

  26. [34]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  27. [35]

    Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2018. https://arxiv.org/abs/1807.02478 Cail2018: A large-scale legal dataset for judgment prediction . Preprint, arXiv:1807.02478

    2018 arXiv

  28. [36]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. https://arxiv.org/abs/2309.07597 C-pack: Packaged resources to advance general chinese embedding . Preprint, arXiv:2309.07597

    2023 arXiv

  29. [37]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  30. [38]

    Zeyuan Yang, Peng Li, and Yang Liu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.109 Failures pave the way: Enhancing large language models through tuning-free rule accumulation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p...

    2023 doi

  31. [39]

    Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2024 b . https://aclanthology.org/2024.eacl-long.13 Language models as inductive reasoners . In Proceedings of the 18th Conference of the European Chapter of the Association for...

    2024

  32. [40]

    Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. 2024. https://doi.org/10.18653/v1/2024.acl-long.197 Self-contrast: Better reflection through inconsistent solving perspectives . In Proceedings of the 62nd Annual Meeting of the As...

    2024 doi

  33. [41]

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024 a . https://doi.org/10.1609/aaai.v38i17.29936 Expel: Llm agents are experiential learners . Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19632--19642

    2024 doi

  34. [42]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2024 arXiv

  35. [43]

    Yu, and Lichao Sun

    Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. 2023. https://arxiv.org/abs/2302.09419 A comprehensive survey o...

    2023 arXiv

  36. [44]

    Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. 2024. https://openreview.net/forum?id=tAmfM1sORP Large language models can learn rules

    2024

  37. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.