Pith. sign in

REVIEW 4 major objections 6 minor 164 references

An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models could move automated implementation generation beyond algorithmic specifications, into open-ended problem solving.

desk verdict A roadmap paper: broad survey of LLM techniques mapped to a five-strategy framework from analog design; the mapping is speculative but the survey and research-needs list are useful. read the letter →

arxiv 2501.00562 v2 pith:IITO7CI5 submitted 2024-12-31 cs.CL cs.AI

Hashmath Shaik , Alex Doboli This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelsimplementationgenerationopen-endedproblemsolvingpromptengineeringretrieval-augmentedreinforcementlearningkernel-basedelaborationframing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This report argues that large language models, especially when combined with prompting, retrieval-augmented generation, and reinforcement learning, could move automated implementation generation beyond traditional methods that require algorithmic specifications and static knowledge bases. The intended contribution is a capability map: which of the five implementation-creation strategies from the authors' earlier cognitive architecture can be realized with current LLM techniques, and which cannot. The report concludes that LLMs could help with problem framing, exploring solution approaches, feature elaboration and combination, qualitative assessment, and handling unexpected situations, while kernel creation, scalable decomposition, dynamic schemas, and quantitative performance prediction remain open. The value, if the argument holds, is a research agenda for building LLM-based systems that tackle open-ended problems rather than only well-defined ones.

What carries the argument

The load-bearing framework is a set of five implementation-creation strategies borrowed from the authors' earlier cognitive-architecture work [32]. Each strategy starts from a kernel, the invariant set of features shared by all implementations in a cluster, and then creates solutions through elaboration, which explores detailing alternatives. Strategy 1 keeps the kernel fixed while elaborating it; Strategy 2 imports features from another cluster; Strategy 3 fuses kernels from two clusters; Strategy 4 deliberately excludes some detailing alternatives to form a niche; Strategy 5 builds a kernel bottom-up by generalizing features of individual implementations. The paper uses this five-way decomposition as the template for judging which LLM techniques, such as prompt engineering, retrieval-augmented generation, and reinforcement learning, can cover which parts of implementation generation.

What would settle it

One concrete test: take an open-ended design brief, such as a novel analog circuit or algorithm for which no algorithmic specification exists, and run an LLM-based pipeline with prompting, RAG, and RL; if the pipeline never produces a working implementation that combines features from two distinct solution clusters, while human designers do, then the claimed capability for feature combination is not realized. Equally, if a traditional synthesis tool given only an algorithmic specification matches the LLM pipeline's output quality on the same brief, the claim that LLMs add capabilities beyond traditional methods is undercut.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that large language models offer new opportunities to devise automated implementation generation methods that can tackle problem solving activities beyond traditional methods. The report surveys the LLM toolkit and maps it onto five strategies for implementation creation: elaborating a fixed kernel, elaborating with features migrated from another solution cluster, combining kernels from two clusters, exploring an excluded niche of detailing alternatives, and building a kernel bottom-up from individual implementations. It finds that prompting can approximate elaboration, decomposition, and self-assessment; RAG can supply dynamic memory and grounding; and RL can tune behavior from feedback. It also finds that current methods do not yet create kernels, combine features across kernels, or produce quantitative performance predictions for novel implementations, and that RL-based fine tuning for the full space of decompositions is likely infeasible.

Load-bearing premise

The argument depends on the premise that the five implementation-creation strategies from the authors' earlier cognitive-architecture work are a valid and complete decomposition of open-ended implementation generation, and that LLM prompting, retrieval, and reinforcement learning can in principle be mapped onto those strategies.

Editorial extensions

If this is right

  • If the mapping holds, automated systems could generate implementations from vague, multi-modal problem descriptions without an algorithmic specification, starting with problem framing and approach exploration.
  • Prompting methods such as chain-of-thought and tree-of-thoughts could supply the elaboration and self-assessment steps that traditional high-level synthesis and evolutionary methods lack.
  • Retrieval-augmented generation with long-term memory could give implementation generation the evolving domain knowledge that static building-block libraries do not have.
  • Reinforcement learning from human or AI feedback could tune elaboration priorities and implementation assessment, but the paper expects it to be infeasible for the huge space of possible decompositions.
  • Because static schemas and static kernels are insufficient, new methods for dynamic kernel creation and feature combination are the central research needs the report identifies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper does not take is to benchmark LLM-assisted implementation generation on an open-ended design task, using the five strategies as a scoring rubric for which activities the model actually performs.
  • If the framework is right, the main bottleneck is not generation but evaluation: predicting quantitative performance of a novel combination of features is a task LLMs cannot yet do, so hybrid systems that pair LLM qualitative judgment with traditional simulation may be the fastest route to practical tools.
  • The paper's skepticism about RL for decomposition suggests that progress will come from modular prompting and retrieval rather than end-to-end fine tuning, which is a testable prediction for future systems.
  • The report's emphasis on kernels implies that human designers will remain in the loop for kernel identification for a long time, so LLM assistance may augment rather than replace the designer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript is an overview/position paper arguing that large language models (LLMs) can extend automated implementation generation to open-ended problem solving. It surveys traditional automated methods (high-level specifications, evolutionary algorithms, agents, cognitive architectures), reviews prompt engineering, retrieval-augmented generation, and reinforcement learning, and then maps these LLM techniques onto a five-strategy framework taken from the authors' prior InnovA cognitive architecture. The paper concludes with a list of research needs. It contains no experiments or formal derivations; its central contribution is a proposed research agenda rather than an established result.

Significance. If accepted, the paper would provide a potentially useful organizing framework for research on LLM-assisted open-ended problem solving, and it does assemble a broad catalog of prompting, RAG, and RL methods. The paper deserves credit for explicitly acknowledging limitations (e.g., the likely infeasibility of RL-based task decomposition and the static nature of schemas) and for naming concrete research needs. However, the significance is currently limited by the unvalidated transfer of the InnovA framework to general open-ended problems, by internal tensions between stated LLM limitations and the proposed mappings, and by a survey taxonomy that contains duplicated and overlapping categories. The manuscript is best read as a position statement whose central claim is plausible but not yet established.

major comments (4)
  1. [Section IV, Figure 1] The central argument rests on the claim, stated as "Similar to [32], this report considers that devising an implementation for a problem specification utilizes the five strategies shown in Figure 1." No evidence is offered that a framework developed for analog circuit design (InnovA) generalizes to the full range of open-ended implementation generation. The paper itself lists six "additional activities" after the five strategies (divide and conquer, kernel creation, elaboration, assessment, memory, adaptive process), and Section I lists problem framing and exploration of solution approaches as central activities; neither is captured by the five-strategy taxonomy. This makes the taxonomy look incomplete as a decomposition, so the subsequent gap analysis is not justified as stated. Please either provide an argument for completeness or explicitly reframe the five strategies as one possible perspective, deriving the research needs conditional on that perspective.
  2. [Section IV.1 and IV.3] The proposed mapping from LLM techniques to implementation-generation activities is undercut by the paper's own feasibility statements. Section IV.1 says that LLM fine tuning through RL for decomposition is "likely infeasible due to the huge space of possible decompositions," and Section IV.3 says that schemas are static structures and that "problem solving often requires performing new sequences of decisions beyond a static schema." These are exactly the mechanisms proposed for kernel creation and elaboration in the surrounding discussion. The paper needs to explain why the proposed methods can still contribute despite these limitations, or it must adjust the research requirements accordingly.
  3. [Section III.A vs. IV.4] There is an internal tension between the claimed limitations of LLMs in Section III.A (no memory structures, no ability to build causally grounded explanations, output produced in a single forward pass) and the claimed capabilities in Section IV.4 (self-critique, qualitative assessment, RLHF fine tuning, and improved correctness through self-consistency and cross-referencing). Since the central claim depends on LLMs being able to support these activities, the paper should reconcile these statements by specifying which limitations are inherent to the architecture and which are addressable through prompting, retrieval, or fine-tuning.
  4. [Section III.C] The survey taxonomy is unreliable because of duplicated and overlapping content. Metacognitive Prompting is described with the same five stages in both Section III.B.d and Section III.C.e, and the categories "Self-Aware and Adaptive Retrieval" (Section III.C.b), "Self-Reflection Methods" (Section III.C.e), and "Self-Critique Methods" (Section III.C.f) overlap substantially. This undermines the paper's claim to offer a clear, extensive presentation and obscures the mapping onto implementation-generation activities. Please consolidate these subsections, cross-reference shared methods, and make the organizing criteria explicit.
minor comments (6)
  1. [Throughout] The manuscript contains numerous typos, including "textS" and "sb-problems" in Section IV.1, "nocel" in Section V, "dicussed" in Section III.B, and broken spacing in the abstract ("c an", "meth ods").
  2. [References [16] and [28]] References [16] and [28] appear to have swapped titles: [16] (Aho et al.) should cite "Compilers: Principles, Techniques, and Tools," while [28] (Laird) should cite "The SOAR Cognitive Architecture."
  3. [Section III.C.e] The term "Implicit Retrieval-Augmented Generation (RAG)" is nonstandard, and the cited works [78], [101], [102] include the original RAG paper rather than a distinct "implicit" method; please rename or justify the category.
  4. [Section IV (GA analogy)] The eight-point genetic-algorithm analogy (selection, crossover, mutation, inversion, elitism, replacement, fitness evaluation, exploration/exploitation) is metaphorical and is not used to derive the five strategies or any research need; statements such as "Crossover is like an LLM writing poetry about quantum physics" are not technical claims. Please remove the passage or clearly label it as an illustrative remark that does not form part of the argument.
  5. [Section III.B.a] The method name "Basic + Annotation Guideline-Based Prompting + Error Analysis-Based Prompting" is unwieldy; consider introducing a shorter name and referring the reader to [43].
  6. [Figure 1] Figure 1 is referenced before its notation is explained; the text should define terms such as "envelope fragment," "implementation cluster," and "niche" prior to describing the five strategies.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; one minor self-citation to the authors' InnovA framework shapes the discussion but does not force the central claim.

full rationale

This paper is a narrative survey and research-needs discussion, not a derivation with fitted parameters, predictions, or equations whose outputs are equivalent to their inputs. The central claim, that LLMs offer new opportunities for automated implementation generation for open-ended problems, is a plausibility argument supported by an extensive literature survey; no quantity is fitted and then reported as a prediction, and no formal derivation reduces to its assumptions. The only notable self-citation is the InnovA framework [32], authored by the current authors, which supplies the five implementation-creation strategies used to organize Section IV: 'Similar to [32], this report considers that devising an implementation for a problem specification utilizes the five strategies shown in Figure 1.' This self-citation shapes the discussion, but it is not load-bearing in a circular way: the paper explicitly supplements the five strategies with six additional activities, acknowledging that the framework is incomplete, and the central opportunity claim does not require the framework to be either complete or uniquely justified. InnovA is also a published prior artifact rather than an unpublished uniqueness theorem. The paper's own admissions, such as 'LLM fine tuning through RL is likely infeasible' and schemas being 'static structures,' weaken the proposed LLM-to-strategy mapping, but that is a feasibility/correctness concern, not circularity. Because the paper is otherwise self-contained against external benchmarks and the self-citation is minor and non-equational, the appropriate circularity score is 2.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or new entities are introduced. The paper's argument rests on domain assumptions inherited from prior work, chiefly the InnovA five-strategy framework, and on a postulated mapping from LLM techniques to open-ended problem-solving activities. Neither is validated in this preprint.

assumptions (3)
  • domain assumption The five strategies for implementation creation (kernel elaboration, feature migration, kernel combination, niche exclusion, bottom-up kernel identification) are a valid decomposition of open-ended implementation generation.
    Section IV states 'Similar to [32], this report considers that devising an implementation for a problem specification utilizes the five strategies shown in Figure 1.' This framework is taken from the authors' prior InnovA work and is not validated here.
  • domain assumption LLMs, possibly with prompting, RAG, and RL, can realize the problem-solving activities absent in traditional methods.
    The entire research-needs discussion in Section IV assumes the mapping between LLM methods and activities such as kernel creation, elaboration, and assessment is feasible. The paper itself notes scalability and coupling problems, so this mapping is not established.
  • domain assumption Self-attention-based LLMs are fundamentally limited for open-ended problem solving because they rely on statistical correlations rather than genuine logical or conceptual understanding.
    Section III-A asserts this limitation and cites references [35]-[42]. It is a background assumption about LLM capabilities and is not demonstrated in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems." pith.science (2026). https://pith.science/paper/IITO7CI5

@misc{pith2026250100562,
  author       = {Pith},
  title        = {Pith review of: An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IITO7CI5}},
  note         = {Machine review of arXiv:2501.00562}
}
read the original abstract

Large Language Models offer new opportunities to devise automated implementation generation methods that can tackle problem solving activities beyond traditional methods, which require algorithmic specifications and can use only static domain knowledge, like performance metrics and libraries of basic building blocks. Large Language Models could support creating new methods to support problem solving activities for open-ended problems, like problem framing, exploring possible solving approaches, feature elaboration and combination, more advanced implementation assessment, and handling unexpected situations. This report summarized the current work on Large Language Models, including model prompting, Reinforcement Learning, and Retrieval-Augmented Generation. Future research requirements were also discussed.

Figures

Figures reproduced from arXiv: 2501.00562 by the authors.

Figure 1
Figure 1. Five strategies for automated implementation creat [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

164 extracted references · 6 canonical work pages

  1. [32]

    Innova: A cognitive architecture for computational innovation through robust divergence and its application for analog circuit design,

    H. Li, X. Liu, F. Jiao, A. Doboli, and S. Doboli, “Innova: A cognitive architecture for computational innovation through robust divergence and its application for analog circuit design,” IEEE Transactions on CADICS, vol. 37, no. 10, pp. 1943–1956, 2018

    1943

  2. [1]

    Toward an understanding of macrocognition in teams: Pre- d icting processes in complex collaborative contexts,

    S. Fiore, M. Rosen, K. Smith-Jentsch, E. Salas, L. M., and N. Warner, “Toward an understanding of macrocognition in teams: Pre- d icting processes in complex collaborative contexts,” Human Factors, vol. 52, no. 2, pp. 203–224, 2010

    2010

  3. [2]

    The process of solvi ng complex problems,

    A. Fischer, S. Greiff, and J. Funke, “The process of solvi ng complex problems,” Journal of Problem Solving , vol. 4, 2012

    2012

  4. [3]

    Towards a generalized competency model of collaborative p roblem solving,

    C. Sun, V . Shute, A. Stewart, J. Y onehiro, N. Duran, and S. D’Mello, “Towards a generalized competency model of collaborative p roblem solving,” Computers & Education , vol. 143, p. 103672, 2020

    2020

  5. [4]

    Problem-solving phase transitions during team collaboration,

    T. Wiltshire, J. Butner, and S. Fiore, “Problem-solving phase transitions during team collaboration,” Cognitive Science, vol. 42, no. 1, pp. 129– 167, 2018

    2018

  6. [5]

    Cognitive pr ocesses in well-defined and ill-defined problem solving,

    G. Schraw, M. E. Dunkle, and L. D. Bendixen, “Cognitive pr ocesses in well-defined and ill-defined problem solving,” Applied Cognitive Psychology, vol. 9, no. 6, pp. 523–538, 1995

    1995

  7. [6]

    The role of precedents in incr easing creativity during iterative design of electronic embedded systems,

    A. Doboli and A. Umbarkar, “The role of precedents in incr easing creativity during iterative design of electronic embedded systems,” Design Studies , vol. 35, no. 3, pp. 298–326, 2014

    2014

  8. [7]

    Modeling semantic knowledge structures for creative problem solving: Studie s on express- ing concepts, categories, associations, goals and context ,

    A. Doboli, A. Umbarkar, S. Doboli, and J. Betz, “Modeling semantic knowledge structures for creative problem solving: Studie s on express- ing concepts, categories, associations, goals and context ,” Knowledge- based Systems , vol. 78, pp. 34–50, 2015

    2015

Show all 164 references
  1. [8]

    Reuse , parameterized reuse, and hierarchical reuse of substructu res in evolving electrical circuits using genetic programming,

    J. R. Koza, F. H. Bennett, D. Andre, and M. A. Keane, “Reuse , parameterized reuse, and hierarchical reuse of substructu res in evolving electrical circuits using genetic programming,” in Evolvable Systems: From Biology to Hardware: First International Conference, ICES96 Tsuku...

    1996

  2. [9]

    Problem frame p atterns: an exploration of patterns in the problem space,

    R. Wirfs-Brock, P . Taylor, and J. Noble, “Problem frame p atterns: an exploration of patterns in the problem space,” Proc. Conference on Pattern Languages of Programs , 2006

    2006

  3. [10]

    High-level synthesis of delta-s igma modu- lators optimized for complexity, sensitivity and power con sumption,

    H. Tang and A. Doboli, “High-level synthesis of delta-s igma modu- lators optimized for complexity, sensitivity and power con sumption,” IEEE Transactions on CADICS , vol. 25, no. 3, pp. 597–607, 2006

    2006

  4. [11]

    Systematic methodology for design- ing reconfigurable delta sigma modulator topologies for mul timode communication systems,

    Y . Wei, H. Tang, and A. Doboli, “Systematic methodology for design- ing reconfigurable delta sigma modulator topologies for mul timode communication systems,” IEEE Transactions on CADICS , vol. 26, no. 3, pp. 480–496, 2007

    2007

  5. [12]

    Improvement of skills f or solving-ill- defined problems,

    G. A. Klein and J. Weitzenfeld, “Improvement of skills f or solving-ill- defined problems,” Educational Psychologist, vol. 13, no. 1, pp. 31–41, 1978

    1978

  6. [13]

    Assessm ent of student problem-solving on ill-defined tasks,

    J. P . Leighton, W. T. Rogers, and T. O. Maguire, “Assessm ent of student problem-solving on ill-defined tasks,” Alberta Journal of Educational Research, vol. 45, no. 4, 1999

    1999

  7. [14]

    A novel agent-based, evolutio nary model for expressing the dynamics of creative open-problem solvi ng in small groups,

    A. Doboli and S. Doboli, “A novel agent-based, evolutio nary model for expressing the dynamics of creative open-problem solvi ng in small groups,” Applied Intelligence, vol. 51, pp. 2094–2127, 2021

    2021

  8. [15]

    Enhanced poet: Open-ended reinforcement learning th rough un- bounded invention of learning challenges and their solutio ns,

    R. Wang, J. Lehman, A. Rawal, J. Zhi, Y . Li, J. Clune, and K . Stan- ley, “Enhanced poet: Open-ended reinforcement learning th rough un- bounded invention of learning challenges and their solutio ns,” in International conference on machine learning . PMLR, 2020, pp. 9940–9951

    2020

  9. [16]

    A. Aho, J. Ullman, R. Sethi, and M. Lam, The SOAR Cognitive Architecture. Addison Wesley, 2006

    2006

  10. [17]

    A library- based approach to analog synthesis from vhdl-ams specificat ions,

    A. Doboli, N. Dhanwada, A. Nunez-Aldana, and R. V emuri, “A library- based approach to analog synthesis from vhdl-ams specificat ions,” ACM Transactions on Design Automation , vol. 9, no. 2, pp. 238–271, 2004

    2004

  11. [18]

    Fingeroff, High-Level Synthesis Blue Book

    M. Fingeroff, High-Level Synthesis Blue Book . Xlibris Us, 2010

    2010

  12. [19]

    McConaghy, P

    T. McConaghy, P . Palmers, P . Gao, M. Steyaert, and G. Gie len, V ariation-aware Analog Structural Synthesis. Springer, 2009

    2009

  13. [20]

    Behavioral modeling for high- level syn- thesis of analog and mixed-signal systems from vhdl-ams,

    A. Doboli and R. V emuri, “Behavioral modeling for high- level syn- thesis of analog and mixed-signal systems from vhdl-ams,” IEEE Transactions on CADICS , vol. 22, no. 11, 2003

    2003

  14. [21]

    Exploration-based high-level synthesis of linea r analog systems operating at low/medium frequencies,

    ——, “Exploration-based high-level synthesis of linea r analog systems operating at low/medium frequencies,” IEEE Transactions on CADICS , vol. 22, no. 22, 2003

    2003

  15. [22]

    When concepts combine,

    E. Wisniewski, “When concepts combine,” Psychonomic Bulletin & Review, vol. 4, no. 2, pp. 167–183, 1997

    1997

  16. [23]

    Darwin: Cmos opamp synt hesis by means of genetic algorithm,

    W. Kruiskamp and D. Leenaerts, “Darwin: Cmos opamp synt hesis by means of genetic algorithm,” Proc. Design Automation Conference, pp. 433–438, 1995

    1995

  17. [24]

    Research direc tions in agent communication,

    A. Chopra, A. Artikis, J. Bentahar, M. Colombetti, F. Di gnum, N. Fornara, A. Jones, M. Singh, and P . Y olum, “Research direc tions in agent communication,” ACM Trans. Intell. Syst. Technol. , vol. 4, no. 2, 2013

    2013

  18. [25]

    Agent-based modeling: methods and techn iques for simulating human systems,

    E. Bonabeau, “Agent-based modeling: methods and techn iques for simulating human systems,” Proceedings of the National Academy of Sciences, vol. 9, no. 3, 7280

  19. [26]

    Collaborating wit h style: Using an agent-based model to simulate cognitive style dive rsity in problem solving teams,

    S. Lapp, K. Jablokow, and C. McComb, “Collaborating wit h style: Using an agent-based model to simulate cognitive style dive rsity in problem solving teams,” in Proc. ASME International Design En- gineering Technical Conferences and Computers and Informa tion in Engineering Co...

    2017

  20. [27]

    Act: A simple theory of complex cognition ,

    J. Anderson, “Act: A simple theory of complex cognition ,” American Psychologist, vol. 51, pp. 355–365, 1996

    1996

  21. [28]

    Laird, Compilers: Principles, Techniques, and Tools

    J. Laird, Compilers: Principles, Techniques, and Tools . The MIT Press, 2012

    2012

  22. [29]

    The sigma cogn itive ar- chitecture and system: towards functionally elegant grand unification,

    P . Rosenbloom, A. Demski, and U. V olkan, “The sigma cogn itive ar- chitecture and system: towards functionally elegant grand unification,” Journal of Artificial General Intelligence , 2016

    2016

  23. [30]

    An overview of the epic architec ture for cognition and performance with application to human-co mputer interaction,

    D. Kieras and D. Meyer, “An overview of the epic architec ture for cognition and performance with application to human-co mputer interaction,” Journal Human-Computer Interaction , vol. 12, no. 4, pp. 391–438, 1997

    1997

  24. [31]

    Sun, A tutorial on clarion 5.0

    R. Sun, A tutorial on clarion 5.0 . Cognitive Science Department, Rensselaer Polytechnic (http://www.cogsci. rpi.edu/ rsun/sun.tutorial.pdf), 2003

    2003

  25. [33]

    Attention is all you need,

    A. V aswani, “Attention is all you need,” Advances in Neural Informa- tion Processing Systems , 2017

    2017

  26. [34]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . D hariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  27. [35]

    Representati on learning: A review and new perspectives,

    Y . Bengio, A. Courville, and P . Vincent, “Representati on learning: A review and new perspectives,” IEEE transactions on pattern analysis and machine intelligence , vol. 35, no. 8, pp. 1798–1828, 2013

    2013

  28. [36]

    Building machines that learn and think like people,

    B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershm an, “Building machines that learn and think like people,” Behavioral and brain sciences, vol. 40, p. e253, 2017

    2017

  29. [37]

    Ethical and social risks of harm from language models,

    L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P .-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh et al., “Ethical and social risks of harm from language models,” arXiv preprint arXiv:2112.04359, 2021

    2021 arXiv

  30. [38]

    Sparks of artificial general intelligence: Early experiments with gp t-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Ho rvitz, E. Kamar, P . Lee, Y . T. Lee, Y . Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gp t-4,” arXiv preprint arXiv:2303.12712, 2023

    2023 arXiv

  31. [39]

    Defending against neural fake news,

    R. Zellers, A. Holtzman, H. Rashkin, Y . Bisk, A. Farhadi , F. Roesner, and Y . Choi, “Defending against neural fake news,” Advances in neural information processing systems , vol. 32, 2019

    2019

  32. [40]

    Fast lexically constrained decod ing with dy- namic beam allocation for neural machine translation,

    M. Post and D. Vilar, “Fast lexically constrained decod ing with dy- namic beam allocation for neural machine translation,” arXiv preprint arXiv:1804.06609, 2018

    2018 arXiv

  33. [41]

    Parameter-efficie nt transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. D e Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficie nt transfer learning for nlp,” in International conference on machine learning . PMLR, 2019, pp. 2790–2799

    2019

  34. [42]

    Shortcut learning in deep neu ral networks,

    R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. B rendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neu ral networks,” Nature Machine Intelligence , vol. 2, no. 11, pp. 665–673, 2020

    2020

  35. [43]

    Improving large language models for clinical named entity recognition via prompt engineering,

    Y . Hu, Q. Chen, J. Du, X. Peng, V . K. Keloth, X. Zuo, Y . Zhou , Z. Li, X. Jiang, Z. Lu et al. , “Improving large language models for clinical named entity recognition via prompt engineering, ” Journal of the American Medical Informatics Association , p. ocad259, 2024

    2024

  36. [44]

    Medical transcriptions,

    T. Boyle, “Medical transcriptions,” 2018, accessed: 2024-12-26. [Online]. Available: https://www.kaggle.com/datasets/tboyle10/medicaltranscriptions

    2018

  37. [45]

    V accine adverse event reporting system (vaers),

    “V accine adverse event reporting system (vaers),” https://vaers.hhs.gov/data/datasets.html, Centers for Disease Control and Prevention (CDC) and U.S. Food and Drug Administration ( FDA), 2024, accessed: 2024-12-26

    2024

  38. [46]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q . V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  39. [47]

    Comp lexity- based prompting for multi-step reasoning,

    Y . Fu, H. Peng, A. Sabharwal, P . Clark, and T. Khot, “Comp lexity- based prompting for multi-step reasoning,” in The Eleventh Interna- tional Conference on Learning Representations , 2022

    2022

  40. [48]

    Thread of thought unraveling chaotic contexts,

    Y . Zhou, X. Geng, T. Shen, C. Tao, G. Long, J.-G. Lou, and J. Shen, “Thread of thought unraveling chaotic contexts,” arXiv preprint arXiv:2311.08734, 2023

    2023 arXiv

  41. [49]

    Chain-of-knowledge: Grounding large language mo dels via dynamic knowledge adapting over heterogeneous sources ,

    X. Li, R. Zhao, Y . K. Chia, B. Ding, S. Joty, S. Poria, and L. Bing, “Chain-of-knowledge: Grounding large language mo dels via dynamic knowledge adapting over heterogeneous sources ,” arXiv preprint arXiv:2305.13269, 2023

    2023 arXiv

  42. [50]

    Chain of code: R easoning with a language model-augmented code emulator,

    C. Li, J. Liang, A. Zeng, X. Chen, K. Hausman, D. Sadigh, S. Levine, L. Fei-Fei, F. Xia, and B. Ichter, “Chain of code: R easoning with a language model-augmented code emulator,” arXiv preprint arXiv:2312.04474, 2023

    2023 arXiv

  43. [51]

    Enhancing zero-shot chain-of-thought reasoning in large language models through logic,

    X. Zhao, M. Li, W. Lu, C. Weber, J. H. Lee, K. Chu, and S. Wer mter, “Enhancing zero-shot chain-of-thought reasoning in large language models through logic,” arXiv preprint arXiv:2309.13339 , 2023

    2023 arXiv

  44. [52]

    Chain-of-event prompting for multi- document summarization by large language models,

    S. Bao, T. Li, and B. Cao, “Chain-of-event prompting for multi- document summarization by large language models,” International Journal of W eb Information Systems , no. ahead-of-print, 2024

    2024

  45. [53]

    Chain-of-table: Evolving tables in the reasoning chain for table understand ing,

    Z. Wang, H. Zhang, C.-L. Li, J. M. Eisenschlos, V . Perot, Z. Wang, L. Miculicich, Y . Fujii, J. Shang, C.-Y . Lee et al. , “Chain-of-table: Evolving tables in the reasoning chain for table understand ing,” arXiv preprint arXiv:2401.04398, 2024

    2024 arXiv

  46. [54]

    Self-consistency improves chain of thou ght rea- soning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A . Chowd- hery, and D. Zhou, “Self-consistency improves chain of thou ght rea- soning in language models,” arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  47. [55]

    Con trastive chain-of-thought prompting,

    Y . K. Chia, G. Chen, L. A. Tuan, S. Poria, and L. Bing, “Con trastive chain-of-thought prompting,” arXiv preprint arXiv:2311.09277 , 2023

    2023 arXiv

  48. [56]

    Federated prompting and cha in-of- thought reasoning for improving llms answering,

    X. Liu, T. Pang, and C. Fan, “Federated prompting and cha in-of- thought reasoning for improving llms answering,” in International Conference on Knowledge Science, Engineering and Manageme nt. Springer, 2023, pp. 3–11

    2023

  49. [57]

    Tree of thoughts: Deliberate problem solvi ng with large language models,

    S. Y ao, D. Y u, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, an d K. Narasimhan, “Tree of thoughts: Deliberate problem solvi ng with large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  50. [58]

    Maieutic prompting: Logically consistent rea soning with recursive explanations,

    J. Jung, L. Qin, S. Welleck, F. Brahman, C. Bhagavatula, R. L. Bras, and Y . Choi, “Maieutic prompting: Logically consistent rea soning with recursive explanations,” arXiv preprint arXiv:2205.11822 , 2022

    2022 arXiv

  51. [59]

    Plan-and-solve prompting: Improving zero-shot chai n-of-thought reasoning by large language models,

    L. Wang, W. Xu, Y . Lan, Z. Hu, Y . Lan, R. K.-W. Lee, and E.-P . Lim, “Plan-and-solve prompting: Improving zero-shot chai n-of-thought reasoning by large language models,” arXiv preprint arXiv:2305.04091, 2023

    2023 arXiv

  52. [60]

    Program of thoug hts prompting: Disentangling computation from reasoning for n umerical reasoning tasks,

    W. Chen, X. Ma, X. Wang, and W. W. Cohen, “Program of thoug hts prompting: Disentangling computation from reasoning for n umerical reasoning tasks,” arXiv preprint arXiv:2211.12588 , 2022

    2022 arXiv

  53. [61]

    C hain- of-symbol prompting elicits planning in large langauge mod els,

    H. Hu, H. Lu, H. Zhang, Y .-Z. Song, W. Lam, and Y . Zhang, “C hain- of-symbol prompting elicits planning in large langauge mod els,” arXiv preprint arXiv:2305.10276, 2023

    2023 arXiv

  54. [62]

    Structured chain-of-tho ught prompting for code generation,

    J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-tho ught prompting for code generation,” ACM Transactions on Software Engineering and Methodology, 2023

    2023

  55. [63]

    Reas oning implicit sentiment with chain-of-thought prompting,

    H. Fei, B. Li, Q. Liu, L. Bing, F. Li, and T.-S. Chua, “Reas oning implicit sentiment with chain-of-thought prompting,” arXiv preprint arXiv:2305.11255, 2023

    2023 arXiv

  56. [64]

    Towards expert- level medical question answering with large language model s,

    K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal et al. , “Towards expert- level medical question answering with large language model s,” arXiv preprint arXiv:2305.09617, 2023

    2023 arXiv

  57. [65]

    Automatic chain of thought prompting in large language models,

    Z. Zhang, A. Zhang, M. Li, and A. Smola, “Automatic chain of thought prompting in large language models,” arXiv preprint arXiv:2210.03493, 2022

    2022 arXiv

  58. [66]

    React: Synergizing reasoning and acting in language model s,

    S. Y ao, J. Zhao, D. Y u, N. Du, I. Shafran, K. Narasimhan, a nd Y . Cao, “React: Synergizing reasoning and acting in language model s,” arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  59. [67]

    Act ive prompting with chain-of-thought for large language models ,

    S. Diao, P . Wang, Y . Lin, R. Pan, X. Liu, and T. Zhang, “Act ive prompting with chain-of-thought for large language models ,” arXiv preprint arXiv:2302.12246, 2023

    2023 arXiv

  60. [68]

    Mathprompter: Mat hematical rea- soning using large language models,

    S. Imani, L. Du, and H. Shrivastava, “Mathprompter: Mat hematical rea- soning using large language models,” arXiv preprint arXiv:2303.05398, 2023

    2023 arXiv

  61. [69]

    Large language models as analogical reaso ners,

    M. Y asunaga, X. Chen, Y . Li, P . Pasupat, J. Leskovec, P . L iang, E. H. Chi, and D. Zhou, “Large language models as analogical reaso ners,” arXiv preprint arXiv:2310.01714 , 2023

    2023 arXiv

  62. [70]

    Syn- thetic prompting: Generating chain-of-thought demonstra tions for large language models,

    Z. Shao, Y . Gong, Y . Shen, M. Huang, N. Duan, and W. Chen, “ Syn- thetic prompting: Generating chain-of-thought demonstra tions for large language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 30 706–30 775

    2023

  63. [71]

    System 2 attention (is som ething you might need too),

    J. Weston and S. Sukhbaatar, “System 2 attention (is som ething you might need too),” arXiv preprint arXiv:2311.11829 , 2023

    2023 arXiv

  64. [72]

    Metacognitive prompting improves understand- ing in large language models,

    Y . Wang and Y . Zhao, “Metacognitive prompting improves understand- ing in large language models,” arXiv preprint arXiv:2308.05342 , 2023

    2023 arXiv

  65. [73]

    Least-to-most prompting enables complex reasoning in large language models,

    D. Zhou, N. Sch¨ arli, L. Hou, J. Wei, N. Scales, X. Wang, D . Schu- urmans, C. Cui, O. Bousquet, Q. Le et al. , “Least-to-most prompting enables complex reasoning in large language models,” arXiv preprint arXiv:2205.10625, 2022

    2022 arXiv

  66. [74]

    Decomposed prompting: A modular approac h for solving complex tasks,

    T. Khot, H. Trivedi, M. Finlayson, Y . Fu, K. Richardson, P . Clark, and A. Sabharwal, “Decomposed prompting: A modular approac h for solving complex tasks,” arXiv preprint arXiv:2210.02406 , 2022

    2022 arXiv

  67. [75]

    Pal: Program-aided language models,

    L. Gao, A. Madaan, S. Zhou, U. Alon, P . Liu, Y . Y ang, J. Cal lan, and G. Neubig, “Pal: Program-aided language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 10 764–10 799

    2023

  68. [76]

    Binding language models in symbolic languages,

    Z. Cheng, T. Xie, P . Shi, C. Li, R. Nadkarni, Y . Hu, C. Xion g, D. Radev, M. Ostendorf, L. Zettlemoyer et al. , “Binding language models in symbolic languages,” arXiv preprint arXiv:2210.02875 , 2022

    2022 arXiv

  69. [77]

    Large la nguage models are versatile decomposers: Decompose evidence and q uestions for table-based reasoning,

    Y . Y e, B. Hui, M. Y ang, B. Li, F. Huang, and Y . Li, “Large la nguage models are versatile decomposers: Decompose evidence and q uestions for table-based reasoning,” arXiv preprint arXiv:2301.13808 , 2023

    2023 arXiv

  70. [78]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P . Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K¨ uttler, M. Lewis, W.-t. Yih, T. Rockt¨ aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020

    2020

  71. [79]

    Reducing hallucination in struc- tured outputs via retrieval-augmented generation,

    P . B´ echard and O. M. Ayala, “Reducing hallucination in struc- tured outputs via retrieval-augmented generation,” arXiv preprint arXiv:2404.08189, 2024

    2024 arXiv

  72. [80]

    Sbi-rag: Enhancing math word pro blem solving for students through schema-based instruction and retriev al-augmented generation,

    P . Dixit and T. Oates, “Sbi-rag: Enhancing math word pro blem solving for students through schema-based instruction and retriev al-augmented generation,” arXiv preprint arXiv:2410.13293 , 2024

    2024 arXiv

  73. [81]

    Kragen: a knowledge graph-enhance d rag framework for biomedical problem solving using large la nguage models,

    N. Matsumoto, J. Moran, H. Choi, M. E. Hernandez, M. V enk atesan, P . Wang, and J. H. Moore, “Kragen: a knowledge graph-enhance d rag framework for biomedical problem solving using large la nguage models,” Bioinformatics, vol. 40, no. 6, 2024

    2024

  74. [82]

    Gram: Generative r etrieval augmented matching of data schemas in the context of data sec urity,

    X. Liu, R. Wang, Y . Song, and L. Kong, “Gram: Generative r etrieval augmented matching of data schemas in the context of data sec urity,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 5476–5486

    2024

  75. [83]

    Table rag: Million-token table understanding with language models,

    S.-A. Chen, L. Miculicich, J. M. Eisenschlos, Z. Wang, Z . Wang, Y . Chen, Y . Fujii, H.-T. Lin, C.-Y . Lee, and T. Pfister, “Table rag: Million-token table understanding with language models,” arXiv preprint arXiv:2410.04739, 2024

    arXiv

  76. [84]

    Seakr: Self-aware knowledge retrieval for adaptive retrieval aug mented gen- eration,

    Z. Y ao, W. Qi, L. Pan, S. Cao, L. Hu, W. Liu, L. Hou, and J. Li , “Seakr: Self-aware knowledge retrieval for adaptive retrieval aug mented gen- eration,” arXiv preprint arXiv:2406.19215 , 2024

    2024 arXiv

  77. [85]

    Self- rag: Self- reflective retrieval augmented generation,

    A. Asai, Z. Wu, Y . Wang, A. Sil, and H. Hajishirzi, “Self- rag: Self- reflective retrieval augmented generation,” in NeurIPS 2023 W orkshop on Instruction Tuning and Instruction F ollowing , 2023

    2023

  78. [86]

    Can we further elicit reasoning in llms? critic-guided plannin g with retrieval-augmentation for solving challenging tasks,

    X. Li, W. Xu, R. Zhao, F. Jiao, S. Joty, and L. Bing, “Can we further elicit reasoning in llms? critic-guided plannin g with retrieval-augmentation for solving challenging tasks,” arXiv preprint arXiv:2410.01428, 2024

    2024 arXiv

  79. [87]

    Simrag: Self-improving retrieval-augmented generatio n for adapting large language models to specialized domains,

    R. Xu, H. Liu, S. Nag, Z. Dai, Y . Xie, X. Tang, C. Luo, Y . Li, J. C. Ho, C. Y anget al., “Simrag: Self-improving retrieval-augmented generatio n for adapting large language models to specialized domains, ” arXiv preprint arXiv:2410.17952, 2024

    arXiv

  80. [88]

    SeRTS: Self-rewarding tree search for bi omedical retrieval-augmented generation,

    M. Hu, L. Zong, H. Wang, J. Zhou, J. Li, Y . Gao, K.-F. Wong, Y . Li, and I. King, “SeRTS: Self-rewarding tree search for bi omedical retrieval-augmented generation,” in Findings of the Association for Computational Linguistics: EMNLP 2024 , Y . Al-Onaizan, M. Bansal, and Y .-...

    2024

  81. [89]

    Speculative rag: Enhanc- ing retrieval augmented generation through drafting,

    Z. Wang, Z. Wang, L. Le, H. S. Zheng, S. Mishra, V . Perot, Y . Zhang, A. Mattapalli, A. Taly, J. Shang et al. , “Speculative rag: Enhanc- ing retrieval augmented generation through drafting,” arXiv preprint arXiv:2407.08223, 2024

    2024 arXiv

  82. [90]

    H ipporag: Neurobiologically inspired long-term memory for large lan guage mod- els,

    B. J. Guti´ errez, Y . Shu, Y . Gu, M. Y asunaga, and Y . Su, “H ipporag: Neurobiologically inspired long-term memory for large lan guage mod- els,” arXiv preprint arXiv:2405.14831 , 2024

    2024 arXiv

  83. [91]

    Augmenting language models with long-term memory,

    W. Wang, L. Dong, H. Cheng, X. Liu, X. Y an, J. Gao, and F. We i, “Augmenting language models with long-term memory,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  84. [92]

    Enhancing long-term memory using hierarchi- cal aggregate tree for retrieval augmented generation,

    A. Aadhithya A et al. , “Enhancing long-term memory using hierarchi- cal aggregate tree for retrieval augmented generation,” arXiv e-prints , pp. arXiv–2406, 2024

    2024

  85. [93]

    Memorag: Mo ving towards next-gen rag via memory-inspired knowledge discov ery,

    H. Qian, P . Zhang, Z. Liu, K. Mao, and Z. Dou, “Memorag: Mo ving towards next-gen rag via memory-inspired knowledge discov ery,” arXiv preprint arXiv:2409.05591, 2024

    2024 arXiv

  86. [94]

    Pistis-rag: Enhancing retrieval-augmented ge neration with human feedback,

    Y . Bai, Y . Miao, L. Chen, D. Wang, D. Li, Y . Ren, H. Xie, C. Y ang, and X. Cai, “Pistis-rag: Enhancing retrieval-augmented ge neration with human feedback,” arXiv preprint arXiv:2407.00072 , 2024

    2024 arXiv

  87. [95]

    Similarity is not all you need: Endowing retrieval augmented generation with multi layered thought s,

    C. Gan, D. Y ang, B. Hu, H. Zhang, S. Li, Z. Liu, Y . Shen, L. J u, Z. Zhang, J. Gu et al. , “Similarity is not all you need: Endowing retrieval augmented generation with multi layered thought s,” arXiv preprint arXiv:2405.19893, 2024

    2024 arXiv

  88. [96]

    Rag-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement,

    J. Jiang, J. Chen, J. Li, R. Ren, S. Wang, W. X. Zhao, Y . Son g, and T. Zhang, “Rag-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement,” arXiv preprint arXiv:2412.12881, 2024

    2024 arXiv

  89. [97]

    Multihop-rag: Benchmarking retri eval- augmented generation for multi-hop queries,

    Y . Tang and Y . Y ang, “Multihop-rag: Benchmarking retri eval- augmented generation for multi-hop queries,” arXiv preprint arXiv:2401.15391, 2024

    2024 arXiv

  90. [98]

    Retrie val- augmented multi-modal chain-of-thoughts reasoning for la rge language models,

    B. Liu, C. Lyu, Z. Min, Z. Wang, J. Su, and L. Wang, “Retrie val- augmented multi-modal chain-of-thoughts reasoning for la rge language models,” arXiv preprint arXiv:2312.01714 , 2023

    2023 arXiv

  91. [99]

    Hop, union , generate: Explainable multi-hop reasoning without rationale superv ision,

    W. Zhao, J. T. Chiu, C. Cardie, and A. M. Rush, “Hop, union , generate: Explainable multi-hop reasoning without rationale superv ision,” arXiv preprint arXiv:2305.14237, 2023

    2023 arXiv

  92. [100]

    Multimodal chain-of-thought reasoning in language model s,

    Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smo la, “Multimodal chain-of-thought reasoning in language model s,” arXiv preprint arXiv:2302.00923, 2023

    2023 arXiv

  93. [101]

    Can gpt improve t he state of prior authorization via guideline based automated question answ ering?

    S. V atsal, A. Singh, and S. Tafreshi, “Can gpt improve t he state of prior authorization via guideline based automated question answ ering?” in AI for Health Equity and Fairness: Leveraging AI to Address S ocial Determinants of Health . Springer, 2024, pp. 147–158

    2024

  94. [102]

    Can gpt redefine medical unders tanding? evaluating gpt on biomedical machine reading comprehensio n,

    S. V atsal and A. Singh, “Can gpt redefine medical unders tanding? evaluating gpt on biomedical machine reading comprehensio n,” arXiv preprint arXiv:2405.18682, 2024

    2024 arXiv

  95. [103]

    Chain-of-verification reduces hallucinati on in large language models,

    S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston, “Chain-of-verification reduces hallucinati on in large language models,” arXiv preprint arXiv:2309.11495 , 2023

    2023 arXiv

  96. [104]

    V erify-and -edit: A knowledge-enhanced chain-of-thought framework,

    R. Zhao, X. Li, S. Joty, C. Qin, and L. Bing, “V erify-and -edit: A knowledge-enhanced chain-of-thought framework,” arXiv preprint arXiv:2305.03268, 2023

    2023 arXiv

  97. [105]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning,

    Y . Zhai, H. Bai, Z. Lin, J. Pan, S. Tong, Y . Zhou, A. Suhr, S. Xie, Y . LeCun, Y . Ma et al. , “Fine-tuning large vision-language models as decision-making agents via reinforcement learning,” arXiv preprint arXiv:2405.10292, 2024

    2024 arXiv

  98. [106]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018

    2018

  99. [107]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P . Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  100. [108]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P . Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  101. [109]

    Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback,

    W. Shen, R. Zheng, W. Zhan, J. Zhao, S. Dou, T. Gui, Q. Zha ng, and X. Huang, “Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback,” arXiv preprint arXiv:2310.05199, 2023

    2023 arXiv

  102. [110]

    Secrets of rlhf in large language models part ii: Reward modeling,

    B. Wang, R. Zheng, L. Chen, Y . Liu, S. Dou, C. Huang, W. Sh en, S. Jin, E. Zhou, C. Shi et al., “Secrets of rlhf in large language models part ii: Reward modeling,” arXiv preprint arXiv:2401.06080 , 2024

    2024 arXiv

  103. [111]

    trlx: A framework fo r large scale reinforcement learning from human feedback,

    A. Havrilla, M. Zhuravinskyi, D. Phung, A. Tiwari, J. T ow, S. Bi- derman, Q. Anthony, and L. Castricato, “trlx: A framework fo r large scale reinforcement learning from human feedback,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Langu age Processing...

    2023

  104. [112]

    Ultrafeedback: Boosting language models with high -quality feedback,

    G. Cui, L. Y uan, N. Ding, G. Y ao, W. Zhu, Y . Ni, G. Xie, Z. L iu, and M. Sun, “Ultrafeedback: Boosting language models with high -quality feedback,” 2023

    2023

  105. [113]

    Learning to sum marize with human feedback,

    N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. V o ss, A. Radford, D. Amodei, and P . F. Christiano, “Learning to sum marize with human feedback,” Advances in Neural Information Processing Systems, vol. 33, pp. 3008–3021, 2020

    2020

  106. [114]

    Nash learning from human feedback,

    R. Munos, M. V alko, D. Calandriello, M. G. Azar, M. Rowl and, Z. D. Guo, Y . Tang, M. Geist, T. Mesnard, A. Michi et al. , “Nash learning from human feedback,” arXiv preprint arXiv:2312.00886 , 2023

    2023 arXiv

  107. [115]

    Skywork-reward: Bag of tricks for reward model ing in llms,

    C. Y . Liu, L. Zeng, J. Liu, R. Y an, J. He, C. Wang, S. Y an, Y . Liu, and Y . Zhou, “Skywork-reward: Bag of tricks for reward model ing in llms,” arXiv preprint arXiv:2410.18451 , 2024

    2024 arXiv

  108. [116]

    Camels in a changing climate: Enhancing lm adaptation with tulu 2,

    H. Ivison, Y . Wang, V . Pyatkin, N. Lambert, M. Peters, P . Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy et al. , “Camels in a changing climate: Enhancing lm adaptation with tulu 2,” arXiv preprint arXiv:2311.10702, 2023

    2023 arXiv

  109. [117]

    Tool- augmented reward modeling,

    L. Li, Y . Chai, S. Wang, Y . Sun, H. Tian, N. Zhang, and H. W u, “Tool- augmented reward modeling,” arXiv preprint arXiv:2310.01045 , 2023

    2023 arXiv

  110. [118]

    Optimal design for reward modeli ng in rlhf,

    A. Scheid, E. Boursier, A. Durmus, M. I. Jordan, P . M´ en ard, E. Moulines, and M. V alko, “Optimal design for reward modeli ng in rlhf,” arXiv preprint arXiv:2410.17055 , 2024

    2024 arXiv

  111. [119]

    Scaling laws for re ward model overoptimization,

    L. Gao, J. Schulman, and J. Hilton, “Scaling laws for re ward model overoptimization,” in International Conference on Machine Learning . PMLR, 2023, pp. 10 835–10 866

    2023

  112. [120]

    A survey of reinforcement learning from human feedback,

    T. Kaufmann, P . Weng, V . Bengs, and E. H¨ ullermeier, “A survey of reinforcement learning from human feedback,” arXiv preprint arXiv:2312.14925, 2023

    2023

  113. [121]

    Rlaif: Scali ng reinforcement learning from human feedback with ai feedbac k,

    H. Lee, S. Phatale, H. Mansoor, K. R. Lu, T. Mesnard, J. F erret, C. Bishop, E. Hall, V . Carbune, and A. Rastogi, “Rlaif: Scali ng reinforcement learning from human feedback with ai feedbac k,” 2023

    2023

  114. [122]

    Hrlaif: Improvements in helpfulness and harm- lessness in open-domain reinforcement learning from ai fee dback,

    A. Li, Q. Xiao, P . Cao, J. Tang, Y . Y uan, Z. Zhao, X. Chen, L. Zhang, X. Li, K. Y ang et al. , “Hrlaif: Improvements in helpfulness and harm- lessness in open-domain reinforcement learning from ai fee dback,” arXiv preprint arXiv:2403.08309 , 2024

    2024 arXiv

  115. [123]

    Magpie: Alignment data synthesis from scratch by prom pting aligned llms with nothing,

    Z. Xu, F. Jiang, L. Niu, Y . Deng, R. Poovendran, Y . Choi, and B. Y . Lin, “Magpie: Alignment data synthesis from scratch by prom pting aligned llms with nothing,” arXiv preprint arXiv:2406.08464 , 2024

    2024 arXiv

  116. [124]

    Helpsteer2-preference: Complem enting ratings with preferences,

    Z. Wang, A. Bukharin, O. Delalleau, D. Egert, G. Shen, J . Zeng, O. Kuchaiev, and Y . Dong, “Helpsteer2-preference: Complem enting ratings with preferences,” arXiv preprint arXiv:2410.01257 , 2024

    2024 arXiv

  117. [125]

    Guiding pretraining in reinforcement lear ning with large language models,

    Y . Du, O. Watkins, Z. Wang, C. Colas, T. Darrell, P . Abbe el, A. Gupta, and J. Andreas, “Guiding pretraining in reinforcement lear ning with large language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 8657–8677

    2023

  118. [126]

    Reward de sign with language models,

    M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh, “Reward de sign with language models,” arXiv preprint arXiv:2303.00001 , 2023

    2023 arXiv

  119. [127]

    Eureka: Human- level reward design via coding large language models,

    Y . J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. J a- yaraman, Y . Zhu, L. Fan, and A. Anandkumar, “Eureka: Human- level reward design via coding large language models,” arXiv preprint arXiv:2310.12931, 2023

    2023 arXiv

  120. [128]

    Self -refined large language model as automated reward function designer for deep reinforcement learning in robotics,

    J. Song, Z. Zhou, J. Liu, C. Fang, Z. Shu, and L. Ma, “Self -refined large language model as automated reward function designer for deep reinforcement learning in robotics,” arXiv preprint arXiv:2309.06687 , 2023

    2023 arXiv

  121. [129]

    Self-rewarding language models,

    W. Y uan, R. Y . Pang, K. Cho, S. Sukhbaatar, J. Xu, and J. W eston, “Self-rewarding language models,” arXiv preprint arXiv:2401.10020 , 2024

    2024 arXiv

  122. [130]

    Exploration v ersus exploita- tion in reinforcement learning: A stochastic control appro ach,

    H. Wang, T. Zariphopoulou, and X. Zhou, “Exploration v ersus exploita- tion in reinforcement learning: A stochastic control appro ach,” arXiv preprint arXiv:1812.01552, 2018

    2018 arXiv

  123. [131]

    Guarantees for epsilon-greedy reinforcement learning wi th function approximation,

    C. Dann, Y . Mansour, M. Mohri, A. Sekhari, and K. Sridha ran, “Guarantees for epsilon-greedy reinforcement learning wi th function approximation,” in International conference on machine learning . PMLR, 2022, pp. 4666–4689

    2022

  124. [132]

    Asynchronous methods for deep reinforcemen t learning,

    V . Mnih, “Asynchronous methods for deep reinforcemen t learning,” arXiv preprint arXiv:1602.01783 , 2016

    2016 arXiv

  125. [133]

    Adaptive ε-greedy exploration in reinforcement learning based on value differences,

    M. Tokic, “Adaptive ε-greedy exploration in reinforcement learning based on value differences,” in Annual conference on artificial intelli- gence. Springer, 2010, pp. 203–210

    2010

  126. [134]

    Bo ltzmann exploration done right,

    N. Cesa-Bianchi, C. Gentile, G. Lugosi, and G. Neu, “Bo ltzmann exploration done right,” Advances in neural information processing systems, vol. 30, 2017

    2017

  127. [135]

    Explorllm : Guiding exploration in reinforcement learning with large language models,

    R. Ma, J. Luijkx, Z. Ajanovic, and J. Kober, “Explorllm : Guiding exploration in reinforcement learning with large language models,” arXiv preprint arXiv:2403.09583 , 2024

    2024 arXiv

  128. [136]

    Epo: Hierarch ical llm agents with environment preference optimization,

    Q. Zhao, H. Fu, C. Sun, and G. Konidaris, “Epo: Hierarch ical llm agents with environment preference optimization,” arXiv preprint arXiv:2408.16090, 2024

    2024 arXiv

  129. [137]

    Balancing exploration and exploitation in llm using soft rllf for enhanced negation understanding,

    H.-T. Nguyen and K. Satoh, “Balancing exploration and exploitation in llm using soft rllf for enhanced negation understanding,” arXiv preprint arXiv:2403.01185, 2024

    2024 arXiv

  130. [138]

    Empower large language model to perform better on industrial domain-specific question answ ering,

    F. Y ang, P . Zhao, Z. Wang, L. Wang, J. Zhang, M. Garg, Q. L in, S. Rajmohan, and D. Zhang, “Empower large language model to perform better on industrial domain-specific question answ ering,” arXiv preprint arXiv:2305.11541, 2023

    2023 arXiv

  131. [139]

    Direct preference optimization: Y our language mo del is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. E rmon, and C. Finn, “Direct preference optimization: Y our language mo del is secretly a reward model,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  132. [140]

    Smaug: Fixing failure modes of preference optimisation wi th dpo- positive,

    A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, a nd C. White, “Smaug: Fixing failure modes of preference optimisation wi th dpo- positive,” arXiv preprint arXiv:2402.13228 , 2024

    2024 arXiv

  133. [141]

    J. Wu, Y . Xie, Z. Y ang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He, arXiv preprint arXiv:2407.08639 , 2024

    2024 arXiv

  134. [142]

    sdpo: Don’t use your data all at once,

    D. Kim, Y . Kim, W. Song, H. Kim, Y . Kim, S. Kim, and C. Park , “sdpo: Don’t use your data all at once,” arXiv preprint arXiv:2403.19270 , 2024

    2024 arXiv

  135. [143]

    Weighted -reward preference optimization for implicit model fusion,

    Z. Y ang, F. Wan, L. Zhong, T. Shi, and X. Quan, “Weighted -reward preference optimization for implicit model fusion,” arXiv preprint arXiv:2412.03187, 2024

    2024 arXiv

  136. [144]

    Provably r obust dpo: Aligning language models with noisy feedback,

    S. R. Chowdhury, A. Kini, and N. Natarajan, “Provably r obust dpo: Aligning language models with noisy feedback,” arXiv preprint arXiv:2403.00409, 2024

    2024 arXiv

  137. [145]

    A comprehensive survey of datas ets, theories, variants, and applications in direct preference optimization,

    W. Xiao, Z. Wang, L. Gan, S. Zhao, W. He, L. A. Tuan, L. Che n, H. Jiang, Z. Zhao, and F. Wu, “A comprehensive survey of datas ets, theories, variants, and applications in direct preference optimization,” arXiv preprint arXiv:2410.15595 , 2024

    arXiv

  138. [146]

    A general theoretical paradigm to unders tand learning from human preferences,

    M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. V a lko, and D. Calandriello, “A general theoretical paradigm to unders tand learning from human preferences,” in International Conference on Artificial Intelligence and Statistics . PMLR, 2024, pp. 4447–4455

    2024

  139. [147]

    Rethinking bradley-te rry models in preference-based reward modeling: Foundations, theory, a nd alterna- tives,

    H. Sun, Y . Shen, and J.-F. Ton, “Rethinking bradley-te rry models in preference-based reward modeling: Foundations, theory, a nd alterna- tives,” arXiv preprint arXiv:2411.04991 , 2024

    2024 arXiv

  140. [148]

    Self-pl ay preference optimization for language model alignment,

    Y . Wu, Z. Sun, H. Y uan, K. Ji, Y . Y ang, and Q. Gu, “Self-pl ay preference optimization for language model alignment,” arXiv preprint arXiv:2405.00675, 2024

    2024 arXiv

  141. [149]

    Preference optimization with multi-sample comparisons,

    C. Wang, Z. Zhao, C. Zhu, K. A. Sankararaman, M. V alko, X . Cao, Z. Chen, M. Khabsa, Y . Chen, H. Ma et al. , “Preference optimization with multi-sample comparisons,” arXiv preprint arXiv:2410.12138 , 2024

    2024 arXiv

  142. [150]

    Robust preference optimization thr ough reward model distillation,

    A. Fisch, J. Eisenstein, V . Zayats, A. Agarwal, A. Beir ami, C. Nagpal, P . Shaw, and J. Berant, “Robust preference optimization thr ough reward model distillation,” arXiv preprint arXiv:2405.19316 , 2024

    2024 arXiv

  143. [151]

    Pace: Impro ving prompt with actor-critic editing for large language model,

    Y . Dong, K. Luo, X. Jiang, Z. Jin, and G. Li, “Pace: Impro ving prompt with actor-critic editing for large language model,” arXiv preprint arXiv:2308.10088, 2023

    2023 arXiv

  144. [152]

    Fine-tuning langu age models from human preferences,

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radfor d, D. Amodei, P . Christiano, and G. Irving, “Fine-tuning langu age models from human preferences,” arXiv preprint arXiv:1909.08593 , 2019

    1909 arXiv

  145. [153]

    Open-ended reinforcement lea rning with neural reward functions,

    R. Meier and A. Mujika, “Open-ended reinforcement lea rning with neural reward functions,” Advances in Neural Information Processing Systems, vol. 35, pp. 2465–2479, 2022

    2022

  146. [154]

    Hdflow: Enhancing llm complex p roblem- solving with hybrid thinking and dynamic workflows,

    W. Y ao, H. Mi, and D. Y u, “Hdflow: Enhancing llm complex p roblem- solving with hybrid thinking and dynamic workflows,” arXiv preprint arXiv:2409.17433, 2024

    2024 arXiv

  147. [155]

    E nhancing multi-step reasoning abilities of language models through direct q- function optimization,

    G. Liu, K. Ji, R. Zheng, Z. Wu, C. Dun, Q. Gu, and L. Y an, “E nhancing multi-step reasoning abilities of language models through direct q- function optimization,” arXiv preprint arXiv:2410.09302 , 2024

    2024 arXiv

  148. [156]

    Llms as debate partners: Utilizing genetic algorithms and adversarial search for adaptive arguments,

    P . Aryan, “Llms as debate partners: Utilizing genetic algorithms and adversarial search for adaptive arguments,” arXiv preprint arXiv:2412.06229, 2024

    2024 arXiv

  149. [157]

    Evolution ary computation in the era of large language model: Survey and ro admap,

    X. Wu, S.-h. Wu, J. Wu, L. Feng, and K. C. Tan, “Evolution ary computation in the era of large language model: Survey and ro admap,” arXiv preprint arXiv:2401.10034 , 2024

    2024 arXiv

  150. [158]

    Cont rolling the mutation in large language models for the efficient evolu tion of algorithms,

    H. Yin, A. V . Kononova, T. B¨ ack, and N. van Stein, “Cont rolling the mutation in large language models for the efficient evolu tion of algorithms,” arXiv preprint arXiv:2412.03250 , 2024

    2024 arXiv

  151. [159]

    Connecting large language models with evoluti on- ary algorithms yields powerful prompt optimizers,

    Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. B ian, and Y . Y ang, “Connecting large language models with evoluti on- ary algorithms yields powerful prompt optimizers,” arXiv preprint arXiv:2309.08532, 2023

    2023 arXiv

  152. [160]

    Larg e language model-based evolutionary optimizer: Reasoning w ith elitism,

    S. Brahmachary, S. M. Joshi, A. Panda, K. Koneripalli, A. K. Sagotra, H. Patel, A. Sharma, A. D. Jagtap, and K. Kalyanaraman, “Larg e language model-based evolutionary optimizer: Reasoning w ith elitism,” arXiv preprint arXiv:2403.02054 , 2024

    2024 arXiv

  153. [161]

    A ma tch made in consistency heaven: when large language models meet evol utionary algorithms,

    W. Chao, J. Zhao, L. Jiao, L. Li, F. Liu, and S. Y ang, “A ma tch made in consistency heaven: when large language models meet evol utionary algorithms,” arXiv preprint arXiv:2401.10510 , 2024

    2024 arXiv

  154. [162]

    Large language models as surrogate models in evolutionary algorithms: A preliminary study,

    H. Hao, X. Zhang, and A. Zhou, “Large language models as surrogate models in evolutionary algorithms: A preliminary study,” arXiv preprint arXiv:2406.10675, 2024

    2024 arXiv

  155. [163]

    Wese: Weak exploration to strong exploitation for llm agents,

    X. Huang, W. Liu, X. Chen, X. Wang, D. Lian, Y . Wang, R. Ta ng, and E. Chen, “Wese: Weak exploration to strong exploitation for llm agents,” arXiv preprint arXiv:2404.07456 , 2024

    2024 arXiv

  156. [164]

    Towards understanding grokking: An effectiv e theory of representation learning,

    Z. Liu, O. Kitouni, N. S. Nolte, E. Michaud, M. Tegmark, and M. Williams, “Towards understanding grokking: An effectiv e theory of representation learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 34 651–34 663, 2022. This figure "fig1.png" is available...

    2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.