{"id":"69fa59fa-e63a-46a9-b27b-13120e498e8a","arxiv_id":"2506.15421","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A structured survey of reward modeling in deep RL, proposing a three-axis taxonomy and reviewing applications and evaluation methods.","lead":"This paper surveys how reward models are built and used in deep reinforcement learning, sorting methods by reward source, learning mechanism, and feedback type. It offers a single reference map for a field that underpins LLM alignment, robot learning, and goal-conditioned control.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The three-axis taxonomy omits algorithmic/agent-generated rewards: Section 4.2's intrinsic methods are labeled 'human' in Table 1, so the central classification is not a faithful partition.","rationale":"The reader's weakest assumption is that the three-axis taxonomy is exhaustive and mutually exclusive, with the 'human intrinsic' cell as a concrete failure. My audit of the full text lands on the same point: the source dimension is not a faithful partition because algorithmic intrinsic rewards are neither human-provided nor AI-generated in the paper's own sense. This is load-bearing because the survey's central contribution is the structured map, not any new empirical or theoretical result. If the source axis misassigns an entire mechanism class, then Table 1's summary and the abstract's 'comprehensive review' claim are both compromised, even where individual method descriptions are accurate. I do not see a separate concern that would change the verdict: the paper gives useful descriptions and covers many representative methods, but the taxonomy's internal inconsistency and the absence of explicit inclusion criteria keep the claim conditional. My recommendation therefore leaves the reader's CONDITIONAL verdict unchanged.","tokens_in":14250,"tokens_out":2812,"duration_ms":28729,"concrete_test":"Audit all entries in Table 1's 'human intrinsic' row against the original papers: for each, determine whether the reward signal is produced by a human annotator/expert, by an LLM/VLM, or by a deterministic or learned function of agent experience. If zero of the entries are human-provided and none are AI foundation-model outputs, then the published taxonomy has no valid source category for them; re-code the row as 'algorithmic intrinsic' and recompute Table 1. A second decisive check: re-derive Section 3's dichotomy after adding this category; if any Section 4.2 method cannot be placed, the exhaustive-partition claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 1 and the abstract claim a comprehensive review and a categorization by source, mechanism, and learning paradigm (Table 1). For this claim to hold, each dimension must be a faithful partition of reward-modeling approaches. Table 1's 'human intrinsic' row violates this. The intrinsic-motivation methods in Section 4.2—Pathak et al. 2017 (prediction error), Houthooft et al. 2016 (information gain), Burda et al. 2018 (random network distillation), Tang et al. 2017 (counts), Bellemare et al. 2016 (density), Eysenbach et al. 2018 / Mazzaglia et al. 2022 (empowerment mutual information)—compute rewards from agent-environment statistics, not from human expertise and not from AI foundation-model outputs. No human provides these rewards, and the AI-generated category in Section 3.2 is explicitly about foundation models. Thus the source axis has no bucket for the dominant class of intrinsic rewards. The inconsistency is structural, not a typo: 'intrinsic' is a mechanism, while the source question is 'where does the reward come from?'; placing these methods under 'human' guarantees the row's counts are wrong. The same gap appears in the learning-paradigm axis: Section 5 defines paradigms only for human feedback, leaving intrinsic and AI rows with '-' and no account of how their rewards are learned. Without a fourth source category (algorithmic/agent-generated) or a redefinition of 'source', the headline claim of a systematic map is undercut.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey reviews reward modeling in deep reinforcement learning. It proposes a taxonomy on three axes — source (human-provided vs. AI-generated), mechanism (extrinsic vs. intrinsic), and learning paradigm (demonstrations, goals, preferences) — and operationalizes this taxonomy in Table 1. It covers background on MDPs and reward design, describes MaxEnt-IRL, AIRL, goal-conditioned rewards, preference models (Bradley-Terry, Plackett-Luce, regret-based), applications (control, generative model post-training, other domains), and evaluation methods (EPIC, DARD, STARC, interpretable representations). The paper's stated contribution is a comprehensive, systematic map of reward-modeling techniques.","tokens_in":14551,"tokens_out":9596,"duration_ms":86429,"significance":"If the taxonomy were sound, the survey would fill a genuine gap: no prior survey treats reward modeling as a standalone topic, and the paper usefully connects RLHF with older IRL and intrinsic-motivation literature. The mathematical exposition is generally reliable: Eq. (3) (MaxEnt-IRL), Eq. (6) (AIRL), Eq. (9) (BT), and Eq. (13) (PL) are standard and correctly stated, and the evaluation section is a helpful resource. However, the central classification — the reason the survey exists — is not currently a faithful partition of the literature: Table 1 places agent-computed intrinsic rewards under \"human\", and Section 5's learning paradigms cover only human feedback while Table 1 includes AI-generated variants. These issues are load-bearing for the paper's main claim, although they are fixable by restructuring the taxonomy and the table.","major_comments":[{"comment":"The row \"human intrinsic\" in Table 1 is internally inconsistent. The methods listed there — Pathak et al. 2017, Houthooft et al. 2016, Burda et al. 2018, Tang et al. 2017, Bellemare et al. 2016, Eysenbach et al. 2018, and Mazzaglia et al. 2022 — generate rewards from agent-environment statistics (prediction error, count-based exploration, density estimation, or mutual information), not from human expertise or supervision, and they do not match the Section 3.1 definition of human-provided rewards. Section 4.2 itself describes intrinsic motivation as the agent's innate motivation to explore. Consequently, the source axis in the proposed taxonomy has no faithful category for a major class of intrinsic rewards, which invalidates the claim that the taxonomy is a partition. I recommend adding an algorithmic/agent-generated source category (or redefining \"source\" to accommodate these methods) and moving the affected entries from the \"human\" row.","section":"Table 1 / Section 4.2"},{"comment":"The learning-paradigm axis is defined and developed only for human feedback: Section 5 begins by saying the paradigms learn \"from different kinds of human feedback\" and then covers demonstrations, goals, and preferences. However, Section 3.2 and Table 1 explicitly include AI-generated rewards, including \"AI extrinsic goal\" (Sontakke et al. 2023; Fan et al. 2022; Rocamonde et al. 2023) and \"AI extrinsic preference\" (Bai et al. 2022; Lee et al. 2024; Wang et al. 2024). These entries have no corresponding learning-paradigm discussion, and the two intrinsic rows have \"-\" in the Feedback column. The third axis therefore does not classify all methods in the survey. Please extend Section 5 to cover AI-generated feedback and intrinsic-reward mechanisms, or restrict the stated scope of the learning-paradigm axis and adjust the paper's claims accordingly.","section":"Section 5 / Table 1"},{"comment":"The paper repeatedly claims a \"comprehensive\" and \"systematic\" review, but no inclusion criteria, search protocol, or other selection mechanism is given for Table 1, and Section 8 concedes that \"the design and variations of reward models still extend beyond the scope of this discussion.\" Moreover, manual reward engineering, which is discussed as a human-provided source in Section 3.1, appears nowhere in Table 1. This makes the comprehensiveness claim unverifiable and the categorization non-exhaustive with respect to the paper's own text. Please state the scope and inclusion criteria explicitly, or qualify the claims in the abstract and Section 1.","section":"Section 1, Section 3.1, Section 8, Table 1"}],"minor_comments":[{"comment":"The caption says the framework categorizes by \"sources, feedback types, and mechanisms\", but Section 1 and Table 1 use \"source, mechanism, and learning paradigm\"; align the caption with the actual three axes.","section":"Figure 1 caption"},{"comment":"The phrase \"In the below\" should be \"In the following\".","section":"Section 5.2"},{"comment":"The text contains many citation spacing errors (e.g., \"Silveret al.\" and \"Towerset al.\"); a careful proofreading pass is needed.","section":"Throughout"},{"comment":"Wan et al. 2024 appears in the \"human intrinsic\" row but is not discussed in Section 4.2; the caption says the table summarizes algorithms mentioned in Sections 3-5, so either add the corresponding discussion or remove the entry.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a survey; my main concern is that the taxonomy, which is the paper's central contribution, is currently not a faithful partition. I do not see grounds for rejection if the authors are willing to restructure the source and learning-paradigm axes and Table 1. The self-citations in Table 1 (Xu et al. 2023, Wan et al. 2024, Wang et al. 2025) are relevant in principle, but with Wan et al. currently misclassified the authors should re-check those entries as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on arXiv:2506.15421. The survey does two things well. It proposes a three-axis taxonomy—source, mechanism, learning paradigm—that is genuinely useful for cross-linking IRL, RLHF, goal-conditioned RL, and the newer LLM/VLM-as-reward work. And it reproduces the standard math correctly: MaxEnt-IRL, AIRL, Bradley-Terry, Plackett-Luce, EPIC/DARD/STARC are all stated without errors I could find. The evaluation section is a plus; most surveys in this space skip it.\n\nThe soft spot is Table 1, and it's not cosmetic. The row 'human intrinsic' lists Pathak, Houthooft, Burda, Bellemare, Eysenbach—methods whose rewards come from prediction errors, counts, density, and mutual information computed by the agent itself. No human supplies those rewards. The paper's own Section 4.2 defines intrinsic motivation as agent-innate, and Section 3.2's AI-generated category is explicitly about foundation models. So the source axis has no bucket for algorithmic/agent-generated rewards. That means the taxonomy is not a faithful partition of the literature, which undercuts the abstract's claim of a comprehensive and systematic map. The 'AI intrinsic' row is also a mix: Klissarov, Xu, Du use LLMs or logic, but the mechanism axis is about what drives learning, not where the reward comes from—so the crossing of source × mechanism isn't being applied consistently.\n\nThere are two smaller issues. No search protocol or inclusion criteria are given, so the comprehensiveness of Table 1 can't be audited; the Section 8 admission that the topic 'extends beyond the scope' actually undercuts the 'filling the vacancy' headline. And Section 5 defines learning paradigms only for human feedback, leaving the intrinsic and AI rows with '-' and no explanation of how those rewards are learned.\n\nNone of this is fatal for the survey's purpose. The writing is clear, the coverage is broad, and the three-axis framing is a reasonable starting point even if the execution is inconsistent. The reader's conditional verdict seems right to me. I'd send this to peer review—it deserves a serious referee who can push the authors to fix Table 1, define the source categories more carefully, and either add a fourth source (algorithmic) or reclassify the intrinsic methods. If those revisions land, it becomes a genuinely useful reference for people entering reward modeling or looking for a map of the field. I'd cite it once the table is fixed; for now, I'd be cautious.","headline":"A useful survey with a solid taxonomy, but Table 1's 'human intrinsic' row is a structural error that undercuts the comprehensiveness claim.","tokens_in":15084,"tokens_out":3267,"would_cite":false,"duration_ms":28870,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that all reward-modeling methods in deep RL can be organized by a three-axis classification: source, mechanism, and learning paradigm.","keywords":["reward modeling","deep reinforcement learning","reward design","inverse reinforcement learning","preference-based reinforcement learning","intrinsic motivation","reinforcement learning from human feedback","foundation models"],"falsifier":"Take a random sample of roughly one hundred recently published reinforcement-learning papers that introduce or use a reward model, and have two independent coders assign each to a cell of Table 1 using the survey's definitions; if a substantial fraction cannot be placed uniquely, or if the authors' own representative table contains cells that conflict with those definitions, the partition claim fails.","tokens_in":14053,"feed_emoji":"🧭","tokens_out":10099,"duration_ms":87603,"temperature":0.7,"pith_summary":"Reward models translate a task designer's intentions into the numerical signals an RL agent optimizes, and this survey treats them as a subject in their own right rather than a side topic inside inverse RL or RLHF. It tries to establish that the field can be systematically mapped by three questions: where the reward comes from (human-provided or AI-generated), what drive makes the agent learn (extrinsic or intrinsic motivation), and how the reward is learned from feedback (demonstrations, goals, or preferences). The survey then uses this map to organize established and emerging methods, to review applications in control and generative-model post-training, and to lay out how reward models should be evaluated. A sympathetic reader would take away a structured route into the literature and a shared vocabulary for comparing methods.","feed_headline":"Three axes map reward-modeling methods in deep reinforcement learning","feed_subtitle":"It spans human- and AI-generated rewards, intrinsic and extrinsic drives, and all three feedback paradigms.","key_machinery":"The carrying mechanism is the taxonomy itself: a three-axis grid of source, mechanism, and learning paradigm, drawn as Figure 1 and populated as Table 1. Each surveyed method is assigned to one cell of this grid, so that, for example, preference-based RLHF occupies the human-extrinsic-preference cell while a vision-language reward method like RoboCLIP sits in the AI-extrinsic-goal cell. This grid is what converts a list of techniques into a map, and it is also the paper's main original contribution. The mathematical preference models (Bradley-Terry, Plackett-Luce, regret-based) serve as the concrete machinery inside the preference cell, connecting observed human comparisons to reward parameters.","core_discovery":"The survey's central claim is that reward modeling deserves a standalone systematic review and that its literature can be cleanly organized by three axes. The source axis distinguishes human-provided rewards, ranging from hand-crafted functions to human-in-the-loop supervision, from AI-generated rewards, where LLMs and VLMs supply reward functions or feedback. The mechanism axis separates extrinsic rewards tied to the task objective from intrinsic motivation, such as curiosity, empowerment, and knowledge-driven bonuses. The learning-paradigm axis groups methods by the feedback they consume: demonstrations, goals, or preferences, with preference learning covering Bradley-Terry models, regret-based models, ordinal feedback, and listwise rankings. The paper presents this framework as covering both established methods and emerging foundation-model-based approaches, and supplements it with a table of representative algorithms and a discussion of evaluation via policy performance, distance metrics, and interpretable representations.","pith_inferences":["A testable extension the survey leaves implicit: if the taxonomy is truly exhaustive, a random sample of recent reinforcement-learning papers should each fit exactly one cell of Table 1, so coding such a sample would validate or challenge the partition.","The source axis suggests a comparative research program the survey does not pursue: reward-hacking failures should cluster differently for human-provided rewards (misspecified objectives) than for AI-generated rewards (inherited model biases), and that difference could be measured empirically.","The preference-learning section implicitly orders feedback types by information content—pairwise, ordinal, listwise—hinting at a quantifiable trade-off between annotator effort and statistical efficiency that the survey does not formalize.","The taxonomy could be extended with a fourth axis, evaluation regime, by pairing each cell with its most appropriate metric from Section 7; this would turn the map into a decision procedure."],"forward_implications":["Practitioners can position any reward-modeling method by asking three questions—source, mechanism, learning paradigm—which makes method selection and comparison more systematic.","Reward-model evaluation is presented as inherently multi-perspective: policy performance, distance-based metrics, and interpretable representations each expose different failure modes and are best used together.","Foundation-model-based reward generation is classified as a distinct and growing source, so future reviews and research agendas should treat LLM and VLM rewards as a first-class category alongside human feedback.","The named future directions—vectorized rewards, interpretable reward models, ethical constraints, and reward foundation models—follow directly from gaps left open by the taxonomy."],"supporting_citations":[{"why":"Supplies the foundational apprenticeship and inverse RL method that the demonstration-based learning paradigm builds on.","marker":"[Abbeel and Ng, 2004]"},{"why":"Provides the maximum-entropy IRL objective connecting demonstrations to a Boltzmann distribution over trajectories.","marker":"[Ziebartet al., 2008]"},{"why":"Establishes deep RL from pairwise human preferences, the core of the preference-based learning paradigm.","marker":"[Christianoet al., 2017]"},{"why":"Defines curiosity-driven intrinsic reward as prediction error, grounding the intrinsic-motivation mechanism axis.","marker":"[Pathaket al., 2017]"},{"why":"Exemplifies reward modeling for LLM alignment, motivating the preference and RLHF paradigm and the AI-generated source category.","marker":"[Ouyanget al., 2022]"},{"why":"Introduces EPIC, the canonical distance metric the survey uses for reward-model evaluation.","marker":"[Gleaveet al., 2020]"},{"why":"A prior RLHF survey the paper positions its contribution against, showing the gap it aims to fill.","marker":"[Kaufmannet al., 2023]"},{"why":"A prior inverse RL survey the paper distinguishes itself from in scope.","marker":"[Arora and Doshi, 2021]"}],"fun_headline_variants":["Survey maps reward models across source, mechanism, learning","Reward modeling in deep RL: a three-axis survey","Three axes organize reward model methods in RL survey","Reward models surveyed: humans, AI, intrinsic, extrinsic, preferences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The map's usefulness assumes the three-axis scheme—source, mechanism, learning paradigm—covers the reward-modeling literature without leaving out a major family and without letting methods fall into more than one cell.","fun_headline_variants_meta":{"raw":{"variants":["Survey maps reward models across source, mechanism, learning","Reward modeling in deep RL: a three-axis survey","Three axes organize reward model methods in RL survey","Reward models surveyed: humans, AI, intrinsic, extrinsic, preferences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000326,"raw_usage":{"total_tokens":1800,"prompt_tokens":893,"completion_tokens":907,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":840}},"tokens_in":509,"tokens_out":907,"duration_ms":8419,"temperature":1.0,"reasoning_tokens":840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:34:28.723789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of roughly one hundred recently published reinforcement-learning papers that introduce or use a reward model, and have two independent coders assign each to a cell of Table 1 using the survey's definitions; if a substantial fraction cannot be placed uniquely, or if the authors' own representative table contains cells that conflict with those definitions, the partition claim fails.","supporting_citations":[],"review_version":2}