Pith. sign in

REVIEW 3 cited by

Inverse Reward Design

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1711.02827 v2 pith:XAR2G4OO submitted 2017-11-08 cs.AI cs.LG

Dylan Hadfield-Menell , Smitha Milli , Pieter Abbeel , Stuart Russell , Anca Dragan This is my paper · ORCID
classification cs.AIcs.LG
keywords rewardbehaviordesignscenariostheywhatactuallyagents
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers

    cs.CY 2026-04 unverdicted novelty 6.0 of 10

    Survey experiment finds that people apply more deontological standards to AI described as human-programmed and to the programmers themselves than to unaided humans or unprogrammed robots in a moral dilemma.

  2. Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    CRA trains sparse autoencoders on PRM activations and applies backdoor adjustment to estimate true rewards, reducing reward hacking in math reasoning.

  3. Residual Reward Models for Preference-based Reinforcement Learning

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Combining a hand-designed or learned prior reward with a preference-trained residual improves sample efficiency and final performance in preference-based reinforcement learning.

Pith tools