REVIEW 4 major objections 4 minor 19 references
TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that an offline reinforcement learning policy can be trained from reward-free data using only a handful of ranked trajectories, matching the performance of a policy trained with the ground-truth reward.
desk verdict A clean T-REX + TD3+BC combination with solid MuJoCo results, but the 'human preferences' claim rests on ground-truth-reward rankings and needs a real human study before the headline holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the learned state-based reward model $\hat{r}_{\theta}(s)$, produced by T-REX from ranked trajectory pairs through the softmax preference loss $\mathcal{L}(\theta) = -\sum_{\tau_i \prec \tau_j} \log \frac{\exp\sum_{s\in\tau_j} \hat{r}_{\theta}(s)}{\exp\sum_{s\in\tau_i} \hat{r}_{\theta}(s) + \exp\sum_{s\in\tau_j} \hat{r}_{\theta}(s)}$. This reward is then used to relabel the entire offline dataset, converting reward-free transitions $(s,a,s')$ into reward-labeled tuples $(s,a,s',\hat{r}_{\theta}(s))$, which TD3+BC consumes in its policy update $\pi = \arg\max_{\pi} \mathbb{E}_{(s,a)\sim D}[\lambda Q(s,\pi(s)) - (\pi(s)-a)^2]$. The mechanism that carries the argument is that trajectory rankings provide preference supervision without reward values or optimal demonstrations, and the relabeled dataset makes an ordinary offline RL algorithm applicable.
What would settle it
A concrete way to settle the central claim would be to collect real human rankings over trajectories from the D4RL medium datasets, train TROFI on those rankings, and compare with TROFI trained on the automated episodic-return rankings used in the paper; a large and systematic performance gap would falsify the assumption that return-based rankings stand in for human preferences, as would an inspection showing the learned policy tracks the proxy return rather than the preferences humans express.
Extended reading notes
Core claim
The paper's central discovery is that the reward function used to label an offline dataset is not just a source of signal: its learnability shapes how well a temporal-difference value function can align with the future discounted reward, and this in turn determines the final policy. TROFI learns a state-based reward $\hat{r}_{\theta}(s)$ from ranked trajectory pairs via the T-REX softmax ranking loss, labels every transition in the offline dataset with it, and then optimizes a policy with TD3+BC. On the D4RL MuJoCo tasks and a 3D game navigation benchmark, the authors report that TROFI consistently outperforms offline imitation and inverse-RL baselines such as ORIL and DWBC, and matches or exceeds the ground-truth-reward TD3+BC baseline. Their reward analysis attributes the cases where TROFI beats the ground-truth reward to a better-aligned value function: the learned reward makes the critic more correlated with the actual discounted return and better at ranking optimal actions over random ones, whereas the same optimizer with the engineered reward learns a less well-aligned critic.
Load-bearing premise
The method stands or falls on the assumption that ordering trajectories by their dataset-provided episodic return is equivalent to ordering them by human preference; the paper does not test this with real people, and its simulated noisy ranking in Appendix E is only an approximation.
Editorial extensions
If this is right
- A reward-free offline dataset with mixed skill levels becomes usable for policy training after a human (or automated proxy) ranks only a small subset of trajectories, with as few as 5% of the dataset providing the preference signal.
- On the D4RL MuJoCo tasks and the 3D game environment, TROFI's policies match or exceed the ground-truth-reward TD3+BC baseline, so for these domains a hand-crafted reward is not a prerequisite for strong offline performance.
- Because TROFI does not need optimal expert demonstrations, it avoids the failure mode of discriminator-based methods on low-variance expert datasets and can exploit suboptimal, multi-skill data.
- The reward analysis implies that offline RL algorithms benefit when the labeling reward is easy for the value function to regress; in the reported cases, the learned TROFI reward yields a value function more correlated with the discounted return than the engineered reward does.
Reading between the lines
- If the central claim holds, TROFI could turn raw playtest recordings into game NPC policies with no reward engineering at all, since only a handful of trajectory rankings are needed; the 3D game experiment already points in this direction.
- The authors' value-alignment analysis suggests a design principle broader than TROFI: offline RL reward shaping should target how easily the critic can regress the discounted return, not only the ordinal content of the reward.
- One testable extension the paper does not run is a real-human ranking study on the same benchmarks; the noise-injection experiment in Appendix E is a substitute, so the human-preference assumption remains open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TROFI, an offline inverse reinforcement learning method that first learns a reward model with T-REX from pairwise trajectory rankings, then uses that model to label an unlabeled offline dataset, and finally trains a policy with TD3+BC on the relabeled data. The experiments cover D4RL MuJoCo tasks, a 3D game environment, and (in the appendix) Adroit tasks, and report that TROFI performs comparably to or better than TD3+BC with ground-truth reward and better than several offline IL/IRL baselines. The paper also presents a reward analysis claiming that the learned reward can outperform the ground-truth reward because it makes the value function easier to fit to the discounted reward.
Significance. If the central claim were fully supported, TROFI would be a practical contribution: it shows that offline RL can be driven by coarse trajectory preferences rather than per-transition rewards, and it highlights an under-appreciated interaction between reward scaling and offline value learning. The method is simple, the experimental comparison is broad, and the reward analysis in Section 3.4 raises an important and testable point about value-function alignment. However, the current evidence does not establish the human-preference claim: all rankings come from ground-truth episodic return, and no human study or systematic deviation model is provided. The paper also contains an unexplained dependence on the size of the ranked subset, and the appendix reports substantial failures on Adroit that are absent from the main text's claims.
major comments (4)
- [Section 3.1 and Abstract] The central claim that TROFI learns a reward function from human preferences without a pre-defined reward function is not supported by the experiments. Section 3.1 states: 'we approximate human preferences through an automated ranking process using the episodic rewards provided by the dataset.' Thus the ranking signal, which is the only supervision for the reward model, is computed from the very ground-truth reward the method claims to avoid. The noise experiment in Appendix E only swaps 20% of ranked pairs on two datasets; it does not test systematic differences between human and return-based orderings, such as preferences for safer or more human-like behavior. The paper itself defers human exploration to future work. The authors should either provide a human-subject study or a credible model of human preferences, or substantially reframe the contribution as learning from trajectory rankings rather than 'without a reward function.'
- [Appendix B, Section 3.2] Appendix B states that 'we used the best number of |M| determined in preliminary experiments for both TROFI and baselines.' If |M| was selected using the same test tasks reported in Tables 1-2, this is a form of test-set selection and can inflate TROFI's results, especially because Table 2 shows that performance varies non-monotonically with |M| and that no single |M| dominates. Please specify the selection protocol (e.g., a single fixed budget for all tasks, or a validation-based rule) and report results for a fixed |M| as well as the best-per-task values.
- [Appendix E, Table 4] The broad claim that TROFI 'consistently outperforms baselines' is contradicted by the Adroit results reported only in the appendix. In Table 4, TROFI's scores on the Adroit human and cloned tasks (e.g., pen-human -2.8 vs BC 99.7, door-human -0.3 vs BC 9.4, relocate-human -0.3 vs BC 0.6) are far below the BC baseline, and BC also outperforms all offline RL methods in that setting. The main text should either restrict the claim to the MuJoCo and 3D-game environments or include and discuss the Adroit failure, since it materially affects the generality claim in the introduction and abstract.
- [Section 3.4, Figure 1(f)] The reward-transformation experiment that allegedly explains why TROFI outperforms GT relies on a scaling factor that is 'chosen doing some preliminar experiments' and whose value is not reported. This makes the claim that simple linear transformations of the GT reward improve performance unfalsifiable as presented. Report the exact transformation, the range of values tried, the selection rule, and the resulting scores across seeds; otherwise the explanation in Section 3.4 remains post hoc.
minor comments (4)
- [Section 2.1] The sentence 'To solve this problem we introduce Trajectory-Ranked OFfline Inverse reinforcement learning (TROFI). consists of the following steps' contains a typo: the period after 'TROFI' should be a comma or the sentence should be restructured.
- [Appendix E] The appendix cross-references 'Section 4.1 of the main paper' and 'Section 4.2', but the main paper has no Section 4; the relevant sections are 3.3 and 3.2.
- [Section 3.4] The word 'utlimately' should be 'ultimately'.
- [Appendix B] The code repository is 'published upon acceptance,' so experimental reproducibility cannot currently be verified; consider releasing the code or providing a detailed hyperparameter table in the supplement.
Circularity Check
The preference signal is generated from the ground-truth reward, so the claimed reward-free result reduces to fitting the very reward the method says it avoids.
-
fitted input called prediction
[Section 3.1 (Environments and Datasets) with Eq. (1); empirical claim in Section 3.3/Table 1]
"For all the environments, we rank a subset M of the trajectories from the original dataset D. Similar to the oracle used by Brown et al. (2019), we approximate human preferences through an automated ranking process using the episodic rewards provided by the dataset. ... While we argue that the rankings generated through this process would resemble human preferences, the exploration using humans within more accessible environments is left for future research."
The preference labels are not independent human judgments; they are defined as the order of episodic ground-truth reward. Eq. (1) fits r_hat so that sums of learned rewards preserve the ranking tau_i ≺ tau_j, so the learned reward function is, by construction, a fitted approximation of the ground-truth reward's ordinal content. The later claim that TROFI 'performs comparably to using the ground truth reward' (Section 3.3, Table 1) is therefore not an independent test of reward-free learning: the input labels already encode the reward the method claims to avoid. Since Appendix E's 'human-generated ranking' is only random noise applied to the GT-reward ordering, no experiment tests systematic divergence between human preferences and reward returns.
full rationale
TROFI's algorithmic pipeline (T-REX reward fitting on ranked trajectories, then TD3+BC on the relabeled dataset) is not internally circular: Eq. (1) is a standard preference-based reward-fitting loss and Eq. (2) is standard TD3+BC. No uniqueness theorem or load-bearing self-citation is used. The circularity is in the evaluation of the central claim. Section 3.1 defines the preference ranking as an automated ranking by the dataset's episodic rewards, i.e. the ground-truth reward the method claims to avoid. T-REX then fits r_hat to reproduce exactly that ordering, so the learned reward is a fitted surrogate of the GT reward. Consequently, Table 1's finding that TROFI 'performs comparably to using the ground truth reward' is largely a consistency check on fitting the GT reward, not an independent demonstration of reward-free learning. The paper's own limitation statement in Section 3.1 ('the exploration using humans within more accessible environments is left for future research') and Appendix E's noise simulation, which starts from GT-reward rankings and swaps 20%, do not test systematic human-vs-reward preference divergence. The method itself may work with genuine human rankings, but that claim is not established by the reported experiments; the demonstrated result reduces to fitting the reward that the headline says is unavailable.
Assumptions & free parameters
free parameters (3)
- Per-task ranked-trajectory budget |M| as a percentage of D =
5%, 10%, 50%, or 100%, selected per task
- Reward transformation scaling factor in Section 3.4 =
Not reported; selected via preliminary experiments
- T-REX partial-trajectory length and number of subtrajectories =
Not reported
assumptions (4)
- ad hoc to paper Rankings built from ground-truth episodic return approximate human preference rankings.
- domain assumption D4RL normalized scores and datasets are an acceptable stand-in for game development offline datasets.
- domain assumption T-REX can extrapolate a usable dense reward from partial ranked trajectories.
- standard math Suitable affine transformations of a dense reward preserve the optimal policy (Ng et al., 1999).
Cite this review
Pith. "Pith review of TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learning." pith.science (2026). https://pith.science/paper/2APNLYBI
@misc{pith2026250622008,
author = {Pith},
title = {Pith review of: TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/2APNLYBI}},
note = {Machine review of arXiv:2506.22008}
}
read the original abstract
In offline reinforcement learning, agents are trained using only a fixed set of stored transitions derived from a source policy. However, this requires that the dataset be labeled by a reward function. In applied settings such as video game development, the availability of the reward function is not always guaranteed. This paper proposes Trajectory-Ranked OFfline Inverse reinforcement learning (TROFI), a novel approach to effectively learn a policy offline without a pre-defined reward function. TROFI first learns a reward function from human preferences, which it then uses to label the original dataset making it usable for training the policy. In contrast to other approaches, our method does not require optimal trajectories. Through experiments on the D4RL benchmark we demonstrate that TROFI consistently outperforms baselines and performs comparably to using the ground truth reward to learn policies. Additionally, we validate the efficacy of our method in a 3D game environment. Our studies of the reward model highlight the importance of the reward function in this setting: we show that to ensure the alignment of a value function to the actual future discounted reward, it is fundamental to have a well-engineered and easy-to-learn reward function.
Figures
Reference graph
Works this paper leans on
-
[2]
It is an open-world city simulation originally proposed by Sestini et al
Algorithm 1 Training with TROFI Input: unlabeled data D, number of sub-trajectories N, sub-trajectory length L Output: a trained policy π and a reward model ˆrθ π ← initialize policy ˆrθ ← initialize reward model Normalize si ∈ D, i = 0, ..,|D| Sample M ∼ D Rank trajectories in M, from best to worst while not converged do ▷ Training with T -REX Sample tra...
work page 2023
-
[5]
Semi-supervised reward learning for offline reinforcement learning
Reinforcement Learning and Video Games Workshop 2025 Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando de Freitas. Semi-supervised reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06899,
work page Pith review arXiv 2025
-
[7]
URL http://example.com/. Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating hu- man behaviour with diffusion models. arXiv preprint arXiv:2301.10677,
-
[8]
Advantage-weighted regression: Simple and scalable off-policy reinforcement learning
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177,
arXiv 1910
-
[11]
Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensembles
Fangzhou Yu, Ryan Batke, Jeremy Dao, Jonathan Hurst, Kevin Green, and Alan Fern. Dynamic bipedal turning through sim-to-real reinforcement learning. In International Conference on Humanoid Robots (IROS), 2022a. Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement ...
-
[12]
Offline learning from demonstrations and unlabeled experience
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011.13885,
arXiv 2011
-
[13]
Offline Reinforcement Learning
Reinforcement Learning and Video Games Workshop 2025 Appendix A Related Work Here we review work from the recent literature most relevant to our contributions. Offline Reinforcement Learning. In ORL settings, also referred to as batch RL, we optimize a policy without relying on interactions with the environment but rather using a fixed dataset de- rived f...
work page 2025
-
[14]
As we describe in Section 3, it is especially suitable for offline settings
is a preference- based IRL approach capable of learning a reward function from sub-optimal data. As we describe in Section 3, it is especially suitable for offline settings. Other recent notable approaches train a policy offline with just a few labeled trajectories. However, in our work we assume we have no pre-labeled trajectories (Li et al., 2023a; Hu e...
work page 2023
Show all 19 references
-
[15]
All of these approaches require optimal expert demonstrations – that is, demonstrations generated by an optimal policy
that learns a reward function prior to agent optimization. All of these approaches require optimal expert demonstrations – that is, demonstrations generated by an optimal policy. Other methods, such as Soft Q Imitation Learning (SQIL) (Konyushkova et al., 2020), achieves simil...
2020
-
[17]
An episode is marked a success if the agent reaches the goal before the timeout
In this environment, an episode consists of a maximum of 1000 steps. An episode is marked a success if the agent reaches the goal before the timeout. The environment is particularly meaningful for this study because it uses standard state- and action-spaces for developing RL a...
2022
-
[18]
We provide more details about the baselines in Section 4.2 of the main paper
and ORIL (Zolna et al., 2020). We provide more details about the baselines in Section 4.2 of the main paper. We test our approach in two different set of tasks: MuJoCo and Adroit. TROFI outperforms both baselines and state-of-the-art algorithms for the MuJoCo tasks, while for ...
2020
-
[19]
we report the results from the original paper (which do not include expert tasks). Reinforcement Learning and Video Games Workshop 2025 Dataset IQL Fisher-BRC TD3+BC (ours) hopper-medium-v2 20.7 ± 4.5 89.7± 1.2 80.3± 1.2 halfcheetah-medium-v2 42.7 ± 0.1 43.3 ± 0.3 55.1± 0.3 wa...
2025
-
[1995]
Efficient active imitation learning with random network distillation
Emilien Biré, Anthony Kobanda, Ludovic Denoyer, and Rémy Portelas. Efficient active imitation learning with random network distillation. arXiv preprint arXiv:2411.01894,
-
[2016]
The provable benefits of unsuper- vised data sharing for offline reinforcement learning
Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang. The provable benefits of unsuper- vised data sharing for offline reinforcement learning. arXiv preprint arXiv:2302.13493,
-
[2018]
Technical challenges of deploying reinforcement learning agents for game testing in aaa games
Jonas Gillberg, Joakim Bergdahl, Alessandro Sestini, Andrew Eakins, and Linus Gisslén. Technical challenges of deploying reinforcement learning agents for game testing in aaa games. In 2023 IEEE Conference on Games (CoG), pp. 1–8. IEEE,
2023
-
[2019]
Demonstration-efficient inverse reinforcement learning in procedurally generated environments
Alessandro Sestini, Alexander Kuhnle, and Andrew D Bagdanov. Demonstration-efficient inverse reinforcement learning in procedurally generated environments. In 2021 IEEE Conference on Games (CoG),
2021
-
[2020]
Offline reinforcement learning with fisher divergence critic regularization
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning (ICML), 2021a. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement lear...
-
[2022]
Towards informed design and validation assistance in computer games using imitation learn- ing
Alessandro Sestini, Joakim Bergdahl, Konrad Tollmar, Andrew D Bagdanov, and Linus Gisslén. Towards informed design and validation assistance in computer games using imitation learn- ing. In 2023 IEEE Conference on Games (CoG), pp. 1–8. IEEE,
2023
-
[2023]
it’ s unwieldy and it takes a lot of time
Mikhail Jacob, Sam Devlin, and Katja Hofmann. “it’ s unwieldy and it takes a lot of time”—challenges and opportunities for creating agents in commercial games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2020a. Mikhai...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.