{"id":"e82e038f-3b7f-4d5f-9247-69a8db0f8323","arxiv_id":"2602.15206","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MAVRL trains one reward model from demonstrations, comparisons, ratings, and stops using a single evidence lower bound, and shows in simulation that combining types improves reward recovery and robustness.","lead":"MAVRL is a method that learns a single reward function from several kinds of human feedback — demonstrations, pairwise comparisons, ratings, and stop signals — by treating each as a probabilistic observation in one variational inference objective. The paper reports that combining feedback types improves reward recovery and policy robustness in simulated grid-world and continuous-control tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic-feedback validation is circular: MAVRL is tested only on data generated by the same likelihood models it uses (A.2), so joint-inference gains may reflect likelihood self-consistency, not robust complementarity under realistic feedback.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the synthetic feedback is generated from the same probability models MAVRL uses as likelihoods, making the empirical validation circular. This is the most serious threat to the central claim because all three headline outcomes—improved reward recovery, policy performance, and robustness—are demonstrated only under this closed loop. If the likelihoods are misspecified for real human feedback, the claimed complementarity may not transfer, and could even become harmful if modalities inject conflicting biases. Other issues, such as the absence of existing multi-type baselines and test-performance hyperparameter selection, are secondary: they weaken the quantitative strength of the comparison but do not directly attack the existence of complementarity. I also considered whether the auxiliary Q-function is genuinely Q*_R: the TD constraint in Eq. (4) uses the actual next action rather than a max, so it enforces a behavior-policy Bellman consistency, not optimality; this is an internal approximation worth examining, but it is less central than the synthetic-data circularity because the empirical results would still need to survive misspecification testing. Given the paper's own Limitations section concedes the synthetic-only evidence, the appropriate verdict remains CONDITIONAL rather than ACCEPT or REJECT: the framework is plausible, but the central empirical claim requires additional validation on data or generative models not identical to the assumed likelihoods.","tokens_in":27540,"tokens_out":6281,"duration_ms":69411,"concrete_test":"Run a misspecification stress test on the same environments and budgets: generate feedback from held-out generative models different from the likelihoods in Section 4—e.g., ratings via ordinal probit with random cutpoints, stops via a fixed cumulative-regret threshold, preferences via Bradley-Terry with heteroscedastic noise or epsilon-flipped labels—while keeping MAVRL's likelihoods unchanged. Then compare PDR against the best single-type baseline using the same normalized returns and EPIC metrics as in Tables 1–2. If the multi-type combination no longer consistently outperforms or is hurt by the misspecified modalities, the central complementarity claim is not robust to realistic feedback deviations; if it remains best, the concern is largely mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that jointly inferring rewards from multiple feedback types improves reward recovery, policy performance, and robustness—rests entirely on experiments whose feedback is generated by the same probabilistic models that MAVRL uses as likelihoods. Appendix A.2 states that preferences are sampled from Bradley-Terry with logistic sigmoid, ratings are assigned via quantile-based cutpoints of normalized returns, stops are simulated from a discrete-time cumulative-regret hazard model, and demonstrations are Boltzmann-rational rollouts; these are exactly the likelihoods defined in Section 4. Thus Tables 1–2 and the robustness figures validate that MAVRL can invert its own generative process, not that the likelihoods match human feedback or that the observed complementarity transfers to real settings. If human ratings are not ordered-logistic, if stops are triggered by a fixed threshold rather than an exponential hazard, or if preferences contain systematic label noise, the joint likelihoods are misspecified and combining feedback types can in principle inject bias rather than complementary signal. The paper acknowledges this in Section 7, but the conclusion still states the transfer as if established. The absence of any comparison to existing multi-type integration methods (e.g., Mehta and Losey 2024) further means even the closed-loop result does not establish that joint Bayesian inference is better than heuristic integration. This is a validation gap, not a proof-of-concept failure: the algorithmic framework could still be correct, but the central empirical claim is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAVRL, a method that frames reward learning from multiple feedback types (preferences, demonstrations, ratings, and stops) as Bayesian inference over a shared latent reward, combining feedback-specific likelihood models into a single amortized variational ELBO. The authors claim that joint inference exploits complementary information across feedback types, improving reward recovery, downstream policy performance, and robustness to environment perturbations. Experiments on grid-world and continuous-control benchmarks compare single-type and multi-type feedback configurations, with all feedback generated synthetically from the same likelihood models used by the method.","tokens_in":1392,"tokens_out":1534,"duration_ms":76012,"significance":"The methodological contribution is well-motivated and principled: replacing heuristic loss weighting with a unified ELBO is a natural extension of AVRIL, and the framework is extensible to new feedback types. The paper provides code and thorough ablation-style tables across multiple environments. However, the empirical validation is entirely internal to the model's own generative assumptions, and the reported results are not compared against existing multi-type integration approaches. If the claims were supported by external feedback or misspecification-robust experiments, the framework would be a valuable contribution to reward learning; in its current form, the central claims outrun the evidence.","major_comments":[{"comment":"The evaluation is circular with respect to the likelihood models. Preferences, demonstrations, ratings, and stops are simulated using exactly the likelihood models defined in Section 4 (Bradley-Terry, Boltzmann-rational, quantile-cutpoint ordered logit, cumulative-regret hazard). Thus Tables 1-3 and Figure 3 show that MAVRL can invert its own generative process, not that it combines realistic human feedback. The conclusion in Section 7 that joint inference 'improves reward recovery, downstream policy performance, and robustness' is not supported by this in-sample validation. Please add experiments with human feedback or with misspecified generators (e.g., threshold-based stops, non-logistic ratings, label noise) to show the benefits survive model mismatch.","section":"Section 6, Appendix A.2"},{"comment":"The hyperparameters λ_KL and λ_TD are selected per environment from {0.5, 1.0} using 'the same tuning procedure applied uniformly to all methods.' If this selection is based on the evaluation metric (normalized return or EPIC distance) on the test environments, the reported numbers are optimistically biased. The protocol does not specify a validation split. This is load-bearing because all comparisons in Tables 1-2 and Figure 3 depend on these hyperparameters. Please state whether a validation set was used; if not, report results with fixed default hyperparameters.","section":"Appendix A.1"},{"comment":"The claim that PDR 'achiev[es] either the best performance or results within 1% of the best performance ... in five out of six environments, with the exception of CartPole-v1' is contradicted by the table. In grid_sparse, PDR (70.0) is 6.7% below the best (75.0); in LunarLander-v3, PDR (80.4) is far below demonstrations alone (115.6); in CartPole-v1 PDR (87.5) is below the best (97.7). Thus the 'strongest overall performance' conclusion is overstated. The analysis should acknowledge these exceptions and discuss when combining feedback types helps rather than hurts.","section":"Section 6.2, Table 1"},{"comment":"No comparison to existing multi-type feedback integration methods is provided. Baselines are single-type instances of MAVRL (e.g., demonstrations only, ratings only), but the paper's contribution is about combining feedback types. Without a baseline such as additive loss weighting (Mehta and Losey 2024), post-hoc ensemble of single-type models (Metz et al. 2025), or the Bıyık et al. (2022) demonstration+preference method, the experiments do not establish that joint Bayesian inference is superior to heuristic integration. Please add at least one existing multi-type method as a comparison.","section":"Section 6"}],"minor_comments":[{"comment":"In the stop likelihood, the hazard function is defined as h_{λ,ρ}^R(ξ, τ), but the product in Eq. (3) uses h_λ^R(ξ, t) with ρ omitted. Align the notation.","section":"Equation (3)"},{"comment":"Figure 2 says '2 demonstrations', while Section 6 states nd=1 for grid worlds. Clarify the budget used in the figure.","section":"Figure 2"},{"comment":"The claim of 'no manual loss balancing' is misleading because Eq. (4) includes λ_KL and λ_TD, which are manually tuned. Rephrase to say there is no manual balancing among feedback types.","section":"Section 5.2"},{"comment":"The column headers in the supplementary tables are inconsistent and hard to parse (e.g., repeated 'D' and 'PR'). Use explicit labels such as D, R, P, S, D+R, etc.","section":"Table 3"},{"comment":"In the preferences likelihood, 'exp βR(ξ1)' should be 'exp(β R(ξ1))' for readability.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and likely of interest to the reward-learning community, but the empirical evidence as presented does not justify the strong claims. The circular feedback generation and the absence of multi-type baselines are the main gaps. A revision that adds human or misspecified feedback experiments and compares against existing integration methods would substantially strengthen the paper. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MAVRL is a legitimate extension of AVRIL that swaps the single demonstration likelihood for a set of feedback-specific likelihoods inside one variational objective. The rating and stop likelihoods are the genuinely new pieces. But read the experiments with caution: every piece of synthetic feedback is generated from exactly the likelihood models the method trains with (Appendix A.2), so the complementarity and robustness results are in-sample self-consistency, not evidence about real human feedback. The paper admits this in Section 7, yet the abstract and conclusion still state the transfer claim as established.\n\nWhat is well done: The formulation is clean. A shared reward encoder plus a collection of likelihood decoders, trained by maximizing a single ELBO, is a natural and sensible unification, and the TD-regularization from AVRIL carries over smoothly. The rating likelihood uses ordered-logit cutpoints, which is appropriate for Likert-scale data, and the stop likelihood as a discrete-time hazard over discounted regret is a reasonable first model for a genuinely under-studied feedback channel. The experimental sweep is broad: multiple grid worlds and continuous-control tasks, single-type ablations, EPIC distances, robustness plots, standard errors, and the code is supplied. That is solid, reproducible work by the standards of this area.\n\nThe soft spots are in the empirical claims, not the algorithm. First, circularity: because the simulator uses the same Bradley-Terry, ordered-logit, Boltzmann-rational, and hazard models, Tables 1 and 2 mostly show that MAVRL can invert its own generative process. If human ratings are not ordinal-logistic or stops follow a fixed threshold, the gains may shrink or become negative. That gap is addressable — run a mismatched-generator sensitivity study or a small human study — but as it stands the paper's central conclusion is under-supported. Second, there is no comparison to any existing multi-type integration method, including the additive loss-weighting approach of Mehta and Losey (2024), which is cited and closest. Without that baseline, we cannot tell whether joint Bayesian inference actually beats heuristic fusion. Third, the claim of no manual loss balancing is diluted by the fact that lambda_KL and lambda_TD are tuned per environment on the evaluation metric; this is common, but should be acknowledged. Fourth, several combinations underperform their single-type constituents (e.g., CartPole), so \"complementary strengths\" is a tendency, not a guarantee.\n\nVerdict: worth a serious referee. The algorithmic contribution is real and the paper is honestly written. I'd send it to review with the request that the authors add a mismatched-model sensitivity check and at least one heuristic-fusion baseline before publication. For a reading group on reward learning, yes, bring it.","headline":"A credible, clean extension of AVRIL to multiple feedback types, but the empirical center is an in-sample self-consistency check: all synthetic feedback is generated from the same likelihood models the method learns with.","tokens_in":28389,"tokens_out":3060,"would_cite":true,"duration_ms":33239,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MAVRL claims that learning a reward function from demonstrations, comparisons, ratings, and stops as a single Bayesian posterior is feasible, and that the combined posterior beats any single feedback type in policy return, reward recovery,","keywords":["reward learning","multi-type feedback","amortized variational inference","Bayesian inverse reinforcement learning","pairwise comparisons","ordinal regression","hazard model","reward robustness"],"falsifier":"Run MAVRL on feedback whose generative model is deliberately mismatched to the assumed likelihoods—for example, ratings generated by context-dependent thresholds or stops triggered by a fixed safety rule rather than discounted regret—and compare reward recovery and policy return against single-type baselines; the central claim weakens if the matched-model synthetic gains disappear under mismatched generators. A complementary check is to collect a modest amount of real human feedback in one of the grid environments and test whether combined feedback still outperforms the best single type.","tokens_in":27475,"feed_emoji":"🤖","tokens_out":6330,"duration_ms":63194,"temperature":0.7,"pith_summary":"This paper tries to establish that reward functions for reinforcement-learning agents can be inferred jointly from several qualitatively different kinds of human feedback—demonstrations, pairwise comparisons, ratings, and emergency stops—by treating each feedback type as a probabilistic observation of one shared latent reward, rather than training separate models and merging them. The proposed method, MAVRL, learns a single variational reward encoder plus feedback-specific likelihood decoders and optimizes one evidence lower bound, so no manual loss balancing is needed. On grid-world and continuous-control benchmarks, jointly inferred reward posteriors are reported to outperform single-type baselines in downstream policy return and reward recovery, and to give policies that degrade more gracefully under environment perturbations. A sympathetic reader would care because human feedback in practice is heterogeneous, and current practice either uses one modality or combines modalities with heuristics; if this Bayesian formulation works, it provides a principled way to pool all available feedback and to read model uncertainty as a signal of which parts of the reward are well constrained.","feed_headline":"Combining four feedback types outperforms single-type rewards","feed_subtitle":"Demonstrations, comparisons, ratings, and stops: one objective combines them without manual weights.","key_machinery":"The central object is a shared variational reward encoder qθ(R|s,a,s′) modeled as a Gaussian whose mean and variance are neural-network functions of state, action, and next state; sampling from it induces distributions over trajectory returns. Alongside it, a Q-value network supplies action values, and a temporal-difference regularization term enforces that encoder rewards match one-step Bellman differences. Each feedback type contributes an explicit likelihood decoder—softmax-rational demonstrations, logistic pairwise comparisons, ordered-logit ratings, and a discounted-regret hazard model for stops—so the whole system is trained by maximizing one evidence lower bound with a KL regularizer.","core_discovery":"MAVRL claims that the shared-latent Bayesian formulation is both tractable and practically effective: by optimizing a single evidence lower bound whose per-modality likelihood terms are a softmax-rational demonstration model, a logistic pairwise-comparison model, an ordered-logit rating model, and a cumulative-regret hazard model for stops, a single reward posterior can exploit complementary information across feedback types. Concretely, the paper reports that combining all four types yields strong normalized returns in five of six environments, best or second-best reward-recovery distance (a shaping-invariant measure) in two of three tabular environments, and more robust performance under d","pith_inferences":["Editorial inference: because the likelihoods are explicit, the same posterior confidence map could drive active data collection; a direct extension would query whichever feedback type is expected to reduce reward uncertainty most.","Editorial inference: if a feedback type in real deployment consistently lowers the evidence lower bound, that is a diagnostic signal that its assumed likelihood is misspecified, which the paper does not explore.","Editorial inference: the observed per-modality uncertainty signatures suggest a practical collection heuristic—use stops and ratings to anchor reward extremes, comparisons to cover the state space, and demonstrations to pin down trajectories—though the paper does not test this protocol."],"forward_implications":["Practitioners can add a new feedback type by defining one likelihood decoder, without retraining the encoder or manually reweighting losses.","Combined-feedback reward models should produce policies that keep more return under dynamics perturbations than policies trained on any single feedback type, as reported across the three perturbation scenarios.","The reward posterior's variance offers an interpretable map of where the reward is constrained—demonstrations pin down trajectories, ratings locate goals, stops mark unsafe regions, comparisons fill the middle ground—usable for inspecting confidence and consistency.","In sparse-reward settings, added modalities can sometimes add noise: the paper reports a LunarLander-v3 case where standalone demonstrations outperform the full combination, so combining all available feedback is not always optimal.","Because the objective factorizes over feedback types, training can proceed asynchronously and in any order, with no staged curriculum."],"fun_headline_variants":["Single ELBO fuses four feedback types into one reward","Bayesian fusion of four feedback types beats single-type baselines","No manual loss weights: one ELBO fuses four feedback types","Unified reward learning from four feedback types achieves better robustness","Joint reward posteriors from four feedback types outperform single-type"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central assumption is that human feedback of each type is generated by the specific probabilistic models the paper uses as likelihoods (softmax-rational demonstrations, logistic pairwise comparisons, ordered-logit ratings, and discounted-regret hazard stops); if real humans deviate from these models, the claimed complementarity and robustness benefits may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Single ELBO fuses four feedback types into one reward","Bayesian fusion of four feedback types beats single-type baselines","No manual loss weights: one ELBO fuses four feedback types","Unified reward learning from four feedback types achieves better robustness","Joint reward posteriors from four feedback types outperform single-type"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001235,"raw_usage":{"total_tokens":4878,"prompt_tokens":682,"completion_tokens":4196,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":4122}},"tokens_in":426,"tokens_out":4196,"duration_ms":27163,"temperature":1.0,"reasoning_tokens":4122,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:54:00.986449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MAVRL on feedback whose generative model is deliberately mismatched to the assumed likelihoods—for example, ratings generated by context-dependent thresholds or stops triggered by a fixed safety rule rather than discounted regret—and compare reward recovery and policy return against single-type baselines; the central claim weakens if the matched-model synthetic gains disappear under mismatched generators. A complementary check is to collect a modest amount of real human feedback in one of the grid environments and test whether combined feedback still outperforms the best single type.","supporting_citations":[],"review_version":1}