{"id":"c533b5dd-93c6-44d1-aa3e-c4e59b85d8d2","arxiv_id":"2602.10635","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HARPO reweights per-sample and per-task advantages in GRPO to balance multitask learning, yielding a 7B model with the best average rank across 10 social-behavior tasks and top zero-shot scores on two held-out benchmarks.","lead":"A new reinforcement-learning method, HARPO, reweights the influence of each sample and task during training so that no single social-behavior task dominates the update, and the resulting 7B model reports top scores on most of 10 benchmark tasks. The paper is relevant for anyone building multi-task AI that must handle heterogeneous data and prediction targets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HARPO omits KL regularization while GRPO baseline retains it (App. C.1), confounding the attribution of gains to the advantage-modulation mechanism.","rationale":"The reader's verdict identifies the proxy assumption in Eqs. (6)-(7) as the weakest point, but the more decisive issue is an experimental control flaw. Appendix C.1 explicitly says HARPO omits KL regularization while baselines reuse their original configurations. Since GRPO's original configuration includes a KL penalty, the main comparison in Table 1 conflates the effect of HARPO's advantage modulation with the removal of KL regularization. This directly undermines the strongest_claim that the advantage-modulation mechanism drives performance improvements. The proposed concrete test—running GRPO with the same no-KL settings—would resolve the ambiguity. The reader did mention 'potentially unfair comparison to untuned baselines' in their rationale, so there is partial agreement, but the specific KL mismatch is not isolated in the reader's weakest_assumption. The paper has real strengths: a clear motivation, a well-specified algorithm, and a reasonable ablation design within HARPO. However, the central attribution claim is not yet supported by cleanly controlled comparisons, and the internal inconsistency between Eq. (12) and App. C.1 must be resolved. Hence the verdict remains CONDITIONAL: the authors should run the missing control and report all KL coefficients.","tokens_in":23571,"tokens_out":5833,"duration_ms":55191,"concrete_test":"Run GRPO on the same Human Behavior Atlas benchmark, same Qwen2.5-Omni-7B base, same reward design, batch size 256, rollouts 5, learning rate 1e-6, AdamW, early stopping, and—critically—with the same omitted KL regularization (beta=0) exactly matching HARPO's C.1 settings. Compare task-level results in Table 1. If GRPO-noKL attains average rank and per-task scores comparable to HARPO (e.g., SAR >60), then the advantage-modulation mechanism is not necessary for the reported gains. Conversely, run HARPO with the GRPO default KL coefficient (e.g., beta from DeepSeekMath) and check whether its advantage over GRPO persists. Also report the KL coefficient actually used for each baseline in the final paper.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparison in Table 1 is confounded. Appendix C.1 states that HARPO 'omit[s] explicit KL regularization' and justifies this by HARPO's internal modulation, while all RL baselines 'reuse the hyperparameter configurations reported in the original papers.' The original GRPO (DeepSeekMath) includes a KL divergence penalty (beta > 0 in Eq. 14). Therefore the HARPO-vs-GRPO comparison changes two variables simultaneously: the advantage-modulation mechanism and the presence/weight of KL regularization. The reported gains—up to +42.29% on SAR—could be caused entirely by removing the KL penalty, not by heterogeneity-aware reweighting. The ablations in Table 3 are all within HARPO's no-KL setting, so they cannot isolate the modulation contribution; a 'GRPO with the same no-KL settings' cell is missing. The internal inconsistency between Eq. (12), which displays a beta-KL term, and C.1's statement that KL is omitted needs clarification. This is more load-bearing than the proxy-accuracy concern about Eqs. (6)-(7), because even a perfect contribution proxy cannot support the central claim if the baseline itself is not controlled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HARPO, an extension of GRPO for multitask behavioral training. HARPO computes sample- and task-level contribution signals as average absolute group-normalized advantages, converts them into inverse-geometric-mean modulation factors, and applies inertial smoothing before the PPO-style clipped update. The authors train Qwen2.5-Omni-7B on the 10-task Human Behavior Atlas benchmark, reporting that the resulting model, OmniSapiens-7B 2.0, achieves the best average rank across existing models and held-out AUT/SER transfer, and that HARPO outperforms GRPO, RLOO, RE++, and GPG with the same reward design and base model. The paper also includes ablations of the modulation components and qualitative reasoning-trace analyses.","tokens_in":23888,"tokens_out":5343,"duration_ms":57379,"significance":"If the attribution is supported, HARPO is a useful and generally applicable contribution to critic-free multitask reasoning RL: it gives a concrete recipe for reweighting advantages at two levels, provides an algorithm block (Algorithm 1), includes ablations of each design choice, and evaluates on an external held-out task (AUT, SER) in addition to the in-distribution benchmark. The paper is also transparent about experimental details and plans to release code and reasoning traces. However, the central causal claim — that the advantage-modulation mechanism, rather than the accompanying removal of KL regularization or other hyperparameter changes, drives the gains — is not currently established by the reported experiments. The main comparison against GRPO is confounded, and the absence of multiple seeds leaves rank differences of the size used to support the ablations within the range of training noise.","major_comments":[{"comment":"The headline HARPO-vs-GRPO comparison is confounded by KL regularization. Eq. (12) displays a KL penalty term, but App. C.1 states that HARPO 'omit[s] explicit KL regularization' because its modulation acts as regularization, while all RL baselines 'reuse the hyperparameter configurations reported in the original papers' — which, for GRPO, includes the KL penalty. Thus the comparison changes two variables at once, and the reported gains (e.g., +42.29% on SAR in Table 1) could arise from removing the KL term alone. The ablations in Table 3 are all within HARPO's no-KL setting, so they cannot isolate the modulation mechanism. The authors should add a GRPO run with KL removed under the same reward and early-stopping settings, and ideally a HARPO run with KL enabled, before claiming that heterogeneity-aware advantage modulation is responsible for the improvement. They should also reconcile E","section":"§3.2, Eq. (12); App. C.1; Table 1"},{"comment":"No error bars, multiple seeds, or significance tests are reported for any training run. On-policy RL with only 5 rollouts per sample is high-variance, and the rank differences used to support the method and its ablations are small in several cases (e.g., HARPO 1.90 vs. w/o structured modulation 2.00 in Table 3; HARPO 1.70 vs. RLOO/GPG 2.50 in Table 1). A single seed is insufficient to distinguish the proposed mechanism from noise. The authors should report mean ± std over at least three seeds for HARPO, for GRPO, and for the ablations, and should state whether differences are statistically reliable. Additionally, the hyperparameter policy is asymmetric: baselines use their original-paper hyperparameters while HARPO uses beta_rho = beta_s = 0.95 and no KL; a fair comparison should also tune or at least vary the baselines' KL weights.","section":"§4.2–§4.5, Tables 1–3"},{"comment":"The load-bearing proxy assumption is not directly validated. The paper equates a sample's or task's contribution to the policy update with the average absolute group-normalized advantage (Eqs. (6)–(7)). The actual per-rollout gradient is A * grad log pi (Eq. (5)), whose norm varies across tokens and sequence lengths, so the average |A| may be a poor ranking of true gradient contributions. Figure 4 shows a correlation between relative task advantage and reward for one task, but it does not test whether the proxy ranks samples correctly. I request a diagnostic: on a small training subset, compute the actual gradient-norm contribution per sample/task and compare it with p(t), or replace p(t) with a fixed/random/ordinal signal in the algorithm. Without such evidence, the reported gains could come from the geometric centering, the inertial smoothing, or the no-KL setting rather than from vali","section":"§3.2, Eqs. (6)–(7) and Eq. (5); §4.5"}],"minor_comments":[{"comment":"The abstract reports gains of up to +12.02% and +9.37%, while the introduction reports +16.85% and +9.37% for multitask and held-out settings. Please clarify which number corresponds to which comparison and make the abstract consistent.","section":"Abstract vs. §1"},{"comment":"There are several typographical issues: 'balances leaning' in §1, 'simulateneously' in App. D.2, and 'of of training' in §4.5. Please proofread.","section":"Throughout"},{"comment":"Several numeric entries run together, e.g., '98.3990.68' and '64.7030.40'. The table also lists HARPO under both Models and Training Algorithms, which is understandable but should be labelled clearly to avoid the appearance of duplicated rows.","section":"Table 1"},{"comment":"In §4.5, 'App. E.1 Fig. 5' should presumably refer to the main-text Fig. 5 or to the appropriate appendix figure; several appendix references are ambiguous because the main text already has a Fig. 5.","section":"Figure/Section cross-references"}],"recommendation":"major_revision","confidential_remarks":"The benchmark used for training and in-distribution evaluation is from the same group (Ong et al., 2026), which is not circular because the comparison is against external baselines and held-out datasets. The main blocker is the experimental control issue: the KL confound must be fixed before the central attribution claim can be accepted. If the authors add the missing GRPO-no-KL control and at least basic variance reporting, I would be willing to reconsider."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the idea is worth engaging with, but the headline result is not yet supported. HARPO's per-sample and per-task reweighting of GRPO advantages is a legitimate extension of difficulty-aware RL, and the paper does the right thing by training all algorithms on the same benchmark with the same reward design. The held-out AUT and SER results are a nice addition. But the HARPO-vs-GRPO comparison changes two things at once: Appendix C.1 says HARPO omits explicit KL regularization while GRPO baselines reuse their original configurations, which include the KL penalty. That means the +42% on SAR and the rank improvements could come from removing KL, not from the modulation mechanism. The ablations in Table 3 are all inside HARPO's no-KL setting, so they cannot isolate the contribution. There is also an internal inconsistency: Eq. (12) still displays a beta-KL term while C.1 says it is omitted.\n\nOn the evidence side: no error bars or multiple seeds anywhere, and baselines use their original hyperparameters while HARPO's hyperparameters (beta_rho, beta_s, reward weights) are effectively chosen on the benchmark. That is a mild double standard, and it means the significance of the Table 1 differences is unclear. The proxy assumption behind Eqs. (6)-(7) — that average absolute group-normalized advantage approximates gradient contribution — is a reasonable first cut, but it is an assumption; the KL confound matters more.\n\nWhat is genuinely good: HARPO is a new combination of geometric-mean-centered, inertially smoothed advantage modulation, and the empirical study, while flawed in comparison, covers a broad set of behavioral tasks and includes held-out generalization. The model is released. The writing is clear about most design choices. The benchmark being from the same group is a small concern, mitigated by external held-out sets and baseline comparisons.\n\nBottom line: this deserves a serious referee, not a desk reject. A careful reviewer should ask for a no-KL GRPO control, multiple seeds with variance, and a fix to the Eq. (12)/C.1 inconsistency. I would not cite the results until those are resolved, but I would read the revision.","headline":"HARPO's advantage modulation is a sensible idea, but the comparison against GRPO is confounded because HARPO drops the KL term, and the paper needs a no-KL GRPO control, seeds, and variance before the central claim holds.","tokens_in":24393,"tokens_out":3031,"would_cite":false,"duration_ms":31667,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OmniSapiens-7B 2.0, trained with HARPO—a reinforcement-learning method that modulates policy-update advantages to balance heterogeneous tasks and samples—achieves the strongest multitask and held-out performance among compared behavioral mo","keywords":["HARPO","social behavior processing","multitask reinforcement learning","advantage modulation","heterogeneous learning signals","behavioral foundation model","reasoning traces","zero-shot generalization"],"falsifier":"Train HARPO and a control version that uses the same smoothing and geometric centering but with randomly assigned per-sample and per-task modulation weights; if the control matches HARPO's average rank across tasks, the contribution-signal mechanism is not the driver. A more direct check: during training, measure the correlation between the contribution signal and the actual per-sample gradient norm; a near-zero correlation would falsify the proxy on which the method is built.","tokens_in":23483,"feed_emoji":"🧠","tokens_out":10656,"duration_ms":98692,"temperature":0.7,"pith_summary":"This paper tries to establish that a single model can be trained across many diverse social-behavior tasks without letting any one task or sample dominate the learning, and that doing so yields better accuracy and generalization than current practice. To this end it introduces HARPO (Heterogeneity-Aware Relative Policy Optimization), a critic-free reinforcement-learning method that estimates each sample's and task's contribution to the policy update and reweights the associated reward-based advantages—using a geometric-mean reference and inertial smoothing—so that strong signals are damped and weak ones amplified. The trained 7-billion-parameter model, OmniSapiens-7B 2.0, outperforms previous unified behavior models on 9 of 10 benchmark tasks and on held-out emotion and behavior-recognition tests, with reported gains up to +16.85% on multitask settings and +9.37% on held-out settings, while producing more explicit, evidence-grounded reasoning traces. If true, this offers a general recipe for multitask learning under heterogeneous data, relevant beyond social behavior to any domain where reward scales and task difficulties differ sharply.","feed_headline":"New RL method rebalances learning and wins 9 of 10 behavioral tasks","feed_subtitle":"The same reweighting that stabilizes weak tasks also boosts zero-shot transfer to new social domains.","key_machinery":"The central object is the HARPO-modulated advantage, formed by multiplying the group-normalized advantage of each sampled response by a sample-level and a task-level modulation factor. Each factor is the reciprocal of a contribution signal (average absolute group-normalized advantage of a sample or task) divided by a geometric-mean reference; this construction guarantees the factors have geometric mean 1, so the overall scale of the policy update is unchanged. The ratio-based design tempers contribution signals that can vary by orders of magnitude, and the inertial control—exponential moving average on signals, multiplicative updates on factors—lets the reweighting evolve more slowly than th","core_discovery":"The central claim is that HARPO rebalances the gradient contributions of heterogeneous behavioral tasks by modulating group-normalized reward advantages before the policy update. For every sample and every task, the method computes a contribution signal—the average absolute group-normalized advantage across that sample's or task's rollouts—and rescales advantages by the reciprocal of the signal's ratio to a geometric-mean reference (one reference within each task, one across tasks). Because the modulation factors are constructed as reciprocals of ratios to their geometric mean, they multiply to 1, so upscaling some advantages is exactly offset by downscaling others and the global update scal","pith_inferences":["The same two-level reweighting recipe could apply to any multitask reinforcement-learning setting with heterogeneous reward scales—for example, combining coding, math, and dialogue rewards—though the paper demonstrates it only on social-behavior tasks.","A natural stress test: apply HARPO to a synthetic benchmark where one task's reward variance is artificially inflated while its informativeness is held fixed; if the model still avoids collapse, the contribution-signal proxy is doing real work rather than the smoothing.","If balanced training indeed yields more transferable features, HARPO-trained models may serve as strong initializations for data-scarce clinical and social domains; the zero-shot results reported here are suggestive but would need broader validation.","The contribution proxy could be replaced by actual per-token gradient-norm estimates; comparing the two would reveal whether the proxy or the modulation structure is the bottleneck."],"forward_implications":["Unified behavior models can be trained across heterogeneous tasks without hand-crafted per-task weights or a separate value network.","Weak tasks that collapse under the base group-based method—sarcasm and humor detection in the reported runs—keep improving under HARPO (sarcasm F1 rises from about 54% to about 71%).","The rebalanced model transfers zero-shot to held-out behavior domains such as autism behavior recognition and speech emotion recognition, outperforming previous unified behavior models.","Because the method only reuses rewards already present in policy-gradient training, it can be added to existing on-policy optimization loops with two extra smoothing hyperparameters."],"fun_headline_variants":["HARPO rebalances RL gradients to top 9 of 10 social tasks","New RL method wins 9/10 tasks and all held-out benchmarks","OmniSapiens 2.0 tops 9 tasks and generalizes to new domains","Rebalancing gradients makes social AI win 9 tasks and all held-out","HARPO stabilizes training, winning 9 tasks and 5/5 benchmarks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a sample's or task's average absolute group-normalized advantage faithfully reflects how much it contributes to the policy update; the true contribution also depends on the size of the log-probability gradient, which can vary across tokens and samples.","fun_headline_variants_meta":{"raw":{"variants":["HARPO rebalances RL gradients to top 9 of 10 social tasks","New RL method wins 9/10 tasks and all held-out benchmarks","OmniSapiens 2.0 tops 9 tasks and generalizes to new domains","Rebalancing gradients makes social AI win 9 tasks and all held-out","HARPO stabilizes training, winning 9 tasks and 5/5 benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001322,"raw_usage":{"total_tokens":5211,"prompt_tokens":726,"completion_tokens":4485,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":4379}},"tokens_in":470,"tokens_out":4485,"duration_ms":33369,"temperature":1.0,"reasoning_tokens":4379,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T01:00:41.962034+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train HARPO and a control version that uses the same smoothing and geometric centering but with randomly assigned per-sample and per-task modulation weights; if the control matches HARPO's average rank across tasks, the contribution-signal mechanism is not the driver. A more direct check: during training, measure the correlation between the contribution signal and the actual per-sample gradient norm; a near-zero correlation would falsify the proxy on which the method is built.","supporting_citations":[],"review_version":1}