{"id":"07a9526b-bb49-43bd-9460-63e6e5e14d9c","arxiv_id":"2509.07006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A framework combining GRPO, LLM-as-judge rewards, and OPA-style policy checks improves a 1B medical assistant's domain-scope adherence by 70.9% in a 100-scenario LLM-judge evaluation.","lead":"ArGen trains a small language model with an automated judge-based reward signal plus explicit Python policy checks, and reports that a medical assistant trained this way stays in scope 70.9% better than the baseline. The case study shows how culturally specific rules such as Dharmic ethics can be encoded as code and used during reinforcement learning, which matters for auditable AI compliance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'hard policy' layer is internally an LLM judge: the claimed OPA-style governance is not hard policy, so the central policy-as-code claim is unsupported.","rationale":"The reader's weakest assumption was that LLM-as-judge scores validly measure the ethical qualities being optimized, citing Section 7.5's uncertainty-marker bias. My concern is adjacent but more structural: in the actual implementation, the 'OPA-inspired Python policy engine'—the component that supposedly provides hard, auditable governance—delegates its core scope and safety classifications to an LLM. Thus the framework's central differentiator (policy-as-code with hard P_scope/P_sev constraints) is not what the code shows. This is internally checkable from Listings 1–8 in Appendix A.1 and does not rely on disputing consensus. I did not select the post-hoc best-checkpoint selection or the lack of human evaluation as the primary concern, although both are real secondary weaknesses; the policy-engine-is-an-LLM issue undermines the claimed architecture itself, not just the strength of the evidence. Credit is due: the paper provides a GitHub repository, full training command, and a candid limitations section, and a deterministic re-run would be a decisive and feasible test. Since the concern can be settled by a concrete ablation and the reader already issued CONDITIONAL, I recommend keeping the verdict unchanged rather than escalating to REJECT on the basis of code inspection alone.","tokens_in":43071,"tokens_out":7362,"duration_ms":69292,"concrete_test":"In the released argen-demo repository, replace every LLM call inside dharma_scope_check and ahimsa_safety_check (Listings 1–8) with deterministic rule-based code—keyword lists, regex/Rego predicates, and fixed penalty tables—that returns the same penalty factors without any Gemini/OpenAI API call. Re-run the Policy-Only ablation from Section 5.5.8 on the same training data and evaluate the resulting model on the same 100-scenario benchmark with Claude 3.5 Sonnet. Compare the Dharma score and violation rate against the reported Policy-Only and Full ArGen numbers: if the deterministic-policy model matches or exceeds them, the policy-as-code layer is doing the work; if the gain collapses or changes sign, the headline effect depends on LLM-as-judge scoring inside the 'policy engine', not on explicit hard policies.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ArGen's central novelty is an OPA-inspired Python policy engine that enforces explicit, auditable hard constraints (P_scope, P_sev) inside the GRPO reward. The paper's own implementation listings contradict this. Listing 2 (evaluate_dharma_with_gemini) classifies response_scope and computes domain_adherence_score by calling an LLM (Gemini), and Listing 4 multiplies that LLM score by a penalty factor derived from the LLM's scope classification. Listings 5–8 do the same for Ahimsa: harm_avoidance_score, safety_context_score, and referral_strength all come from an LLM call. Therefore P_scope and P_sev are not deterministic code checks; they are outputs of another stochastic LLM-as-judge. The formal guarantee in Appendix B.6.1 (R_total ≤ 0 under hard violations, repulsive gradient) applies only to a binary P_scope, not to the implemented LLM-derived penalty. The 'policy-as-code' transparency and auditability claims also weaken: the auditable artifact is a prompt, not a rule. Section 7.5 admits reliance on and biases of the evaluator LLM but does not flag that this reliance is inside the supposed hard policy engine. The 70.9% Dharma gain could thus be caused by a second LLM reward channel (scope scored by Gemini) rather than by the OPA-inspired governance layer, reducing the empirical result to GRPO with LLM-judge rewards.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ArGen, a framework that combines LLM-as-a-judge reward scoring, GRPO, and an OPA-inspired Python policy layer to align a small language model with explicit governance policies. The central case study trains Llama-3.2-1B-Instruct as a medical assistant guided by Dharmic principles (Ahimsa, Dharma, Helpfulness) and reports a 70.9% relative improvement in Dharma scope-adherence score over the baseline, measured by Claude 3.5 Sonnet on a 100-scenario benchmark. The paper also reports an ablation study comparing reward-only and policy-only variants, a cross-evaluator validation with Gemini 2.0 Flash, adversarial red-team prompts, and an open-source implementation repository.","tokens_in":43358,"tokens_out":4669,"duration_ms":37802,"significance":"If the central claims held, ArGen would be a meaningful step toward configurable, policy-as-code alignment: the paper ships machine-checkable code listings, an open-source repository, a cross-evaluator analysis, and an ablation study, and it demonstrates that a 1B model can be steered by a multi-component reward signal. The Dharmic case study also addresses a genuinely underrepresented cultural perspective in AI alignment. However, the load-bearing 'hard policy' claim is contradicted by the paper's own implementation, and the headline metric rests on LLM-judge scores without human validation. These issues are substantial but, in my view, fixable through re-framing and additional analysis rather than requiring rejection.","major_comments":[{"comment":"The OPA-inspired 'hard policy' layer is not hard, and the formal guarantee does not apply to the implemented reward. In Listing 2, response_scope and domain_adherence_score are outputs of the Gemini call; Listing 4 multiplies that LLM score by a penalty factor; Listings 5–8 obtain harm_avoidance_score, safety_context_score, and referral_strength from an LLM call. Therefore P_scope and P_sev are stochastic LLM-derived quantities, not deterministic code checks. Appendix B.6.1's theorem (R_total ≤ 0 under hard violations) assumes P_scope is a binary hard flag; it does not hold for the implemented continuous LLM output. The 'policy-as-code' transparency claim is correspondingly weakened: what is auditable is an evaluation prompt, not a rule. The authors should either implement deterministic checks or consistently re-frame the governance layer as soft LLM-judged penalties and remove the hard-guarantee language.","section":"§3.4, §3.6, Appendix B.6.1; Listings 2–8"},{"comment":"The headline 70.9% improvement is measured by an LLM judge on the same construct that the training reward optimizes; no human-expert validation establishes that either judge measures safety or scope. Section 7.5 concedes that LLM judges can systematically penalize uncertainty markers, the very cautious phrasing that Ahimsa training aims to produce. Cross-evaluator agreement (Table 5) rules out some same-judge circularity, but it does not rule out both judges sharing stylistic preferences. The divergence on helpfulness (Gemini +7.8%, Claude −7.4%) shows evaluator sensitivity. Please add human-validated labels or a judge-calibration analysis showing that score changes track expert-rated safety and scope rather than refusal-style lexical markers.","section":"§5.5.4, §7.5"},{"comment":"Table 3 reports the best-performing of 15 evaluated checkpoints, selected post hoc by Combined Score, and the 70.9% Dharma gain is for that model. With 15 candidates, some selection inflation is expected. The median seed model used in ablations (GRPO7 Seed 3, Combined Score 0.7825) is below the best model (0.7947), so the selection rule matters. Report the full distribution of checkpoints or pre-register a selection rule with a held-out test set, and qualify the abstract and Section 5.5.2 claims accordingly.","section":"§5.5.1, Table 3"}],"minor_comments":[{"comment":"The reward weights are inconsistent: Section 5.3 assigns Ahimsa 0.4, Dharma 0.3, Helpfulness 0.3, while Section 3.8 states that Dharma/scope adherence receives 40% priority; reconcile the two descriptions.","section":"§5.3 vs §3.8"},{"comment":"Section 5.4 says the model was trained for 2 epochs, while Appendix A.3 and Figure 5 report 3 training epochs; clarify which configuration produced the reported results.","section":"§5.4 vs Appendix A.3"},{"comment":"The sentence that 'both evaluators confirmed substantial improvements' overstates agreement on helpfulness, since Gemini reports +7.8% while Claude reports −7.4%; the discussion should not describe that as confirmation of the same effect.","section":"§5.5.4"},{"comment":"The claim of an 89.7% reduction in violation rate is inconsistent with the 34% to 4% reduction shown in Table 5, which is an 88.2% relative decrease; check the arithmetic.","section":"§7.1, Table 5"},{"comment":"The caption claims 'superior performance across all metrics,' but Table 3 shows helpfulness declined by 7.4% for the best-performing model; the caption should reflect the trade-off.","section":"Figure 4 caption"},{"comment":"The discussion of Gato-style multi-objective fine-tuning refers to DeepMind's Gato without a citation; add a reference or remove the un-cited entity.","section":"Appendix B.4"},{"comment":"The abstract contains a typo, 'Governable Al', which should read 'Governable AI'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper has a real empirical pipeline and the authors are transparent about many limitations, but the gap between the 'hard policy-as-code' framing and the actual LLM-based policy implementation is the central issue. If the authors cannot replace the LLM-derived scope/safety classifications with deterministic checks, they should substantially weaken the governance claims and reposition the contribution as GRPO with soft LLM-judged policy penalties. This is fixable, but it requires a genuine re-framing of the paper's headline contribution; I would not accept the current framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is an honest, unusually transparent integration of GRPO with LLM-judge rewards and OPA-style penalties, on a genuinely interesting case study. Its central 'policy-as-code' claim is oversold: the 'hard' policy engine is itself an LLM judge, so the formal guarantee does not apply to the implementation.\n\nWhat is new is the synthesis—reward functions that call Gemini, a Python 'policy' layer that feeds penalties into GRPO, and a 1B medical assistant aligned to Dharmic principles. The repo, training command, and code listings are all there. The authors used a held-out judge (Claude 3.5 Sonnet), ran reward-only and policy-only ablations, and explicitly discuss LLM-judge bias in Section 7.5. That is real credit.\n\nBut the main claim does not survive its own appendix. Listings 2–8 show that response_scope, domain_adherence_score, harm_avoidance_score, etc. are all produced by Gemini; the penalty tables then operate on those LLM outputs. So P_scope and P_sev are not deterministic, auditable checks; they are a second stochastic LLM reward. The formal theorem in B.6.1 assumes a binary P_scope enforced by code. As implemented, ArGen is GRPO with two LLM-judge channels, not policy-as-code with hard constraints. An auditor inspecting the Python would still have to audit a prompt.\n\nThe reported 70.9% Dharma improvement is plausible but selected post hoc: the Best-Performing Model is the best of 15 checkpoints. That is peak picking unless the authors report all checkpoints or a preselected rule. Helpfulness is also inconsistent across judges (-7.4% Claude, +7.8% Gemini), which undercuts the 'both judges agree' narrative.\n\nWho should read it: people building small, regulated LLMs who want a concrete recipe and are comfortable with LLM-judge-based evaluation. It is not a formal governance contribution. Still, it deserves a real peer review—the engineering is serious and reproducible. A good referee should require revised claims about hard vs. learned constraints, full checkpoint data, and some human-expert validation before the 'verifiably compliant' language is warranted.","headline":"ArGen is a transparent, reproducible integration of GRPO with LLM-judge rewards, but its central 'policy-as-code' claim is undermined by the implementation itself, where the supposed hard policy layer is just another LLM judge.","tokens_in":43889,"tokens_out":2682,"would_cite":false,"duration_ms":25531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ArGen: aligning a 1B language model to explicit governance policies via LLM-judge rewards, GRPO, and OPA-style Python penalties.","keywords":["Policy as Code","AI Governance","Group Relative Policy Optimisation","LLM-as-a-Judge","Dharmic ethics","AI alignment","Open Policy Agent","Reinforcement Learning"],"falsifier":"Evaluate the best-performing ArGen checkpoint on a fresh set of out-of-scope prompts whose vocabulary does not overlap with the medical training set, and compare the LLM-judge Dharma scores with a panel of human clinicians' ratings of scope and safety; the central claim is falsified if the ~0.96 Dharma score does not reproduce on the fresh set, or if clinicians flag responses as unsafe or out-of-scope that the judges rate as compliant.","tokens_in":42851,"feed_emoji":"🩺","tokens_out":9780,"duration_ms":74960,"temperature":0.7,"pith_summary":"The paper tries to establish that AI alignment can be recast as a configurable, policy-as-code process rather than a fixed preference-tuning exercise. It claims that a 1B language model can be trained to obey explicit governance rules by combining LLM-as-judge reward scoring against stated principles, Group Relative Policy Optimisation (GRPO), and a Python policy engine inspired by Open Policy Agent that injects scope and safety penalties directly into the reward. The headline evidence is the MedGuide-AI case study, where the trained model improved its domain-scope (Dharma) score by 70.9% over the baseline (from 0.5640 to 0.9641, judged by a held-out LLM) while also improving safety and sacrificing only modest helpfulness. A sympathetic reader would care because, if the claim is right, the same machinery could turn regulations and culturally specific ethical codes into auditable, swappable training constraints for small open models in regulated settings.","feed_headline":"Policy-as-code training lifts a 1B model's scope adherence 70.9%","feed_subtitle":"Configurable ethics and OPA-style penalties keep a small Llama model in its medical lane and deferring to clinicians.","key_machinery":"The load-bearing mechanism is a modular reward composition with hard-constraint penalties embedded in a GRPO training loop. The Python policy engine defines a four-tier scope classification (S0 clinical core, S1 para-clinical, S2 tangential, S3 outside medicine) with a penalty matrix that nullifies or scales the reward when a response leaves scope, plus a three-tier urgency system (A emergency, B urgent/specialist, C routine) that penalises referral-strength mismatches. These penalty factors multiply the weighted sum of LLM-evaluated principle scores (Ahimsa, Dharma, Helpfulness), and the resulting scalar reward drives policy updates through DR-GRPO with KL regularisation toward a reference model. The essential work of this composition is to make 'stay in scope' and 'refer appropriately' hard gradient signals rather than soft preferences, which is how the paper explains the large, judge-confirmed Dharma gains.","core_discovery":"The paper's central claim is that governance policies can be written as interpretable code artifacts, converted into reward penalties, and internalised by a small model through GRPO, so that the model learns to resign from out-of-scope requests. Concretely, ArGen composes the total reward as $R_{\\text{total}} = P_{\\text{scope}} \\sum_i \\lambda_i R_i + P_{\\text{sev}}$, where $P_{\\text{scope}}$ multiplicatively zeroes the reward on hard scope violations and $P_{\\text{sev}}$ adds a negative severity penalty; each $R_i$ is a principle score produced by an evaluator LLM. On the 100-scenario medical benchmark, the best ArGen checkpoint raised the average Dharma score from 0.5640 to 0.9641 (+70.9%), cut scope violations from 34% to 4%, and improved Ahimsa (safety) by 5.2%, with a 25.0% rise in the combined score; both the training evaluator (Gemini) and a held-out judge (Claude 3.5 Sonnet) confirmed the direction of these gains. The paper further argues that the same policy overlay can govern inference-time behaviour and be hot-swapped without retraining, turning compliance updates into a software-operations task rather than a retraining cycle.","pith_inferences":["Beyond the paper: a sharp test of whether the Dharma gain is genuine concept learning rather than keyword or refusal-pattern matching would be to evaluate the trained model on out-of-scope prompts whose vocabulary does not overlap with the medical training set.","Beyond the paper: the disagreement between the two judges on helpfulness (−7.4% with the held-out judge, +7.8% with the training judge) suggests that multi-objective trade-off claims need human preference data before being accepted.","Beyond the paper: because the framework itself notes that LLM judges may penalise uncertainty markers, the Ahimsa gains should be validated against clinician safety ratings on the same benchmark scenarios; otherwise part of the improvement could be stylistic.","Beyond the paper: the live hot-swap claim implies a continuous-alignment deployment loop, but the study only demonstrates training-time effects; a longitudinal deployment test measuring whether policy updates propagate to behaviour without drift would be the natural follow-up."],"forward_implications":["If the claim holds, alignment becomes a software-operations task: a compliance team can edit policy files and redeploy an overlay, and new constraints take effect without retraining the model.","The reported drop in scope violations from 34% to 4% implies that even a 1B open model can internalise hard domain boundaries when penalties are injected directly into the reward.","The ablations imply that neither component alone suffices: reward-only degrades slightly, policy-only degrades substantially on Dharma, and the hybrid achieves the best balanced performance.","Because the framework treats principles as configurable modules, the same reward-and-penalty machinery could encode other ethical traditions or regulatory rule sets beyond the Dharmic medical example.","The claimed minutes-to-compliance adaptation timeline, if correct, would give regulated deployments a way to respond to new rules without multi-day retraining."],"supporting_citations":[{"why":"Supplies the GRPO algorithm that ArGen uses to optimise the policy from the composed reward signal.","marker":"[Shao et al., 2024]"},{"why":"Constitutional AI is the principle-based RLAIF baseline that ArGen extends by externalising policies as code.","marker":"[Bai et al., 2022]"},{"why":"RLHF/InstructGPT is the preference-based alignment approach against which ArGen positions itself.","marker":"[Ouyang et al., 2022]"},{"why":"OPA's declarative Rego governance model inspires the Python policy engine that produces scope and severity penalties.","marker":"[Open Policy Agent Maintainers, 2024]"},{"why":"The concept of algorithmic resignation motivates the framework's goal of teaching models to disengage from out-of-scope requests.","marker":"[Bhatt and Sargeant, 2024]"},{"why":"Demonstrates GRPO's effectiveness on large language models, supporting the choice of GRPO as the optimization engine.","marker":"[DeepSeek-AI, 2025]"},{"why":"Documents LLM-as-judge capabilities and reliability, the basis for the principle-based automated reward scoring.","marker":"[Dou et al., 2024]"}],"fun_headline_variants":["Policy-as-code GRPO: 1B model scope adherence up 70.9%","GRPO + policy-as-code: medical AI follows Dharma, 70.9% better","Small model internalizes written policies via GRPO, 70.9% gain","ArGen: OPA-style rules and GRPO boost medical scope 70.9%","Policy-as-code governance: LLM learns scope limits, 70.9% improvement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the LLM judges used for training (Gemini) and for reporting results (Claude 3.5 Sonnet) score scope and safety as intended, rather than rewarding easy-to-recognize refusal phrasing; the paper offers no human expert validation of these scores.","fun_headline_variants_meta":{"raw":{"variants":["Policy-as-code GRPO: 1B model scope adherence up 70.9%","GRPO + policy-as-code: medical AI follows Dharma, 70.9% better","Small model internalizes written policies via GRPO, 70.9% gain","ArGen: OPA-style rules and GRPO boost medical scope 70.9%","Policy-as-code governance: LLM learns scope limits, 70.9% improvement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1721,"prompt_tokens":1056,"completion_tokens":665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":552}},"tokens_in":672,"tokens_out":665,"duration_ms":5871,"temperature":1.0,"reasoning_tokens":552,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:21:56.537325+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the best-performing ArGen checkpoint on a fresh set of out-of-scope prompts whose vocabulary does not overlap with the medical training set, and compare the LLM-judge Dharma scores with a panel of human clinicians' ratings of scope and safety; the central claim is falsified if the ~0.96 Dharma score does not reproduce on the fresh set, or if clinicians flag responses as unsafe or out-of-scope that the judges rate as compliant.","supporting_citations":[{"cited_title":"Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E","cited_arxiv_id":null,"evidence_quote":"RLHF/InstructGPT is the preference-based alignment approach against which ArGen positions itself."},{"cited_title":"Open Policy Agent Documentation","cited_arxiv_id":null,"evidence_quote":"OPA's declarative Rego governance model inspires the Python policy engine that produces scope and severity penalties."}],"review_version":2}