{"id":"079d3426-3a09-4122-b309-d0e0ae2fec7a","arxiv_id":"2412.13053","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A sparse top-1 mixture of linear experts, trained with SAC and distilled into decision trees, matches or beats interpretable baselines and narrows the gap to opaque policies on MuJoCo tasks.","lead":"SMOSE trains a set of simple linear controllers, called experts, and a simple router that picks one expert at each step, then distills decision trees to explain the router. On six MuJoCo continuous control tasks it beats three interpretable baselines in most environments and approaches the scores of larger opaque neural policies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison with CGP/LGP baselines rests on visually transcribed numbers from plots; if those values are inaccurate or from different environment versions, the headline outperformance claim is unverified.","rationale":"The reader's verdict identified the visually transcribed CGP/LGP numbers as the weakest assumption; I agree. The central claim of the paper is empirical: a sparse top-1 mixture of linear experts can beat recent interpretable baselines on continuous control. The evidence for this is Table 1, where the two main interpretable competitors (CGP and LGP) are not run locally but taken from plots in a prior paper. This is a double risk: transcription error and protocol mismatch. The HalfCheetah environment illustrates the first risk, because SMOSE's margin over CGP is only ~900 points (~12%); a modest misread of a plot's y-axis could erase it. The second risk is more structural: if the original papers used different MuJoCo versions (e.g., -v2 or -v3 with different reward coefficients), the absolute rewards are not comparable regardless of transcription accuracy. The table's duplicate Metric-40 value for Ant and HalfCheetah further undermines confidence in the table's integrity. I considered the load-balancing loss in Eq. (3), which uses softmax(pi_m(...)) where pi_m is a Gaussian density, a notational error that would make the formula ill-defined. However, if the implementation used the correct router logits, the empirical results could still be valid; thus it is secondary to the baseline comparison. Similarly, the lack of decision-tree fidelity metrics and the tuning of M and lambda on the same benchmarks are concerns, but they do not directly challenge the outperform-interpretable-baselines claim. Therefore the load-bearing concern is the validity of the CGP/LGP comparison, and the concrete test is to rerun those baselines under the exact SMOSE protocol. The conditional verdict is appropriate until that check is done.","tokens_in":40082,"tokens_out":11514,"duration_ms":102790,"concrete_test":"Obtain the original CGP and LGP implementations from Nadizar, Medvet, and Wilson (2024) (or request code from the authors) and run them on the exact same Gymnasium 0.29.1 -v4 environments used for SMOSE, with the same 1000-step episode horizon and 100 deterministic evaluation episodes. Compare the resulting mean rewards to the values in Table 1; if any environment's true value differs by more than one standard deviation reported by the original paper, the outperformance claim is suspect. Additionally, check the Metric-40 values directly in Akrour et al. (2021) to confirm whether the HalfCheetah-v4 entry is indeed 2210.50 or a duplication of the Ant-v4 value; if duplicated, re-read the correct value and reassess the margin.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that SMOSE 'outperforms recent interpretable baselines' rests on Table 1, where the CGP and LGP entries are 'visually derived from the plots reported in the original papers.' This introduces two uncontrolled sources of error. First, reading means and error bars from figures is imprecise; a 10–20% misread on HalfCheetah-v4 (SMOSE 7310 vs CGP 6375) would narrow the margin to near parity. Second, the original CGP/LGP papers may have used different environment versions (e.g., Gym 'v2'/'v3' rather than Gymnasium 'v4'), different episode horizons, or different evaluation procedures (number of episodes, deterministic vs stochastic policies), which would make the absolute reward values non-comparable even if perfectly transcribed. The table also contains an internal copy-paste anomaly: Metric-40 is listed as 2210.50 for both Ant-v4 and HalfCheetah-v4, which is either a duplicated-row error or an unlikely coincidence; this weakens confidence in the table's accuracy generally. Because the comparison to CGP/LGP is the empirical basis for the paper's main contribution, this is the most load-bearing assumption. If the true baseline values are higher or were produced under different protocols, the central outperformance claim is not established. No code is provided to rerun either the baselines or SMOSE, so the numbers cannot currently be independently verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMOSE, an interpretable reinforcement learning policy architecture in which a sparse top-1 mixture of linear experts is selected by a linear router, trained end-to-end with SAC and load-balancing losses, and followed by distillation of decision trees from the router. The method is evaluated on six MuJoCo continuous control tasks. The authors report that SMOSE outperforms the interpretable baselines CGP, LGP, and Metric-40 in five of six environments, and that its performance is close to neural SAC policies of comparable size. The interpretability of the learned policies is demonstrated through per-expert weight visualizations and decision trees for Reacher-v4, with additional interpretations for the other environments in the appendix.","tokens_in":40352,"tokens_out":5407,"duration_ms":51079,"significance":"If the reported results are correct, SMOSE would be a valuable contribution to interpretable RL: a simple, trainable top-1 mixture-of-linear-experts policy that narrows the gap to neural policies while remaining directly interpretable through its linear router weights. The paper's strengths include a 10-seed evaluation protocol, a dedicated ablation on the number of experts, parameter-count-aware comparisons with SAC variants, and unusually detailed qualitative interpretations for all six environments. However, the central outperformance claim is contingent on baseline numbers that were visually transcribed from plots in other papers, and no code or data is provided for independent verification. The absence of a fidelity measure for the distilled decision trees also leaves part of the interpretability contribution unsupported.","major_comments":[{"comment":"The paper's headline claim that SMOSE 'consistently outperforms its competitors' among interpretable baselines rests on CGP and LGP values that the authors state were 'visually derived from the plots reported in the original papers.' Plot transcription is imprecise, and the original papers may have used different environment versions (e.g., v2/v3 vs. v4), different episode horizons, or different evaluation protocols, which would make the absolute rewards non-comparable even with perfect transcription. Moreover, Table 1 contains a clear internal anomaly: the Metric-40 entry is identical (2210.50 ± 175.50) for both Ant-v4 and HalfCheetah-v4, which is either a copy-paste error or an extremely unlikely coincidence. Since the comparison to CGP/LGP/Metric-40 is the empirical basis for the main contribution, the authors should re-run these baselines under their own protocol (or otherwise obtain trustworthy numbers with confirmed environment versions) and correct Table 1. Without this, the primary outperformance claim is not established.","section":"Table 1, Policy evaluation"},{"comment":"The load-balancing losses in Eqs. (3) and (4) use π_m(s|θ_m, σ_m) as if it were a scalar router logit: Eq. (3) applies softmax to it, and Eq. (4) compares it to τ(s_k)=max_m π_m(...). However, π_m is defined just above as a Gaussian distribution N(θ_m·s, σ_m^2). As written, softmax of a Gaussian distribution and the threshold comparison in the load term are not well-defined, so the training objective cannot be reproduced from the text. The authors should clarify whether ĝ(s|Θ), the inner router output, or the expert mean θ_m·s is intended, and rewrite the equations accordingly.","section":"Method, Eqs. (3)-(4)"},{"comment":"The paper distills binary decision trees from the learned router and presents these trees as a tool that 'significantly improves the ease of interpretation.' However, no fidelity metric is reported: the authors do not measure how often the depth-limited DTs agree with the continuous router, nor the performance impact of replacing the router with the DTs. Since the trees are post-hoc surrogates, the reader cannot tell whether the depicted splits faithfully represent the router's actual decisions. Please report at least one agreement/fidelity metric (e.g., percentage of states where the DT selects the same expert as the router, or the average return when the DT router is deployed), or moderate the claim accordingly.","section":"Decision Trees as a support for interpretation"}],"minor_comments":[{"comment":"The abstract says SMOSE 'outperforms recent interpretable baselines' without the caveat that appears in the Results section, where Table 1 shows SMOSE is far worse than CGP/LGP on Swimmer-v4 (45.40 vs. 280.00/278.50). Please either add the caveat in the abstract or weaken the wording.","section":"Abstract / Results"},{"comment":"The caption reads 'SMoSE' but the method is consistently spelled 'SMOSE' elsewhere; please unify the spelling.","section":"Figure 2 caption"},{"comment":"The decision-tree figures for Reacher-v4 are extremely low-resolution and essentially unreadable in the manuscript; they also appear to contain 'Powered by TCPDF' artifacts. Please re-render these figures at higher quality.","section":"Figures 7-14"},{"comment":"The notation in the distillation description is inconsistent: the text says 'DT_m(Θ_m·s)=1' but the discussion of inputs and labels could be clarified, particularly whether the trees are trained on raw states s or on the projected router scores Θ_m·s. Please state this explicitly.","section":"Appendix, Decision Trees"},{"comment":"The footnote says 'magenta = best score per environment,' but the table appears monochrome in the submitted PDF; please use boldface or another visible marker.","section":"Table 1 footnote"}],"recommendation":"major_revision","confidential_remarks":"The duplicated Metric-40 values in Table 1 for Ant-v4 and HalfCheetah-v4 are particularly concerning because they suggest insufficient table checking; I would ask the authors to audit the entire table. The lack of code/data is also problematic given that the key comparison numbers were transcribed from plots. The method itself is plausible and the writing is generally clear, so I see major revision rather than rejection as the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SMOSE is a genuinely neat architecture — a top-1 sparse MoE with linear experts and a linear router trained end-to-end with SAC, then distilled into shallow decision trees. That specific combination is new as far as I can tell, and the interpretability analysis for the learned weights is thorough and useful. If the numbers hold, the method is a real step toward interpretable continuous control.\n\nWhat I like: The design is simple and well-motivated; the top-1 hard selection with load-balancing is standard, but applying it with fully linear experts and router is a sensible way to retain interpretability without sacrificing too much performance. The comparison against SAC-M and SAC-S is internally controlled — they trained those baselines with the same protocol — and the results look plausible. The error bars are reasonable, and the training curves in Fig. 2 match the tables.\n\nSoft spots, in order of concern. First, the headline claim that SMOSE outperforms recent interpretable baselines rests on Table 1, where CGP and LGP values are 'visually derived from plots' in the original papers. That is a fragile basis. A 10–20% misread on HalfCheetah would put SMOSE at parity. More importantly, those papers may have used different environment versions or evaluation protocols; the absolute scores may not be comparable. Second, the table has an internal red flag: Metric-40 is listed as 2210.50 ± 175.50 for both Ant-v4 and HalfCheetah-v4. That looks like a copy-paste error, and it undercuts confidence in the table's accuracy generally. Third, no code or data is provided, so nothing can be independently verified. Fourth, the decision trees distilled from the router are presented as an interpretability aid, but the paper never measures how faithfully they represent the router's decisions. That is a gap, though minor since the linear policy itself is already interpretable.\n\nNone of these issues kills the central idea. The method is coherent, the internal comparisons are solid, and the interpretation section is a genuine contribution. But the 'outperforms interpretable baselines' claim should be treated as provisional until the authors either rerun the baselines under their own protocol or provide exact numbers with environment versions.\n\nWho should read it: anyone working on interpretable RL, policy distillation, or safe control. It deserves a serious referee, but the review should ask for code/data and a clarification of Table 1's provenance.","headline":"SMOSE is a clean, plausible interpretable-RL architecture, but its headline outperformance claim rests on visually transcribed baselines and a suspicious duplicated table entry.","tokens_in":40924,"tokens_out":3076,"would_cite":true,"duration_ms":28244,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A top-1 sparse mixture of shallow linear experts can outperform recent interpretable controllers on continuous control benchmarks and approach the performance of neural Soft Actor-Critic policies.","keywords":["interpretable reinforcement learning","mixture of experts","continuous control","Soft Actor-Critic","decision trees","MuJoCo","sparse activation","linear policies"],"falsifier":"Re-run CGP, LGP, and Metric-40 under the same MuJoCo v4 environments, ten seeds, one million training timesteps, and 100 evaluation episodes, then compare mean episodic rewards; if the transcribed baseline values are materially higher or lower than the original runs, the claimed five-of-six dominance over interpretable methods is not established.","tokens_in":1669,"feed_emoji":"🤖","tokens_out":2660,"duration_ms":53821,"temperature":0.7,"pith_summary":"SMOSE is a policy architecture for continuous control that decomposes behavior into a small set of linear sub-policies, or experts, and a linear router that selects exactly one expert per state. The paper argues that this sparse, fully linear mixture can be trained with Soft Actor-Critic and load-balancing penalties, and that the resulting controller is both high-performing and directly interpretable. On six MuJoCo benchmarks, SMOSE outperforms the interpretable baselines CGP, LGP, and Metric-40 in five of six environments, and its mean returns land close to neural SAC policies of comparable active parameter count. The authors also distill decision trees from the router weights to make the task-allocation logic readable without sacrificing the policy's inherent transparency.","feed_headline":"Sparse linear experts beat interpretable RL baselines on 5 of 6 tasks","feed_subtitle":"SMOSE activates one shallow linear controller per state, closing the gap to neural SAC with readable router weights.","key_machinery":"The central object is the top-1 sparse mixture $\\pi(s)=\\sum_{m=1}^M [\\mathrm{TOP1}(\\mathrm{softmax}(\\Theta s))]_m \\, (\\theta_m s)$, where each expert is a linear map from state to action and the router is also linear, so every decision is a single linear control law selected by a linear preference score. Training uses Soft Actor-Critic for exploration and adds the importance loss and load-balancing loss from sparse MoE literature, with Gaussian noise on the router logits, to prevent expert collapse and keep the experts evenly used. After training, the router is distilled into depth-limited decision trees by labeling replay-buffer states, producing a readable approximation of when each expert takes control.","core_discovery":"The central claim is that a sparse Mixture-of-Experts policy with a top-1 hard gate, linear experts $\\pi_m(s)=\\theta_m s$, and a linear router $\\hat{g}(s|\\Theta)=\\Theta s$ can be trained end-to-end with SAC to reach competitive performance on continuous control tasks while remaining interpretable. The router uses $g(s|\\Theta)=\\mathrm{TOP1}(\\mathrm{softmax}(\\Theta s))$, so in any state only a single linear controller is active. This design outperforms recent interpretable baselines on Walker2d-v4, Hopper-v4, Ant-v4, HalfCheetah-v4, and Reacher-v4, and it narrows the gap to non-interpretable SAC networks of comparable size, sometimes matching or exceeding them. The paper further shows that the learned router weights can be read as scoring functions that assign each expert to a recognizable regime, and that decision trees distilled from those router scores provide an additional compact explanation layer.","pith_inferences":["Editorial inference: the same router-plus-linear-experts decomposition may transfer to higher-dimensional control if the router remains sparse and the experts are kept shallow, since the interpretability burden stays localized in the router's decision boundaries rather than in a monolithic network.","Editorial inference: the failure on Swimmer-v4, where all SAC-based policies are weak, hints that the top-1 gating structure struggles when the optimal behavior requires smooth blending of several velocity-dependent strategies; a testable extension would be to compare TOP1 against TOP2 or TOP3 on velocity-dominated environments.","Editorial inference: distilling decision trees from the router rather than from the whole policy creates a path toward formal verification: a bounded-depth tree over linear thresholds could be model-checked or converted to rules for certification, which the paper only gestures at by citing imitation-learning extensions as future work.","Editorial inference: one could replace the linear experts with other interpretable classes, such as single-step lookahead or proportional-derivative controllers, and keep the same training pipeline, which would test whether linearity or modularity is the main driver of the observed performance."],"forward_implications":["SMOSE shows that a sparse combination of linear policies, with only one expert active per state, can outperform the selected interpretable baselines on five of six MuJoCo tasks.","With far fewer active parameters than the neural SAC baselines, SMOSE achieves returns close to SAC-M and SAC-S, narrowing the gap between interpretable and closed-box controllers.","The linear router weights give a state-space decomposition into identifiable skill regimes, which suggests a practical route for auditing or debugging a learned controller at a per-expert level.","The router distillation into decision trees provides a compact, human-readable map of when each expert is invoked, supporting verification and human oversight without needing to inspect the full weight matrices.","Because the architecture allows TOP k with $k>1$, the same training scheme can trade interpretability for expressiveness while keeping the policy linear and inspectable."],"supporting_citations":[{"why":"Supplies the Metric-40 interpretable MoE baseline whose reported performance SMOSE must beat, and the comparison context for mixture-of-experts interpretable control.","marker":"(Akrour, Tateo, and Peters 2021)"},{"why":"Supplies the CGP and LGP baseline results that SMOSE compares against in Table 1, even though those numbers were visually derived from plots.","marker":"(Nadizar, Medvet, and Wilson 2024)"},{"why":"Provides the Soft Actor-Critic algorithm and hyperparameter setup used to train SMOSE and the SAC-based neural baselines.","marker":"(Haarnoja et al. 2018)"},{"why":"Provides the importance loss, load-balancing loss, and the softmax-before-TOP1 gradient propagation ordering that SMOSE adopts for the router.","marker":"(Riquelme et al. 2021)"},{"why":"Introduces sparse MoE with top-k gating, the architectural inspiration for SMOSE's sparsely activated experts.","marker":"(Shazeer et al. 2017)"},{"why":"Defines the MuJoCo environments used as the six continuous-control benchmarks in the evaluation.","marker":"(Todorov, Erez, and Tassa 2012)"},{"why":"Provides the CleanRL SAC parameters and the PPO baseline numbers used as non-interpretable comparison points.","marker":"(Huang et al. 2022)"}],"fun_headline_variants":["Sparse linear experts beat interpretable RL baselines","Sparse linear experts beat 5 of 6 interpretable baselines","One linear controller per state: interpretable, near-SAC","Readable router, linear experts: beats interpretable baselines"],"cache_read_input_tokens":43008,"weakest_assumption_plain":"The headline comparison against interpretable baselines rests on performance numbers transcribed by eye from plots in earlier papers, which may have been produced under different environment versions, episode counts, or seeds than SMOSE's runs.","fun_headline_variants_meta":{"raw":{"variants":["Sparse linear experts beat interpretable RL baselines","Sparse linear experts beat 5 of 6 interpretable baselines","One linear controller per state: interpretable, near-SAC","Readable router, linear experts: beats interpretable baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001679,"raw_usage":{"total_tokens":6667,"prompt_tokens":965,"completion_tokens":5702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":5631}},"tokens_in":581,"tokens_out":5702,"duration_ms":42658,"temperature":1.0,"reasoning_tokens":5631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:27:46.248899+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run CGP, LGP, and Metric-40 under the same MuJoCo v4 environments, ten seeds, one million training timesteps, and 100 evaluation episodes, then compare mean episodic rewards; if the transcribed baseline values are materially higher or lower than the original runs, the claimed five-of-six dominance over interpretable methods is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Metric-40 interpretable MoE baseline whose reported performance SMOSE must beat, and the comparison context for mixture-of-experts interpretable control."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CGP and LGP baseline results that SMOSE compares against in Table 1, even though those numbers were visually derived from plots."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MuJoCo environments used as the six continuous-control benchmarks in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CleanRL SAC parameters and the PPO baseline numbers used as non-interpretable comparison points."}],"review_version":1}