{"id":"4015aa27-00b1-4892-a68a-1bb03f65e8d9","arxiv_id":"2509.02737","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Action Collapse Policy Gradient (ACPG) fixes the action-selection layer to a simplex ETF and claims improved discrete-action RL performance, with a theory that only covers a weighted optimal-action imitation objective.","lead":"This paper reports a neural-collapse-like pattern in policy-gradient networks called Action Collapse, and proposes fixing the action-selection layer to an equiangular tight frame to induce it. The authors report faster, more stable training in several Gym environments, but the theoretical proof optimizes an imitation objective rather than the actual policy-gradient loss.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 optimizes Eq.(8), a weighted log-likelihood of optimal actions, not the policy-gradient objective minimized in Sec. 5; Appendix B's assertion that both are valid losses does not establish equivalence, so the theorem does not show ACPG induces collapse under REINFORCE, PPO, TRPO, or A3C.","rationale":"The reader's weakest assumption identifies exactly the gap I find most load-bearing: Eq.(8) is not the policy-gradient objective minimized in the experiments. The paper's own Appendix B admits the two equations differ but merely asserts both are valid loss functions, and no derivation shows equivalence. The proof of Theorem 1 is internally coherent for the imitation objective—the reduction from log(1+S) to S preserves argmins by monotonicity, and the KKT argument with a fixed ETF gives h=√(EH/EW)w_k—but that only makes the mismatch more visible: a correct theorem about the wrong objective cannot support the claim that ACPG induces Action Collapse in REINFORCE, PPO, TRPO, or A3C. The additional issues raised by the reader (log-sum-exp reduction, layer-peeled simplification, untested collapse in real environments) are secondary or partly defensible; the imitation-vs-PG gap is the single point on which the central mechanism depends. I would keep the reader's REJECT verdict: the empirical results may be useful, but the theoretical support for the central claim is not established. No ad hominem; the concern is about the argument. If the proposed concrete test shows the PG fixed point does coincide with Eq.(8)'s minimizer in representative cases, the verdict could be revisited.","tokens_in":16114,"tokens_out":7222,"duration_ms":69967,"concrete_test":"Under the same LPM, write the true PG objective J_PG(H)=Σ_s dπ_H(s) Σ_a π_H(a|s)Ψ(s,a) with H the last-layer activations and W fixed as the ETF. Compute ∂J_PG/∂h_s and compare with the KKT condition Eq.(21) used in the proof. Concretely, on a two-state, two-action tabular MDP with a fixed ETF head, solve for the global maximizer of Eq.(8), h*=√(EH/EW)w_k, and separately run REINFORCE/PPO to convergence from the same initialization. If the converged h_s from the PG run is not h* (in particular, if ∂J_PG/∂h_s at h* is nonzero), then Theorem 1 does not govern the ACPG training and the theoretical link fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that Eq.(8) is the objective actually optimized by the PG algorithms in Sec. 5.1. Eq.(8) is a weighted log-likelihood of the optimal action a* over observed states, with dπ(s) and Ψπ treated as fixed; this is a behavioral-cloning (supervised imitation) objective. The true PG objective, Eq.(1), is J(θ)=Σ_s dπ(s)Σ_a πθ(a|s)Ψπ(s,a), whose score-function gradient uses the sampled action a∼πθ, not δ(a=a*). Appendix B's final note concedes Eq.(1) and Eq.(2) differ and asserts both are valid loss functions, but 'valid loss function' does not imply 'same objective' or 'same global minimizer'. Moreover, in PG, dπ(s) is the stationary distribution of the policy being updated, so it depends on H; Eq.(8) freezes it. Theorem 1 therefore characterizes the global minimizer of an imitation loss, not a stationary point of REINFORCE, PPO, TRPO, or A3C. The empirical improvements of ACPG may still hold, but the paper's central claim that ACPG 'induces' Action Collapse through Theorem 1 is unsupported. The log/exp reduction in the proof is not the bottleneck; monotonicity preserves the argmin.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the geometry of last-layer features and action-selection weights in discrete policy-gradient (PG) networks. It reports that in idealized environments (fully explored, balanced state subsets, uniform stationary distribution) an 'Action Collapse' (AC) phenomenon occurs: state-action activations and action-selection weights converge to a simplex equiangular tight frame (ETF). To induce this structure in realistic environments, the authors propose Action Collapse Policy Gradient (ACPG), which fixes the action-selection layer to a randomly oriented simplex ETF and trains only the backbone. Theorem 1 claims that any global minimizer of Eq.(8) is a simplex ETF aligned with the fixed weights, irrespective of exploration, balance, or state-subset size. Experiments with REINFORCE, PPO, TRPO, and A3C on several Gym environments report faster and more robust convergence when ACPG is applied.","tokens_in":16521,"tokens_out":9383,"duration_ms":91498,"significance":"The empirical study is broad: four PG algorithms, multiple environments, and 20 seeds, and the proposed intervention is simple and easy to integrate. The paper also identifies a genuine gap, since neural-collapse results from supervised classification do not automatically transfer to reinforcement learning. If Theorem 1 were about the actual objective minimized by PG algorithms, the result would be valuable, giving a provable geometric target for policy networks. However, the theorem optimizes a per-state weighted log-likelihood of the optimal action, which is an imitation/classification loss rather than the PG objective used in the experiments. Appendix B explicitly acknowledges that Eq.(1) and Eq.(2) differ but only asserts that both are valid losses, without proving equivalence of their optimizers. The central conceptual claim, that ACPG 'induces' Action Collapse under PG, is therefore not supported by the presented theory. The empirical improvements may still hold, but the theoretical bridge is missing.","major_comments":[{"comment":"Theorem 1 optimizes Eq.(8), which is a weighted log-likelihood of the optimal action a* for each state, i.e., a supervised imitation/classification objective, not the policy-gradient objective in Eq.(1) that REINFORCE, PPO, TRPO, and A3C actually minimize in Section 5. Eq.(1) contains a sum over all actions weighted by πθ(a|s), whereas Eq.(8) contains only the log-probability of a*. Appendix B's final note concedes that Eq.(1) and Eq.(2) differ but merely asserts that both are valid loss functions; it does not show that they share global minimizers or that the algorithms in Section 5 minimize Eq.(8). Moreover, dπ(s) and Ψπ in Eq.(8) are treated as fixed even though in PG they depend on the current policy and hence on H. Consequently, Theorem 1 does not establish that ACPG induces Action Collapse under the PG algorithms evaluated.","section":"§4.2 (Theorem 1) and Appendix B"},{"comment":"The definition of a* is not operational for PG. In Eq.(2), a* is 'the optimal action predicted by the model', while Theorem 1 treats a* as the true optimal action of state s. In the experiments, the agent does not know optimal actions and samples actions from πθ, so the loss in Eq.(8) is not the loss being minimized. If a* were the model's own argmax, the theorem would describe a self-imitation objective and would still not apply to the sampled-action policy-gradient loss used by the compared algorithms.","section":"§2.1 and §4.2"},{"comment":"The reduction from Eq.(15) to Eq.(16) silently drops the factors dπ(s) and Ψπ and replaces the problem by minimizing Σ_{j≠k} exp(h^T(w_j-w_k)). This equivalence holds only if Ψπ > 0 for all states (and dπ(s) > 0), which is not stated in Theorem 1 and is not guaranteed for returns or advantages in PG. If Ψπ < 0 for some state, the sign of the term flips and the global minimizer of Eq.(8) need not align h with w_k. The theorem therefore requires an explicit positivity assumption or a different argument.","section":"Appendix B, Eqs. (15)-(16)"},{"comment":"The theorem's conclusion is largely built into the setup: once the action-selection layer is fixed to a simplex ETF, the unconstrained-features optimum of the weighted softmax loss aligns each activation with the corresponding ETF vector. This is a known property of fixed-ETF classifiers in the neural-collapse literature (e.g., Fang et al. 2021; Yang et al. 2022), not a statement about PG training dynamics. The paper should present Theorem 1 as a property of the chosen target geometry and should not use it to claim that Action Collapse 'emerges' under PG updates.","section":"§4.2"}],"minor_comments":[{"comment":"In Eq.(1), Ψπ should be written Ψπ(s,a); as printed, the quantity inside the sum over a appears not to depend on a, which is confusing given the summation index.","section":"§2.1, Eq. (1)"},{"comment":"The notation a* is used inconsistently: 'optimal action predicted by the model' in Eq.(2) versus 'true optimal action of state s' in Theorem 1. Please define one convention and use it throughout.","section":"§2.1 and §4.2"},{"comment":"It is unclear how Car-Racing, which has a continuous state space, satisfies Condition 3 (dπ(s) = dπ(s′) for all states). Please specify how states are discretized and how the stationary distribution is estimated; without this, the claimed evidence of Action Collapse in Car-Racing is hard to verify.","section":"§3"},{"comment":"Table 1 is difficult to parse: the 'Best Final' columns mix best and final rewards, and some cells contain '--' without explanation. Please restructure the table or add a clear legend explaining each block of columns.","section":"Table 1"},{"comment":"The notation S_k and n_k is introduced in the proof but is not connected to the state subsets S_k defined in Definition 1; also, the summation index in Eq.(14) is inconsistent ('nk' versus 'n_k').","section":"Appendix B, Eqs. (13)-(16)"},{"comment":"There is a typo: 'Theorem 1 1 states...' should be 'Theorem 1 states...'.","section":"Appendix E"}],"recommendation":"reject","confidential_remarks":"The central problem is the disconnect between Theorem 1 and the experiments: the theorem concerns a supervised imitation objective, while the experiments use standard PG algorithms with sampled actions and the PG objective in Eq.(1). I do not see a local fix for this; a revision would require either a new convergence analysis for PG with a fixed ETF action head or a redefinition of ACPG as an imitation-learning method with matching experiments. The empirical results may be of interest, but the paper's main theoretical claim, as stated, is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper makes a real empirical observation and a simple practical modification: freezing the action-selection layer to a simplex ETF makes discrete-action policy gradient methods (REINFORCE, PPO, TRPO, A3C) converge faster and more stably across a range of Gym environments. Second, the theory they bolt onto this does not support the headline claim that ACPG 'induces' Action Collapse under policy gradient training. The theorem is about a different objective: a weighted log-likelihood of the optimal action, i.e., supervised imitation, not the policy gradient objective used in the experiments.\n\nWhat is new: the observation of a neural-collapse-like pattern in the last-layer activations of optimal policy networks in 'ideal' fully explored environments is new, and transferring the fixed-ETF classifier idea from classification to RL is a legitimate new application. The paper documents this carefully with metric like equinormness, equiangularity, and variance convergence. The experimental work is solid in scope: 20 seeds, multiple algorithms, both MLP and CNN, and the gains are credible. The limitations section is unusually honest, explicitly conceding that Action Collapse has not been tested in complex environments and that the theorem's assumptions are ideal.\n\nSoft spots, in proportion. The load-bearing issue is the objective mismatch. Theorem 1 optimizes Eq.(8), which drops the sum over actions and keeps only the optimal action's log-probability. That is behavioral cloning, not policy gradient. The appendix's claim that 'both are valid loss functions' does not make them the same objective or give them the same global minimizers. Also, Eq.(8) freezes dπ(s), while in PG it depends on the current policy. So the theorem characterizes an imitation-learning optimum, not a stationary point of the PG algorithms they run. The claim that ACPG naturally induces Action Collapse in realistic environments is therefore unsupported by the theory. The empirical improvements may still be real and reproducible, but they stand as a practical trick plus intuition, not as a proven mechanism. The log/exp reduction in the proof is fine; that is not the issue. The circularity concern is also real: with the ETF fixed as target, the unconstrained-features optimum aligning to that same ETF is a direct consequence of the setup, not a surprising result.\n\nWho this is for: someone working on representation learning in RL, or on neural collapse transfer to new domains. The paper is worth a serious referee, but only if the authors are pushed to either prove the theorem for the actual PG objective or drop the induction claim and present ACPG as an empirical method with a motivating geometry. As is, I would not accept without major revision.\n\nRecommendation: send it to review, with clear instructions that the theory and experiments need to be aligned. If the authors can fix the mismatch or reframe the claims, there is a useful paper here.","headline":"The empirical trick of freezing the policy head to an ETF is worth knowing, but the theory proves something about an imitation loss, not the policy gradient objective actually trained, so the induced-collapse claim is unsupported.","tokens_in":16966,"tokens_out":2610,"would_cite":true,"duration_ms":24525,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that optimal policy networks collapse last-layer features into a simplex ETF, and fixing the action layer to that ETF makes the collapse a guaranteed global optimum while speeding up training.","keywords":["Action Collapse","Neural Collapse","policy gradient","simplex equiangular tight frame","reinforcement learning","action selection layer","representation geometry","ACPG"],"falsifier":"Take a small finite MDP with $K=2$, unequal class sizes $|S_1| \\ne |S_2|$, and fixed ETF weights; solve Eq. (8) exactly and confirm the global optimum is $h_s \\propto w_k$. Then compute the actual REINFORCE or PPO objective for those same activations with Monte-Carlo returns and the implied state distribution; if any non-collapsed activation set gives a strictly higher value, the equivalence the proof relies on fails.","tokens_in":15856,"feed_emoji":"🎮","tokens_out":9081,"duration_ms":81022,"temperature":0.7,"pith_summary":"Policy-gradient agents normally learn the last action-selection layer along with the backbone. This paper identifies a terminal geometry called Action Collapse: state-action activations sharing the same optimal action converge to one mean, within-group variance goes to zero, and both those means and the action-selection weights form a simplex equiangular tight frame (ETF), the most angularly separated configuration of $K$ vectors. The paper proves that if the action-selection layer is fixed to such an ETF, the simplified weighted-log-likelihood objective has a unique global optimum in which every state activation aligns with its optimal action's weight vector at common norm $\\sqrt{E_H}$, regardless of whether exploration is complete or state subsets are balanced. The resulting method, ACPG, trains only the backbone and reports faster, more stable reward improvement across several discrete-action benchmarks and policy-gradient algorithms. If the proof transfers, it turns Action Collapse from an ideal-condition observation into a cheap architectural prescription.","feed_headline":"Frozen simplex action head speeds policy-gradient RL up to 199%","feed_subtitle":"Fixing the last layer to an equiangular frame induces Action Collapse: faster, steadier policy-gradient rewards.","key_machinery":"The load-bearing object is the simplex equiangular tight frame (ETF): a set of $K$ vectors with equal norm and pairwise angle $\\cos\\theta = -1/(K-1)$, the maximal equiangular separation. The paper fixes the action-selection layer $W^*$ to a randomly oriented ETF scaled to norm $\\sqrt{E_W}$ and, following the layer-peeled model, treats the last-layer activations as the only variables under the constraint $\\|h_s\\|_2 \\le \\sqrt{E_H}$. The proof uses Karush-Kuhn-Tucker conditions; the ETF's defining identities $\\sum_k w_k = 0$ and $w_i^T w_j = -E_W/(K-1)$ make the alignment $h = \\sqrt{E_H/E_W}\\,w_k$ a stationary point, and convexity of the objective makes it the global optimum for every class simultaneously, independent of class sizes.","core_discovery":"The central claim is Theorem 1: for a layer-peeled policy network with a fixed simplex ETF action-selection layer, every global optimizer $H^*$ of Eq. (8) satisfies $h^{*T}_{s_k} w^*_{k'} = \\sqrt{E_H E_W}\\left(\\frac{K}{K-1}\\delta_{k,k'}-\\frac{1}{K-1}\\right)$ for all $k,k'$ and all states $s\\in S_k$. That is, activations collapse to the direction of their optimal action's weight, with length $\\sqrt{E_H}$, so Action Collapse is guaranteed by the objective rather than by balanced data or full exploration. The paper also reports the empirical counterpart: in ideal discrete environments, the geometry appears naturally, while in harder environments it does not, and imposing the ETF layer both induces the geometry and improves rewards.","pith_inferences":["A testable extension: if the theorem's mechanism is the binding one, then deliberately limiting backbone capacity should erase ACPG's gains, since the proof assumes activations can be optimized independently.","The epsilon-greedy results hint that ACPG converts exploration into geometric alignment; an untested corollary is that pairing ACPG with an exploration schedule tuned per environment would widen the reported improvements.","The same fixed-frame idea could transfer to other finite large action spaces, such as token-selection heads in sequence models, where a frozen ETF might induce a similar collapse; the paper does not test this."],"forward_implications":["A discrete policy-gradient agent with a fixed ETF action-selection layer will drive its last-layer activations to the ETF configuration even under partial exploration and imbalanced state sampling, because that geometry is the unique global optimum of the simplified objective.","ACPG can be layered onto existing discrete policy-gradient algorithms without changing their update rules; in the tested environments it improves best and final rewards and reduces run-to-run variance.","The fixed ETF gives maximal pairwise angular separation between actions, so the policy's decision boundary is as spread out as possible, which the paper connects to faster convergence and robustness under imbalanced states.","Because the action-selection layer is frozen, ACPG removes a set of learned parameters and reduces the training objective to aligning activations, which is why the paper reports earlier stopping and lower standard deviation."],"supporting_citations":[{"why":"Supplies the neural-collapse definitions and ETF geometry that Action Collapse adapts from classification to policy networks.","marker":"Papyan et al. (2020)"},{"why":"Contributes the layer-peeled model used to isolate last-layer activations in the proof of Theorem 1.","marker":"Fang et al. (2021)"},{"why":"Supplies the simplex ETF construction used to define the fixed action-selection layer.","marker":"Strohmer & Heath Jr (2003)"},{"why":"Defines the policy-gradient objective and the REINFORCE baseline that the experiments modify.","marker":"Sutton et al. (1999)"},{"why":"Provides TRPO, one of the policy-gradient baselines integrated with ACPG in the comparisons.","marker":"Schulman et al. (2015)"},{"why":"Provides PPO, the main baseline for performance comparisons and the exploration experiments.","marker":"Schulman et al. (2017)"},{"why":"Provides A3C, one of the baselines integrated with ACPG.","marker":"Mnih et al. (2016)"},{"why":"Supplies the benchmark environments used for the empirical evaluations.","marker":"Brockman et al. (2016)"}],"fun_headline_variants":["Fixed simplex ETF head guarantees Action Collapse in PG","Frozen equiangular frame in policy net yields Action Collapse","Simplex ETF head induces Action Collapse, speeds policy-gradient RL","Equiangular action head collapses features, boosts PG robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof applies to a simplified weighted-log-likelihood objective (Eq. 8), and the paper's own appendix notes that this differs from the general policy-gradient objective without proving equivalence; if the two are not equivalent, Theorem 1 does not cover REINFORCE, PPO, TRPO, or A3C.","fun_headline_variants_meta":{"raw":{"variants":["Fixed simplex ETF head guarantees Action Collapse in PG","Frozen equiangular frame in policy net yields Action Collapse","Simplex ETF head induces Action Collapse, speeds policy-gradient RL","Equiangular action head collapses features, boosts PG robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000463,"raw_usage":{"total_tokens":2365,"prompt_tokens":1050,"completion_tokens":1315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":1245}},"tokens_in":666,"tokens_out":1315,"duration_ms":10252,"temperature":1.0,"reasoning_tokens":1245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:34:48.090236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small finite MDP with $K=2$, unequal class sizes $|S_1| \\ne |S_2|$, and fixed ETF weights; solve Eq. (8) exactly and confirm the global optimum is $h_s \\propto w_k$. Then compute the actual REINFORCE or PPO objective for those same activations with Monte-Carlo returns and the implied state distribution; if any non-collapsed activation set gives a strictly higher value, the equivalence the proof relies on fails.","supporting_citations":[],"review_version":2}