{"id":"1f97d59f-d613-40bf-8d6c-3aa66b68cfeb","arxiv_id":"2501.10529","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Low-rank tensor decomposition of multi-task Q-functions lets related reinforcement learning tasks share data and learn faster than independent training.","lead":"This paper proposes a way for an AI to learn several related tasks at once by storing all their value functions in a single low-rank tensor, which makes similar tasks share a small set of learned patterns. The authors test the method on balancing inverted pendulums and on wireless transmission scheduling, and report that it learns faster than training each task separately.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eqs. (6a)-(6c) give the semi-gradient with the wrong sign for minimizing the squared TD loss, so Algorithm 1 as written would maximize the loss and diverge; the experiments in Section IV therefore do not validate the central claim.","rationale":"The reader's stated weakest_assumption is the low-rank representability of the Q-tensor, which is never validated. However, the reader's rationale also identifies the sign error in Eqs. (6a)-(6c), and that is the most load-bearing concern: the central algorithm as written is internally inconsistent, so the numerical results cannot be reproduced from the manuscript. Re-deriving the gradient from Eq. (5b), the derivative of the squared TD loss with respect to Q(s,a) carries a negative sign; the factor Jacobian is positive, so the printed expressions have the wrong sign. Because Algorithm 1 subtracts these gradients, it performs ascent on the loss and should diverge. This is a correctness failure independent of the empirical claims. A corrected manuscript with the proper negative sign, disclosed hyperparameters (rank K, λ_m, learning rate schedule, ε), error bars, and a validation of the low-rank assumption against Monte Carlo Q-estimates could support a revised claim, but the current submission does not. The reader's REJECT verdict remains appropriate; hence verdict_should_be is UNCHANGED.","tokens_in":9454,"tokens_out":3823,"duration_ms":39996,"concrete_test":"Implement Algorithm 1 exactly as printed, with the positive signs in Eqs. (6a)-(6c), on a small tabular MDP (e.g., a 2-state, 2-action chain with known optimal Q) and record the batch TD loss over episodes. If the loss does not decrease, the sign is wrong. Then rerun with the signs flipped to negative and verify the loss decreases. As a control, inspect the code in the cited GitHub repository to determine which sign is actually used in the reported experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III, Eqs. (6a)-(6c): the objective in Eq. (5b) is a sum of squared TD errors, with TD error δ = r + γ max_a Q(s',a,m) − Q(s,a,m). Minimizing this by stochastic semi-gradient descent requires the update direction −2λ_m δ ∇_Θ Q(s,a), ignoring the derivative of the max term. Eq. (6a) instead reports +2λ_m δ [Q2]_{at,k}[Q3]_{m,k} I{is=ist}, and Eqs. (6b) and (6c) have the same positive sign. Since Algorithm 1 subtracts these quantities in Eqs. (9a)-(9c), the printed rule is gradient ascent on the squared TD error: the parameters move in the direction that increases the loss, so the Q-estimates drift away from targets and training should diverge on any nontrivial task. This is an internal algebraic inconsistency, not a disagreement with any external consensus. If the authors' code uses the corrected negative sign, the paper has a transcription error in all three core gradient equations; but as submitted, the algorithm cannot be reproduced from the text, and the empirical claims in Section IV are not supported by the presented method. Secondary issues—rank K and λ_m never reported, no error bars, and the low-rank assumption in Eq. (7) never checked against true Q-values—would remain even after the sign fix.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-task reinforcement learning method in which the Q-functions of M related tabular MDPs are stacked into a third-order tensor and approximated by a low-rank PARAFAC decomposition of rank K. The authors formulate a joint squared Bellman-error objective over the tensor factors, propose a stochastic semi-gradient block-coordinate algorithm called S-TLR-Q, and report two experiments (a four-task inverted-pendulum benchmark and a four-task wireless scheduling setup) comparing against independent and fully shared low-rank baselines. The central claim is that the low-rank tensor structure shares statistical strength across tasks, yielding faster learning and competitive final rewards compared with learning each task separately or forcing a single shared Q-function.","tokens_in":9741,"tokens_out":6164,"duration_ms":66845,"significance":"If the algorithm worked as claimed, the paper would offer a simple and interpretable way to exploit task relatedness in tabular multi-task RL: low-rank tensor factorization reduces the number of parameters from |S||A|M to (|S|+|A|+M)K and provides a latent representation of task similarity without explicit similarity constraints. The authors also provide a public code repository, which is a positive reproducibility feature. However, the current manuscript does not establish these claims: the core update equations are wrong as printed, the experiments omit essential hyperparameters and error bars, and the central low-rank assumption is never validated against true Q-values.","major_comments":[{"comment":"The printed update direction is inconsistent with minimization of the objective in Eq. (5b). For a transition with TD error delta = r + gamma max_a Q(s',a,m) - Q(s,a,m), the loss contribution is (target - Q)^2, and its semi-gradient with respect to the factors is -2 lambda_m delta dQ/dtheta. Stochastic gradient descent therefore requires adding +2 lambda_m eta delta dQ/dtheta to the parameters. Equations (6a)-(6c) report the positive expression +2 lambda_m delta (...), and Algorithm 1 subtracts these quantities in Eqs. (9a)-(9c), giving an increment of -2 lambda_m eta delta (...). This is gradient ascent on the squared Bellman error and would be expected to diverge on any nontrivial problem. As submitted, the algorithm cannot be reproduced from the text, and the experiments in Section IV cannot be taken as validation of the proposed method; the equations need a sign correction and a consistent convention with Eq. (3).","section":"III, Eqs. (6a)-(6c) and (9a)-(9c)"},{"comment":"The empirical evaluation is incomplete in load-bearing ways. The paper never reports the rank K, the task weights lambda_m, the learning-rate schedule eta(n), or the exploration probability epsilon used in the experiments, so the results in Fig. 2 cannot be reproduced or properly compared. The curves are averaged over 100 experiments, but no error bars, confidence intervals, or variance statistics are shown, and the claim that S-TLR-Q converges faster is not supported by a defined convergence criterion or a statistical test. These omissions are particularly important because the comparison is empirical and the baselines are only described by reference to the authors' earlier work [21].","section":"IV"},{"comment":"The central modeling assumption, that the multi-task Q-tensor is well approximated by a small-rank PARAFAC decomposition, is never validated. Equation (7) presumes Q(i_s,i_a,m) is close to sum_k [Q1]_{i_s,k}[Q2]_{i_a,k}[Q3]_{m,k}, but no experiment compares the learned low-rank Q with ground-truth Q-values or reports the approximation error as a function of K. Without such evidence, the claimed sample-efficiency gains could be specific to the chosen benchmark tasks, and the suitability of the model for the wireless scenario remains unsupported.","section":"II-A and IV"}],"minor_comments":[{"comment":"In the tensor constraint, the factor [Q3]_{i1,k} should be [Q3]_{i3,k}; otherwise the decomposition does not index the task mode correctly.","section":"II, Eq. (5a)"},{"comment":"Line 8 says 'Update [Q(n+1)_m]m,:' but should update the task-factor matrix Q3; additionally, lines 7-8 refer to sigma_l^m where the surrounding loop uses sigma_t^m.","section":"III, Algorithm 1"},{"comment":"The text says that the only hyperparameter is the rank K, but eta(n), epsilon, and lambda_m are also hyperparameters that must be set; this statement should be corrected.","section":"II-B and IV"},{"comment":"The figure panels lack axis labels, and the text does not define whether the reported 'average return' is the discounted cumulative reward or the undiscounted sum over the test episode.","section":"IV, Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The sign error in Eqs. (6) and (9) is likely a transcription issue rather than a conceptual impossibility, and it can be fixed in revision. However, as submitted it invalidates the presented algorithm, and the missing hyperparameters and lack of validation of the low-rank assumption are substantial. The paper also depends heavily on the authors' own prior work for its baselines, so the editor may wish to consider fit with the journal's broader readership."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is reasonable: stack the tabular Q-matrices of M tasks into a 3-way tensor and impose a PARAFAC low-rank structure, with one factor matrix for tasks. This is a natural extension of the authors' earlier low-rank RL work, and the writing is mostly clear. The update equations are a different story. For the squared TD loss in (5b), the semi-gradient in (6) has the wrong sign. Minimizing that loss requires updating parameters in the direction +2λ_m δ ∇Q, not −2λ_m δ ∇Q. Since Algorithm 1 subtracts the (6) quantities, the printed algorithm maximizes the squared TD error and should diverge. That is not a minor typo; it is in all three core equations and it means the experiments in Section IV, as described, do not support the claims. The code may well use the corrected sign, in which case this is a transcription error, but the paper as submitted is not reproducible.\n\nBeyond the sign, the empirical support is thin: rank K, learning rates, and λ_m are never reported; the plots have no error bars; and the only baselines are the authors' own LR-Q and C-LR-Q. The low-rank assumption itself is never checked against true Q-values, so we do not know whether the claimed sample-efficiency gains come from the assumption or from something else. The paper also says the only hyperparameter is K, which is not true (η, ε, λ_m are all free). What the paper does well: the tensor model is simple and interpretable, the task-factor viewpoint is a sensible way to share information, and the two environments (pendulum, wireless) are at least different in flavor. But a plausible idea with a wrong sign in the core algorithm and missing experimental details is still a desk-reject level issue in its current form. If the authors correct the sign, report the hyperparameters, add error bars, and make the code available, this could be a modest but useful workshop paper. As it stands, I would not send it out for review; the central algorithm as written is internally inconsistent. For a reading group, it might be worth a quick look as a cautionary example of how sign errors in semi-gradient RL can slip through, but I would not cite it in its current form.","headline":"Plausible tensor low-rank multi-task idea, but the core update has a sign error that makes the printed algorithm divergent, so the experiments do not validate the claims.","tokens_in":10294,"tokens_out":2769,"would_cite":false,"duration_ms":27026,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-rank tensor factorization of the multi-task Q-function yields faster learning with fewer samples in two tabular RL tasks.","keywords":["multi-task reinforcement learning","low-rank tensor approximation","PARAFAC decomposition","value function approximation","tabular reinforcement learning","sample efficiency","tensor factorization","Q-learning"],"falsifier":"Build a family of M related tabular tasks whose true Q-functions are known or estimable to high accuracy and whose joint Q-tensor has high PARAFAC rank; run S-TLR-Q at the paper's rank K and measure both the normalized tensor reconstruction error and the early-training return gap against independent per-task learning. If the reconstruction error is large and the early-training advantage over independent learning vanishes, the low-rank assumption is the mechanism and its stated benefit is confined to task families that satisfy it.","tokens_in":9233,"feed_emoji":"🤖","tokens_out":7566,"duration_ms":68027,"temperature":0.7,"pith_summary":"The paper proposes to learn the state-action value functions of several related reinforcement-learning tasks at once by stacking them into a single tensor — one mode for states, one for actions, one for tasks — and forcing that tensor to have small PARAFAC rank. The claim is that this low-rank constraint encodes task similarity without anyone prescribing which tasks resemble which, and that a stochastic block-coordinate algorithm over the tensor factors lets every task borrow data from every other task. In two small tabular experiments, an inverted-pendulum family and a wireless transmission scheduling problem, the authors report faster convergence and higher early-training rewards than learning each task independently, while beating a baseline that forces all tasks to share one value function. If true, this gives a parameter-lean route to multi-task RL in data-scarce regimes.","feed_headline":"Shared low-rank Q-tensor cuts samples in multi-task RL","feed_subtitle":"Stacking related tasks' value functions into one low-rank tensor lets each task learn from all tasks' data.","key_machinery":"The load-bearing object is the third-order Q-tensor with PARAFAC structure, $\\mathcal{Q} \\in \\mathbb{R}^{|S|\\times|A|\\times M}$, whose rank-$K$ factorization $\\mathcal{Q}(i_s,i_a,m)=\\sum_k [\\mathbf{Q}_1]_{i_s,k}[\\mathbf{Q}_2]_{i_a,k}[\\mathbf{Q}_3]_{m,k}$ collapses the degrees of freedom from $|S||A|M$ to $(|S|+|A|+M)K$. This factorization is the mechanism that turns multi-task sharing into a model: a transition sampled from any task updates the shared state and action factors, so information flows into every task's value function, while the task factor $\\mathbf{Q}_3$ keeps the tasks distinct. The accompanying algorithm S-TLR-Q updates the three factor matrices with a block-coordinate stochastic step, treating the max over actions in the Bellman target as a fixed quantity, which is the standard semi-gradient maneuver in value-based RL.","core_discovery":"The paper's central discovery is that a collection of tabular Q-functions can be modeled as a three-way tensor $\\mathcal{Q}(i_s,i_a,m)$ and approximated by a low-rank PARAFAC form $\\sum_{k=1}^{K}[\\mathbf{Q}_1]_{i_s,k}[\\mathbf{Q}_2]_{i_a,k}[\\mathbf{Q}_3]_{m,k}$, where the three factor matrices are shared across states, actions, and tasks. Each task's Q-matrix becomes a weighted sum of $K$ common rank-one state-action layers, with the task's row of $\\mathbf{Q}_3$ supplying the weights, so task similarity is inferred from data rather than imposed as a constraint. The authors derive block-coordinate stochastic semi-gradient updates for the factors, collect them in an algorithm they call S-TLR-Q, and report that it reaches the final performance of independent per-task Q-learning in both experiments while needing noticeably fewer samples; the fully shared baseline converges faster but settles at a worse solution.","pith_inferences":["Beyond the paper's tests, the factorization suggests a built-in transfer mechanism the authors do not exercise: hold out one task's Q-tensor slice, learn factors on the remaining tasks, and treat the fitted task rows as a basis for predicting the held-out task's value function; observing small prediction error would establish few-shot adaptation to a new related task.","The paper never validates the rank-$K$ assumption against true Q-values, so a natural diagnostic is to compute an oracle $Q^*$ on a family of small tabular tasks and plot normalized PARAFAC reconstruction error versus $K$; that curve would separate genuine multi-task structure from the bias of a compact model.","The same PARAFAC-sharing prior should apply to any multi-task regression with partially shared latent factors, making the method a general interpolation between full pooling and independent fits; this generalization is an inference from the mechanism, not a claim the paper states."],"forward_implications":["If the low-rank tensor model is correct, the parameter count for a multi-task problem drops from $|S||A|M$ to $(|S|+|A|+M)K$, so sample complexity should scale with the rank rather than with the full state-action-task grid.","Task similarity needs no explicit graph, clustering, or distance constraint; the factor matrices encode which tasks align in latent structure, and any transition from any task updates all tasks' value functions.","In data-scarce regimes, S-TLR-Q should dominate independent per-task learning on final accuracy and dominate fully shared learning whenever tasks differ in ways that are structurally aligned.","In the two tested environments, the same rank-$K$ representation achieves per-task final rewards comparable to independent learning while converging faster in the early stages of training.","Higher-order versions for multi-dimensional state and action vectors should yield even larger savings, because the rank-one outer-product form keeps the parameter count linear in the number of modes."],"supporting_citations":[{"why":"Defines PARAFAC rank and the rank-one outer-product decomposition on which the Q-tensor model is built.","marker":"[28]"},{"why":"Source of the LR-Q and C-LR-Q baselines and of the prior low-rank value-function approximation this work extends.","marker":"[21]"},{"why":"The Q-learning update whose Bellman target the semi-gradient updates are built from.","marker":"[23]"},{"why":"Supplies the standard justification for treating the max operator with a semi-gradient in parametric value-function approximation.","marker":"[29]"},{"why":"Provides the inverted-pendulum environment used in the first experiment.","marker":"[30]"},{"why":"Supports the premise that low-rank structure is prevalent in RL value functions.","marker":"[22]"},{"why":"Earlier evidence that low-rank value-function models help in RL, motivating the tensor formulation.","marker":"[20]"}],"fun_headline_variants":["Low-rank Q-tensor shares data across RL tasks","Infer task similarity with low-rank Q-tensor","Sample-efficient multi-task RL via tensor low-rank","Tensor low-rank lets tasks learn from each other","Multi-task RL needs fewer samples with low-rank Q"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Q-functions of the M tasks really are close to a small-rank PARAFAC tensor; if related tasks fail to share a common low-rank latent structure, the compact factorization cannot represent the true value functions, and the reported sample-efficiency gains should disappear.","fun_headline_variants_meta":{"raw":{"variants":["Low-rank Q-tensor shares data across RL tasks","Infer task similarity with low-rank Q-tensor","Sample-efficient multi-task RL via tensor low-rank","Tensor low-rank lets tasks learn from each other","Multi-task RL needs fewer samples with low-rank Q"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000935,"raw_usage":{"total_tokens":3981,"prompt_tokens":909,"completion_tokens":3072,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2998}},"tokens_in":525,"tokens_out":3072,"duration_ms":24839,"temperature":1.0,"reasoning_tokens":2998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:09:26.846537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a family of M related tabular tasks whose true Q-functions are known or estimable to high accuracy and whose joint Q-tensor has high PARAFAC rank; run S-TLR-Q at the paper's rank K and measure both the normalized tensor reconstruction error and the early-training return gap against independent per-task learning. If the reconstruction error is large and the early-training advantage over independent learning vanishes, the low-rank assumption is the mechanism and its stated benefit is confined to task families that satisfy it.","supporting_citations":[{"cited_title":"Tensor decomposition for signal processing and machine learning,","cited_arxiv_id":null,"evidence_quote":"Defines PARAFAC rank and the rank-one outer-product decomposition on which the Q-tensor model is built."},{"cited_title":"Tensor and matrix low- rank value-function approximation in reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Source of the LR-Q and C-LR-Q baselines and of the prior low-rank value-function approximation this work extends."},{"cited_title":"Algorithmic survey of parametric value function approximation,","cited_arxiv_id":null,"evidence_quote":"Supplies the standard justification for treating the max operator with a semi-gradient in parametric value-function approximation."},{"cited_title":"Tensor low-rank approximation of finite- horizon value functions,","cited_arxiv_id":null,"evidence_quote":"Supports the premise that low-rank structure is prevalent in RL value functions."},{"cited_title":"Matrix low-rank approximation for policy gradient methods,","cited_arxiv_id":null,"evidence_quote":"Earlier evidence that low-rank value-function models help in RL, motivating the tensor formulation."}],"review_version":1}