{"id":"2f1a63a0-ab9f-456c-947d-28ab9292cc8d","arxiv_id":"2507.16983","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Adding general-value-function predictions of future lower-limb sensor signals improved a policy network's terrain classification accuracy during simulated online learning.","lead":"Researchers tested whether predictions of future sensor signals, learned by general value functions, help a neural network classify walking terrain from body-worn sensors, and found that they improve accuracy on several terrains. The work is a step toward exoskeletons that adapt their control automatically to ramps, stairs, and uneven ground.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No capacity- or input-count-matched control; the observed GVF accuracy gains are confounded with added features/parameters, so the causal claim that predictive information improves accuracy is not established.","rationale":"The reader's weakest assumption correctly identifies the missing control for input dimensionality and network capacity as the load-bearing issue. The paper's empirical result is only that two GVF-augmented networks outperform a smaller, lower-capacity control; this does not establish that the predictive content of GVFs, rather than the extra features or parameters, caused the improvement. A matched baseline is the minimal check needed to support the Abstract's causal phrasing. The concern is addressable and does not invalidate the collected data or the statistical comparisons; it undermines the interpretation. Therefore the reader's CONDITIONAL verdict remains appropriate, with the condition being the addition of capacity- and input-count-matched controls (ideally including a time-delayed feature control to match information content). The concrete test proposed would settle whether the concern lands: if a noise or delayed-feature control reproduces the gains, the GVF-specific claim fails; if it does not, the claim is supported.","tokens_in":5591,"tokens_out":4407,"duration_ms":50674,"concrete_test":"Run the same training and evaluation pipeline with two additional control conditions, keeping all hyperparameters fixed: (a) Control-Noise: the original 30 signals plus 30 independent white-noise features, so input dimension equals the Input GVF net (60). (b) Control-Delay: the original 30 signals plus the same signals delayed by 17 timesteps, matching input dimension and providing a non-learned temporal feature comparable in information content to a GVF prediction. Apply the same Kruskal-Wallis/Dunn procedure to compare end-of-training accuracies. If either matched control achieves accuracy not significantly different from the Input GVF Policy Net (and both remain significantly above the 30-input Policy Net), the claim that GVF predictions specifically drive the improvement is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'the addition of predictions made from GVFs increased overall network accuracy' requires isolating the effect of predictive content from the effect of additional input dimensions and model capacity. The control Policy Net uses 30 sensor inputs; the Input GVF Policy Net uses 60 inputs (30 actual plus 30 GVF predictions); the Latent GVF Policy Net appends predictions after an encoding layer, increasing parameters through a parallel pathway. No baseline matches total input count or capacity while withholding the predictive semantics. This matters because walking signals are quasi-periodic: a GVF prediction roughly 0.5 s ahead carries phase information similar to a time-delayed copy of the same signals. Thus the accuracy gains (66.9% vs. 73.4% and 72.0%) could arise simply from providing more features or more model capacity, rather than from the predictive nature of GVFs. The paper's Section 2 defines the control net without GVF inputs and never introduces a feature-count-matched or capacity-matched comparator, so the mechanistic conclusion stated in the Abstract and Conclusion is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hierarchical reinforcement learning (HRL) pipeline for adaptive exoskeleton control, in which lower-limb sensor signals (EMG, goniometer, pressure insoles) are used to learn general value functions (GVFs) that predict signal values roughly 0.5 s into the future. These GVF predictions are then integrated into a terrain-classification policy network in one of two ways: appended to the raw input stream or injected into a latent layer. Using data from ten participants walking over seven terrains, the authors compare these two GVF-augmented networks against a control policy net that receives only raw sensor signals. They report that both GVF-augmented networks achieve significantly higher end-of-training classification accuracy (73.4% and 72.0%) than the control (66.9%), with particularly large gains on down-ramp classification, and they conclude that predictive GVF information improves decision-making during terrain uncertainty.","tokens_in":5832,"tokens_out":4358,"duration_ms":47243,"significance":"If the central claim is valid, the paper provides a useful demonstration that learned temporal abstractions can improve online terrain classification from wearable sensor data, which is a plausible step toward adaptive lower-limb exoskeleton control. The work has concrete strengths: it uses human gait data, implements continual learning with a replay buffer, and reports per-terrain statistical comparisons. However, the empirical comparison does not isolate the predictive content of GVFs from the effects of added input dimensions and network capacity, and the evaluation appears to be confined to the training data stream with no held-out protocol. These issues must be resolved before the mechanistic conclusion that GVF predictions themselves improve accuracy can be accepted.","major_comments":[{"comment":"The central comparison is confounded by input dimensionality and model capacity. The control Policy Net uses 30 raw sensor inputs, the Input GVF Policy Net uses 60 inputs (30 raw plus 30 GVF predictions), and the Latent GVF Policy Net adds an additional pathway after the encoding layer, increasing the number of parameters in subsequent layers. No control condition matches the input count or capacity while withholding the predictive semantics of GVFs. The reported significant accuracy differences (p = 0.001 and p = 0.01) could therefore arise simply from providing more features or more model capacity. A control using 30 time-delayed copies of the raw signals, or a larger-capacity network without GVF inputs, would directly address this concern. This is load-bearing because the Abstract and Conclusion attribute the improvement specifically to 'the addition of predictions made from GVFs.'","section":"Section 2 (Policy Network for Terrain Control Decisions) and Section 3 (Table 1)"},{"comment":"All reported accuracies appear to be end-of-training values computed on the same data stream used to train the policy networks. The paper does not describe any held-out evaluation, such as a per-subject holdout, a temporal train/test split, or cross-validation. Without such a protocol, the accuracy improvements may reflect the networks' ability to memorize the training stream rather than generalize to unseen walking data. The manuscript should specify the evaluation protocol explicitly and report at least one held-out metric, even if the authors intend the paper as a demonstration of online learning rather than generalization.","section":"Section 3 (Figure 2 and Table 1)"},{"comment":"The statistical analysis lacks a clear statement of the unit of analysis and the number of independent samples used in the Kruskal-Wallis and Dunn tests. It is unclear whether the reported accuracies are per-participant end-of-training values (n = 10), per-gait-cycle values, or pooled timestep-level results. The number of training runs per participant, random seeds, and whether the same data are reused across networks are also not stated. This matters because the validity of the p-values depends on the independence and sample size of the accuracies being compared.","section":"Section 2 (Statistics) and Section 3"},{"comment":"The paper frames the work as an HRL framework for adaptive exoskeleton control, but the experiments stop at terrain classification; no exoskeleton control task, actuator policy, or closed-loop user study is presented. The abstract's statement that the goal is 'adaptive control strategies for lower-limb exoskeletons' and Figure 1's inclusion of an exoskeleton block overstate the demonstrated scope. This is acceptable as a preliminary study if the claims are softened or if a limitations paragraph explicitly states that exoskeleton control is future work, but as written the title and framing promise more than the experiments deliver.","section":"Section 1, Section 2, and Figure 1"}],"minor_comments":[{"comment":"There are typographical spacing errors in 'General V alue Functions' (title) and 'T rue Online Temporal-difference Learning' (Section 2 heading). These should be corrected to 'Value' and 'True'.","section":"Title and Section 2 headings"},{"comment":"The comment 'Offest (m - 1) * K' should read 'Offset'; more importantly, the intended indexing of the active prototype features is unclear and should be clarified in the text or pseudocode.","section":"Algorithm 1"},{"comment":"The equation for the number of prediction timesteps is written as 'timesteps = 1 1−γ', which is missing an explicit division and is visually ambiguous. It should be typeset as 1/(1−γ).","section":"Section 2 (GVFs)"},{"comment":"The phrase 'filtered with a 2nd-order low-pass filter at 5 Hz' should specify the filter type (e.g., Butterworth) and, ideally, the reason for the 5 Hz cutoff, to aid reproducibility.","section":"Section 2 (data preprocessing)"},{"comment":"The convergence curves are shown without a caption describing whether they are averaged over participants, how many runs are included, and what the shaded regions represent. Adding this information would make the figure interpretable.","section":"Figure 2"},{"comment":"The total number of training timesteps or epochs and the number of replay-buffer updates are not specified; adding these details would allow readers to understand the duration of the online learning process and the convergence behavior.","section":"Section 2 (Policy Network)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and addresses a timely topic, but the main empirical claim needs stronger experimental support before publication. The lack of a capacity-matched control and a held-out evaluation are the key blockers. I am not concerned about the citation pattern; the self-citations to prior GVF work are appropriate given the niche topic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on GVF-based assistive devices, but keep the real story in view. The genuinely new piece is the systematic comparison between input-level and latent-level GVF injection for terrain classification on seven terrains, using real wearable sensor data from ten walkers. The per-terrain results are reported cleanly, with appropriate nonparametric tests and honest caveats about which differences are not significant. The authors also explicitly stop short of claiming input beats latent — that restraint is good.\n\nThe main soft spot is the one the stress-test flagged: the control net has 30 inputs, the Input GVF net has 60, and the Latent GVF net adds a parallel pathway with extra parameters. No capacity- or input-count-matched baseline exists. Because walking signals are quasi-periodic, a GVF prediction ~0.5 s ahead can carry much of the same information as a time-delayed copy of the current signal. So the accuracy gains (66.9% vs 73.4% and 72.0%) may just reflect more features or more parameters, not the predictive semantics. This is the central mechanistic claim in the abstract, and it is underdetermined. That flaw is fixable: add a baseline that receives the same extra inputs but with the predictive ordering scrambled, and a capacity-matched control.\n\nSecond, all accuracies appear to be end-of-training values on the training stream itself. That is a legitimate online-learning metric, but it is not out-of-sample generalization. A held-out participant or session split would make the practical relevance far stronger. Minor related point: the title and abstract say \"adaptive walking control\" for exoskeletons, but no exoskeleton is involved; the actual contribution is terrain classification. That framing should be tightened.\n\nThe statistical reporting and the data collection protocol are sound enough. I would send this to a serious referee, mainly to force the capacity-matched baseline and a held-out evaluation. With those additions it becomes a solid incremental paper in the Pilarski line of work. My own verdict is skeptical on the causal interpretation, but the empirical comparison is genuine and worth engaging with.","headline":"A modest empirical comparison of where to inject GVF predictions into a terrain classifier, but the headline claim about predictive information is confounded by unmatched input width and capacity, and the title promise of walking control is untested.","tokens_in":6360,"tokens_out":1205,"would_cite":false,"duration_ms":14954,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feeding a terrain classifier forecasts of future sensor signals, learned online by general value functions, improves its accuracy across walking surfaces, with the largest gains on the most error-prone terrains.","keywords":["hierarchical reinforcement learning","general value functions","exoskeleton control","terrain classification","continual deep learning","wearable sensors","temporal-difference learning","policy network"],"falsifier":"Train the same policy network with the GVF output streams replaced one-for-one by non-predictive signals (fixed or shuffled values) while keeping identical input size and parameter count; if such a control matches the 73.4% and 72.0% accuracies, the specific contribution of predictive content would be disproved.","tokens_in":5396,"feed_emoji":"🦿","tokens_out":5590,"duration_ms":50627,"temperature":0.7,"pith_summary":"This paper tests whether giving a terrain-classifying controller a preview of future lower-limb sensor signals helps it decide what ground a person is walking on. In a two-level reinforcement learning setup, general value functions learned online from electromyography, pressure-insole, and goniometer signals produced temporally abstracted predictions, which were then fed either into the input layer or the latent layer of a policy network. Across ten participants walking on seven terrains, both GVF-augmented networks finished training with higher overall accuracy than a control network without predictions (73.4% and 72.0% versus 66.9%). The biggest gains appeared on exactly the terrains that were most often misclassified, such as down ramps, which improved by up to 13.5 percentage points. The authors take this as evidence that predictive information supports decision-making under uncertainty in exoskeleton control.","feed_headline":"Predicting future sensor readings boosts exoskeleton terrain accuracy","feed_subtitle":"Learned forecasts helped controllers sort ramps, uneven ground, and turns, lifting accuracy by up to 13.5 points.","key_machinery":"The machinery is a two-level hierarchy built around general value functions (GVFs), which are value functions whose cumulant can be any sensor signal rather than a reward. A fast predictive state uses Selective Kanerva Coding to turn the 30 sensor channels into sparse binary prototype activations, and true online temporal-difference learning (TOTD) updates GVFs that predict the cumulative discounted value of each signal about 0.5 seconds ahead (17 timesteps at $\\gamma = 0.94$). A slower policy network for terrain decisions is trained continually with a replay buffer, and receives these GVF predictions either at its input or in its latent space. The GVF outputs are the information-carrying component being tested; the policy network is what converts actual plus predicted signals into a terrain label.","core_discovery":"The central claim is that learned predictions of future sensor values, not just current sensor values, improve a policy network's ability to classify walking terrain during simulated online learning. The paper shows this by adding general value function outputs to a continual deep learning policy network in two ways: concatenated with the actual sensor signals at the input, and injected after the encoding layers in the latent space. Both additions raised end-of-training accuracy significantly relative to the no-GVF control (Input GVF vs. Policy Net: $p = 0.001$; Latent GVF vs. Policy Net: $p = 0.01$), while the two integration sites were not statistically distinguishable ($p = 0.42$). The authors interpret the terrain-specific gains, especially the 13.5-point improvement on down ramps, as predictive information helping when the classifier is most uncertain.","pith_inferences":["A testable extension not run in the paper would be a capacity-matched and input-count-matched baseline; without it, part of the gain may be attributable to the larger effective feature space rather than prediction content.","The predictive-information benefit is probably strongest when the classifier's uncertainty is high; one could test this by correlating per-timestep prediction error with misclassification rate.","The same GVF-plus-policy architecture could be applied to upper-limb prosthetics or gait-phase detection, where anticipating the next sensor state is similarly useful.","Future adaptive controllers could use the GVF prediction error itself as a signal for when the terrain has changed, triggering a strategy switch before misclassification occurs."],"forward_implications":["A GVF-augmented policy net can be paired with an exoskeleton controller to choose terrain-appropriate walking strategies without manual mode switching.","Terrain-specific gains on ramps, uneven ground, even ground, and turns suggest predictive information is most valuable exactly where classifiers are most error-prone.","Since input and latent integration performed similarly overall, the simpler input-concatenation design may be sufficient for exoskeleton control, pending further analysis.","Continual replay-based training combined with online GVF learning forms a plausible architecture for controllers that adapt as a user encounters new environments."],"supporting_citations":[{"why":"Supplies the hierarchical RL design in which a higher-level agent learns from GVF-style predictions.","marker":"[4]"},{"why":"Established that GVF predictions can act as inputs to a policy network, the technique this paper extends.","marker":"[5]"},{"why":"Motivates injecting predictions into latent layers of a deep network, one of the two integration methods tested.","marker":"[6]"},{"why":"Provides the experience-replay sampling strategy used to train the continual policy network.","marker":"[7]"},{"why":"Introduces Selective Kanerva Coding for representing high-dimensional sensor states to wearable assistive robots.","marker":"[8]"},{"why":"Supports the use of coarse and fine prototype activations as state features in similar walking-control settings.","marker":"[9]"},{"why":"Defines GVFs as value functions of arbitrary cumulants, the basis for predicting sensor signals.","marker":"[12]"},{"why":"Supplies the true online temporal-difference algorithm that updates the GVFs online.","marker":"[14]"}],"fun_headline_variants":["Exoskeleton AI uses predictions to master ramps, turns, uneven ground","Predicting sensor signals helps exoskeleton read terrain accurately","AI forecasts next steps to improve walking on tricky terrains","Hierarchical learning uses predictive signals to power exoskeleton control","Predictive sensors boost exoskeleton accuracy by 13.5 points on ramps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the accuracy gains come from the predictive meaning of the GVF signals, not from giving the network more inputs or more parameters, since the control network had fewer inputs and the GVF networks had extra capacity without a matched control.","fun_headline_variants_meta":{"raw":{"variants":["Exoskeleton AI uses predictions to master ramps, turns, uneven ground","Predicting sensor signals helps exoskeleton read terrain accurately","AI forecasts next steps to improve walking on tricky terrains","Hierarchical learning uses predictive signals to power exoskeleton control","Predictive sensors boost exoskeleton accuracy by 13.5 points on ramps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001357,"raw_usage":{"total_tokens":5546,"prompt_tokens":1020,"completion_tokens":4526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":4434}},"tokens_in":636,"tokens_out":4526,"duration_ms":32527,"temperature":1.0,"reasoning_tokens":4434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:58:41.982919+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same policy network with the GVF output streams replaced one-for-one by non-predictive signals (fixed or shuffled values) while keeping identical input size and parameter count; if such a control matches the 73.4% and 72.0% accuracies, the specific contribution of predictive content would be disproved.","supporting_citations":[{"cited_title":"Learning how to Interact with a Complex Interface using Hierarchical Reinforcement Learning","cited_arxiv_id":"2204.10374","evidence_quote":"Supplies the hierarchical RL design in which a higher-level agent learns from GVF-style predictions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Established that GVF predictions can act as inputs to a policy network, the technique this paper extends."},{"cited_title":"Gamma-Nets: Generalizing value estimation over timescale,","cited_arxiv_id":null,"evidence_quote":"Motivates injecting predictions into latent layers of a deep network, one of the two integration methods tested."},{"cited_title":"Experience replay for continual learning,","cited_arxiv_id":null,"evidence_quote":"Provides the experience-replay sampling strategy used to train the continual policy network."},{"cited_title":"Representing high-dimensional data to intelligent prostheses and other wearable assistive robots: A first comparison of tile coding and selective Kanerva coding,","cited_arxiv_id":null,"evidence_quote":"Introduces Selective Kanerva Coding for representing high-dimensional sensor states to wearable assistive robots."},{"cited_title":"Pavlovian control of intraspinal microstimulation to produce over-ground walking,","cited_arxiv_id":null,"evidence_quote":"Supports the use of coarse and fine prototype activations as state features in similar walking-control settings."},{"cited_title":"Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction,","cited_arxiv_id":null,"evidence_quote":"Defines GVFs as value functions of arbitrary cumulants, the basis for predicting sensor signals."},{"cited_title":"True online temporal-difference learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the true online temporal-difference algorithm that updates the GVFs online."}],"review_version":1}