{"id":"0e3a9358-9c77-4ef8-b994-11ff15fdc71b","arxiv_id":"2608.10829","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An online-trained PPO controller with a finite return horizon aligned to convective time stabilizes a low-duty-cycle actuation pattern and yields a forward-flow fraction of about 53% at the feedback sensor, about one percentage point above optimized periodic open-loop control.","lead":"Researchers trained a reinforcement-learning agent live in a turbulent wind tunnel to control flow separation on a ramp, using one wall-shear-stress sensor and simple on/off air valves. The agent learned faster and steadier when rewards were summed over a short window matched to the flow response time, and it slightly outperformed optimized periodic blowing at that one sensor.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The finite-horizon agent differs from the infinite-horizon agent in both horizon and discount factor (Eqs. 4-5, Table 1); the claimed causal role of convective-time alignment is therefore not isolated.","rationale":"The reader's weakest assumption is exactly the confound I identify: the comparison changes both the return horizon and the discount factor. This is the most load-bearing issue because the paper's distinctive contribution is the claim that a horizon aligned with convective time improves online learning; if that is not isolated, the main design conclusion is unsupported, even though the feasibility of online DRL in a turbulent wind tunnel remains demonstrated. The paper itself is careful to localize the claimed benefit to the feedback sensor and to note that training-time comparisons should be interpreted cautiously, which supports a conditional acceptance rather than rejection. The proposed concrete test is feasible in the same facility and would settle the causal question directly. I do not see a stronger objection: the experimental demonstration, reproducible code repository, and the qualitative stability difference between the two return formulations are real evidence, and the remaining issues are addressable with additional control conditions and uncertainty quantification.","tokens_in":12485,"tokens_out":6448,"duration_ms":71462,"concrete_test":"Run two additional agents in the same wind-tunnel configuration: (i) finite-horizon return with N_c=4 but gamma=0.995 instead of 1; (ii) full-trajectory return with gamma=1 instead of 0.995. For each, use the same hyperparameters as Table 1 and repeat at least 3 times; record the episode at which the smoothed mean gamma+ at RS4 first exceeds 0.52 and the variance of the mean duty cycle over episodes 100-500. If agent (i) converges as fast and as stably as the reported finite-horizon agent, and agent (ii) remains as slow and variable as the reported infinite-horizon agent, then horizon truncation is the causal factor. If the behavior follows gamma rather than horizon, the central claim must be weakened to 'discounting/truncation jointly improve online learning,' and the paper should be revised accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central methodological claim is that a return horizon matched to the convective response time (N_c=4, Eq. 5) causes faster convergence and a more stable policy than the standard discounted full-trajectory return (Eq. 4). However, the two agents are not compared at constant discounting: the finite-horizon agent uses gamma=1 while the infinite-horizon agent uses gamma=0.995 (Table 1). These two changes affect PPO's advantage estimates in different ways: truncation shortens the credit-assignment interval, while gamma=1 removes the exponential down-weighting of later rewards and changes the scale and variance of returns and the critic target. With gamma=0.995, a reward 24 ms after an action is still weighted 0.98, so the 'standard' baseline also emphasizes near-term rewards; the main difference is the long tail (a reward 1 s later has weight ~0.44). The observed improvement in learning stability (Sec. III.A) could therefore be due to removing the noisy long tail, to removing discounting altogether, or to the interaction of both, rather than specifically to aligning the horizon with the convective time. The paper's explanation in terms of a 'shorter local reward observation window' is plausible but untested. The performance comparison to optimized periodic open-loop control is a secondary concern: since the reward is the sign of the same RS4 signal used to define gamma+ (Eq. 2, Sec. III.B), the closed-loop controller is trained on the metric against which it is later judged, so the 1% margin is not an independent validation of control quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an experimental implementation of Proximal Policy Optimization (PPO) for active separation control on a one-sided diffuser in a fully turbulent wind tunnel. The policy maps the voltage of a single wall-shear-stress sensor to a Bernoulli distribution over binary valve commands for three in-phase pulse-jet actuators. Two return formulations are compared: a discounted full-trajectory return (Eq. 4, gamma=0.995) and a truncated undiscounted return over four steps (Eq. 5, gamma=1), corresponding to 24 ms, which the authors relate to the convective response time of the flow. The finite-horizon agent is reported to converge in about 20 episodes to a low-duty-cycle policy, while the infinite-horizon agent continues to fluctuate; the deployed finite-horizon controller achieves a forward-flow fraction of about 0.53 at sensor RS4, roughly one percentage point above periodic open-loop control at a 12.5% duty cycle. The authors conclude that a return horizon matched to the flow response time improves the efficiency and stability of online learning.","tokens_in":12832,"tokens_out":6027,"duration_ms":67146,"significance":"If the results hold, the paper is a valuable demonstration that model-free DRL can be trained directly in a fully turbulent experiment with minimal sensing and actuation. The experimental effort is substantial, the setup is realistic, a public code repository is cited (Ref. [45]), and the comparison against open-loop control provides a useful reference point. The central feasibility claim—that PPO can identify a low-duty-cycle separation-control policy from online interactions with a turbulent wind-tunnel flow—is credible and supported by the reported data. The more specific methodological claim about the benefit of a convective-time-aligned return horizon is, however, not yet cleanly established because the two agents differ in both horizon length and discount factor, and the performance metric is the same sensor signal used in the reward. With additional experiments or a more modest framing, the paper could make a solid contribution to experimental DRL-based flow control.","major_comments":[{"comment":"The comparison between the two agents varies two quantities at once: the finite-horizon agent uses N_c=4 with gamma=1, whereas the infinite-horizon agent uses the full trajectory with gamma=0.995. With gamma=0.995, a reward 24 ms after an action is still weighted by approximately 0.98, so the practical difference between the two returns is mainly the removal of the long tail rather than a direct contrast between aligned and misaligned horizons. The faster convergence and stability of the finite-horizon agent could therefore be due to truncation, to the removal of discounting, or to an interaction of the two, rather than specifically to aligning the horizon with the convective time. The interpretation offered in Section III.A is plausible but untested; an additional experiment or analysis that changes one factor at a time (for example, gamma=0.995 with N_c=4, or gamma=1 with a full trajectory) is needed to support the causal claim.","section":"II.B, Eqs. (4)-(5), Table 1"},{"comment":"The reward is the sign of the RS4 sensor signal, and the reported performance metric gamma+ at RS4 is the fraction of positive samples of the same sensor. The DRL controller is therefore trained to maximize exactly the metric against which it is later compared with open-loop control. The roughly 1% advantage over periodic open-loop control in Fig. 7a is thus partly a statement about matching the reward to the benchmark, not an independent validation of closed-loop superiority. The authors should either benchmark against a metric not used in the reward (for example, gamma+ at the downstream sensors or an integrated quantity such as pressure recovery) or explicitly frame the result as an optimization of the RS4 metric rather than a general performance advantage.","section":"II.B, Eq. (2); III.B, Fig. 7"},{"comment":"The faster-convergence and stability claims are based on a single 500-episode run for each agent, with only two additional 100-episode repetition runs. Given the run-to-run variability typical of online turbulent-flow experiments, this is a thin basis for the quantitative convergence comparison. Reporting the mean and spread of the learning curves across several independent runs, or at least a clear statistical summary of the existing repeats at matching episode counts, would substantially strengthen the claim that the finite-horizon agent converges more reliably.","section":"III.A, Fig. 4"}],"minor_comments":[{"comment":"The entry 'Return upper limit' lists 'T' for the infinite-horizon agent and '4 steps' for the finite-horizon agent; since Eq. (4) uses the remaining trajectory length T-t, the label 'T' is misleading and should be rephrased as 'remaining trajectory'.","section":"Table 1"},{"comment":"The phrase 'more robust control strategy' is stronger than what the data support, given the limited number of independent runs; consider 'more stable in the tested runs' or similar.","section":"Abstract"},{"comment":"The statement that 'both agents have 50% of all visited states lying within the same narrow interval around zero' is ambiguous because the intervals are specific to each agent; please report the numerical 50% and 90% intervals for each agent.","section":"III.A, Fig. 6"},{"comment":"The claim that consecutive valve closures in the learned policy exceed the characteristic separation time should be supported by the measured distribution of off-intervals in the deployed policy; currently this is a qualitative statement.","section":"III.B"},{"comment":"The sentence reporting '20% skin-friction drag reduction at Re=100' should be checked for consistency with the original reference, as the Reynolds number and flow configuration are not clearly specified.","section":"Introduction, Ref. [26]"},{"comment":"The gray curves labeled 'loss per update' are hard to distinguish in the printed figure; a logarithmic scale or smoothed curves with shaded confidence bands would improve readability.","section":"Fig. 3"},{"comment":"The GitHub repository is cited without a version or commit identifier; adding a commit hash or release tag would improve reproducibility.","section":"Refs. [45]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's feasibility claim is credible and the experimental effort is substantial. The main concern is analytical: the finite- versus infinite-horizon comparison is confounded by the simultaneous change in discount factor, and the reward/metric circularity limits the open-loop comparison. These issues are fixable with additional experiments or a more cautious framing of the conclusions. The paper appears to fit the journal's scope and shows no citation or integrity concerns beyond the points raised for the authors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine experimental demonstration of online DRL for separation control in a fully turbulent wind tunnel, using one shear-stress sensor and binary valve commands. That alone is worth putting on record, because most prior DRL control work has been numerical or low-Re. The authors are also honest about the limits: the learned controller improves the local forward-flow fraction at the sensing point, does worse upstream, and matches open-loop performance further downstream.\n\nThe advertised methodological point is the comparison between a truncated return (N_c=4, gamma=1) and a full-trajectory discounted return (gamma=0.995). The stress-test note is correct that this comparison changes two variables at once, so the claimed causal role of the convective-time-aligned horizon is not isolated. The authors' explanation is plausible, and they have the right intuition about credit assignment in a noisy environment, but the design cannot support the causal reading. This is the main soft spot, and it is addressable: run the finite-horizon agent with gamma=0.995, or the infinite-horizon agent with gamma=1, and see whether the difference persists.\n\nA second, related concern is that the reward is the sign of the same RS4 signal used to define gamma+, so the control policy is explicitly trained on the metric against which it is later benchmarked. That is by design, and for a feasibility study it is acceptable, but it means the reported 1% edge over optimized periodic actuation is not an independent validation of control quality. The confidence intervals in Fig. 7 are there, but the text never states whether the margin is significant. Minor, but worth a referee asking.\n\nThe reproducibility posture is good: hyperparameters are in Table 1, the code is linked, and the actuator flow physics is grounded in the authors' prior work. Those self-citations are appropriate. The two repeat runs stop at 100 episodes, which is thin for claims about long-run stability, but the main 500-episode run is shown.\n\nThis paper deserves a serious referee. The confound is real but the feasibility result holds up, and the horizon question is a good topic for a follow-up ablation. I would send it out.","headline":"Genuine online DRL in a fully turbulent wind tunnel, with a useful but confounded return-horizon comparison.","tokens_in":13362,"tokens_out":3138,"would_cite":true,"duration_ms":32264,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A PPO agent trained directly in a turbulent wind tunnel learns a low-duty-cycle separation-control policy in about two minutes; truncating the reward horizon to the convective time is what makes the learning fast and stable.","keywords":["deep reinforcement learning","active separation control","turbulent boundary layer","wind tunnel experiment","Proximal Policy Optimization","temporal credit assignment","wall shear stress sensor","pulsed jet actuators"],"falsifier":"Run the full-trajectory agent with gamma=1 (no discounting) and check whether it still converges within about 20 episodes; if it does, the advantage the paper attributes to the convective-time horizon actually comes from removing discounting, not from truncation.","tokens_in":12292,"feed_emoji":"🌀","tokens_out":7542,"duration_ms":74595,"temperature":0.7,"pith_summary":"This paper shows that a reinforcement-learning controller can be trained online, in real time, inside a fully turbulent wind-tunnel flow over a one-sided diffuser, using only the signal from a single wall-shear-stress sensor and binary open/close commands to three pulsed jet actuators. The central methodological claim is that the temporal credit-assignment horizon matters: an agent whose return is truncated to the convective response time of the flow (four interaction steps, about 24 ms) converges to a stable low-duty-cycle control policy within roughly 20 episodes (about two minutes of interaction), whereas an agent using a discounted return over the full trajectory keeps fluctuating. Deployed, the fast-converging controller achieves a local forward-flow fraction of about 53% at the feedback sensor, about one percentage point above optimized periodic open-loop forcing at a comparable duty cycle of 12.5%. The authors argue this makes reinforcement learning a practical tool for discovering robust and interpretable separation-control strategies directly from physical experiments.","feed_headline":"Live-trained RL beats periodic flow control in a wind tunnel","feed_subtitle":"A 24 ms reward horizon gives faster, steadier convergence and a 53% forward-flow fraction.","key_machinery":"The mechanism is the finite-horizon return of Eq. (5), which sums the next N_c=4 instantaneous sign rewards after an action without discounting, constraining credit assignment to the convective time scale of 24 ms; this is contrasted with the infinite-horizon discounted return of Eq. (4). This truncation is what reduces the variance of the PPO advantage estimates and, according to the paper, what allows the actor-critic updates to settle quickly on a stable policy. The rest of the machinery is the experimental loop: a single bidirectional MEMS wall-shear-stress sensor at the transitory detachment point supplies the scalar state, a Bernoulli policy over the binary valve action is sampled each 6 ms step, and the reward is the sign of the next sensor state.","core_discovery":"The discovery is that when Proximal Policy Optimization is coupled directly to a wind-tunnel experiment, restricting the return to a finite horizon aligned with the characteristic convective response time of the flow (N_c=4 steps, gamma=1, Eq. 5) yields faster convergence, lower policy entropy, and a steadier low-duty-cycle actuation pattern than the standard discounted full-trajectory return (gamma=0.995, Eq. 4). The resulting policy commands valve openings with a mean duty cycle near 12.5%, matching the previously identified optimal open-loop duty cycle for this configuration, and produces a local forward-flow fraction at the reference sensor of approximately 0.53, slightly exceeding the best periodic open-loop control. The controller is interpretable: it maps the measured wall-shear-stress sign to a probability of valve opening, and for the majority of visited states it keeps that probability below 0.5. The authors therefore claim that a model-free policy can be learned directly from interactions with a fully turbulent experimental flow, and that temporal credit assignment is the key design choice that makes that learning efficient.","pith_inferences":["The paper compares two agents that differ in both horizon length and discount factor (N_c=4 with gamma=1 versus full trajectory with gamma=0.995), so the causal role of horizon alignment is not isolated; a sweep over (N_c, gamma) would be needed to confirm the mechanism.","Because the reward is the sign of the same RS4 sensor used to define the performance metric, the agent is optimizing the metric on which it is later judged; a reward based on a downstream sensor or spanwise-integrated shear stress could change the discovered policy and the comparison with open-loop control.","The roughly 1% margin over periodic open-loop control may or may not be statistically robust across the reported autocorrelation-corrected confidence intervals; replicating the deployment with more trajectories would sharpen whether the closed-loop advantage is real.","The same truncation principle could be transferred to other flow-control tasks with a known convective or shedding time scale, with N_c set by that physically estimated time rather than by trial and error."],"forward_implications":["If the finite-horizon return is the cause of faster convergence, then online RL for turbulent flow control should set the return horizon by the physical response time rather than by the episode length.","The trained DRL policy discovers the same optimal duty cycle (about 12.5%) that open-loop parametric sweeps identified, suggesting that the agent recovers known actuation physics without a model.","The roughly 1% improvement over optimized periodic forcing at the sensor location is accompanied by lower forward-flow fractions upstream, meaning the current single-sensor objective trades global performance for local attachment.","Training in about two minutes of flow interaction makes online experimental RL practical for other turbulent-flow control tasks with sparse sensing and binary actuation.","The low entropy and narrow duty cycle of the converged policy indicate that the learned controller is deterministic and interpretable rather than a fragile stochastic artifact."],"supporting_citations":[{"why":"Supplies the characteristic convective and transient wall-shear-stress response times that motivate N_c=4, and the optimized duty-cycle baseline (12.5%) used as the open-loop reference.","marker":"[39]"},{"why":"Defines Proximal Policy Optimization, the algorithm used for online training and the subject of the hyperparameter choices (clip factor, learning rates).","marker":"[44]"},{"why":"Describes the MEMS calorimetric wall-shear-stress sensor that provides the single scalar state and the sign reward at RS4.","marker":"[43]"},{"why":"Demonstrates experimental DRL control of a turbulent boundary layer with plasma actuators, providing the closest recent precedent for online experimental RL that this paper extends to separation control with pulsed jets.","marker":"[37]"},{"why":"Shows DRL active flow control in a turbulent separation bubble and motivates the multi-actuator extension and short-training-time claims.","marker":"[36]"},{"why":"Provides the one-sided diffuser configuration and the baseline forward-flow fraction distribution along the ramp used to locate the transitory detachment point and select RS4.","marker":"[38]"}],"fun_headline_variants":["Flow-time horizon makes RL wind-tunnel separation control robust","Wind-tunnel RL: 53% forward flow, beats periodic control","Real-time RL learns steady separation control in a diffuser","Shorter reward horizon stabilizes RL-controlled wind-tunnel flow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison that anchors the central claim changes both the return horizon and the discount factor at the same time (N_c=4 with gamma=1 versus full trajectory with gamma=0.995), so attributing the faster, more stable learning to the horizon length alone is an untested assumption; the reward is also the sign of the same sensor used as the performance metric.","fun_headline_variants_meta":{"raw":{"variants":["Flow-time horizon makes RL wind-tunnel separation control robust","Wind-tunnel RL: 53% forward flow, beats periodic control","Real-time RL learns steady separation control in a diffuser","Shorter reward horizon stabilizes RL-controlled wind-tunnel flow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001058,"raw_usage":{"total_tokens":4447,"prompt_tokens":960,"completion_tokens":3487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":3416}},"tokens_in":576,"tokens_out":3487,"duration_ms":29587,"temperature":1.0,"reasoning_tokens":3416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:31:28.612046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full-trajectory agent with gamma=1 (no discounting) and check whether it still converges within about 20 episodes; if it does, the advantage the paper attributes to the convective-time horizon actually comes from removing discounting, not from truncation.","supporting_citations":[{"cited_title":"Simulation and Testing of a MEMS Calorimetric Shear-Stress Sensor,","cited_arxiv_id":null,"evidence_quote":"Describes the MEMS calorimetric wall-shear-stress sensor that provides the single scalar state and the sign reward at RS4."},{"cited_title":"Deep reinforcement learning for active flow control in a turbulent separation bubble,","cited_arxiv_id":null,"evidence_quote":"Shows DRL active flow control in a turbulent separation bubble and motivates the multi-actuator extension and short-training-time claims."}],"review_version":1}