{"id":"5b1dec47-7328-4eeb-abb5-7221242da3dc","arxiv_id":"2505.23355","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"In simulated winter greenhouse climate control, interactive RL guided by imperfect grower advice improves profit when the advice shapes actions (policy shaping +8.4%, control sharing +6.8%) but reduces profit when it shapes rewards (-9.4%).","lead":"This paper tests three ways of letting a human grower guide an AI greenhouse climate controller: shaping rewards, shaping actions, and directly sharing control. In simulation, the action-based methods raised profit by about 6 to 8 percent, while reward shaping actually hurt, a caution for designers of human-in-the-loop control.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported improvements use beta values chosen on the test set, so the algorithm comparison may be overfit and the central claim is not yet supported.","rationale":"The reader's weakest_assumption concerns the representativeness of the simulated grower inputs, which is an external-validity question. Even if real growers behave differently, the simulation could still support a qualitative ranking of the three interaction mechanisms. The more immediate threat is internal: Section 5.3 explicitly says beta values were chosen on the test set, and Section 5.4 demonstrates that beta has a strong effect on test reward. With only seven test trajectories, choosing beta per algorithm on that same set means the numbers in Table 9 are selected maxima, not honest estimates. This directly undermines the abstract's quantitative claims (8.4%, 6.8%, -9.4%) and the relative robustness conclusion. The issue is fixable by re-running beta selection on a validation split and reporting seed-level statistics; therefore the verdict remains CONDITIONAL, matching the reader's overall decision, but the condition should be sharpened to prioritize validation-based hyperparameter selection and uncertainty reporting over the grower-representativeness discussion.","tokens_in":14449,"tokens_out":4843,"duration_ms":43954,"concrete_test":"Withhold two of the seven test trajectories (e.g., the 2021-01-01 and 2024-12-01 runs) before any tuning. For each of the four interactive algorithms and each beta in {0.05, 0.1, 0.2, 0.5}, train the 15 seeds as before, choose the beta with the best average reward on the remaining five trajectories (or on a separate validation set), then evaluate that single beta on the two held-out trajectories and report the mean and the 15-seed distribution. If the rankings or the sign of the improvement over baseline PPO do not match Table 9, the central claim is a test-set selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 states: 'the input weights (likelihood) β are chosen to be the ones that work best in the test.' Table 9 then reports each algorithm's cumulative reward at its own best beta (0.2, 0.1, 0.05, 0.2). Because Section 5.4 and Figure 6 show that test performance is highly sensitive to beta, and only seven test trajectories are used, selecting beta on the same trajectories used to compute the headline numbers makes the reported improvements optimistic and confounds algorithm ranking with beta tuning. For example, policy shaping's +8.4% is the maximum over beta on the test set; a different beta chosen fairly could yield a smaller or even negative improvement. Thus the claim that action-selection mechanisms (policy shaping, control sharing) are robust to imperfect inputs while reward shaping is not is not established unless beta is selected on a validation set independent of the test trajectories. This internal-validity issue affects the quantitative conclusion even before considering whether the simulated grower inputs are representative of real growers, which is why it is the more load-bearing concern.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes and evaluates three interactive RL mechanisms for greenhouse climate control: reward shaping, policy shaping, and control sharing, all built on a PPO baseline. The authors define three simulated grower input types (feedback, precise action advice, and constraint advice), analyze trade-offs among input availability, cognitive bias, latency, and knowledge level, and introduce auxiliary neural networks with a pool-based selection strategy to handle limited input availability. In a simulated Van Henten lettuce greenhouse, they report that policy shaping with precise advice improves test reward by 8.4%, control sharing by 6.8%, and policy shaping with constraints by 3.1%, while reward shaping decreases reward by 9.4% relative to baseline PPO. They conclude that action-selection mechanisms are robust to imperfect grower input while reward shaping is not, and they attribute this to PPO's GAE estimator. The paper also studies sensitivity to the input weight beta and to input availability.","tokens_in":14643,"tokens_out":8183,"duration_ms":87824,"significance":"The paper addresses a relevant and under-explored application: interactive RL for greenhouse climate control. It provides a useful taxonomy of input types and imperfect-input characteristics, and it implements three representative integration mechanisms in a standard PPO framework with 15 training seeds and seven winter test trajectories. If the reported ordering is robust, the practical recommendation to prefer action-selection mechanisms over reward shaping under imperfect human guidance would be valuable for greenhouse automation. The strengths are the careful decomposition of input characteristics, the explicit comparison of three incorporation mechanisms, and the analysis of beta sensitivity. The main uncertainty is not the derivation but the empirical validity of the headline comparison, because beta is selected on the test set and no interval estimates are provided; the representativeness of the simulated grower is also untested. These concerns are fixable and do not invalidate the framework proposed.","major_comments":[{"comment":"The reported ranking is based on beta values 'chosen to be the ones that work best in the test' (Section 5.3). Because Table 9 reports each algorithm at its own best beta (0.2, 0.1, 0.05, 0.2) and Figure 6 shows that test reward is highly sensitive to beta (e.g., beta=0.5 degrades performance while beta=0.05-0.2 improves it), the headline improvements of +8.4%, +6.8%, +3.1% and the decline of -9.4% are best-case values computed on the same seven test trajectories used for evaluation. This selection procedure can inflate the gains and confounds the algorithm comparison with beta tuning; the claimed robustness ordering is therefore not established. Please either select beta on a validation split independent of the test trajectories, or report the full beta sweep for every algorithm and show that the ordering is stable across reasonable beta values.","section":"Section 5.3, Table 9, Figure 6"},{"comment":"Point estimates are reported without any measure of uncertainty. The paper trains each configuration with 15 seeds (Section 5.2) and evaluates on seven test trajectories (Section 5.1), but no standard deviations, confidence intervals, or significance tests are reported for the cumulative rewards or for the relative changes. Given the small test set and the demonstrated sensitivity to beta, the observed differences (e.g., 1.91 versus 2.07 Hfl/m2) may be within seed or trajectory noise. The authors should report per-seed and per-trajectory statistics, or at least standard errors, before claiming that the interactive methods improve over baseline and that reward shaping is worse.","section":"Table 9 and Figure 4"},{"comment":"The simulated grower inputs are generated from pre-calculated optimal values and ranges derived from the same Van Henten greenhouse model that defines the training and evaluation environment, with a deliberate bias toward maximizing current crop growth. The 'imperfect grower' is therefore a model-derived oracle with an injected bias structure, not an independent source of expertise or a sample of real grower behavior. The paper's conclusions about robustness to imperfect inputs and the ranking of the three mechanisms may depend on this specific bias pattern. This limitation should be stated prominently, and ideally the analysis should include a sensitivity study over alternative input error distributions (e.g., stochastic feedback, mis-specified ranges, or risk-averse advice) or a small real-grower validation. The current Section 6 limitation paragraph mentions only the choice of PPO, not this representativeness issue.","section":"Section 3.3 and the Appendix"}],"minor_comments":[{"comment":"Equation (5) is hard to parse: pi_agent(s,a) is a probability, so the phrase a = pi_agent(s,a) should be a = arg max over actions of pi_agent(s, .), and the second line should be normalized explicitly. Please rewrite for readability.","section":"Equation (5)"},{"comment":"The symbol beta is called the 'input weight (likelihood)' throughout. Since 'likelihood' has a standard statistical meaning, consider renaming it 'blending weight' or 'input weight' to avoid ambiguity.","section":"Throughout"},{"comment":"Please state whether the averages are over seeds, test trajectories, or both, and add the corresponding standard deviation in parentheses.","section":"Table 9"},{"comment":"The reported 'test loss' is defined only in the text. Please give the exact loss definition (MSE for feedback, KL for advice) and state whether the curves are averaged over seeds and/or test trajectories.","section":"Section 5.5 and Figures 8-9"},{"comment":"The model equations in the Appendix contain formatting ambiguities (for example, the term x1(t)^2 x3(t)/10^-5/2 is unclear). A cleaned equation set would improve reproducibility.","section":"Appendix"},{"comment":"Reference [5] appears incomplete, and the DOI in reference [32] is line-broken; please check the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid framework and an interesting practical question, but the empirical headline is currently selected on the test set. I would encourage the editor to ask for a validation-based beta selection and interval estimates; the simulated-grower representativeness issue should also be addressed as a limitation. I do not see a need to reject, as the concerns are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a competent simulation study, and the section you'd care about is the three-way comparison of reward shaping, policy shaping, and control sharing under imperfect grower input. The qualitative finding — action-selection mechanisms are more robust to imperfect input than reward shaping — is coherent and I suspect it will hold up. But the paper's headline numbers are not yet supported, because the input weight β is selected on the same test trajectories used to compute them, and the paper reports no variance except for a passing note about 15 seeds.\n\nWhat's new: applying interactive RL to greenhouse climate control is a sensible, under-explored direction. The trade-off discussion (availability, cognitive bias, latency, knowledge level) is genuinely useful, and the auxiliary-network/active-learning approach for sparse input is a reasonable contribution. The authors are also honest about the main limitation: the result may be PPO/GAE-specific, and they say so in the conclusion.\n\nThe soft spots are three. First and most important, Section 5.3 states β is 'chosen to be the ones that work best in the test.' Table 9 then reports each algorithm's reward at its own best β. Given Figure 6 shows test performance is sensitive to β, this is tuning on the evaluation set; the 8.4% and 6.8% improvements are maxima over β, not fair comparisons. The fix is straightforward: pick β on a validation set, then evaluate on held-out trajectories. Second, despite 15 seeds, there are no error bars or significance tests, so we cannot tell whether the ordering is stable. This is minor but easy to fix. Third, the simulated grower inputs are derived from the same Van Henten model used for training, with a deliberate short-term bias. That's a mild circularity; the qualitative ordering may transfer, but the magnitudes probably won't. The paper should say more about how real grower behavior might differ.\n\nVerdict: the central claim is plausible and worth testing, but the quantitative claims are overfit in the current form. This deserves peer review — a good referee will send it back for a validation-set β selection and variance reporting. I'd bring it to a reading group for the trade-off discussion.\n\nBest.","headline":"Solid simulation study whose qualitative comparison is plausible but whose headline numbers are undermined by beta tuning on the test set and missing uncertainty reporting.","tokens_in":15187,"tokens_out":2355,"would_cite":true,"duration_ms":22496,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Interactive RL with imperfect grower input can improve greenhouse climate control when the input steers action selection, but fails when it reshapes the reward.","keywords":["interactive reinforcement learning","grower-in-the-loop learning","greenhouse climate control","policy shaping","reward shaping","control sharing","imperfect human input","PPO"],"falsifier":"Run the same three interactive RL mechanisms in the same simulated lettuce greenhouse but with advice collected from real, experienced growers, then measure profit over full 56-day test episodes; the central claim fails if reward shaping matches or exceeds policy shaping, or if policy shaping no longer improves profit over baseline PPO.","tokens_in":14233,"feed_emoji":"🌱","tokens_out":3612,"duration_ms":39331,"temperature":0.7,"pith_summary":"The paper asks whether interactive reinforcement learning, in which a grower's advice is mixed into training, can improve greenhouse climate control when that advice is imperfect. It tests three integration mechanisms (policy shaping, control sharing, and reward shaping) with simulated grower inputs in a lettuce greenhouse model. The central finding is that mechanisms influencing action selection improve profit by 8.4% and 6.8% over baseline PPO, while reward shaping decreases profit by 9.4%. The authors argue that this ordering follows from PPO's advantage estimation, because biased input that reshapes the reward corrupts the value baseline, whereas input that steers action selection injects exploration without destroying the reward signal. A sympathetic reader would care because grower domain knowledge is otherwise discarded, and this is a first demonstration that it can be reused even when imperfect.","feed_headline":"Reward shaping fails, action shaping wins in grower-in-the-loop RL","feed_subtitle":"Policy shaping and control sharing lift greenhouse profit 8.4% and 6.8% over PPO; reward shaping loses 9.4%.","key_machinery":"The central object is the mechanism by which grower input enters the PPO agent: reward shaping modifies the reward to r' = r + beta*F(s,a); control sharing overrides the agent's action selection with probability beta toward the action that maximizes estimated feedback; and policy shaping mixes the agent's action distribution with the grower-advice distribution using weight beta. Auxiliary neural networks estimate the grower's feedback or advice, and a discrepancy-based selective strategy, inspired by SafeDAgger, chooses which time steps receive input. The generalized advantage estimator (GAE) is the explanatory mechanism that makes action-selection methods robust to imperfect input, because underestimating the value function under biased guidance can accelerate policy improvement instead of distorting the learning objective.","core_discovery":"The paper's central claim is that interactive RL with imperfect grower inputs can improve the performance of an RL agent in greenhouse climate control, provided the integration mechanism influences action selection rather than the reward function. Specifically, policy shaping with precise action advice achieves an 8.4% profit improvement over baseline PPO, control sharing achieves 6.8%, policy shaping with constraint advice achieves 3.1%, and reward shaping loses 9.4%. The mechanism difference is traced to the generalized advantage estimator: action-selection methods keep the reward signal intact and use the input to guide exploration, while reward shaping changes the objective the critic estimates, so biased input propagates into the value function and misguides learning.","pith_inferences":["If GAE is the causal mechanism, the ordering of the three methods may not transfer to value-based RL algorithms or to policy-gradient methods without advantage estimation, a scope the paper explicitly leaves open.","Real grower advice is likely more biased toward yield quality, risk aversion, and long-term crop health than the simulated short-term-growth signals; the qualitative advantage of action-selection mechanisms would likely persist, but the reported profit magnitudes may shrink or shift.","The discrepancy-based selective querying strategy could be combined with inverse reinforcement learning to ask growers only for advice on states where the agent's uncertainty is highest, improving sample efficiency further.","The 14-day truncated training episodes may favor short-sighted advice; training on full 56-day cycles could change the balance between methods that accelerate short-term reward and those that preserve long-term profit."],"forward_implications":["Greenhouse RL systems can leverage grower knowledge without requiring perfect advice, making human-in-the-loop automation more practical.","Practitioners should prefer policy shaping or control sharing over reward shaping when human inputs are noisy or biased.","The input weight beta must be tuned and annealed to zero during training, since too high an early weight slows later learning.","Action advice tolerates limited input availability better than feedback, especially with discrepancy-based selection, enabling query-efficient grower involvement.","Reward shaping should be avoided in PPO-based interactive RL unless the human input is nearly optimal, because imperfect rewards propagate into the value estimate."],"supporting_citations":[{"why":"Supplies the lettuce greenhouse dynamic model that defines the simulation environment and the constraint set.","marker":"[21]"},{"why":"PPO is the baseline RL algorithm whose generalized advantage estimate is central to the explanation of mechanism robustness.","marker":"[22]"},{"why":"DQN-TAMER provides the reward shaping and control sharing formulations adapted with feedback-estimation networks.","marker":"[25]"},{"why":"TAMER introduces the F-function framework used to estimate human feedback in reward shaping and control sharing.","marker":"[27]"},{"why":"Policy shaping is defined here as the method that mixes agent and human action probabilities for action advice.","marker":"[18]"},{"why":"Extends policy shaping to human-teacher settings and supports the implementation used for both precise and constraint action advice.","marker":"[26]"},{"why":"Supplies the taxonomy of input characteristics (accuracy, availability, cognitive bias, latency, knowledge level) that justifies modeling inputs as imperfect.","marker":"[28]"},{"why":"SafeDAgger inspires the selective strategy that queries inputs at steps with high predicted discrepancy.","marker":"[30]"},{"why":"Provides the price parameters for lettuce, heating, and CO2 used in the profit reward function and the RL versus MPC comparison context.","marker":"[16]"}],"fun_headline_variants":["Grower input best used to guide actions, not rewards","Action shaping beats reward shaping in greenhouse RL","Imperfect grower advice: use it to pick actions, not rewards","Policy shaping lifts greenhouse profit 8.4%, reward shaping fails","For greenhouse RL, let growers guide actions, not rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simulated grower inputs, generated from the same optimal-control model that defines the training environment and deliberately biased toward short-term growth, are representative of how real growers would actually advise an RL agent.","fun_headline_variants_meta":{"raw":{"variants":["Grower input best used to guide actions, not rewards","Action shaping beats reward shaping in greenhouse RL","Imperfect grower advice: use it to pick actions, not rewards","Policy shaping lifts greenhouse profit 8.4%, reward shaping fails","For greenhouse RL, let growers guide actions, not rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1713,"prompt_tokens":995,"completion_tokens":718,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":648}},"tokens_in":611,"tokens_out":718,"duration_ms":6411,"temperature":1.0,"reasoning_tokens":648,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:47:21.917098+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three interactive RL mechanisms in the same simulated lettuce greenhouse but with advice collected from real, experienced growers, then measure profit over full 56-day test episodes; the central claim fails if reward shaping matches or exceeds policy shaping, or if policy shaping no longer improves profit over baseline PPO.","supporting_citations":[{"cited_title":"Van Henten, Greenhouse climate management: an optimal control ap- proach, Wageningen University and Research, 1994","cited_arxiv_id":null,"evidence_quote":"Supplies the lettuce greenhouse dynamic model that defines the simulation environment and the constraint set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DQN-TAMER provides the reward shaping and control sharing formulations adapted with feedback-estimation networks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TAMER introduces the F-function framework used to estimate human feedback in reward shaping and control sharing."},{"cited_title":"Griffith, K","cited_arxiv_id":null,"evidence_quote":"Policy shaping is defined here as the method that mixes agent and human action probabilities for action advice."},{"cited_title":"Cederborg, I","cited_arxiv_id":null,"evidence_quote":"Extends policy shaping to human-teacher settings and supports the implementation used for both precise and constraint action advice."},{"cited_title":"Bignold, F","cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy of input characteristics (accuracy, availability, cognitive bias, latency, knowledge level) that justifies modeling inputs as imperfect."},{"cited_title":"Morcego, W","cited_arxiv_id":null,"evidence_quote":"Provides the price parameters for lettuce, heating, and CO2 used in the profit reward function and the RL versus MPC comparison context."}],"review_version":1}