pith. sign in

arxiv: 2606.01665 · v1 · pith:IFUK7QEEnew · submitted 2026-06-01 · 💻 cs.LG

Quantifying the Energy Floor: Direct Measurement and Replay Buffer Bias in SAC-Based HVAC Control on sbsim

Bo Li , Chen Zhang This is my paper

Pith reviewed 2026-06-28 15:31 UTC · model grok-4.3

classification 💻 cs.LG
keywords SACHVAC controlreplay buffer biasenergy floorbuilding simulatorreinforcement learningsub-optimalityminimum power constraint
0
0 comments X

The pith

Direct measurement shows replay buffer initialization from a schedule policy accounts for 96% of the sub-optimality gap in SAC HVAC control, with equipment minimum power setting the true energy floor at 35.51 USD per day.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper directly measures the lowest daily operating cost achievable for HVAC control under the given action constraints in the sbsim building simulator. Minimum-action experiments establish this energy floor at 35.51 USD per day, driven almost entirely by continuous electrical loads. The standard SAC agent, started with a replay buffer filled from a schedule policy, reaches 37.18 USD per day. Starting instead with an empty buffer brings performance to 35.57 USD per day and closes 96% of the gap to the floor. Ablations confirm that pre-filled buffer variants all perform similarly and that further widening of temperature ranges adds negligible savings, indicating that hardware limits rather than algorithmic details set the binding constraint.

Core claim

Through minimum-action experiments on the sbsim simulator, the energy floor is directly measured at USD 35.51/day, dominated by continuous electrical loads. The standard SAC baseline initialized with schedule-policy replay buffer transitions converges to USD 37.18/day. Training from an empty buffer reduces cost to USD 35.57/day, eliminating 96% of the gap. Expanding the supply water temperature range by 10 K yields negligible additional savings, and further expansion triggers physical constraint violations. Systematic ablation across planning horizon, reward weights, and observation enrichment shows all pre-filled-buffer configurations cluster within 0.7%, demonstrating that equipment minimu

What carries the argument

Direct measurement of the energy floor through minimum-action experiments on sbsim, combined with controlled comparisons of replay buffer initialization strategies.

If this is right

  • Training SAC from an empty buffer reaches performance within 0.06 USD of the directly measured floor.
  • Widening the supply water temperature range by 10 K produces only 0.03 USD per day in additional savings.
  • All configurations that begin with a pre-filled buffer perform within 0.7% of one another.
  • The observed discount factor coupling reduces the effective planning horizon from 8.3 hours to 46 minutes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same replay buffer initialization bias may limit performance in other reinforcement learning applications that seed buffers with expert or schedule policies.
  • The identified discount factor coupling indicates that effective planning horizons should be audited in other RL-based building or process control studies.
  • Direct energy floor measurement offers a general method to separate hardware limits from algorithmic performance in simulator-based control research.

Load-bearing premise

The minimum-action experiments correctly identify the absolute energy floor without missing any feasible lower-cost actions or unmodeled physical constraints.

What would settle it

A feasible control policy on sbsim that achieves a daily cost below 35.51 USD while satisfying all physical and action constraints would falsify the measured energy floor.

Figures

Figures reproduced from arXiv: 2606.01665 by Bo Li, Chen Zhang.

Figure 2
Figure 2. Figure 2: Gap decomposition from Schedule Baseline ($41.27) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation sensitivity analysis. Each point represents [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
read the original abstract

We quantify the energy floor -- the minimum achievable cost given action space constraints -- for Soft Actor-Critic (SAC) HVAC control on the sbsim calibrated building simulator. Through minimum-action experiments, we directly measure this floor at USD 35.51/day, dominated by continuous electrical loads (USD 35.44, 99.8%) with negligible gas consumption. The standard SAC baseline, initialized with schedule-policy replay buffer transitions, converges to USD 37.18/day, 4.7% above the floor. We identify buffer initialization as the dominant source of sub-optimality in this scenario: training from an empty buffer reduces cost to USD 35.57/day, eliminating 96% of the gap. Expanding the supply water temperature range by 10 K yields negligible additional savings (USD 0.03/day), and further expansion triggers physical constraint violations. We additionally uncover a discount factor coupling (gamma_eff = 0.891) shrinking the effective planning horizon from 8.3 h to 46 min -- a benchmark-wide issue warranting audit. Systematic ablation across planning horizon, reward weights, and observation enrichment confirms all pre-filled-buffer configurations cluster within 0.7% (USD 37.18--USD 37.42), demonstrating that equipment minimum power -- not algorithmic design -- imposes the binding constraint.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims to directly quantify the energy floor for SAC-based HVAC control on the sbsim simulator at USD 35.51/day via minimum-action experiments (dominated by continuous electrical loads at USD 35.44/day). It reports that a standard SAC baseline with schedule-policy replay buffer reaches USD 37.18/day (4.7% above floor), while training from an empty buffer achieves USD 35.57/day and closes 96% of the gap; supply-water-temperature range expansion yields only USD 0.03/day further savings before constraint violations; a discount-factor coupling (gamma_eff=0.891) shrinks the effective horizon from 8.3 h to 46 min; and systematic ablations show all pre-filled-buffer variants cluster within 0.7%, implying equipment minimum power—not algorithmic design—is the binding constraint.

Significance. If the direct-measurement protocol is shown to return the true minimum cost under the given action-space constraints, the work supplies a concrete, falsifiable benchmark for RL HVAC control that isolates the contribution of replay-buffer initialization from other sources of sub-optimality and demonstrates the dominance of physical equipment limits. The explicit ablations across planning horizon, reward weights, and observation enrichment, together with the identification of a discount-factor coupling that may affect other SAC deployments, constitute reproducible experimental protocols that strengthen the result.

major comments (2)
  1. [Abstract (direct measurement paragraph)] Abstract (paragraph on direct measurement): the claim that minimum-action experiments identify the absolute energy floor at USD 35.51/day rests on the assumption that setting actions to per-step lower bounds exhausts all feasible lower-cost trajectories; without an explicit constrained trajectory optimization or exhaustive verification that no lower-cost sequence exists under the action-space constraints, the reported floor may be loose, which directly undermines the 96% gap-closure interpretation of the empty-buffer result (USD 35.57/day) and the conclusion that equipment minimum power is the sole binding constraint.
  2. [Abstract (supply water temperature paragraph)] Abstract (supply-water-temperature expansion): the observation that a 10 K expansion yields only USD 0.03/day savings and further expansion triggers violations is presented as supporting evidence that the floor is tight, yet this single-axis perturbation does not rule out other feasible action combinations (e.g., coordinated changes in multiple continuous variables) that could produce lower cost while remaining within constraints.
minor comments (2)
  1. [Abstract] The derivation or numerical procedure used to obtain gamma_eff = 0.891 and the effective planning horizon reduction (8.3 h to 46 min) is not shown; a short appendix or equation block would allow independent verification.
  2. The manuscript should state the precise definition of the reward function and the numerical values of all reward weights used in the ablations, as these are required to reproduce the reported cost figures.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the insightful comments on the energy floor measurement. We agree that the minimum-action protocol provides a practical but not formally verified lower bound, and we will revise the manuscript to clarify this distinction while preserving the main empirical findings on replay buffer bias. Point-by-point responses follow.

read point-by-point responses

  1. Referee: [Abstract (direct measurement paragraph)] Abstract (paragraph on direct measurement): the claim that minimum-action experiments identify the absolute energy floor at USD 35.51/day rests on the assumption that setting actions to per-step lower bounds exhausts all feasible lower-cost trajectories; without an explicit constrained trajectory optimization or exhaustive verification that no lower-cost sequence exists under the action-space constraints, the reported floor may be loose, which directly undermines the 96% gap-closure interpretation of the empty-buffer result (USD 35.57/day) and the conclusion that equipment minimum power is the sole binding constraint.

    Authors: We acknowledge the referee's point: applying per-step lower bounds independently yields an empirical lower bound rather than a provably optimal trajectory, as temporal dependencies in building dynamics could allow coordinated sequences to achieve lower cost. The manuscript presents this as a 'direct measurement' because the protocol exhausts the per-step action bounds in a single rollout, and the 99.8% dominance of fixed electrical loads makes further reduction improbable without constraint violation. The empty-buffer result (35.57) being only 0.06 above supports the interpretation in practice. We will revise the abstract, methods, and discussion to describe the quantity as an 'empirical energy floor via minimum-action rollouts' and explicitly note the absence of constrained trajectory optimization as a limitation. This is a partial revision that tempers the wording without changing the reported numbers or the buffer-bias conclusion. revision: partial

  2. Referee: [Abstract (supply water temperature paragraph)] Abstract (supply-water-temperature expansion): the observation that a 10 K expansion yields only USD 0.03/day savings and further expansion triggers violations is presented as supporting evidence that the floor is tight, yet this single-axis perturbation does not rule out other feasible action combinations (e.g., coordinated changes in multiple continuous variables) that could produce lower cost while remaining within constraints.

    Authors: We agree that the supply-water-temperature ablation is a single-axis test and does not exhaustively rule out beneficial coordinated changes across multiple variables. It was included as one of several targeted probes (alongside planning horizon, reward weights, and observation enrichment) to check whether relaxing a key continuous constraint could approach lower cost. The negligible savings and subsequent violations indicate the minimum actions already sit near the feasible boundary for that actuator. We will revise the abstract and results section to present the outcome as corroborating rather than conclusive evidence of tightness, and we will add a brief statement that a full multi-variable constrained optimization is left for future work. This is a partial revision that addresses the concern on presentation. revision: partial

Circularity Check

0 steps flagged

No circularity; energy floor from independent direct measurement, gap closed by explicit ablation

full rationale

The paper defines the energy floor via direct minimum-action experiments on the simulator and measures the SAC baseline and empty-buffer ablation against it with explicit numerical comparisons (35.51, 37.18, 35.57). No step reduces a claimed prediction or uniqueness result to a fitted parameter, self-citation, or ansatz imported from prior work by the same authors. All load-bearing claims rest on simulator runs and ablations whose outputs are not forced by the inputs by construction. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Abstract-only; the claims rest on the simulator faithfully capturing all relevant dynamics and on the minimum-action policy being the true lower bound.

axioms (2)
  • domain assumption The sbsim simulator accurately represents building thermal dynamics, equipment constraints, and energy costs.
    All floor and gap measurements depend on this.
  • domain assumption The minimum-action policy is feasible and achieves the absolute lowest cost without violating unstated physical limits.
    Directly invoked to define the energy floor.

pith-pipeline@v0.9.1-grok · 5774 in / 1267 out tokens · 34784 ms · 2026-06-28T15:31:19.727965+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Department of Energy

    U.S. Department of Energy. Buildings energy data book. https://www.energy. gov/eere/buildings/building-technologies-office, 2011

    2011

  2. [3]

    Goldfeder, V

    J. Goldfeder, V. Dean, Z. Jiang, X. Wang, B. Dong, H. Lipson, and J. Sipple. The Smart Buildings Control Suite: A diverse open source benchmark to evaluate and scale HVAC control policies for sustainability.arXiv preprint arXiv:2410.03756, 2025

    work page arXiv 2025

  3. [4]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InProc. ICML, pages 1861–1870, 2018

    2018

  4. [5]

    T. Wei, Y. Wang, and Q. Zhu. Deep reinforcement learning for building HVAC control. InProc. DAC, pages 1–6, 2017

    2017

  5. [6]

    Zhang and K.P

    Z. Zhang and K.P. Lam. Practical implementation and evaluation of deep rein- forcement learning control for a radiant heating system. InProc. BuildSys, pages 148–157, 2018

    2018

  6. [7]

    Biemann, F

    M. Biemann, F. Scheller, X. Liu, and L. Huang. Experimental evaluation of model- free reinforcement learning algorithms for continuous HVAC control.Applied Energy, 298:117164, 2021

    2021

  7. [8]

    Drgoňa et al

    J. Drgoňa et al. All you need to know about model predictive control for buildings. Annual Reviews in Control, 50:190–232, 2020

    2020

  8. [9]

    B. Chen, Z. Cai, and M. Bergés. Gnu-RL: A precocial reinforcement learning solution for building HVAC control using a differentiable MPC policy. InProc. BuildSys, pages 316–325, 2019

    2019

  9. [10]

    Henderson, R

    P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. InProc. AAAI, pages 3207–3214, 2018

    2018

  10. [11]

    Dulac-Arnold, N

    G. Dulac-Arnold, N. Levine, D.J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis.Machine Learning, 110:2419–2468, 2021

    2021

  11. [12]

    de Bruin, J

    T. de Bruin, J. Kober, K. Tuyls, and R. Babuska. Experience selection in deep reinforcement learning for control.Journal of Machine Learning Research, 19(9):1- 56, 2018

    2018

  12. [13]

    Hester, M

    T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Leibo, and A. Gruslys. Deep Q-learning from demonstrations. InProc. AAAI, pages 3223–3230, 2018

    2018

  13. [14]

    Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

    M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.arXiv preprint arXiv:1707.08817, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017