REVIEW 3 major objections 6 minor 29 references
Counterfactual simulations for large scale systems with burnout variables
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that counterfactual simulations of budget-capped advertising systems can be made parallel by first estimating the order in which campaigns exhaust their budgets, with a provable error bound that grows like (1+γ)^K in the nu
desk verdict The sort2aggregate idea is genuinely useful and the experiments are encouraging, but the proof of Theorem 5.2 conflates true and simulated capping orders and needs a fix before the main bound is credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the activation vector a_n, a binary vector of which campaigns are still active (budget not exhausted), together with the capping-out times N_i where the i-th campaign burns out. The argument's engine is an error recursion: whenever a campaign's estimated capping time is off, the smoothness assumption (3.3) lets the spend error on that campaign be charged, up to a factor γ and slack ε, to the errors of the campaigns capped before it; solving the recursion yields the (1+γ)^K exponential factor. Supporting that recursion are a Hoeffding-type concentration lemma for random-order samples (Lemma 5.1) and, for sort2aggregate, a projected fixed-point iteration on the variationa
What would settle it
Take one day of real auction logs with strong intra-day patterns, compute the sequential spend, then run the parallel algorithm under the random-order assumption; if the empirical error exceeds the Theorem 5.2 bound (plugging in measured γ, C, t, ε) by a wide margin, the random-order relaxation is the culprit. Alternatively, in the synthetic setup, shuffle events to destroy order, then re-introduce a time trend in valuations while keeping every other setting fixed, and record the error.
Extended reading notes
Core claim
The paper's central claim is that budget-coupling effects—the reason counterfactual simulation in ad platforms is sequential—can be decoupled if one knows the order in which campaigns exhaust their budgets. Given that order, each block of events between two capping times can be aggregated independently, and the paper proves (Theorem 5.2) that the resulting parallel estimate of final spend differs from the sequential counterfactual by at most (1+γ)^K (C/N + t + γε + ε), with high probability, under a smoothness condition bounding how one campaign's deactivation affects others. The proof works by propagating an error recursion from the first capped campaign to the last, using a concentration l
Load-bearing premise
The whole guarantee rests on the observed auction sequence being a random sample without replacement from the event pool; if the real order of auctions carries information (e.g., campaigns' values change over the day), the concentration and error bounds no longer have their stated support.
Editorial extensions
If this is right
- What-if simulations of auction rule changes (e.g., different cost-per-click or bid multipliers) can run on MapReduce clusters in near-linear time, after one pass to estimate capping times.
- The error bound is actionable: for many campaigns, keep γ small (e.g., γ ≤ D/K) so the exponential factor becomes e^D, making the method practical.
- Capping-out order, not exact spend, is the key information; once the order is known, activation vectors are fixed per block and aggregation is embarrassingly parallel.
- The method extends beyond simple auctions to multi-slot search auctions, nested campaigns, and state-dependent CPC models, as argued in the discussion.
- Algorithm 4 can be implemented with stochastic gradients, meaning it can scale to event streams without storing all data.
Reading between the lines
- The same 'burnout' abstraction should apply to other irreversible-deactivation systems—inventory depletion, coupon redemptions, battery or resource exhaustion—where the deactivation order can be estimated independently of the full trajectory.
- The random-order assumption is the main practical bottleneck; a natural extension is to test the method on data with explicit time-of-day seasonality and measure when the stated bound degrades.
- The single-pass claim suggests a batching protocol: run Algorithm 4 on a small sample to get the capping order, then run one aggregated pass on full data; the paper does not explicitly analyse the sample-to-full error transfer.
- The exponential-in-K factor hints that for large campaign counts the method will need γ to shrink, which the paper argues holds when no single campaign is dominant; a direct test would be to measure γ empirically from logs.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies counterfactual simulation in large-scale systems with 'burnout' variables, motivated by budget-capped advertising campaigns. It proposes a parallel simulation algorithm (Algorithm 2) that processes auction events in blocks until an estimated budget-capping time, and states a concentration bound (Theorem 5.2) comparing the parallel output with the sequential counterfactual spend. The paper then introduces sort2aggregate (Algorithms 3 and 4) to estimate capping times efficiently and reports numerical experiments on synthetic data and on a Yahoo search-marketing dataset.
Significance. If the main bound were established, the paper would offer a principled parallelization strategy for counterfactual estimation in auction platforms, with transparent concentration guarantees. The problem is timely, and the random-order/smoothness modeling is a reasonable way to make the analysis tractable. Strengths include the explicit statement of assumptions rather than hidden fitted constants, and the use of both synthetic and real data. The main weakness is that the proof of the central theorem has a serious gap, and the practically used Algorithm 4 has no convergence theorem under the stated assumptions.
major comments (3)
- [Appendix, proof of Theorem 5.2, around Eq. (24)-(27)] The proof defines eta_i using the true campaign c_i and the simulated horizon \hat N_i, but then bounds the A term using Eq. (23), which is derived only for the algorithm-selected campaign \hat c_{i+1} and its residual b_{\hat c_{i+1}} - \hat s_{\hat c_{i+1}, \hat N_i}. No argument shows that \hat c_i = c_i, i.e., that the estimated capping order matches the true order. If the orders differ, the relation used to bound A does not follow from Algorithm 2's update. The induction leading to Eq. (34) therefore does not control |s_N^c - \hat s_N^c| as claimed.
- [Appendix, B-term estimate: '=0 since \hat a_n = a_n for t small enough'] This assertion is circular. The simulated activation vector \hat a_n and the true activation vector a_n can differ whenever an estimated capping time \hat N_j falls in a different interval than the true capping time N_j. The proof has not established that \hat N_j is close to N_j; indeed, bounding the cumulative effect of such discrepancies is the theorem's goal. The second sum in the B term cannot be set to zero without an additional lemma controlling capping-time deviations.
- [§6, Algorithm 4 and the VI convergence paragraph] Algorithm 4 is the practical heart of the paper, yet no convergence theorem is proved for it. The surrounding variational-inequality discussion assumes that F is monotone and Lipschitz, but these properties are not derived from Assumptions 3.1-3.3 or from the first-price auction model used in Section 7; the text explicitly leaves identification of primitive conditions for future work. Since Figures 3-6 all rely on Algorithm 4, the empirical claims about sort2aggregate are not backed by the theory. The authors should either prove convergence under explicit conditions or clearly frame the numerical section as heuristic and temper the claims.
minor comments (6)
- [Assumption 3.3] The assumption states a \in [0,1]^C while f(\cdot, a) is defined for a \in {0,1}^C; the domain should be clarified.
- [Theorem 5.2 and Corollary 5.3] The phrase 'for t small enough' is not quantified. The proof implicitly requires that the estimated capping times be close to the true capping times; the statement should make explicit the condition on t, e.g., in terms of the minimum capping gap.
- [Algorithm 2, line 5] The conditional expectation E[f(e_{\hat N_i+1}, \hat a_i) | \{ \hat N_i, e_1, \dots, e_{\hat N_i} \}] should clarify the case \hat N_i+1 > N, although the loop condition may exclude it.
- [§6] 'Tasky's fixed-point theorem' should read 'Tarski's fixed-point theorem.'
- [Section 7.2] The Yahoo evaluation does not check or discuss Assumption 3.1. Since the concentration guarantees depend on the exchangeability of the auction order, the authors should justify this assumption for the dataset or state more clearly that the empirical results are conditional on it.
- [Figures 3-6] The figure captions are under-specified: Figure 4 does not state the error metric, and Figures 5-6 do not describe the exact displayed quantities. The synthetic experiments would be easier to evaluate if the reported error numbers were stated in the text.
Circularity Check
No significant circularity: the theorem's constants are assumptions, not fitted values; the only flagged issue is a proof gap, not definitional circularity.
full rationale
The paper's central guarantee (Theorem 5.2) is derived from Assumptions 3.1-3.3 and Lemma 5.1, which uses Hoeffding's inequality for sampling without replacement. No parameter is fitted to the quantity being predicted: C, gamma, epsilon, and delta are assumed inputs, and the bound contains no constant calibrated to the sequential ground truth. Algorithm 2 and sort2aggregate are validated against a sequential simulation using the same auction rule f; this is a self-consistency check of the parallel implementation, not a prediction of external data, so no fitted input is renamed as a prediction. Self-citations ([5], [6], [7], [14], [15], [16]) appear only in related work or as pointers to standard lattice/variational-inequality references; they are not load-bearing for Theorem 5.2, and the convergence of Algorithm 4 is explicitly left for future work. The one genuine concern is in the appendix proof of Theorem 5.2, where the B-term is dismissed as '=0 since ∧a_n = a_n for t small enough'; this asserts, rather than proves, that the simulated and true activation vectors coincide, which is a correctness gap rather than a circular reduction, because it is not obtained by defining the estimator in terms of the ground truth or by fitting a parameter to the target. Overall, the derivation is self-contained against external concentration results and does not reduce to its own inputs.
Assumptions & free parameters
free parameters (5)
- C (small contribution bound)
- γ, δ, ε (smoothness constants)
- Algorithm 4 step size η
- Sampling rate ρ =
0.001 in Figure 3
- Iteration count T
assumptions (8)
- domain assumption Assumption 3.1: realized auction sequence is a random sample without replacement from E.
- domain assumption Assumption 3.2: exists C > 0 with f_c(e,a) < C/N for all events and active sets.
- domain assumption Assumption 3.3: (γ,δ,ε)-smoothness, deactivating one campaign affects others by at most γ times that campaign's spend plus ε, uniformly over intervals and active vectors.
- domain assumption E contains all auction-relevant state variables.
- domain assumption ML models trained under rule f remain valid under counterfactual rule \tilde f.
- ad hoc to paper Algorithm 2 has access to exact conditional expectations F_{i+1}.
- ad hoc to paper The residual map F in Algorithm 4 is monotone and Lipschitz, and the projected iteration converges.
- standard math Hoeffding concentration for sampling without replacement.
Cite this review
Pith. "Pith review of Counterfactual simulations for large scale systems with burnout variables." pith.science (2026). https://pith.science/paper/LO2UJLZP
@misc{pith2026250904038,
author = {Pith},
title = {Pith review of: Counterfactual simulations for large scale systems with burnout variables},
year = {2026},
howpublished = {\url{https://pith.science/paper/LO2UJLZP}},
note = {Machine review of arXiv:2509.04038}
}
read the original abstract
We consider large-scale systems influenced by burnout variables - state variables that start active, shape dynamics, and irreversibly deactivate once certain conditions are met. Simulating what-if scenarios in such systems is computationally demanding, as alternative trajectories often require sequential processing, which does not scale very well. This challenge arises in settings like online advertising, because of campaigns budgets, complicating counterfactual analysis despite rich data availability. We introduce a new type of algorithms based on what we refer to as uncertainty relaxation, that enables efficient parallel computation, significantly improving scalability for counterfactual estimation in systems with burnout variables.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Zoö Abrams, S Sathiya Keerthi, Ofer Mendelevitch, and John A Tomlin. 2008. Ad delivery with budgeted advertisers: A comprehensive LP approach. Journal of Electronic Commerce Research 9, 1 (2008)
work page 2008
-
[2]
Gagan Aggarwal, Ashwinkumar Badanidiyuru, Santiago R Balseiro, Kshipra Bhawalkar, Yuan Deng, Zhe Feng, Gagan Goel, Christopher Liaw, Haihao Lu, Mohammad Mahdian, et al. 2024. Auto-bidding and auctions in online advertising: A survey. ACM SIGecom Exchanges 22, 1 (2024), 159–183
work page 2024
-
[3]
Santiago Balseiro, Haihao Lu, and Vahab Mirrokni. 2020. Dual mirror descent for online allocation problems. In International Conference on Machine Learning . PMLR, 613–628
work page 2020
-
[4]
Rémi Bardenet and Odalric-Ambrym Maillard. 2015. Concentration Inequalities for Sampling without Replacement. https://doi.org/10.48550/arXiv.1309.4029 arXiv:1309.4029
-
[5]
Artem Betlei, Mariia Vladimirova, Mehdi Sebbar, Nicolas Urien, Thibaud Rahier, and Benjamin Heymann. 2024. Maximizing the Success Probability of Policy Allocations in Online Systems. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 11061–11068. 12 Counterfactual simulations for large scale systems with burnout variables Fig. 4. C...
work page 2024
-
[6]
Martin Bompaire, Antoine Désir, and Benjamin Heymann. 2024. Fixed point label attribution for real-time bidding. Manufacturing & Service Operations Management 26, 3 (2024), 1043–1061
work page 2024
-
[7]
Martin Bompaire, Alexandre Gilotte, and Benjamin Heymann. 2021. Causal Models for Real Time Bidding with Repeated User Interactions. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (KDD ’21). Association for Computing Machinery, New York, NY, USA, 75–85. https://doi.org/10.1145/3447548.3467280
arXiv 2021
-
[8]
Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Di- pankar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research 14, 11 (2013)
work page 2013
Show all 29 references
-
[9]
Qinyi Chen, Phuong Ha Nguyen, and Djordje Gligorijevic. 2024. Optimization-Based Budget Pacing in eBay Sponsored Search. In Companion Proceedings of the ACM on Web Conference 2024 (Singapore, Singapore) (WWW ’24). Association for Computing Machinery, New York, NY, USA, 328–337...
2024
-
[10]
Hana Choi, Carl F Mela, Santiago R Balseiro, and Adam Leary. 2020. Online display advertising markets: A literature review and future directions. Information Systems Research 31, 2 (2020), 556–575
2020
-
[11]
Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: simplified data processing on large clusters. Commun. ACM 51, 1 (Jan. 2008), 107–113. https://doi.org/10.1145/1327452.1327492
2008
-
[12]
Nikhil R Devanur and Thomas P Hayes. 2009. The adwords problem: online keyword matching with budgeted bidders under random permutations. In Proceedings of the 10th ACM conference on Electronic commerce . 71–78
2009
-
[13]
Patrick T Harker and Jong-Shi Pang. 1990. Finite-dimensional variational inequality and nonlinear complementarity problems: a survey of theory, algorithms and applications. Mathematical programming 48, 1 (1990), 161–220
1990
-
[14]
Benjamin Heymann, Rémi Chan-Renous-Legoubin, and Alexandre Gilotte. 2024. A pragmatic policy learning approach to account for users’ fatigue in repeated auctions. (July 2024). https://hal.science/hal-04646638 working paper or preprint
2024
-
[15]
Benjamin Heymann, Alexandre Gilotte, and Rémi Chan-Renous. 2024. Repeated Bidding with Dynamic Value. In WINE 2024. 13 Heymann some capping (left) and uncapping campaigns (right) 0 1000 2000 3000 4000 5000 6000spend initial spend final spend Fig. 5. Evolution of the predicted ...
2024
-
[16]
Benjamin Heymann and Alejandro Jofré. 2024. Reverse auctions with transportation and convex costs. Computational Management Science 21, 1 (2024), 20
2024
-
[17]
Wassily Hoeffding. 1994. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding (1994), 409–426
1994
-
[18]
Imbens and Donald B
Guido W. Imbens and Donald B. Rubin. 2015. Causal Inference in Statistics, Social, and Biomedical Sciences . Cambridge university press
2015
-
[19]
Thomas Kesselheim, Andreas Tönnis, Klaus Radke, and Berthold Vöcking. 2014. Primal beats dual on online packing lps in the random-order model. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing . 303–312
2014
-
[20]
Luofeng Liao and Christian Kroer. 2023. Statistical inference and A/B testing for first-price pacing equilibria. In International Conference on Machine Learning . PMLR, 20868–20905
2023
-
[21]
Aranyak Mehta, Amin Saberi, Umesh Vazirani, and Vijay Vazirani. 2007. Adwords and generalized online matching. Journal of the ACM (JACM) 54, 5 (2007), 22–es
2007
-
[22]
Daisuke Moriwaki, Yuta Hayakawa, Akira Matsui, Yuta Saito, Isshu Munemasa, and Masashi Shibata. 2021. A real- world implementation of unbiased lift-based bidding system. In 2021 IEEE International Conference on Big Data (Big Data). IEEE, 1877–1888
2021
-
[23]
Jerzy Neyman. 1923. On the application of probability theory to agricultural experiments. Essay on principles. Ann. Agricultural Sciences (1923), 1–51
1923
-
[24]
Judea Pearl. 1995. Causal diagrams for empirical research. Biometrika 82, 4 (1995), 669–688
1995
-
[25]
Judea Pearl. 2009. Causality. Cambridge university press
2009
-
[26]
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2017. Elements of causal inference: foundations and learning algorithms. The MIT Press. 14 Counterfactual simulations for large scale systems with burnout variables 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.2 0.4 0.6 0.8 1.0Prop. ...
2017
-
[27]
Donald M Topkis. 1998. Supermodularity and complementarity. Princeton university press
1998
-
[28]
Caio Waisman, Harikesh S Nair, and Carlos Carrion. 2024. Online causal inference for advertising in real-time bidding auctions. Marketing Science (2024)
2024
-
[29]
Yahoo! Search Marketing advertiser bidding data, version 1.0
Yahoo! [n.d.]. Yahoo! Search Marketing advertiser bidding data, version 1.0. 15 Heymann PROOFS Lemma .1. Let𝑛⩽𝑁 ,𝑐∈C ,𝛼∈{ 0, 1}C, and𝑡 > 0, set𝐹 = E 𝑓𝑐(𝒆1,𝛼) then P | 𝑛∑︁ 𝑖=1 𝑓𝑐(𝒆𝑖,𝛼)− 𝑛𝐹|⩾𝑡 ! ⩽ 2 exp − 2𝑁𝑡 2 𝐶2 . (14) Proof. By Hoeffding’s inequality [4, 17] and the random or...
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.