Pith. sign in

REVIEW 3 major objections 6 minor 29 references

Counterfactual simulations for large scale systems with burnout variables

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read The paper claims that counterfactual simulations of budget-capped advertising systems can be made parallel by first estimating the order in which campaigns exhaust their budgets, with a provable error bound that grows like (1+γ)^K in the nu

desk verdict The sort2aggregate idea is genuinely useful and the experiments are encouraging, but the proof of Theorem 5.2 conflates true and simulated capping orders and needs a fix before the main bound is credible. read the letter →

arxiv 2509.04038 v1 pith:LO2UJLZP submitted 2025-09-04 cs.DC math.OCstat.ME

classification cs.DCmath.OCstat.ME
keywords burnoutvariablescounterfactualestimationparallelsimulationbudgetcappingonlineadvertisingMapReducevariationalinequalityrandom-ordermodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper addresses a scalability wall in counterfactual analysis: simulating 'what would have happened if' in systems where a state variable irreversibly turns off (a budget burns out) normally requires replaying events in order. The author proposes an uncertainty relaxation that lets the simulation run in parallel: first estimate when each campaign would have run out of budget, then aggregate the auction outcomes block by block. The central result is a concentration bound (Theorem 5.2) showing the parallel simulation's per-campaign spend stays close to the sequential ground truth, with the error growing like (1+γ)^K in the number of campaigns and linearly in the per-event contribution, a tolerance t, and a smoothness slack. The paper also introduces sort2aggregate, a scalable way to estimate the capping-out times via a projected fixed-point iteration, and reports experiments on synthetic auctions and on the Yahoo search-ad bidding dataset.

What carries the argument

The central object is the activation vector a_n, a binary vector of which campaigns are still active (budget not exhausted), together with the capping-out times N_i where the i-th campaign burns out. The argument's engine is an error recursion: whenever a campaign's estimated capping time is off, the smoothness assumption (3.3) lets the spend error on that campaign be charged, up to a factor γ and slack ε, to the errors of the campaigns capped before it; solving the recursion yields the (1+γ)^K exponential factor. Supporting that recursion are a Hoeffding-type concentration lemma for random-order samples (Lemma 5.1) and, for sort2aggregate, a projected fixed-point iteration on the variationa

What would settle it

Take one day of real auction logs with strong intra-day patterns, compute the sequential spend, then run the parallel algorithm under the random-order assumption; if the empirical error exceeds the Theorem 5.2 bound (plugging in measured γ, C, t, ε) by a wide margin, the random-order relaxation is the culprit. Alternatively, in the synthetic setup, shuffle events to destroy order, then re-introduce a time trend in valuations while keeping every other setting fixed, and record the error.

Watch

Extended reading notes

Core claim

The paper's central claim is that budget-coupling effects—the reason counterfactual simulation in ad platforms is sequential—can be decoupled if one knows the order in which campaigns exhaust their budgets. Given that order, each block of events between two capping times can be aggregated independently, and the paper proves (Theorem 5.2) that the resulting parallel estimate of final spend differs from the sequential counterfactual by at most (1+γ)^K (C/N + t + γε + ε), with high probability, under a smoothness condition bounding how one campaign's deactivation affects others. The proof works by propagating an error recursion from the first capped campaign to the last, using a concentration l

Load-bearing premise

The whole guarantee rests on the observed auction sequence being a random sample without replacement from the event pool; if the real order of auctions carries information (e.g., campaigns' values change over the day), the concentration and error bounds no longer have their stated support.

Editorial extensions

If this is right

  • What-if simulations of auction rule changes (e.g., different cost-per-click or bid multipliers) can run on MapReduce clusters in near-linear time, after one pass to estimate capping times.
  • The error bound is actionable: for many campaigns, keep γ small (e.g., γ ≤ D/K) so the exponential factor becomes e^D, making the method practical.
  • Capping-out order, not exact spend, is the key information; once the order is known, activation vectors are fixed per block and aggregation is embarrassingly parallel.
  • The method extends beyond simple auctions to multi-slot search auctions, nested campaigns, and state-dependent CPC models, as argued in the discussion.
  • Algorithm 4 can be implemented with stochastic gradients, meaning it can scale to event streams without storing all data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same 'burnout' abstraction should apply to other irreversible-deactivation systems—inventory depletion, coupon redemptions, battery or resource exhaustion—where the deactivation order can be estimated independently of the full trajectory.
  • The random-order assumption is the main practical bottleneck; a natural extension is to test the method on data with explicit time-of-day seasonality and measure when the stated bound degrades.
  • The single-pass claim suggests a batching protocol: run Algorithm 4 on a small sample to get the capping order, then run one aggregated pass on full data; the paper does not explicitly analyse the sample-to-full error transfer.
  • The exponential-in-K factor hints that for large campaign counts the method will need γ to shrink, which the paper argues holds when no single campaign is dominant; a direct test would be to measure γ empirically from logs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies counterfactual simulation in large-scale systems with 'burnout' variables, motivated by budget-capped advertising campaigns. It proposes a parallel simulation algorithm (Algorithm 2) that processes auction events in blocks until an estimated budget-capping time, and states a concentration bound (Theorem 5.2) comparing the parallel output with the sequential counterfactual spend. The paper then introduces sort2aggregate (Algorithms 3 and 4) to estimate capping times efficiently and reports numerical experiments on synthetic data and on a Yahoo search-marketing dataset.

Significance. If the main bound were established, the paper would offer a principled parallelization strategy for counterfactual estimation in auction platforms, with transparent concentration guarantees. The problem is timely, and the random-order/smoothness modeling is a reasonable way to make the analysis tractable. Strengths include the explicit statement of assumptions rather than hidden fitted constants, and the use of both synthetic and real data. The main weakness is that the proof of the central theorem has a serious gap, and the practically used Algorithm 4 has no convergence theorem under the stated assumptions.

major comments (3)
  1. [Appendix, proof of Theorem 5.2, around Eq. (24)-(27)] The proof defines eta_i using the true campaign c_i and the simulated horizon \hat N_i, but then bounds the A term using Eq. (23), which is derived only for the algorithm-selected campaign \hat c_{i+1} and its residual b_{\hat c_{i+1}} - \hat s_{\hat c_{i+1}, \hat N_i}. No argument shows that \hat c_i = c_i, i.e., that the estimated capping order matches the true order. If the orders differ, the relation used to bound A does not follow from Algorithm 2's update. The induction leading to Eq. (34) therefore does not control |s_N^c - \hat s_N^c| as claimed.
  2. [Appendix, B-term estimate: '=0 since \hat a_n = a_n for t small enough'] This assertion is circular. The simulated activation vector \hat a_n and the true activation vector a_n can differ whenever an estimated capping time \hat N_j falls in a different interval than the true capping time N_j. The proof has not established that \hat N_j is close to N_j; indeed, bounding the cumulative effect of such discrepancies is the theorem's goal. The second sum in the B term cannot be set to zero without an additional lemma controlling capping-time deviations.
  3. [§6, Algorithm 4 and the VI convergence paragraph] Algorithm 4 is the practical heart of the paper, yet no convergence theorem is proved for it. The surrounding variational-inequality discussion assumes that F is monotone and Lipschitz, but these properties are not derived from Assumptions 3.1-3.3 or from the first-price auction model used in Section 7; the text explicitly leaves identification of primitive conditions for future work. Since Figures 3-6 all rely on Algorithm 4, the empirical claims about sort2aggregate are not backed by the theory. The authors should either prove convergence under explicit conditions or clearly frame the numerical section as heuristic and temper the claims.
minor comments (6)
  1. [Assumption 3.3] The assumption states a \in [0,1]^C while f(\cdot, a) is defined for a \in {0,1}^C; the domain should be clarified.
  2. [Theorem 5.2 and Corollary 5.3] The phrase 'for t small enough' is not quantified. The proof implicitly requires that the estimated capping times be close to the true capping times; the statement should make explicit the condition on t, e.g., in terms of the minimum capping gap.
  3. [Algorithm 2, line 5] The conditional expectation E[f(e_{\hat N_i+1}, \hat a_i) | \{ \hat N_i, e_1, \dots, e_{\hat N_i} \}] should clarify the case \hat N_i+1 > N, although the loop condition may exclude it.
  4. [§6] 'Tasky's fixed-point theorem' should read 'Tarski's fixed-point theorem.'
  5. [Section 7.2] The Yahoo evaluation does not check or discuss Assumption 3.1. Since the concentration guarantees depend on the exchangeability of the auction order, the authors should justify this assumption for the dataset or state more clearly that the empirical results are conditional on it.
  6. [Figures 3-6] The figure captions are under-specified: Figure 4 does not state the error metric, and Figures 5-6 do not describe the exact displayed quantities. The synthetic experiments would be easier to evaluate if the reported error numbers were stated in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theorem's constants are assumptions, not fitted values; the only flagged issue is a proof gap, not definitional circularity.

full rationale

The paper's central guarantee (Theorem 5.2) is derived from Assumptions 3.1-3.3 and Lemma 5.1, which uses Hoeffding's inequality for sampling without replacement. No parameter is fitted to the quantity being predicted: C, gamma, epsilon, and delta are assumed inputs, and the bound contains no constant calibrated to the sequential ground truth. Algorithm 2 and sort2aggregate are validated against a sequential simulation using the same auction rule f; this is a self-consistency check of the parallel implementation, not a prediction of external data, so no fitted input is renamed as a prediction. Self-citations ([5], [6], [7], [14], [15], [16]) appear only in related work or as pointers to standard lattice/variational-inequality references; they are not load-bearing for Theorem 5.2, and the convergence of Algorithm 4 is explicitly left for future work. The one genuine concern is in the appendix proof of Theorem 5.2, where the B-term is dismissed as '=0 since ∧a_n = a_n for t small enough'; this asserts, rather than proves, that the simulated and true activation vectors coincide, which is a correctness gap rather than a circular reduction, because it is not obtained by defining the estimator in terms of the ground truth or by fitting a parameter to the target. Overall, the derivation is self-contained against external concentration results and does not reduce to its own inputs.

Assumptions & free parameters 5 free parameters · 8 assumptions · 0 invented entities

The central claim rests on structural assumptions: random order, small per-event spend, smoothness of the auction rule, and full state capture. Algorithm 2 additionally assumes an oracle for conditional expectations, and Algorithm 4 assumes monotone/Lipschitz behavior without proof. No new physical entities are introduced. The hand-chosen hyperparameters are not fitted to the target counterfactual, so they are not a circularity concern, but they are underreported.

free parameters (5)
  • C (small contribution bound)
    Assumption 3.2; existence is assumed, not estimated from data; the main theorem's error scales with C/N.
  • γ, δ, ε (smoothness constants)
    Assumption 3.3; existence is assumed and bound grows as (1+γ)^K and includes γε+ε; not estimated or validated on the real data.
  • Algorithm 4 step size η
    Hand-chosen; the paper does not report values or sensitivity analysis; it controls convergence of the capping-time iteration.
  • Sampling rate ρ = 0.001 in Figure 3
    Hand-chosen; affects quality of N_i estimates; no cross-validation or error bars.
  • Iteration count T
    Hand-chosen; not reported for final experiments; convergence is unspecified.
assumptions (8)
  • domain assumption Assumption 3.1: realized auction sequence is a random sample without replacement from E.
    Needed for Hoeffding concentration and the conditional expectations in Algorithm 2; not empirically validated in the paper.
  • domain assumption Assumption 3.2: exists C > 0 with f_c(e,a) < C/N for all events and active sets.
    Bounds overshoot and powers Lemma .1; plausible but unverified.
  • domain assumption Assumption 3.3: (γ,δ,ε)-smoothness, deactivating one campaign affects others by at most γ times that campaign's spend plus ε, uniformly over intervals and active vectors.
    Central to the theorem's induction; includes an ε slack; authors argue γ scales as 1/|C| without dominant campaigns, but no direct estimation is provided.
  • domain assumption E contains all auction-relevant state variables.
    Section 4 states this is an approximation and rules out long-term dependencies; if false, counterfactual estimates are biased.
  • domain assumption ML models trained under rule f remain valid under counterfactual rule \tilde f.
    Section 4 says these biases are assumed negligible; unquantified and untested.
  • ad hoc to paper Algorithm 2 has access to exact conditional expectations F_{i+1}.
    Line 5 of Algorithm 2 uses exact expected spend conditioned on history; the theorem does not account for estimation error in these expectations.
  • ad hoc to paper The residual map F in Algorithm 4 is monotone and Lipschitz, and the projected iteration converges.
    Needed for sort2aggregate; the paper explicitly says identifying primitive conditions is left for future work, so the guarantee is assumed rather than proven.
  • standard math Hoeffding concentration for sampling without replacement.
    Invoked in Lemma .1 via [4,17]; appropriate background result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Counterfactual simulations for large scale systems with burnout variables." pith.science (2026). https://pith.science/paper/LO2UJLZP

@misc{pith2026250904038,
  author       = {Pith},
  title        = {Pith review of: Counterfactual simulations for large scale systems with burnout variables},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LO2UJLZP}},
  note         = {Machine review of arXiv:2509.04038}
}
read the original abstract

We consider large-scale systems influenced by burnout variables - state variables that start active, shape dynamics, and irreversibly deactivate once certain conditions are met. Simulating what-if scenarios in such systems is computationally demanding, as alternative trajectories often require sequential processing, which does not scale very well. This challenge arises in settings like online advertising, because of campaigns budgets, complicating counterfactual analysis despite rich data availability. We introduce a new type of algorithms based on what we refer to as uncertainty relaxation, that enables efficient parallel computation, significantly improving scalability for counterfactual estimation in systems with burnout variables.

Figures

Figures reproduced from arXiv: 2509.04038 by the authors.

Figure 1
Figure 1. Sampling events before applying directly a sequential simulation might result in very poor estimate. [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Same setup as in Figure [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Same setup as in Figure [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of the output of sort2aggregate with the ground truth. [6] Martin Bompaire, Antoine Désir, and Benjamin Heymann. 2024. Fixed point label attribution for real-time bidding. Manufacturing & Service Operations Management 26, 3 (2024), 1043–1061. [7] Martin Bomp…
Figure 5
Figure 5. Figure 5: Evolution of the predicted spend for a subset of campaigns over the iterations of the algorithm. The [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Cumulative of the relative error for the campaigns (weighted by spend). We see that our algorithm [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 25 canonical work pages

  1. [1]

    Zoö Abrams, S Sathiya Keerthi, Ofer Mendelevitch, and John A Tomlin. 2008. Ad delivery with budgeted advertisers: A comprehensive LP approach. Journal of Electronic Commerce Research 9, 1 (2008)

  2. [2]

    Gagan Aggarwal, Ashwinkumar Badanidiyuru, Santiago R Balseiro, Kshipra Bhawalkar, Yuan Deng, Zhe Feng, Gagan Goel, Christopher Liaw, Haihao Lu, Mohammad Mahdian, et al. 2024. Auto-bidding and auctions in online advertising: A survey. ACM SIGecom Exchanges 22, 1 (2024), 159–183

  3. [3]

    Santiago Balseiro, Haihao Lu, and Vahab Mirrokni. 2020. Dual mirror descent for online allocation problems. In International Conference on Machine Learning . PMLR, 613–628

  4. [4]

    Rémi Bardenet and Odalric-Ambrym Maillard. 2015. Concentration Inequalities for Sampling without Replacement. https://doi.org/10.48550/arXiv.1309.4029 arXiv:1309.4029

  5. [5]

    Artem Betlei, Mariia Vladimirova, Mehdi Sebbar, Nicolas Urien, Thibaud Rahier, and Benjamin Heymann. 2024. Maximizing the Success Probability of Policy Allocations in Online Systems. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 11061–11068. 12 Counterfactual simulations for large scale systems with burnout variables Fig. 4. C...

  6. [6]

    Martin Bompaire, Antoine Désir, and Benjamin Heymann. 2024. Fixed point label attribution for real-time bidding. Manufacturing & Service Operations Management 26, 3 (2024), 1043–1061

  7. [7]

    Martin Bompaire, Alexandre Gilotte, and Benjamin Heymann. 2021. Causal Models for Real Time Bidding with Repeated User Interactions. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (KDD ’21). Association for Computing Machinery, New York, NY, USA, 75–85. https://doi.org/10.1145/3447548.3467280

  8. [8]

    Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Di- pankar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research 14, 11 (2013)

Show all 29 references
  1. [9]

    Qinyi Chen, Phuong Ha Nguyen, and Djordje Gligorijevic. 2024. Optimization-Based Budget Pacing in eBay Sponsored Search. In Companion Proceedings of the ACM on Web Conference 2024 (Singapore, Singapore) (WWW ’24). Association for Computing Machinery, New York, NY, USA, 328–337...

  2. [10]

    Hana Choi, Carl F Mela, Santiago R Balseiro, and Adam Leary. 2020. Online display advertising markets: A literature review and future directions. Information Systems Research 31, 2 (2020), 556–575

  3. [11]

    Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: simplified data processing on large clusters. Commun. ACM 51, 1 (Jan. 2008), 107–113. https://doi.org/10.1145/1327452.1327492

  4. [12]

    Nikhil R Devanur and Thomas P Hayes. 2009. The adwords problem: online keyword matching with budgeted bidders under random permutations. In Proceedings of the 10th ACM conference on Electronic commerce . 71–78

  5. [13]

    Patrick T Harker and Jong-Shi Pang. 1990. Finite-dimensional variational inequality and nonlinear complementarity problems: a survey of theory, algorithms and applications. Mathematical programming 48, 1 (1990), 161–220

  6. [14]

    Benjamin Heymann, Rémi Chan-Renous-Legoubin, and Alexandre Gilotte. 2024. A pragmatic policy learning approach to account for users’ fatigue in repeated auctions. (July 2024). https://hal.science/hal-04646638 working paper or preprint

  7. [15]

    Benjamin Heymann, Alexandre Gilotte, and Rémi Chan-Renous. 2024. Repeated Bidding with Dynamic Value. In WINE 2024. 13 Heymann some capping (left) and uncapping campaigns (right) 0 1000 2000 3000 4000 5000 6000spend initial spend final spend Fig. 5. Evolution of the predicted ...

  8. [16]

    Benjamin Heymann and Alejandro Jofré. 2024. Reverse auctions with transportation and convex costs. Computational Management Science 21, 1 (2024), 20

  9. [17]

    Wassily Hoeffding. 1994. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding (1994), 409–426

  10. [18]

    Imbens and Donald B

    Guido W. Imbens and Donald B. Rubin. 2015. Causal Inference in Statistics, Social, and Biomedical Sciences . Cambridge university press

  11. [19]

    Thomas Kesselheim, Andreas Tönnis, Klaus Radke, and Berthold Vöcking. 2014. Primal beats dual on online packing lps in the random-order model. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing . 303–312

  12. [20]

    Luofeng Liao and Christian Kroer. 2023. Statistical inference and A/B testing for first-price pacing equilibria. In International Conference on Machine Learning . PMLR, 20868–20905

  13. [21]

    Aranyak Mehta, Amin Saberi, Umesh Vazirani, and Vijay Vazirani. 2007. Adwords and generalized online matching. Journal of the ACM (JACM) 54, 5 (2007), 22–es

  14. [22]

    Daisuke Moriwaki, Yuta Hayakawa, Akira Matsui, Yuta Saito, Isshu Munemasa, and Masashi Shibata. 2021. A real- world implementation of unbiased lift-based bidding system. In 2021 IEEE International Conference on Big Data (Big Data). IEEE, 1877–1888

  15. [23]

    Jerzy Neyman. 1923. On the application of probability theory to agricultural experiments. Essay on principles. Ann. Agricultural Sciences (1923), 1–51

  16. [24]

    Judea Pearl. 1995. Causal diagrams for empirical research. Biometrika 82, 4 (1995), 669–688

  17. [25]

    Judea Pearl. 2009. Causality. Cambridge university press

  18. [26]

    Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2017. Elements of causal inference: foundations and learning algorithms. The MIT Press. 14 Counterfactual simulations for large scale systems with burnout variables 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.2 0.4 0.6 0.8 1.0Prop. ...

  19. [27]

    Donald M Topkis. 1998. Supermodularity and complementarity. Princeton university press

  20. [28]

    Caio Waisman, Harikesh S Nair, and Carlos Carrion. 2024. Online causal inference for advertising in real-time bidding auctions. Marketing Science (2024)

  21. [29]

    Yahoo! Search Marketing advertiser bidding data, version 1.0

    Yahoo! [n.d.]. Yahoo! Search Marketing advertiser bidding data, version 1.0. 15 Heymann PROOFS Lemma .1. Let𝑛⩽𝑁 ,𝑐∈C ,𝛼∈{ 0, 1}C, and𝑡 > 0, set𝐹 = E 𝑓𝑐(𝒆1,𝛼) then P | 𝑛∑︁ 𝑖=1 𝑓𝑐(𝒆𝑖,𝛼)− 𝑛𝐹|⩾𝑡 ! ⩽ 2 exp − 2𝑁𝑡 2 𝐶2 . (14) Proof. By Hoeffding’s inequality [4, 17] and the random or...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.