arxiv: 2605.06340 · v1 · submitted 2026-05-07 · 💻 cs.CY · cs.GT· cs.LG

Recognition: unknown

A Benchmark for Strategic Auditee Gaming Under Continuous Compliance Monitoring

Brittany I. Davidson, Florian A. D. Burnat

Authors on Pith no claims yet

Pith reviewed 2026-05-08 04:52 UTC · model grok-4.3

classification 💻 cs.CY cs.GTcs.LG

keywords continuous auditingStackelberg gamestrategic gamingcompliance monitoringauditee strategiescover regimeharm decompositionbenchmark simulator

0 comments

The pith

In continuous compliance monitoring, any noise-aware static auditor policy admits a cover regime where coverage gaps and granularity gaps cannot be closed simultaneously.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper models ongoing audits required by new regulations as a repeated game in which the auditor sets a policy first and the regulated entity adapts its behavior over multiple rounds. It identifies a structural limitation: static policies aware of noise cannot fix both how often they check and how detailed each check is without leaving an opening for the auditee. This matters because real-world systems can delay reports, adjust within noise, drop data, or pick favorable definitions, shifting harm in ways regulators cannot see. The authors provide concrete policies, an advanced counter-strategy, a way to measure hidden harm, and a simulator to test these interactions empirically.

Core claim

Continuous auditing is formalized as a T-round Stackelberg game between a committing auditor and an adaptive auditee; Observation 1 establishes that any noise-aware static-auditor design has a cover regime in which coverage gaps and granularity gaps cannot be closed simultaneously. Two minimal extensions close the regime along orthogonal axes—one addressing granularity via sample-size awareness and one addressing coverage via suspicion escalation—yet an audit-aware OffAuditDrift strategy defeats both, as predicted. A non-additive harm decomposition separates welfare loss from coverage loss, revealing how attrition moves harm to invisible surfaces.

What carries the argument

The cover regime, a structural feature of noise-aware static-auditor designs in which coverage gaps and granularity gaps cannot be closed simultaneously, carrying the argument that static policies are inherently limited against adaptive gaming.

If this is right

Periodic-with-floor policy closes the granularity-failure case but leaves coverage open.
History-conditioned suspicion-escalation policy closes coverage failure for the naive Drift strategy but not both.
An audit-aware OffAuditDrift strategy that exploits the auditor's commitment defeats both minimal extensions.
The non-additive harm decomposition shows attrition shifts harm from regulator-accountable welfare loss to regulator-invisible coverage loss.
The contributed simulator and strategy library enable empirical study of these dynamics calibrated to real audit data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Regulators may need policies that adapt dynamically rather than remaining static to address the identified cover regime.
The benchmark could be used to test the effectiveness of emerging regulations such as the EU AI Act against strategic gaming.
Extensions to the model could incorporate external detection costs to better reflect real-world enforcement.

Load-bearing premise

The model assumes auditees can freely choose among delay, drift within plausible noise, attrition, and cherry-picking without external detection costs or legal penalties beyond the modeled game payoffs.

What would settle it

Demonstrating a single noise-aware static auditor policy that simultaneously eliminates both coverage gaps and granularity gaps against all modeled auditee strategies would falsify the existence of the cover regime.

Figures

Figures reproduced from arXiv: 2605.06340 by Brittany I. Davidson, Florian A. D. Burnat.

**Figure 1.** Figure 1: Why attrition escapes detection. (a) Per-round trajectories under periodic audit (𝑘 = 3): Drift (red) and Attrition (purple) both produce a +0.05 gap, but Drift’s report at 𝑛𝑡 = 1000 sits outside the narrow noise band (flagged on first audit), while Attrition’s 𝑛𝑡 = 100 widens the band so the same gap fits inside (never flagged). (b) WaldCI 95% detection threshold as a function of 𝑛𝑡 at ̂𝑚 = 0.5 (red dash… view at source ↗

**Figure 2.** Figure 2: Cover regime in (𝑛min, 𝛿) space, periodic policy, 𝑚0 = 0.30, 𝑇 = 12, 30 seeds per cell. (a) Heat-map of mean time-to-detection 𝜏𝑑 under uncorrected Wald-CI. Red regions correspond to attrition-evasion (𝜏𝑑 = 𝑇 ), green regions to fast detection. The default operating point (𝑛min = 100, 𝛿 = 0.05) sits firmly in the cover regime, but the boundary is steep — attrition is sensitive to both parameters. (b) Cover… view at source ↗

**Figure 3.** Figure 3: Sensitivity curves for the two adaptive baselines, 30 seeds per point. (a) Min-sample floor: TPR on Attrition (fixed 𝑛𝑡 = 100) is 1.0 for any 𝑛floor > 100 (the policy fires under strict inequality 𝑛𝑡 < 𝑛floor); FPR on a legitimate Honest auditee operating at sample size 𝑛𝑡 collapses step-wise at 𝑛𝑡 = 𝑛floor = 500. The trade-off appears structurally binary in this configuration: a floor that conditions only… view at source ↗

read the original abstract

Continuous post-deployment compliance audits, mandated by emerging regulations such as the EU AI Act and Digital Services Act, create a class of strategic gaming distinct from the one-shot input/output gaming studied in prior work. Regulated systems can delay outcome reporting, drift their reports within plausible noise envelopes, exploit longitudinal sample attrition, and cherry-pick among ambiguous metric definitions. We formalize continuous auditing as a $T$-round Stackelberg game between an auditor that commits to a temporal policy and an adaptive auditee, and identify a structural feature of any noise-aware static-auditor design: a cover regime in which coverage gaps and granularity gaps cannot be closed simultaneously. We make this formal as Observation 1 and show that two minimal extension policies, each derived from the observation, close the regime along orthogonal axes: a sample-size-aware static rule (Periodic-with-floor) closes the granularity-failure case, while a history-conditioned suspicion-escalation policy closes the coverage-failure case for the naive Drift strategy -- and neither closes both, exactly as the observation predicts; an audit-aware OffAuditDrift strategy that exploits Stackelberg commitment defeats both. To support empirical study we contribute a non-additive harm decomposition (welfare loss $W$, coverage loss $C$) that exposes how attrition shifts harm from the regulator-accountable surface to a regulator-invisible one; an initial library of five auditee strategies (Delay, Drift, Cherry-pick, Attrition, OffAuditDrift) and five auditor policies, calibrated to summary statistics from published audits of the DSA Transparency Database; and a reproducible simulator with a small, extensible Python interface.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper shows that static continuous audit policies leave a structural cover regime open to adaptive gaming, and supplies two orthogonal fixes plus an open simulator to study it.

read the letter

The central takeaway is that any noise-aware static auditor policy in this T-round Stackelberg game creates a cover regime where coverage gaps and granularity gaps cannot both be closed at once. Observation 1 follows directly from the game definition, and the two minimal policies each close one axis exactly as predicted while leaving the other exposed. An OffAuditDrift strategy then exploits the auditor's commitment to defeat both extensions. That distinction is the main new piece relative to one-shot gaming work.

Referee Report

0 major / 3 minor

Summary. The paper formalizes continuous post-deployment compliance auditing (as mandated by regulations such as the EU AI Act and DSA) as a T-round Stackelberg game between a committing auditor and an adaptive auditee capable of delay, drift, attrition, and cherry-picking. It identifies a structural 'cover regime' (Observation 1) in any noise-aware static-auditor design where coverage gaps and granularity gaps cannot be closed simultaneously, demonstrates two minimal orthogonal policy extensions (Periodic-with-floor and history-conditioned suspicion-escalation) that each close one gap but not both, introduces an OffAuditDrift strategy that defeats both, and contributes a non-additive harm decomposition (welfare loss W, coverage loss C), a library of five auditee strategies and five auditor policies calibrated to DSA Transparency Database statistics, and a reproducible Python simulator.

Significance. If the structural claim in Observation 1 holds, the work is significant for providing a formal benchmark and extensible simulator to study strategic gaming in continuous auditing settings, distinct from one-shot input/output gaming. Strengths include the first-principles derivation of the cover regime, the orthogonal policy extensions that confirm the observation's predictions, the non-additive harm decomposition exposing how attrition shifts harm to regulator-invisible surfaces, the strategy library, and the reproducible simulator interface calibrated to external DSA summary statistics. These elements support empirical study and falsifiable predictions about policy limitations.

minor comments (3)

The abstract states Observation 1 and the policy results but provides no error bars, sensitivity analysis to parameters such as T or noise levels, or proof sketch for the structural claim, which reduces accessibility and leaves the empirical support for the orthogonal closures harder to assess at a glance.
The non-additive harm decomposition is introduced with welfare loss W and coverage loss C, but the abstract does not define these quantities or their non-additivity explicitly, which would clarify how attrition shifts harm.
The simulator is described as having a small extensible Python interface with calibration to DSA data, but the manuscript would benefit from including at least one concrete code example or API snippet in an appendix or supplementary section to aid reproducibility.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their accurate and positive summary of our work, including the formalization of continuous auditing as a T-round Stackelberg game, the identification of the cover regime (Observation 1), the orthogonal policy extensions, the OffAuditDrift strategy, the non-additive harm decomposition, the strategy library calibrated to DSA data, and the reproducible simulator. The recommendation for minor revision is noted.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper derives its central claim—the T-round Stackelberg game formalization and Observation 1 on the cover regime—from the explicit game definition, auditor policy commitments, and payoff structure without any reduction to fitted parameters, self-citations, or prior-work ansatzes. The observation is stated as a direct structural consequence of noise-aware static designs, with the two minimal extensions and OffAuditDrift strategy constructed explicitly to test its predictions. Simulator calibration draws on external DSA summary statistics rather than internal fits, and no step renames a known result or imports uniqueness from the authors' own prior work. The derivation remains self-contained against the stated model.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The paper rests on standard Stackelberg commitment and perfect-information assumptions from game theory plus domain assumptions about auditee capabilities (delay, drift within noise, attrition, cherry-picking) drawn from regulatory text; no new invented entities are introduced and no free parameters are fitted inside the central observation.

axioms (2)

domain assumption Auditees can choose actions (delay, drift, attrition, cherry-pick) each round without external detection costs beyond the modeled payoffs.
Invoked when defining the five auditee strategies and the OffAuditDrift counter-strategy.
standard math Auditor commits first to a temporal policy that the auditee observes.
Core of the T-round Stackelberg setup stated in the abstract.

pith-pipeline@v0.9.0 · 5599 in / 1550 out tokens · 28536 ms · 2026-05-08T04:52:51.339246+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages · 2 internal anchors

[1]

Procaccia, and Arunesh Sinha

Audit games with multiple defender resources. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1). A vailable at: https://doi.org/10.1609/aaai.v29i1.9317. Burnat, F. A. D., and Davidson, B. I. (May

work page doi:10.1609/aaai.v29i1.9317
[2]

arXiv [cs.CY]

The accountability paradox: How plat- form API restrictions undermine AI transparency mandates. arXiv [cs.CY] . A vailable at: https://doi.org/10.48550/arXiv.2505.11577. Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., Sharkey, L., Krishna, S., Von Hagen, M., Alberti, S., Chan, A.,...

work page internal anchor Pith review doi:10.48550/arxiv.2505.11577
[3]

non-expert uplift

Black-box access is insuﬀicient for rigorous AI audits. In: The 2024 ACM Conference on Fairness, Accountability, and Transparency. New York, NY, USA: ACM, pp. 2254–2272. A vailable at: https://doi.org/10.1145/3630106.3659037. Dong, J., Roth, A., Schutzman, Z., Waggoner, B., and Wu, Z. S. (June

work page doi:10.1145/3630106.3659037 2024
[4]

Strategic classification from revealed preferences

Strategic classification from revealed preferences. In: Proceedings of the 2018 ACM Conference on Economics and Computation . New York, NY, USA: ACM, pp. 55–70. A vailable at: https://doi.org/10.1145/3219166.3219193. Drolsbach, C., and Pröllochs, N. (Dec

work page doi:10.1145/3219166.3219193 2018
[5]

arXiv [cs.SI]

Content moderation on social media in the EU: Insights from the DSA Transparency Database. arXiv [cs.SI] . A vailable at: https://doi.org/10.48550/arXiv.2312.04431. Ganev, G., Annamalai, M. S. M. S., and De Cristofaro, E. (June

work page doi:10.48550/arxiv.2312.04431
[6]

arXiv [cs.LG]

The elusive pur- suit of reproducing PATE-GAN: Benchmarking, auditing, debugging. arXiv [cs.LG] . A vailable at:https://doi.org/10.48550/arXiv.2406.13985. Hardt, M., Megiddo, N., Papadimitriou, C., and Wootters, M. (Jan

work page doi:10.48550/arxiv.2406.13985
[7]

Benjamin Birnbaum, Nikhil R Devanur, and Lin Xiao

Strategic classification. In: Proceedings of the 2016 ACM Conference on Innovations in The- oretical Computer Science . New York, NY, USA: ACM, pp. 111–122. A vailable at: https://doi.org/10.1145/2840728.2840730. He, C., Baranowska, N., Castañeira, J. A. E., Escriba, G., Juentgen, M., Via, A., Borge- sius, F. Z., and Biega, A. (Apr

work page doi:10.1145/2840728.2840730 2016
[8]

Co-designing for Compliance: Multi-party Computation Protocols for Post-Market Fairness Monitoring in Algorithmic Hiring

Co-designing for compliance: Multi-party com- putation protocols for post-market fairness monitoring in algorithmic hiring. arXiv [cs.CY]. A vailable at: https://doi.org/10.48550/arXiv.2602.01837. Heald, D. (Nov

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2602.01837
[9]

Financial Accountability and Management, 34(4), pp

Transparency-generated trust: The problematic theorization of public audit. Financial Accountability and Management, 34(4), pp. 317–335. A vailable at: https://doi.org/10.1111/faam.12175. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove,...

work page doi:10.1111/faam.12175
[10]

Holistic Evaluation of Language Models

Holistic evaluation of language models. arXiv [cs.CL]. A vailable at: https://doi.org/10.48550/arXiv.2211.09110. Parker, L. D., Schmitz, J., and Jacobs, K. (May

work page Pith review doi:10.48550/arxiv.2211.09110
[11]

Financial Accountability and Management, 37(2), pp

Auditor and auditee engagement with public sector performance audit: An institutional logics perspective. Financial Accountability and Management, 37(2), pp. 142–162. A vailable at: https://doi.org/10. 1111/faam.12243. Perdomo, J. C., Zrnic, T., Mendler-Dünner, C., and Hardt, M. (2020) Performative pre- diction. In: Proceedings of the 37th International C...

2020
[12]

Academy of Management Review

Modelling the microfoundations of the audit society: Organizations and the logic of the audit trail. Academy of Management Review . A vailable at: https: //doi.org/10.5465/amr.2017.0212. Raji, I. D., Xu, P., Honigsberg, C., and Ho, D. (July

work page doi:10.5465/amr.2017.0212 2017
[13]

In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society

Outsider oversight: Design- ing a third party audit ecosystem for AI governance. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society . New York, NY, USA: ACM. A vailable at:https://doi.org/10.1145/3514094.3534181. Shore, C., and Wright, S. (June

work page doi:10.1145/3514094.3534181 2022
[14]

Current Anthropology, 56(3), pp

Audit culture revisited: Rankings, ratings, and the reassembling of society. Current Anthropology, 56(3), pp. 421–444. A vailable at: https://doi.org/10.1086/681534. Srivastava, A. (2023) Beyond the imitation game: Quantifying and extrapolating the ca- pabilities of language models. Transactions on Machine Learning Research . A vailable at: https://openre...

work page doi:10.1086/681534 2023
[15]

Proceedings of the ACM on Human-Computer Interaction , 9(2), pp

The DSA Transparency Database: Auditing self-reported moderation actions by social media. Proceedings of the ACM on Human-Computer Interaction , 9(2), pp. 1–28. A vailable at: https://doi.org/10. 1145/3711085. Yang, Y.-T., Zhang, T., and Zhu, Q. (2025) Herd accountability of privacy-preserving algorithms: A Stackelberg game approach. IEEE Transactions on ...

work page doi:10.1109/tifs.2025 2025
[16]

The trade-off appears structurally binary in this configuration: a floor that conditions only on the level of 𝑛𝑡 exposes the same signal under strategic attrition and under a legitimately small operating population, suggesting why richer adaptive frontiers conditioning on 𝑛𝑡 trajectories (not just levels) are the natural follow-on direction. (b) Suspicion...

2024