Measuring Average Treatment Effect from Heavy-tailed Data
read the original abstract
Heavy-tailed metrics are common and often critical to product evaluation in the online world. While we may have samples large enough for Central Limit Theorem to kick in, experimentation is challenging due to the wide confidence interval of estimation. We demonstrate the pressure by running A/A simulations with customer spending data from a large-scale Ecommerce site. Solutions are then explored. On one front we address the heavy tail directly and highlight the often ignored nuances of winsorization. In particular, the legitimacy of false positive rate could be at risk. We are further inspired by the idea of robust statistics and introduce Huber regression as a better way to measure treatment effect. On another front covariates from pre-experiment period are exploited. Although they are independent to assignment and potentially explain the variation of response well, concerns are that models are learned against prediction error rather than the bias of parameter. We find the framework of orthogonal learning useful, matching not raw observations but residuals from two predictions, one towards the response and the other towards the assignment. Robust regression is readily integrated, together with cross-fitting. The final design is proven highly effective in driving down variance at the same time controlling bias. It is empowering our daily practice and hopefully can also benefit other applications in the industry.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Multi-wavelength Emission for a Post-merger Magnetar: The Magnetar-Driven Poynting Jet and Its Associated Pulsar Wind Nebula
A dynamical model of magnetar-driven jet and PWN emission predicts a sequence of thermal, X-ray plateau, and late synchrotron/inverse-Compton radiation that accounts for key features in merger GRBs.
-
Spectral energy-loss bump and $\gamma$-ray pulsar halos
The curved spectrum of the young pulsar halo LHAASO J0248+6021 is explained by a time-dependent energy-loss bump in the electron spectrum that remains close to the cutoff, unifying it with the shifted bump observed in...
-
A 14-year-old Mystery: The Peculiar Case of the Engine-driven SN 2012ap
Late-time radio rebrightening in SN 2012ap is consistent with either progenitor mass-loss variation producing a density enhancement or an off-axis energetic jet viewed at large angle, potentially reclassifying it as G...
-
Guitar Nebula: extreme accelerator in extreme environment
The Guitar Nebula requires extreme acceleration with η_acc ≳ 3/4 and traverses a dense low-ionization shell from an old supernova remnant in the pressure-driven snowplow regime.
-
EP250827b/SN 2025wkm: An X-ray Flash-Supernova Powered by a Central Engine and Circumstellar Interaction
EP250827b/SN 2025wkm is an X-ray flash supernova at z=0.1194 powered by a long-lived magnetar and disk winds interacting with extended circumstellar medium, without an on-axis relativistic jet.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.