REVIEW 4 major objections 5 minor 54 references
Legal aid eligibility and court outcomes: a design-based double-machine-learning approach
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Being denied legal aid and hiring a private lawyer reduces a defendant's probability of incarceration by about 8 to 10 percentage points, according to this study.
desk verdict First credible attempt to quantify legal aid denial effects, with a plausible but not airtight unconfoundedness story; the -9.7pp estimate survives its own checks but not the missing-offense-type worry. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Double machine learning with the Interactive Regression Model, estimated with random forests, is the engine of the analysis. The model learns two unknown functions from more than 70 application fields: the outcome regressions for each treatment arm and the propensity score — the probability that aid is denied given all recorded inputs. Because every input that the legal aid office uses to decide aid is observed, the author argues that unconfoundedness holds, so reweighting by the estimated propensity score removes selection bias. The estimated propensity score is also used to check common support and to test sensitivity to unobserved confounding against benchmark covariates.
What would settle it
Include the free-text statement about the alleged offence (the one application field not in the study dataset) as a covariate in the same double-machine-learning model. If the estimated 9.7 percentage-point reduction in incarceration shrinks materially toward zero once that text or a measure of discretionary judgment is added, then unconfoundedness fails and the causal effect is overstated.
Extended reading notes
Core claim
The central discovery is a performance gap: legal aid applicants who fail the means test and hire a private lawyer end up with better court outcomes on the margin that matters most — avoiding prison. The preferred estimate for private versus public representation is a reduction of 9.7 percentage points in the probability of being incarcerated, while the aid-denial estimate is 8.1 percentage points. The paper also finds that denied applicants are less likely to plead guilty to their highest charge, more likely to be fined, and, if they are incarcerated, tend to serve longer sentences. The author interprets this combination as plea-bargaining behavior by public defenders: they secure guilty pleas and avoid long trials, which lowers average sentence length but raises the chance of any imprisonment.
Load-bearing premise
The whole causal interpretation rests on the assumption that, after controlling for the recorded application fields, whether aid is denied is unrelated to anything else that also affects the court outcome — in particular, that legal aid officers never use unrecorded information such as the applicant's free-text statement or discretionary judgment about lifestyle.
Editorial extensions
If this is right
- If the estimate is correct, denying legal aid to a defendant who can afford a private lawyer does not, on average, worsen their outcome: it lowers the chance of prison by roughly 8 to 10 percentage points.
- The result implies that public representation, as currently funded, produces a different case strategy — more plea bargaining and more incarceration but shorter spells — so increasing resources or time per case could reduce prison rates among legal aid clients.
- Because the private lawyers in the sample are low-fee lawyers hired by people who nearly qualified for aid, the true gap between public representation and the broader private market is likely even larger.
- The design-based double-machine-learning approach — known assignment inputs, latent assignment function — transfers to other public programs where eligibility rules are partly standards-based and all decision inputs are recorded.
Reading between the lines
- The estimated gap may overstate a pure lawyer-quality effect if the denial decision uses unrecorded signals of the defendant's ability to marshal resources, such as family support or access to bail.
- A sharper test of the plea-bargaining mechanism would use the as-good-as-random assignment of cases to in-house lawyers within an office: if the gap persists among randomly allocated lawyers, workload rather than lawyer selection is the driver.
- If public defenders' heavy reliance on plea bargains explains the pattern, capping caseloads or funding more expert reports for public defense would be testable policies to close the incarceration gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper uses linked administrative data from Legal Aid NSW and BOCSAR's Re-Offending Database for serious criminal matters in New South Wales (2012-2021). It estimates the effect of legal aid denial on court outcomes using double machine learning, specifically the Interactive Regression Model with random forests, targeting the average treatment effect on the treated. The headline finding is that denial of aid, and—after dropping self-represented defendants—private representation, reduces the probability of incarceration by 8.1 to 9.7 percentage points, alongside smaller significant effects on guilty pleas and fines and a positive effect on sentence length conditional on incarceration. The paper interprets this pattern as reflecting greater reliance on plea bargaining by public lawyers and concludes that a performance gap exists between public and private legal representation.
Significance. The paper addresses a policy-relevant and understudied question, and it benefits from an unusually rich administrative setting in which the inputs to the aid decision are largely recorded. The internal robustness of the extensive-margin incarceration estimate across the preferred DML specification, the trimmed-overlap specification, the AIPW sanity check, and the whole-sample specification is a genuine strength, as is the transparent reporting of the estimated propensity score and its overlap across treatment groups. If the unconfoundedness assumption can be more convincingly defended, the paper would provide one of the first causal estimates of the effect of legal aid denial on court outcomes and would be of substantial value to both the indigent-defense literature and applied causal machine learning.
major comments (4)
- [Section 5, footnote 7; Section 2.5; Appendix A] The claim that all inputs to the aid decision are observed is not supported by the institutional detail in the manuscript. Section 2.5 lists 'the likely cost of the proceedings' and 'the overall financial position' as discretion factors, and the likely cost depends on the seriousness and complexity of the alleged offence. Appendix A contains no offence-type, offence-severity, or criminal-history variable, and the free-text statement on the circumstances of the offence is explicitly excluded from the dataset. Because discretion can grant aid to applicants who fail the means test when expected costs are high, unobserved case seriousness can induce a correlation between denial/private representation and incarceration even after conditioning on the 70+ covariates. This directly threatens the central ATT estimate in Table 4 (-0.097). The paper should add charge-level controls from ROD (for example, charge category, number of charges, prior record) or provide a convincing argument that the existing covariates fully capture case seriousness.
- [Section 6.5, Figures 5-6] The formal sensitivity analysis does not address the most plausible unobserved confounder, namely case seriousness. The two benchmarks used are private submission, which is strong on the treatment side but nearly zero on the outcome side, and the income variables, which are strong on both sides but have an estimated adversity parameter of rho = 0.014. A confounder with high treatment-side strength, high outcome-side strength, and rho close to one—the natural shape for omitted offence severity—is not represented in the benchmark points, and the displayed contours do not establish that such a confounder could not overturn the estimate. I recommend benchmarking against observable proxies for case seriousness from ROD, or reporting sensitivity bounds over a grid that explicitly includes high cfd, high cfy, and rho = 1.
- [Section 6.2, Table 4] Dropping self-represented defendants conditions on a post-treatment outcome. If aid denial causes some defendants to self-represent, the comparison in Table 4 is no longer the effect of denial but the effect of denial combined with hiring a private lawyer, and the difference between Table 3 (-0.081) and Table 4 (-0.097) could reflect selection on the excluded self-represented group rather than a representation effect. The paper should either state explicitly that the target parameter changes and discuss the selection, or model self-representation as part of the outcome.
- [Section 6.4, Table C8] The parametric AIPW sanity check does not agree with the preferred estimates for several outcomes, despite the text claiming the results are 'similar in magnitude and statistical significance'. In particular, Incarceration (months) changes from -0.124 (s.e. 0.830) in Table 3 to +5.072 (s.e. 1.532) in Table C8, and Reduced charges changes from -0.005 (s.e. 0.013) to -0.034 (s.e. 0.010). Because Section 7's proposed mechanism relies on a null unconditional incarceration duration, this discrepancy should be discussed, or the robustness claim should be restricted to the extensive-margin incarceration outcome.
minor comments (5)
- [Throughout] There are numerous typographical errors that should be corrected, including 'uncounfoundedness' in the introduction, 'asstes' in Table 1, 'moted' in Section 4, 'trail' in the discussion of Dietrich v. The Queen, 'V ariable' in figure titles, and 'This not is an input' in footnote 7.
- [Table 2 notes] The sentence 'the size of the treatment group is 13% of that of the treatment group, of legal aid recipients' is garbled and should be rewritten to state the treatment-control sample-size ratio clearly.
- [Section 6.3, Figure 2] The discussion of the propensity-score trimming test would benefit from a more explicit statement that the trimmed estimate in Table 5 targets a different population than the preferred estimate, even though the authors note this in general terms.
- [Section 5.2] The collider-bias caveat for the intensive-margin estimates is well taken, but the same logic applies to the self-represented sample restriction discussed in Table 4; a cross-reference would be helpful.
- [References and replication] The paper does not mention a data-availability or code-availability statement; given the use of administrative data and the reproducibility ambitions of the DML approach, a statement would strengthen the manuscript.
Circularity Check
No circularity: all target treatment-effect parameters are defined independently of the estimator, and the DML procedure uses cross-fitting rather than refitting the same data.
full rationale
The paper's central quantities are the ATE and ATT of aid denial (or of private versus public representation) on court outcomes. These are defined through potential-outcome contrasts in Equations (1)-(2), not through any fitted quantity. The DML implementation estimates the propensity score and outcome regressions as nuisance functions on cross-fitting folds and then constructs the treatment-effect estimator using orthogonal scores, so the same observations are not used to both fit and evaluate the target parameter. The institutional argument that all aid-decision inputs are recorded in the dataset is an identifying assumption for unconfoundedness; it could be wrong, but it is not a definitional equivalence between the estimand and the inputs. The private-versus-public analysis redefines the treatment, but it still estimates a standard causal contrast under the same unconfoundedness assumption, and no parameter is defined as the fitted value of another parameter. The sensitivity analysis in Section 6.5 benchmarks unobserved confounding against observed covariates such as private submission and income; this is a robustness test, not a circular step. The paper explicitly flags the intensive-margin estimates as correlational because of collider bias (Section 5.2), which shows it does not overclaim causal status for those quantities. The reference list contains no self-citations by the author; the cited methodological results (Chernozhukov et al. 2018a, 2022; Wager 2025; Athey et al. 2019) are external published or widely circulated works, not author-specific results invoked to force a conclusion. No self-definitional, fitted-input-called-prediction, self-citation, uniqueness-importation, ansatz-smuggling, or renaming pattern is present. The derivation chain is therefore self-contained in the relevant sense.
Assumptions & free parameters
free parameters (2)
- Random forest hyperparameters (n_trees=500, min_node_size=2, max_depth=5)
- Propensity score trimming threshold =
0.45
assumptions (6)
- domain assumption Unconfoundedness: conditional on observed covariates X (all inputs to the aid assignment decision), treatment assignment D is independent of potential outcomes Y(0), Y(1).
- domain assumption Common support: no unit has propensity score exactly 0 or 1.
- domain assumption Individualistic treatment / no interference: one applicant's aid denial does not affect another's outcome.
- domain assumption The free-text statement on the application form is not an input to the aid decision for serious crimes.
- domain assumption Case allocation to in-house lawyers within an office is as-good-as-random.
- domain assumption The BOCSAR-ROD linkage correctly matches Legal Aid applications to court outcomes.
Cite this review
Pith. "Pith review of Legal aid eligibility and court outcomes: a design-based double-machine-learning approach." pith.science (2026). https://pith.science/paper/2PM3L4Q3
@misc{pith2026260805211,
author = {Pith},
title = {Pith review of: Legal aid eligibility and court outcomes: a design-based double-machine-learning approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/2PM3L4Q3}},
note = {Machine review of arXiv:2608.05211}
}
read the original abstract
Equality before the law is a human right, and access to high-quality legal aid for indigent defendants is essential to enforce it. In a context where all defendants have access to a lawyer, I study the impact of denying legal aid on court outcomes. I combine double machine learning and a new administrative dataset linking aid to court outcomes in New South Wales, Australia, to learn the assignment function, whose inputs are known. I find that applicants who fail the means test and hire private lawyers are 10 percentage points less likely to be incarcerated than if they passed and relied on legal aid. Given an average incarceration length of nearly four years, this gap is significant. However, I find evidence suggesting that they spend more time in jail if they are incarcerated. A government preference for broad access to aid over allocated time per case could explain this pattern. Keywords: Indigent Defense, Crime, Criminal Justice. JEL: I30, K14, H44.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
8667.0 - Legal Services, Australia, 2007-08
ABS (2008). 8667.0 - Legal Services, Australia, 2007-08 . Technical report. Accessed 2 November 2023
work page 2008
-
[2]
Agan, A., Freedman, M., and Owens, E. (2021). Is Your Lawyer a Lemon? Incentives and Selection in the Public Provision of Criminal Defense . The Review of Economics and Statistics , 103(2):294--309
work page 2021
-
[3]
Anderson, J. M. and Heaton, P. (2012). How much difference does the lawyer make: The effect of defense counsel on murder case outcomes. Yale Law Journal , 122(1):154--217
work page 2012
-
[4]
Athey, S., Tibshirani, J., and Wager, S. (2019). Generalized random forests . The Annals of Statistics , 47(2):1148 -- 1178
2019
-
[5]
S., and Spindler, M
Bach, P., Chernozhukov, V., Kurz, M. S., and Spindler, M. (2022). DoubleML -- A n object-oriented implementation of double machine learning in P ython. Journal of Machine Learning Research , 23(53):1--6
2022
-
[6]
S., Chernozhukov, V., Spindler, M., and Klaassen, S
Bach, P., Kurz, M. S., Chernozhukov, V., Spindler, M., and Klaassen, S. (2024). DoubleML : An object-oriented implementation of double machine learning in R . Journal of Statistical Software , 108(3):1--56
work page 2024
-
[7]
Banack, H. R., Mayeda, E. R., Naimi, A. I., Fox, M. P., and Whitcomb, B. W. (2023). Collider stratification bias i: Principles and structure. American Journal of Epidemiology , 193(2):238--240
work page 2023
-
[8]
Barreca, A. I., Guldi, M., Lindo, J. M., and Waddell, G. R. (2011). Saving Babies? Revisiting the effect of very low birth weight classification* . The Quarterly Journal of Economics , 126(4):2117--2123
work page 2011
Show all 54 references
-
[9]
Bertrand, M., Crépon, B., Marguerie, A., and Premand, P. (2021). Do workfare programs live up to their promises? experimental evidence from cote d’ivoire. Working Paper 28664, National Bureau of Economic Research
2021
-
[10]
Breiman, L. (2001). Random forests. Machine Learning , 45(1):5--32
2001
-
[11]
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018a). Double/debiased machine learning for treatment and structural parameters . The Econometrics Journal , 21(1):C1--C68
2018
-
[12]
K., Sharma, A., and Syrgkanis, V
Chernozhukov, V., Cinelli, C., Newey, W. K., Sharma, A., and Syrgkanis, V. (2022). Long story short: Omitted variable bias in causal machine learning. Working Paper 30302, National Bureau of Economic Research
2022
-
[13]
Chernozhukov, V., Demirer, M., Duflo, E., and Fernández-Val, I. (2018b). Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in india. Working Paper 24678, National Bureau of Economic Research
2018
-
[14]
Davis, J. M. and Heller, S. B. (2020). Rethinking the Benefits of Youth Employment Programs: The Heterogeneous Effects of Summer Jobs . The Review of Economics and Statistics , 102(4):664--677
2020
-
[15]
C., and Lepri, B
De Nadai, M., Xu, Y., Letouzé, E., González, M. C., and Lepri, B. (2020). Socio-economic, built environment, and mobility conditions associated with crime: a study of multiple cities. Scientific Reports , 10(1):13871
2020
-
[16]
H., Molitor, D., and Reif, J
Deryugina, T., Heutel, G., Miller, N. H., Molitor, D., and Reif, J. (2019). The mortality and medical costs of air pollution: Evidence from changes in wind direction. American Economic Review , 109(12):4178--4219
2019
-
[17]
The Queen (1992)
Dietrich v. The Queen (1992). F.C. No. 92/044 . ://www8.austlii.edu.au/cgi-bin/viewdoc/au/cases/cth/HCA/1992/57.html
1992
-
[18]
Forell, S., Cain, M., and Gray, A. (2010). Recruitment and retention of lawyers in regional, rural and remote new south wales. Working paper, Law and Justice Foundation of NSW, Sydney
2010
-
[19]
Greifer, N. (2024). cobalt: Covariate Balance Tables and Plots . R package version 4.5.5
2024
-
[20]
Imbens, G. W. (2004). Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review . The Review of Economics and Statistics , 86(1):4--29
2004
-
[21]
Imbens, G. W. and Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences . Cambridge University Press
2015
-
[22]
and Sawada, M
Ishihara, T. and Sawada, M. (2022). Manipulation-robust regression discontinuity designs. arXiv preprint arXiv:2009.07551
2022 arXiv
-
[23]
Iyengar, R. (2007). An analysis of the performance of federal indigent defense counsel. Working Paper 13187, National Bureau of Economic Research
2007
-
[24]
Jacobsen, G. (2011). Legal aid lawyers buckle under work stress. Sydney Morning Herald. https://www.smh.com.au/national/nsw/legal-aid-lawyers-buckle-under-work-stress-20110905-1ju9v.html. [Online; accessed 27-11-2023]
2011
-
[25]
Robins, A
James M. Robins, A. R. and Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association , 89(427):846--866
1994
-
[26]
Robins, A
James M. Robins, A. R. and Zhao, L. P. (1995). Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the American Statistical Association , 90(429):106--121
1995
-
[27]
Knaus, M. C. (2022). Double machine learning-based programme evaluation under unconfoundedness . The Econometrics Journal , 25(3):602--627
2022
-
[28]
C., Lechner, M., and Strittmatter, A
Knaus, M. C., Lechner, M., and Strittmatter, A. (2022). Heterogeneous employment effects of job search programs. Journal of Human Resources , 57(2):597--636
2022
-
[29]
Lee, A. J. (2021). Flat fee compensation, lawyer incentives, and case outcomes in indigent criminal defense
2021
-
[30]
Legal aid commission act 1979 no 78 (nsw)
Legal Aid Commission Act (1979). Legal aid commission act 1979 no 78 (nsw). ://legislation.nsw.gov.au/view/whole/html/inforce/current/act-1979-078
1979
-
[31]
Who we are
Legal Aid NSW (2017). Who we are. https://www.legalaid.nsw.gov.au/about-us/who-we-are. [Online; accessed 28-03-2023]
2017
-
[32]
Policies in brief
Legal Aid NSW (2022). Policies in brief. https://www.legalaid.nsw.gov.au/__data/assets/pdf_file/0008/7757/Policies-in-Brief.pdf. [Brochure; accessed 28-03-2023]
2022
-
[33]
Policy online
Legal Aid NSW (2023). Policy online. http://www.legalaid.nsw.gov.au/for-lawyers/policyonline. [Online; accessed 28-03-2023]
2023
-
[34]
McCannon, B. C. and Porreca, Z. (2025). The right to counsel: criminal prosecution in 19th century london. Economica , 92(365):285--321
2025
-
[35]
McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics , 142(2):698--714. The regression discontinuity design: Theory and applications
2008
-
[36]
Millane, E., Jackson, A., and Blane, N. (2023). Justice on the brink: Stronger legal aid for a better legal system. https://nationallegalaid.org.au/policy-and-advocacy/reports/justice-on-the-brink Link . Report prepared by Impact Economics and Policy for National Legal Aid. [O...
2023
-
[37]
T., and Maule, M
Moccia, C., Moirano, G., Popovic, M., Pizzi, C., Fariselli, P., Richiardi, L., Ekstrøm, C. T., and Maule, M. (2024). Machine learning in causal inference for epidemiology. European Journal of Epidemiology
2024
-
[38]
I., Mishler, A
Naimi, A. I., Mishler, A. E., and Kennedy, E. H. (2020). Challenges in obtaining valid causal effect estimates with machine learning algorithms
2020
-
[39]
J., Poynton, S., and Weatherburn, D
Ooi, E. J., Poynton, S., and Weatherburn, D. (2019). The impact of private versus public legal representation on criminal proceedings. Crime and Justice Bulletin , 221. Sydney: NSW Bureau of Crime Statistics and Research
2019
-
[40]
Access to justice arrangements: Productivity commission inquiry report, volume 2
Productivity Commission (2014). Access to justice arrangements: Productivity commission inquiry report, volume 2. https://www.pc.gov.au/inquiries/completed/access-justice/report/access-justice-volume2.pdf. [No. 72, 5 September 2014; Online; accessed 27-11-2023]
2014
-
[41]
Roach, M. A. (2014). Indigent Defense Counsel, Attorney Quality, and Defendant Outcomes . American Law and Economics Review , 16(2):577--619
2014
-
[42]
Rosenbaum, P. R. and Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects . Biometrika , 70(1):41--55
1983
-
[43]
Schwall, B. (2017). Getting what you paid for: how payment methods affect attorneys' behaviors. Working paper, Clemson University working paper
2017
-
[44]
Shem-Tov, Y. (2022). Make or Buy? The Provision of Indigent Defense Services in the United States . The Review of Economics and Statistics , 104(4):819--827
2022
-
[45]
Sullivan, K. M. (1992). The supreme court, 1991 term---foreword: The justices of rules and standards. Harvard Law Review , 106:24--118. Cited at p. 58
1992
-
[46]
Criminal proceeds confiscation act 2002 - sect
The Parliament of Australia (2002). Criminal proceeds confiscation act 2002 - sect. 104. ://classic.austlii.edu.au/au/legis/qld/consol_act/cpca2002285/s104.html
2002
-
[47]
History of the Peloponnesian War
Thucydides (1874). History of the Peloponnesian War . Retrieved via Project Gutemberg. Translated by Richard Crawley (1874)
-
[48]
Universal Declaration of Human Rights
United Nations (1948). Universal Declaration of Human Rights
1948
-
[49]
United nations principles and guidelines on access to legal aid in criminal justice systems
United Nations (2012). United nations principles and guidelines on access to legal aid in criminal justice systems. General Assembly resolution . ://www.unodc.org/documents/justice-and-prison-reform/UN_principles_and_guidlines_on_access_to_legal_aid.pdf
2012
-
[50]
Wager, S. (2025). Causal Inference: A Statistical Learning Approach
2025
-
[51]
and Athey, S
Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association , 113(523):1228--1242
2018
-
[52]
and Lind, B
Weatherburn, D. and Lind, B. (1998). Poverty, parenting, peers and crime-prone neighbourhoods. Trends & Issues in Crime & Criminal Justice , (85)
1998
-
[53]
Wright, M. N. and Ziegler, A. (2017). ranger : A fast implementation of random forests for high dimensional data in C++ and R . Journal of Statistical Software , 77(1):1--17
2017
-
[54]
H., Bodnar, L
Zhong, Y., Kennedy, E. H., Bodnar, L. M., and Naimi, A. I. (2021). Aipw: An r package for augmented inverse probability weighted estimation of average causal effects. American Journal of Epidemiology , 190:2690--2699
2021
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.