REVIEW 4 major objections 4 minor 45 references
AI for Scaling Legal Reform: Mapping and Redacting Racial Covenants in Santa Clara County
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A $258 AI run finds 7,500 racial covenants in Santa Clara County deeds.
desk verdict Genuinely useful deployment paper, but the headline metrics come from a non-representative eval set and deployment recall is unmeasured; the contribution survives, the numbers need reframing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a LoRA-finetuned Mistral 7B language model, where LoRA (low-rank adaptation) updates a small set of weight matrices rather than the full model. The model is fed OCR text from deed pages: docTR converts page images to text, the model classifies each page and, when positive, returns the exact covenant passage, and a confidence score based on the softmax of the 'yes' and 'no' tokens, thresholded at 75 percent, filters low-confidence predictions. A separate geolocation pipeline extracts map names and book-and-page references from deed text and matches them to county surveyor maps, which is what turns a document classifier into a map of housing discrimination.
What would settle it
Take a period-stratified random sample of 300 deed pages from 1940 to 1980 in Santa Clara County, have two independent legal annotators label them for racial covenants, and run the released model at the 75 percent confidence threshold with the fair-housing filter. If precision on that sample falls below 95 percent, or recall falls below the claimed 0.99, the deployment-time accuracy claim is falsified.
Extended reading notes
Core claim
The central discovery is that a 7-billion-parameter open language model, finetuned with low-rank adaptation on 3,801 annotated deed pages, can outperform keyword search, fuzzy matching, and zero- or few-shot prompting of an off-the-shelf closed model in detecting racial covenants. On the paper's own evaluation set it achieves perfect precision and 99.4 percent recall, with a span-level BLEU of 0.932, and it runs for a fraction of the cost of a proprietary API. The authors go on to show that the model's outputs can be integrated into a legally required human review workflow, with county counsel confirming more than 4,500 of the identified covenants, and that the resulting dataset reveals sharp geographic clustering and the outsized role of about ten developers. Their population-level estimate is that by 1950 one in four properties in the county was encumbered by a racial covenant.
Load-bearing premise
The evaluation set that produced the precision-1.0 and recall-0.99 numbers contains only pre-1940s Santa Clara County deeds, while the deployed scan covers 1902 to 1980, so the reported accuracy may not hold for the later documents unless the confidence threshold and fair-housing filter fix the drift.
Editorial extensions
If this is right
- Counties facing AB 1466-style mandates can complete a full deed sweep without years of manual review; the paper estimates roughly 86,500 person-hours saved for Santa Clara County's 5.2 million pre-1980 pages.
- Open-weight models make comprehensive deed scanning economical; the paper's comparison puts the full county run at $258, under 2 percent of the cost of an off-the-shelf closed model.
- The roughly 7,500 identified covenant deeds, once geolocated, produce the paper's historical findings: one in four properties covered by 1950, ten developers responsible for about a third of covenants, and continued use after Shelley v. Kraemer.
- Releasing the fine-tuned model and review interface gives other jurisdictions a starting point, provided they validate on their own records as the paper recommends.
- The county's redaction workflow retains an unredacted registry, so removing discriminatory language from active records does not erase the historical evidence.
Reading between the lines
- Editorial inference: the same pipeline should extend to other discriminatory restrictions, such as religion, family status, or national origin, because the paper already reports surfacing a family-status covenant and few-shot prompting or further finetuning would likely retrieve those with the same architecture.
- Editorial inference: a period-stratified audit of 1960s and 1970s deeds would likely show lower precision than the headline 1.0, since the evaluation set is pre-1940s and the paper itself documents fair-housing false positives in later documents.
- Editorial inference: OCR quality, not model quality, is the binding constraint for older records; the paper excludes pre-1902 handwritten deeds for lack of effective OCR, so better handwriting recognition would extend coverage backward in time.
- Editorial inference: the approximately $258 cost and under-2-percent comparison are time-bound to current GPU rental and API prices, though the qualitative advantage of open models at this scale should persist.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes an academic-government partnership that built an OCR plus language-model pipeline to detect racially restrictive covenants in Santa Clara County's pre-1980 deed records. The authors fine-tune Mistral 7B with LoRA and report page-level precision 1.000 and recall 0.994 on a held-out set of 739 annotated pages, compare against keyword and GPT-3.5 baselines, and estimate that running the pipeline over 5.2 million pages costs $258 and saves 86,500 person-hours of manual review. Using the detector's outputs, geolocation, and census data, the paper estimates roughly 7,500 covenant deeds and that one in four 1950 properties were covered by racial covenants, and it derives historical, geographic, and developer-level findings. The paper also documents the integration of the model into the county's AB 1466 redaction workflow and states that the model and review interface will be released.
Significance. If the performance claims held up, this would be a significant contribution: a concrete, deployed demonstration that fine-tuned open LLMs can reduce the cost of legal-record review by orders of magnitude, plus a population-scale quantitative map of racial covenants. The partnership details, the random-sample deployment precision check (200 positives, 96.4-99.7% CI), the manual verification of geolocations on a 50-document sample, and the transparent discussion of deployment false positives are genuine strengths. The manuscript is also careful in Section 8 to list directional biases in the prevalence estimate. However, the central evaluation claim is currently not supported by the evidence as presented, because the evaluation set does not represent the deployment corpus and deployment recall is unmeasured; the historical and prevalence findings inherit that gap. The contribution remains valuable, and the gap is addressable with additional validation rather than being fatal to the approach.
major comments (4)
- [5.2 / Table 2 and Section 6] The headline evaluation is not representative of the deployment population. The 739-page evaluation set described in Section 5.2 is drawn from the annotation pool constructed in Section 3.3, which began with 3,000 keyword-selected pages and then added model predictions and hard negatives; about 70% of the evaluation pages contain covenants, whereas the full corpus has fewer than 2 per 1,000. Section 6 further states that the Santa Clara County evaluation documents were 'exclusively pre-1940s' while the deployed system processes 5.2 million pages from 1902 to 1980, and that later-period documents produced fair-housing false positives requiring a post hoc confidence threshold and a 'fair housing' filter. The deployment precision estimate from 200 sampled positives (96.4-99.7%) is a useful check, but deployment recall was never measured. Therefore the abstract's 'precision of 1.0 and a recall of 0.99' and the downstream 7,500-deed count are not supported as estimates for the full corpus.
- [4.2 and Section 6] The 75% confidence threshold was chosen on the same evaluation set used to report the headline metrics. Section 4.2 says the threshold was 'empirically determined' and that the Section 5.1 results are computed at that threshold; because the same 739 pages were used for threshold selection and for the reported precision/recall, the numbers in Table 2 are selection-optimized. Section 6 then changes the operating point again at deployment by adding the 75% threshold and a 'fair housing' filter, without measuring recall on a sample from the deployment distribution. The paper should either use a separate validation set for threshold selection or present the metrics as conditional on the tuning set, and it should add a deployment recall estimate on a stratified random sample.
- [8] The estimate that one in four properties were covered by a racial covenant in 1950 inherits the unmeasured recall error described above. The count of 24,522 lots is built from detector outputs for neighborhood-wide declarations (412 manually confirmed), multi-lot deeds (1,293, of which only 20 units were verified according to the footnote), and single-lot deeds; no sensitivity analysis varies the detector's recall or the classification of multi-lot deeds. The manuscript lists directional biases that could move the estimate up or down, but a quantitative lower/upper bound or confidence interval is needed for the headline prevalence claim.
- [7.3] The historical periodization, including the 'near 75% decrease' after Shelley and the persistence of covenants after 1948, compares detector counts across decades. Because detection recall is unvalidated outside the pre-1940s Santa Clara evaluation documents, and later-period documents are known to contain different language (e.g., fair-housing covenants), the observed drop could partly reflect a decline in detection sensitivity rather than a decline in covenant usage. A stratified recall evaluation by decade is required before these historical conclusions are stated as results.
minor comments (4)
- [3.2] The text says the external data came from 'seven counties' but then lists six counties (Bexar, Cuyahoga, Denton, Franklin, Hidalgo, and Lawrence); please correct the count or add the missing county.
- [Appendix E] Appendix E states that the total cost for an off-the-shelf LLM would be 'roughly $30,000,' but Table 1 reports $13,634 for GPT-3.5 few-shot and $47,944 for GPT-4 Turbo; please reconcile these numbers.
- [1 and 7] The phrase 'one in four properties the county were covered' is missing the word 'in' in both Section 1 and Section 7's opening; also 'Mistral 7B parameter open source model' should read 'Mistral 7B-parameter open source model.'
- [Table 3] The column header 'Native White' is ambiguous; the table appears to report 'White' and 'Native American' as separate columns, and the 1980 'Others' spike is explained only in the caption; please clarify the column labels in the header.
Circularity Check
No significant circularity: the model is evaluated on held-out human annotations and the downstream historical/cost claims rest on external data sources, not on the model's own fitted parameters.
full rationale
The paper's central empirical chain is not circular. The finetuned Mistral detector is trained on one portion of the human-annotated deed set and evaluated on a held-out 739-page subset, with precision, recall, F1, and BLEU reported directly from that held-out comparison. The claimed operational savings (86,500 person-hours, $258 cost) are computed from stated throughput and cloud-pricing assumptions in Appendix E, not from the detector's own outputs. The historical findings depend on model predictions, but the key downstream quantities are separately anchored: the roughly 7,500 covenant deeds are a model-output count later checked by sampling 200 positives, and the 'one in four properties by 1950' estimate is built from lot counts derived from surveyor's maps, manual confirmation of the 412 neighborhood-wide covenants, and U.S. Census dwelling-unit totals. Several author-team self-citations appear (e.g., Magesh et al., 2024, for LLM hallucination; Engstrom et al., 2020, for government AI), but none carries a load-bearing mathematical or empirical claim that reduces to the paper's own result. The evaluation-set composition concerns raised by the reader are real external-validity limitations, not circularity: a keyword-enriched, pre-1940-dominated evaluation set can overstate deployment performance on later-period documents, but the paper does not define its performance metric in terms of the deployment outputs or fit a parameter and then relabel that parameter as a prediction. The paper even discloses the later-period false positives and separately estimates deployment precision from a fresh random sample, which is the opposite of hiding the input-output equivalence. Therefore, no step in the derivation chain is equivalent by construction to its own inputs.
Assumptions & free parameters
free parameters (2)
- confidence_threshold =
0.75
- fuzzy_keyword_threshold =
0.75
assumptions (5)
- domain assumption OCR text is a sufficient representation of the deed page content for covenant detection
- domain assumption Human annotations constitute ground truth for what a racial covenant is
- domain assumption The evaluation set is representative of the full 5.2 million page corpus
- domain assumption Recorded deed references to surveyor maps can be matched to present-day geography
- domain assumption Census dwelling-unit counts are commensurate with 'properties' for the prevalence denominator
Cite this review
Pith. "Pith review of AI for Scaling Legal Reform: Mapping and Redacting Racial Covenants in Santa Clara County." pith.science (2026). https://pith.science/paper/PU42URIM
@misc{pith2026250303888,
author = {Pith},
title = {Pith review of: AI for Scaling Legal Reform: Mapping and Redacting Racial Covenants in Santa Clara County},
year = {2026},
howpublished = {\url{https://pith.science/paper/PU42URIM}},
note = {Machine review of arXiv:2503.03888}
}
read the original abstract
Legal reform can be challenging in light of the volume, complexity, and interdependence of laws, codes, and records. One salient example of this challenge is the effort to restrict and remove racially restrictive covenants, clauses in property deeds that historically barred individuals of specific races from purchasing homes. Despite the Supreme Court holding such racial covenants unenforceable in 1948, they persist in property records across the United States. Many jurisdictions have moved to identify and strike these provisions, including California, which mandated in 2021 that all counties implement such a process. Yet the scale can be overwhelming, with Santa Clara County (SCC) alone having over 24 million property deed documents, making purely manual review infeasible. We present a novel approach to addressing this pressing issue, developed through a partnership with the SCC Clerk-Recorder's Office. First, we leverage an open large language model, finetuned to detect racial covenants with high precision and recall. We estimate that this system reduces manual efforts by 86,500 person hours and costs less than 2% of the cost for a comparable off-the-shelf closed model. Second, we illustrate the County's integration of this model into responsible operational practice, including legal review and the creation of a historical registry, and release our model to assist the hundreds of jurisdictions engaged in similar efforts. Finally, our results reveal distinct periods of utilization of racial covenants, sharp geographic clustering, and the disproportionate role of a small number of developers in maintaining housing discrimination. We estimate that by 1950, one in four properties across the County were subject to racial covenants.
Figures
Figures from the paper (15 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Ashley Bakelmun and Sarah Jane Shoenfeld. 2019. Open data and racial segregation: Mapping the historic imprint of racial covenants and redlining on american cities. In Open Cities| Open Data: Collaborative Cities in the Information Era, pages 57--83. Springer
work page 2019
-
[4]
Richard R W Brooks. 2011. Covenants without courts: enforcing residential segregation with legally unenforceable agreements. American Economic Review, 101(3):360--365
work page 2011
-
[5]
Richard R. W. Brooks and Carol M. Rose. 2013. Saving the Neighborhood: Racially Restrictive Covenants, Law, and Social Norms . Harvard University Press
work page 2013
-
[6]
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77--91. PMLR
2018
- [7]
-
[8]
City Roots Community Land Trust and Yale Environmental Protection Clinic . 2020. https://law.yale.edu/sites/default/files/area/clinic/document/2020.7.31_-_confronting_racial_covenants_-_yale.city_roots_guide.pdf Confronting Racial Covenants: How They Segregated Monroe County and What to Do About Them
work page 2020
Show all 45 references
-
[9]
Ho, Catherine Sharkey, and Mariano-Florentino Cuéllar
David Freeman Engstrom, Daniel E. Ho, Catherine Sharkey, and Mariano-Florentino Cuéllar. 2020. https://www.acus.gov/sites/default/files/documents/Government Administrative Conference of the United States
2020
-
[10]
Jeffrey D Gonda. 2015. Unjust deeds: The restrictive covenant cases and the making of the civil rights movement. UNC Press Books
2015
-
[11]
Kevin Fox Gotham. 2000. Urban space, restrictive covenants and the origins of racial residential segregation in a US city, 1900--50. International Journal of Urban and Regional Research, 24(3):616--633
2000
-
[12]
Mary L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books
2019
-
[13]
Eunice Grier and George Grier. 1960. Privately Developed Interracial Housing: An Analysis of Experience. University of California Press, Berkeley, CA
1960
-
[14]
April Hope Halberstadt. 1997. The Willow Glen Neighborhood: Then and Now . Renacsi
1997
-
[15]
Lexi Purich Howard. 2021. California swings for the fences to strike racially restrictive covenants from the public record. California Real Property Law Journal, 39(4)
2021
-
[16]
Ocean Howell. 2016. https://doi.org/10.1525/phr.2016.85.3.379 The Merchant Crusaders: Eichler Homes and Fair Housing, 1949–1974 . Pacific Historical Review, 85(3):379--407
2016 doi
-
[17]
Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://doi.org/10.48550/arXiv.2106.09685 LoRA : Low - Rank Adaptation of Large Language Models . arXiv preprint. ArXiv:2106.09685 [cs]
- [18]
-
[19]
Michael Jones-Correa. 2000. The origins and diffusion of racial restrictive covenants. Political Science Quarterly, 115(4):541--568
2000
-
[20]
Ho, Percy Liang, and Arvind Narayanan
Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen K Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, Rumman Chowdhury, Alex Engler, Peter Henderson, Yacine Jernite, Seth Lazar, Stefano Maffulli, Alondra Nelson, Joelle Pi...
2024
-
[21]
Steve Kelman. 1990. Procurement and public management. Technical report, American Enterprise Institute
1990
-
[22]
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565--6576. PMLR
2021
-
[23]
Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. 2020. Real-time scene text detection with differentiable binarization. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11474--11481
2020
-
[24]
Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D Manning, and Daniel E Ho. 2024. Hallucination-free? A ssessing the reliability of leading AI legal research tools. arXiv preprint arXiv:2405.20362
2024 arXiv
-
[25]
Mapping Prejudice Project . 2022. https://mappingprejudice.umn.edu/racial-covenants/what-is-a-covenant What is a Covenant?
2022
-
[26]
Olivia Martin, Faiz Surani, Kit Rodolfa, Amy Perez, and Daniel E. Ho. 2024. The Spectrum of AI Integration : The Case of Benefits Adjudication . In Cynthia H. Cwik, Christopher A. Suarez, and Lucy L. Thomson, editors, AI : Legal Issues , Policy , and Practical Strategies . Ame...
2024
-
[27]
Mindee. 2021. doctr: Document text recognition. https://github.com/mindee/doctr
2021
-
[28]
William R Ming. 1949. Racial restrictions and the fourteenth amendment: The restrictive covenant cases. The University of Chicago Law Review, 16(2):203--238
1949
-
[29]
Jennifer Pahlka. 2023. Recoding America: why government is failing in the digital age and how we can do better. Metropolitan Books
2023
-
[30]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318
2002
-
[31]
Jonathan G Pray. 2005. Congressional reporting requirements: Testing the limits of the oversight power. U. Colo. L. Rev., 76:297
2005
-
[32]
Laura Redford. 2017. https://doi.org/10.1177/1538513216676191 The intertwined history of class and race segregation in los angeles . Journal of Planning History, 16(4):305--322
2017 doi
-
[33]
Loren Rhoads. 2018. https://cemeterytravel.com/2018/05/19/cemetery-of-the-week-167-oak-hill-memorial-park/ Cemetery of the Week \#167: Oak Hill Memorial Park
2018
-
[34]
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156
2018 arXiv
-
[35]
Florence Wagman Roisman. 2022. Stumbling stones at levittown: What to do about racial covenants in the united states. Journal of Affordable Housing Volume, 30(3)
2022
-
[36]
Carol M Rose. 2022. Property law and inequality: Lessons from racially restrictive convenants. Nw. UL Rev., 117:225
2022
-
[37]
Carol M Rose. 2024. General customs and legal institutions: The short, sad example of racially restrictive covenants in the United States . In Samuel L. Bray, John CP Goldberg, Paul B. Miller, and Henry E. Smith, editors, Interstitial Private Law. Oxford University Press
2024
-
[38]
Carol M Rose, Richard RW Brooks, A Brown, and V Smith. 2016. Racial Covenants and Housing Segregation, Yesterday and Today . Race and Real Estate, pages 161--76
2016
-
[39]
Richard Rothstein. 2017. The Color of Law: A Forgotten History of How Our Government egregated America . Liveright Publishing
2017
-
[40]
Larry Santucci. 2020. Documenting Racially Restrictive Covenants in 20th Century Philadelphia . Cityscape, 22(3):241--268
2020
-
[41]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
2014 arXiv
-
[42]
US Census Bureau . 1942. https://www.census.gov/library/publications/1942/dec/population-vol-1.html 1940 Census of Population : Volume 1. Number of Inhabitants . Total Population for States , Counties , and Minor Civil Divisions ; for Urban and Rural Areas ; for Incorporated P...
1942
-
[43]
Clement E Vose. 1967. Caucasians only: The Supreme Court, the NAACP, and the restrictive covenant cases. Univ of California Press
1967
-
[44]
KL Wagstaff. 2012. Machine learning that matters. In Proceedings of the 29th International Conference on Machine Learning, 2012, pages 529--536
2012
-
[45]
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1--34
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.