{"id":"17b59a1b-8bbf-4d82-aed4-ff04e4f8caf9","arxiv_id":"2505.24676","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hand-annotated dataset of 1933 Hamilton County property appraisals is extracted with template-aligned OCR, and a random forest trained on contemporary features estimates those historical values with about 17% MAPE.","lead":"This paper builds a machine-learning pipeline to read handwritten 1933 property appraisal cards from Hamilton County, Ohio, and releases the resulting dataset. It also trains a model that estimates historical home values from today's building records, so researchers can study redlining's financial effects without scanning millions of paper cards.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Regression model's core proxy assumption—that contemporary features reflect 1933 characteristics—is unverified and could bias historical estimates; a direct feature-comparison test on the scanned cards is needed.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the regression model uses contemporary building features as a proxy for 1933 characteristics. This is the most fundamental threat to the paper's central claim because the regression model is presented as a general tool for estimating historical assessments when scanned records are unavailable. If the proxy is biased, the model's estimates are invalid even in the training county, and the subsequent cross-county generalization is moot. The paper's bias analysis does not resolve the concern because it tests only whether errors correlate with contemporary demographics, not whether the proxy features themselves match historical records. The concrete test I propose directly compares historical and contemporary features on a sample of the same property cards, which would settle whether the assumption holds. I agree with the reader's conditional verdict: the paper's contributions—especially the released dataset and OCR pipeline—are real, but the regression model requires additional validation before its estimates can be used confidently. The concern is addressable, so conditional acceptance is appropriate rather than rejection.","tokens_in":19780,"tokens_out":6607,"duration_ms":76088,"concrete_test":"","verdict_should_be":"UNCHANGED","load_bearing_attack":"The regression model's validity rests on the unverified assumption stated in §3.2.1 that 'the vast majority of properties built before 1930 remain the same size,' allowing contemporary parcel data to proxy 1933 building characteristics. This assumption is load-bearing because the model is the paper's proposed solution for counties without scanned records, and it is also used to impute values for the 6,192 Hamilton County parcels where OCR failed (§3.2.4). If renovations or condition changes are systematically correlated with value (e.g., more valuable homes were more likely to be expanded, or disinvested neighborhoods have worse current condition), the model will produce biased historical estimates. The demographic bias check in §3.4 does not test this: it compares prediction errors against contemporary tract demographics, not against actual historical features, and it is computed on the trimmed middle-90% sample (see §3.2.3), potentially masking errors on high-value properties. No evidence is provided for the 'vast majority' claim, and the paper's own data (the scanned cards) could be used to verify it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a two-part pipeline for recovering 1933 property assessments for Hamilton County, Ohio. First, scanned assessment cards are aligned to a template, segmented with classical computer vision, and read with a fine-tuned TrOCR model; the authors manually annotated 12,423 single-cell and 588 comprehensive cards for training and validation and report an OCR MAPE of 5.42% on the top 90% confidence subset. Second, a random forest regressor trained on contemporary parcel features predicts the 1933 building value, with an in-county MAPE of 17.48% and an augmented version using OCR-generated labels reaching 16.12%; the model is applied to Franklin County with a location-scale correction, reaching an MAPE of 22.72%. The authors release code and the Hamilton County dataset.","tokens_in":19999,"tokens_out":4853,"duration_ms":56377,"significance":"The main contribution is practical: a reproducible, relatively cheap pipeline and a new public dataset of historical appraisal values. The OCR component is carefully evaluated on held-out manual annotations, with explicit error analysis of digit insertion/deletion and confidence filtering. The cost comparisons are useful for practitioners. The regression contribution is more tentative: its central proxy assumption is not directly verified, and the cross-county experiment requires target-county calibration, so the generalizability claim is weaker than the title suggests. The paper is a solid systems and dataset contribution for COMPASS if the regression claims are appropriately scoped.","major_comments":[{"comment":"The regression model's validity rests on the assumption that contemporary building characteristics approximate 1933 characteristics for properties built before 1930. The sentence 'the vast majority of properties built before 1930 remain the same size' is asserted without evidence, yet this assumption is load-bearing: the model is offered as the solution for counties without scanned records and is used to impute values for the 6,192 OCR-failed Hamilton parcels (§3.2.4). The demographic bias check in §3.4 does not validate the assumption; it compares prediction errors with contemporary tract demographics and is computed on the middle-90% trimmed sample (§3.2.3), so it can miss systematic feature-driven bias. Since the scanned cards contain historical feature information, the authors should perform a direct stability check (e.g., compare card-recorded rooms, exterior type, and stories with contemporary records on a labeled subsample) or clearly mark the proxy as an unverified limitation.","section":"§3.2.1"},{"comment":"The Franklin County experiment does not test unadjusted transfer. The location-scale correction in Eq. (1) uses the mean and standard deviation of 100 Franklin County ground-truth labels, so the reported MAPE of 22.72% is conditional on target-county calibration data. This is a legitimate calibration strategy, but the paper should say so explicitly and should report performance without the correction and with a sensitivity analysis on calibration sample size. As written, the abstract and conclusion imply the model generalizes to other counties without historical labels, which overstates the evidence.","section":"§3.3.3, Eq. (1)"},{"comment":"The augmented regression training set uses OCR labels produced by a TrOCR model fine-tuned on the same Hamilton County hand annotations. The paper does not state whether the parcels contributing OCR-generated training labels are disjoint from the hand-labeled regression test split. If they overlap, the augmented results in Table 4 and Figure 9 are optimistically biased; if they are disjoint, the split rule should be stated. This bears directly on the claim that OCR augmentation improves regression performance.","section":"§3.2.4 and §2.4.2"},{"comment":"All reported regression metrics are computed on the middle 90% of appraised values, excluding the top and bottom 5%. This trimming is reasonable for robustness, but the headline MAPE values (17.48%, 16.12%, 22.72%) are not overall accuracy measures. The paper should report untrimmed metrics or at least the excluded tails separately, because the stated goal is to estimate historical assessments and high-value properties are exactly where the model in Figure 8 shows larger errors.","section":"§3.2.3"}],"minor_comments":[{"comment":"A two-sample classifier p-value of 0.387 does not 'confirm' missing-at-random status; it only means the test does not reject the null of no detectable difference. Please rephrase.","section":"Appendix A"},{"comment":"The learning curve would benefit from confidence intervals or repeated-seed error bars; as presented, the apparent improvement is small and the claim that additional data would improve performance is speculative.","section":"Figure 7"},{"comment":"The fine-tuning experiments report exact match accuracy but not the evaluation set size or confidence intervals; please add these details.","section":"Table 7"},{"comment":"The cost estimates assume a single per-document rate for scanning and manual entry and provide no sensitivity analysis; please state how the conclusions change with the assumed wage and scanning rates.","section":"§3.5"},{"comment":"There is a typo: 'Howeve' should be 'However'.","section":"§2.5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for COMPASS. The OCR and dataset portion is strong and likely acceptable; the regression and generalization portion needs substantial revision to match its claims. I recommend major revision rather than rejection because the central proxy assumption can be tested using the authors' own scanned cards, and the Franklin calibration can be re-framed honestly as a calibration method requiring a small target-county sample."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth taking seriously for the dataset and the OCR pipeline, but the regression side leans on an assumption the authors never test, and the cross-county generalization is weaker than the word 'generalizability' suggests. I'd send it to a serious referee, expecting the regression claims to be tightened.\n\nWhat's genuinely new and good: they release a public county-wide dataset of 1933 Hamilton County property appraisals, which isn't just a promise -- the code and data are on GitHub and HuggingFace. The OCR pipeline (template alignment via ORB + Hough segmentation + fine-tuned TrOCR) is competently built and honestly benchmarked against Tesseract, LayoutLMv3, LayoutParser, Azure, and GPT-4o. The error analysis is transparent: they report the digit insertion/deletion failure mode and show how confidence filtering trades off coverage and accuracy. At the top 90% confidence threshold, 5.42% MAPE is a real result for handwritten digits on noisy scans. The cost comparison is also reasonable and useful.\n\nThe soft spots are concentrated in Section 3. First, the load-bearing proxy assumption -- that contemporary (2020s) building features stand in for 1933 characteristics -- is stated in Section 3.2.1 with no evidence. The paper says 'the vast majority of properties built before 1930 remain the same size,' but they don't check this on their own data, even though they have the scanned cards and could compare features like square footage or room count on a sample. If renovations are correlated with value, the regression estimates for counties without scans are biased, and so are the imputations for the 6,192 Hamilton parcels where OCR failed. This is fixable, but it's not a minor omission.\n\nSecond, the Franklin County test uses Equation 1, which rescales Hamilton's predictions using the mean and standard deviation of Franklin's own labels (from 100 parcels). That means the test is partly a calibration exercise, not a clean transfer. The paper does acknowledge the limitation, but the framing in the abstract and conclusion overstates what's shown. Third, the regression metrics are reported on the middle 90% of appraised values, which can hide systematic error on high-value homes; the full-sample numbers should be in the table, even if the authors argue outliers are less relevant. The bias check in Section 3.4 is fine as far as it goes, but it compares errors to current tract demographics; it doesn't speak to the proxy assumption.\n\nWho is this for? Researchers in digital history, redlining studies, and computational social science who need the kind of parcel-level historical data this pipeline produces. The OCR contribution is the more solid half; the regression contribution is a suggestive prototype that needs the proxy assumption verified or the claims scaled back. I'd referee it, and I'd ask the authors to run the feature-comparison test on their own cards before accepting the regression as a general tool.","headline":"A useful public dataset and a solid OCR pipeline, undercut by an unverified proxy assumption in the regression half and a cross-county test that is partly calibrated to the target county's own labels.","tokens_in":20533,"tokens_out":2397,"would_cite":true,"duration_ms":31068,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper develops a cost-effective OCR and machine-learning pipeline that digitizes handwritten 1933 property assessment cards at scale, produces the first county-wide dataset of historical appraisals for Hamilton County, Ohio, and…","keywords":["historical property assessments","OCR","handwritten digit recognition","redlining","machine learning imputation","document layout understanding","housing data","random forest regression"],"falsifier":"Compare the contemporary features used by the model (square footage, room counts, wall type) against the features recorded on the original 1933 cards for a random sample of parcels; if parcels with measurable feature changes show prediction errors systematically above those without changes, the proxy assumption fails.","tokens_in":19605,"feed_emoji":"🏠","tokens_out":5943,"duration_ms":69624,"temperature":0.7,"pith_summary":"This paper shows that handwritten 1933 property appraisal cards can be digitized accurately and cheaply enough to reconstruct a county-wide historical dataset, and that a machine-learning model can estimate historical appraisals even where no scan exists. The authors build and release a dataset of pre-1936 housing assessments for Hamilton County, Ohio by aligning each scanned card to a template, cutting out table cells, and reading handwriting with a fine-tuned OCR model, reaching a median absolute percentage error as low as 5.42% on the most confident predictions. They also train a random-forest regressor on contemporary building features that estimates 1933 values with 17.48% error within the county and 22.72% error on a second county after a simple average-and-spread adjustment. If these accuracies hold, researchers can finally quantify the financial damage of redlining for individual parcels and neighborhoods rather than rely on anecdotes.","feed_headline":"Handwritten 1933 house appraisals become a county-wide dataset","feed_subtitle":"OCR and machine learning recover Hamilton County values to quantify redlining's financial toll.","key_machinery":"The workhorse is a two-stage tabular extraction pipeline: first align the scanned card to a blank template using ORB feature matching and a RANSAC homography so every table cell has a known position, then read each cropped cell with a fine-tuned TrOCR model; a separate single-cell variant locates the BUILDINGS column header with Tesseract and uses a Hough transform to isolate the first valuation cell. When a card cannot be read, a random-forest regressor trained on contemporary building features (square footages, stories, grade, wall type, rooms, baths, garage capacity, and similar attributes) predicts the 1933 value, and a location-scale shift, Equation 1, adjusts the prediction to another county's value distribution.","core_discovery":"The first county-wide dataset of historical property assessments, constructed from scanned cards, is the paper's central deliverable, together with a regression model that can extrapolate the same quantity to counties without scans. The authors claim that their pipeline, which aligns a scan to a fixed template using ORB feature matching and a RANSAC homography, segments table cells with a Hough transform, and reads digits with a fine-tuned TrOCR transformer, extracts the 1933 building valuation from handwritten cards with 85% of predictions within 5% of the true value and a 5.42% MAPE on the top 90% of confident outputs. The regression model, a random forest trained on contemporary parcel and building data, reaches 17.48% MAPE on held-out Hamilton County properties, improves to 16.12% when OCR labels are added at a confidence threshold, and transfers to Franklin County at 22.72% MAPE after a location-scale correction estimated from 100 labeled parcels. The authors further report near-zero correlation between prediction error and tract-level income, race, and housing variables, and no bias from the roughly 5.6% of parcels whose cards were missing.","pith_inferences":["The paper does not test how many ground-truth labels are needed per new county beyond the 100 used for the location-scale shift; a low-cost robustness check would be fitting the same model with 25, 50, and 200 Franklin labels and seeing whether the 22.72% error stabilizes.","Because the regression relies on today's features as a proxy for 1933 features, neighborhoods with high rates of renovation, demolition, or postwar construction will likely show larger errors than the reported aggregates suggest, so per-parcel use of the regression estimates should be treated cautiously.","The pipeline could be run on the same cards after 1933, not just the earliest entry, so the method can quantify how appraisal gaps widened after the 1936 Federal Underwriting Manual rather than only establishing the pre-redlining baseline.","The heavy-tailed OCR errors (occasional inserted or deleted digits) make point estimates for individual parcels risky; the confidence-threshold filtering the paper applies is essential for any per-parcel use, and downstream users should report uncertainty intervals."],"forward_implications":["A researcher can now download the released Hamilton County 1933 building appraisals and compare them with redlining maps, sale prices, and demographic data to estimate the policy's immediate financial consequences.","Counties without scanned cards can obtain rough pre-1936 estimates from contemporary assessor data at 17% to 23% typical error, enabling multi-city studies without archival scanning.","The fixed-layout alignment-and-segment recipe transfers to other tabular historical records such as tax rolls, censuses, and business directories at an estimated 78% cost saving over manual entry.","Because the paper finds no meaningful error gradient across census-tract demographics, downstream inequality analyses can use the predicted values without a built-in demographic bias in the errors.","The released code and dataset allow social scientists to reproduce the digitization and extend the regression to other jurisdictions rather than re-scan archives from scratch."],"supporting_citations":[{"why":"TrOCR is the transformer-based OCR model the paper fine-tunes to read handwritten digits from property cards.","marker":"[21]"},{"why":"ORB feature matching supplies the keypoint correspondences used to align each scanned card to the blank template.","marker":"[35]"},{"why":"RANSAC computes the homography matrix that maps scan coordinates to template coordinates for cell extraction.","marker":"[15]"},{"why":"The Hough transform detects the row and column lines that define the boundaries of the target table cell.","marker":"[12]"},{"why":"CAR-B provides supplementary handwritten digit-string samples used together with Hamilton annotations to fine-tune TrOCR.","marker":"[11]"},{"why":"LayoutLMv3 serves as a baseline document-understanding model whose table-cell identification the paper measures against.","marker":"[18]"},{"why":"LayoutParser is the baseline layout-analysis toolkit that struggled on these cards, motivating the custom alignment approach.","marker":"[36]"},{"why":"GPT-4o is benchmarked as a low-effort alternative for card parsing and per-cell OCR, informing the cost-accuracy comparison.","marker":"[30]"},{"why":"The classifier two-sample test is used to argue that parcels with missing scans do not bias the regression training sample.","marker":"[22]"}],"fun_headline_variants":["OCR recovers 1933 appraisals for Hamilton County","Machine learning estimates redlining-era property values","New dataset quantifies redlining's financial toll","Handwritten appraisals become machine-readable data","Predicting historical appraisals with OCR and ML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Contemporary building records from the 2020s are a dependable stand-in for the size and features a house had in 1933, so a model trained on today's features can estimate a 1933 appraisal.","fun_headline_variants_meta":{"raw":{"variants":["OCR recovers 1933 appraisals for Hamilton County","Machine learning estimates redlining-era property values","New dataset quantifies redlining's financial toll","Handwritten appraisals become machine-readable data","Predicting historical appraisals with OCR and ML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1609,"prompt_tokens":970,"completion_tokens":639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":565}},"tokens_in":586,"tokens_out":639,"duration_ms":6971,"temperature":1.0,"reasoning_tokens":565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:16:03.026196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the contemporary features used by the model (square footage, room counts, wall type) against the features recorded on the original 1933 cards for a random sample of parcels; if parcels with measurable feature changes show prediction errors systematically above those without changes, the proxy assumption fails.","supporting_citations":[{"cited_title":"Saavedra, David Contreras, Juan Manuel Barrios, and Luiz S","cited_arxiv_id":null,"evidence_quote":"CAR-B provides supplementary handwritten digit-string samples used together with Hamilton annotations to fine-tune TrOCR."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LayoutLMv3 serves as a baseline document-understanding model whose table-cell identification the paper measures against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4o is benchmarked as a low-effort alternative for card parsing and per-cell OCR, informing the cost-accuracy comparison."}],"review_version":1}