Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Bounding Boxes to Improve Small Language Model Performance on Vision-Based Grading Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cropping handwritten exam answers to a bounding box improves grading accuracy and cuts compute for small vision-language models.

desk verdict Useful empirical preprocessing note, but the significance claim rests on a table I can't reproduce from the paper's own model-level data; fix that before relying on it. read the letter →

arxiv 2607.18767 v1 pith:WDZEUIWR submitted 2026-07-21 cs.CV cs.AIcs.CL

Lachlan McGinness This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords smalllanguagemodelsvision-languageautomatedgradingboundingboxesimagecroppinghandwrittenexamschain-of-thoughtpromptingcomputationalefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a simple preprocessing step — cropping a scanned handwritten exam page to just the box containing the target answer — makes small vision-language models better graders. Across 126 student responses to one physics short-answer question and eight open-weights models from 4B to 72B parameters, every model graded more accurately when shown the cropped region instead of the full double-page spread. The crop also roughly halved the input tokens and cut estimated inference FLOPs from about 41–45 trillion to 17–18 trillion. The authors take this as evidence that removing visual clutter before the model sees the image is a crucial step for making locally deployable small models practical for large-scale educational grading.

What carries the argument

The load-bearing mechanism is the bounding-box crop: a rectangle, chosen once by visual inspection of the exam layout, that isolates the pixels containing the target question's answer. It operates on two fronts: it shrinks the image so the vision encoder produces fewer patches and tokens, and it removes unrelated visual content that competes for the model's attention. The paper also tracks chain-of-thought prompting as a second experimental lever, but the evidence suggests the crop, not the prompt, is what moves accuracy. The cost model used to quantify the savings is the standard approximation C ≈ 2nN, where n is total tokens and N active parameters, which translates token reduction directl

What would settle it

Re-run the eight models on the same 126 scans with bounding boxes generated independently per page (for example, by locating the handwriting), and check whether the fixed boxes ever exclude a written answer. If a nontrivial number of answers fall outside the fixed crops, or if accuracy with per-page crops does not beat full-page accuracy, the paper's central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that presenting a vision-language model with a fixed bounding-box crop of the answer region, rather than the entire scanned page, improves grading accuracy and lowers computational cost across all tested models. In the cleanest comparison (no chain-of-thought prompting), average accuracy rose from 0.728 on full pages to 0.840 on crops, while average tokens fell from 1463 to 679 and estimated inference FLOPs fell from 41 to 17 trillion. The authors attribute the gain to two mechanisms: the crop removes visual distractions (other questions, diagrams, scribbles) that confuse small models, and fewer image patches means fewer visual tokens, which directly cuts compute

Load-bearing premise

The load-bearing assumption is that the fixed bounding box, placed once by visual inspection, actually contains every student's answer on all 126 pages; the paper does not verify per page that the crop captured the response, so a misaligned answer could distort the apparent benefit of cropping.

Editorial extensions

If this is right

  • For short-answer grading of fixed-layout paper exams, cropping to the answer region is a reliable accuracy boost and a token/compute saver, so exam systems should treat layout-aware cropping as a default preprocessing step.
  • Because the benefit held across all eight models from 4B to 72B parameters, the result transfers across architectures and sizes rather than being a quirk of one model.
  • The larger models reached roughly the same accuracy with crops as without, meaning cropping lets smaller, cheaper local models approach the accuracy of much larger cloud-hosted models.
  • Chain-of-thought prompting did not consistently help and sometimes hurt; for simple binary grading tasks, the added tokens buy no accuracy, so simpler prompting is preferable.
  • Reduced FLOPs per graded response lower energy use and cost, strengthening the case for on-premises grading of sensitive student data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: whether the fixed crop would hold up on exams with varied layouts or multi-part questions whose answers move; an adaptive region proposal (for example, a layout model or handwriting locator) would be the natural next test.
  • The token savings imply that, for a fixed compute budget, cropping effectively lets a school run a larger local model at the same cost — the two interventions trade off directly.
  • Since the crop was chosen by human inspection of the same dataset, part of the gain may be layout-specific; applying the same method to a different exam would show how much of the benefit survives.
  • The CoT findings suggest that for low-reasoning binary grading, prompting for step-by-step reasoning mostly wastes tokens; in constructed-response grading that requires actual reasoning, CoT might interact differently with cropping.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports an empirical study of whether cropping scanned handwritten exam pages to a question-specific bounding box improves the accuracy and computational efficiency of small vision-language models on a binary short-answer grading task. Using 126 responses to Question 1a of the 2025 Australian Physics Olympiad, the author evaluates eight open-weight VLMs (4B–72B active parameters) under four conditions defined by the combination of chain-of-thought prompting (CoT vs. no-CoT) and image input (full A3 double-page spread vs. bounding-box crop). Accuracy is measured against human marker agreement, and FLOPs are estimated using the standard 2nN approximation on measured token counts. The paper reports that bounding boxes improve or tie accuracy in 15 of 16 model-condition comparisons and reduce token counts and FLOPs by roughly half, and it claims 'significantly improves grading accuracy' on the basis of a statistical comparison involving Table 3.

Significance. If the central claim holds, the paper offers a simple, actionable preprocessing step for VLM-based grading of handwritten exams: crop the page to the region containing the target response, thereby reducing visual clutter and input tokens. This could make local, privacy-preserving SLM deployment more feasible in educational settings. The study has several strengths: it uses real anonymized student exam data, external human labels with no discrepancies, multiple open-weight models, and measured input/output token counts with a transparent FLOPs approximation. The computational-cost reduction is robust and not dependent on the disputed statistical test. However, the quantitative evidence for the 'significantly improves accuracy' claim is undermined by an internal inconsistency between Table 2 and Table 3, and the significance test is based on a post-hoc extremal contrast. These issues are fixable but must be addressed before the central claim can be considered established.

major comments (4)
  1. [§3, Tables 2 and 3] Table 3 cannot be derived from Table 2. Averaging the eight model accuracies in Table 2 gives CoT+BB=0.796, CoT+no-BB=0.709, no-CoT+BB=0.818, no-CoT+no-BB=0.708, whereas Table 3 reports 0.817, 0.734, 0.840, 0.728. The discrepancies are about 0.02–0.025, which exceed the standard errors reported in Table 3 (0.019–0.024). Moreover, the SEMs in Table 3 (~0.02) are far smaller than the between-model SEM computed from Table 2 (e.g., CoT+BB values range 0.55–0.94, SD≈0.14, SEM≈0.05). Because the paper's significance statement (the '3.08 times the quadrature' comparison) is computed from Table 3, the quantitative support for the accuracy claim is currently unreliable. The authors must report exactly how Table 3 was aggregated, correct it using the per-model data, and base the significance test on the actual trial-level data or on an appropriate model-based analysis.
  2. [§3, significance statement] The significance test uses a post-hoc extremal contrast: it compares the lowest bounding-box mean (0.817) with the highest no-bounding-box mean (0.734) from Table 3, i.e., it selects the most favorable contrast after inspecting the data. Even setting aside the Table 3 inconsistency, this ignores the pairing of conditions within each model and the large between-model variability visible in Table 2. A proper test should compare the marginal BB vs. no-BB means, or fit a mixed-effects model with model as a random effect and CoT/BB as fixed effects, so that the repeated measures on the same eight models are accounted for. The current statistic treats the two group means as independent and provides no justification for the 99.8% confidence claim.
  3. [§2.1, Bounding Boxes] The bounding box is a single fixed crop chosen once by the researchers from visual inspection of the scanned exams, and no per-page verification is reported that the crop contains the student's answer to Q1a in all 126 papers. If any response fell outside the crop, the model would see only a region without the answer and would likely output 0, biasing the BB accuracy downward. While this bias is conservative for the BB-vs-no-BB comparison, the absence of verification leaves an unresolved threat to internal validity, and the transferability of a single hand-picked box to other exam layouts is unexamined. Please report any failed crops or a verification protocol, and discuss how the fixed-box approach would generalize.
  4. [§2.1, dataset description] The text says '126 of the 588 student exam papers were chosen randomly as the dataset for each of our four experimental conditions.' It is unclear whether the same 126 papers were used in all four conditions or whether each condition used an independently sampled 126. If the latter, student-level difficulty is not controlled across conditions, and model-level comparisons between conditions are not matched. This ambiguity also affects which statistical test is appropriate. Please clarify the sampling design.
minor comments (5)
  1. [§2.1, prompts] The CoT and no-CoT prompts describe the page as containing 'a question about snails,' whereas the target question is about human walking (footnote 2 explains this). This wording is likely to confuse models, especially in the cropped condition where the snail diagram is absent; consider using a prompt that matches the actual question content.
  2. [Figure 2] The figure caption describes error bars as 'maximum and minimum values of the Wilson Score interval at the Z=3 (p=99.7%) significance level.' A Wilson interval is at a confidence level, not a significance level; state the confidence level explicitly (e.g., 99.7%) and clarify what 'maximum and minimum values' refers to.
  3. [§2.2] The text says all evaluated models are 'small enough to run on upper-end consumer-grade hardware,' but this is questionable for Qwen2.5-VL 72B and Llama 4 Scout (109B total parameters). Consider revising to avoid overstating hardware accessibility.
  4. [§3] The phrase '3.08 times the squared quadrature of the standard errors' is ambiguous and likely meant '3.08 times the quadrature (root-sum-square) of the standard errors.' Reword for clarity.
  5. [General] There are minor typographical issues, e.g., 'using of CoT prompting' in Section 2, and the run-together 'PromptingTo investigate' in Section 2.1. A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the bounding-box result is an empirical comparison against external labels and measured token counts, with no fitted parameter renamed as a prediction.

full rationale

The paper's central claim — that bounding-box cropping improves grading accuracy and reduces FLOPs — is an empirical comparison, not a derivation from fitted parameters. Grading accuracy is computed against external human markers with no discrepancies; no parameter is fitted to the accuracy data and then renamed as a prediction. The FLOPs figures come from the standard C≈2nN approximation (Kaplan et al. [17], with [18]) applied to measured input/output token counts; the token counts are reported measurements, and the reduction is consistent with the smaller crop but is not itself used to define the crop or the accuracy outcome. The bounding boxes are chosen once by visual inspection of the scans, not tuned to the human labels or to model outputs. Self-citations (refs. [9], [10], [12], [18], [20]) appear only as background or as secondary support for the token-cost approximation, which is independently attributed to Kaplan et al.; none of these self-citations is load-bearing for the bounding-box comparison. One non-circular concern flagged by the reviewer: the Table 3 aggregate accuracies and SEMs do not match the per-model Table 2 values, so the significance statement ('3.08 times the squared quadrature') is not independently reproducible; that is a data-consistency/transcription issue, not circularity. Overall, no step reduces by construction to its inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No fitted constants enter a derivation. The quantitative claims rest on measured token counts, external human labels, and a cited FLOPs approximation. The main hand-chosen element is the bounding-box placement, an experimental preprocessing choice rather than a model parameter.

free parameters (1)
  • Bounding box placement = Coordinates fixed by visual inspection; not reported numerically
    The crop location was chosen by the researchers after inspecting the scans and is central to the bbox condition; no sensitivity analysis is reported.
assumptions (4)
  • domain assumption Human markings of Question 1a are correct and consistent
    Accuracy is defined as agreement with the human marking team; the paper states there were no discrepancies, which is used as ground truth.
  • domain assumption FLOPs ≈ 2 n N approximates total VLM inference cost
    Used in Section 3 to compute FLOPs from token counts; the cited vision-encoder cost literature suggests this scaling-law formula is not the whole cost, so the compute comparison is approximate.
  • ad hoc to paper The fixed bounding box captures every student's answer to Question 1a
    The bbox condition relies on the crop containing the relevant handwritten response; the paper does not verify this per page.
  • domain assumption The 126-response random sample is representative of the 588 consented exams
    The sample is drawn randomly, but only from students who opted in (588 of 1357), and the paper does not compare sample characteristics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bounding Boxes to Improve Small Language Model Performance on Vision-Based Grading Tasks." pith.science (2026). https://pith.science/paper/WDZEUIWR

@misc{pith2026260718767,
  author       = {Pith},
  title        = {Pith review of: Bounding Boxes to Improve Small Language Model Performance on Vision-Based Grading Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WDZEUIWR}},
  note         = {Machine review of arXiv:2607.18767}
}
read the original abstract

The deployment of Small Language Models (SLMs) in educational settings offers significant advantages in terms of privacy, cost, and scalability. However, SLMs often struggle with complex vision-based tasks, such as grading handwritten student exams, due to the high computational cost of processing large images and the visual distractions present on a full page. In this paper, we investigate whether cropping student responses using bounding boxes can improve the accuracy and computational efficiency of SLMs on a short-answer grading task. Using a dataset of scanned handwritten responses from the 2025 Australian Physics Olympiad, we evaluate the performance of several models ranging from 4B to 72B parameters under varying conditions of Chain of Thought (CoT) prompting and image cropping. Our results demonstrate that using bounding boxes significantly improves grading accuracy and reduces computational cost (FLOPs) across models. We conclude that bounding boxes are a crucial pre-processing step for deploying SLMs in large-scale, vision-based educational assessments.

Figures

Figures reproduced from arXiv: 2607.18767 by the authors.

Figure 1
Figure 1. Visual comparison of the bounding box and non-bounding box experimental conditions. Left: Original scan of student exam paper. Right: Cropped bounding box of the same student exam paper. The bounding box reduces the number of visual features that could confuse smaller models. Intuitively, the size of an image is approximately proportional to the number of tokens that it takes to encode it. Therefore presenting a mod… view at source ↗
Figure 2
Figure 2. Plot of accuracy against computational cost for all models and experimental conditions. Error bars indicate the maximum and minimum values of the Wilson Score interval at the Z=3 (𝑝 =99.7%) significance level. 4. Conclusion As educational institutions look to adopt AI, SLMs present a secure, transparent, and cost-effective alternative to large, cloud-based commercial models. However, effectively deploying SLMs requi… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 5 canonical work pages

  1. [1]

    Henkel, L

    O. Henkel, L. Hills, A. Boxer, B. Roberts, Z. Levonian, Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability To Mark Short Answer Questions in K- 12 Education, in: Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, Association for Computing Machinery, New York, NY, USA, 2024, pp. 300–304. URL: https: /...

    arXiv 2024

  2. [2]

    Kortemeyer, Toward AI grading of student problem solutions in introductory physics: A feasibility study, Physical Review Physics Education Research 19 (2023) 020163

    G. Kortemeyer, Toward AI grading of student problem solutions in introductory physics: A feasibility study, Physical Review Physics Education Research 19 (2023) 020163

    2023

  3. [3]

    Kortemeyer, J

    G. Kortemeyer, J. Nohl, Assessing confidence in AI-assisted grading of physics exams through psychometrics: An exploratory study, Physics Review Physics Education Research 21 (2025). doi:https://doi.org/10.1103/PhysRevPhysEducRes.21.010136

    work page doi:10.1103/physrevphyseducres.21.010136 2025

  4. [4]

    Kortemeyer, Performance of the pre-trained large language model GPT-4 on automated short answer grading, Discover Artificial Intelligence 4 (2024) 47

    G. Kortemeyer, Performance of the pre-trained large language model GPT-4 on automated short answer grading, Discover Artificial Intelligence 4 (2024) 47

    2024

  5. [5]

    T. Liu, J. Chatain, G. Kortemeyer, M. Sachan, AI-assisted Automated Short Answer Grading of Handwritten University Level Mathematics Exams, 2024. URL: http://arxiv.org/abs/2408.11728. doi:10.48550/arXiv.2408.11728

    arXiv doi:10.48550/arxiv.2408.11728 2024

  6. [6]

    R. Mok, F. Akhtar, L. Clare, C. Li, J. Ida, L. Ross, M. Campanelli, Using AI Large Language Models for Grading in Education: A Hands-On Test for Physics, 2024. URL: http://arxiv.org/abs/2411.13685. doi:10.48550/arXiv.2411.13685

    work page doi:10.48550/arxiv.2411.13685 2024

  7. [7]

    Z. Chen, T. Wan, Achieving Human Level Partial Credit Grading of Written Responses to Physics Conceptual Question using GPT-3.5 with Only Prompt Engineering, Boston, 2024

    2024

  8. [8]

    Z. Chen, T. Wan, Grading explanations of problem-solving process and generating feedback using large language models at human-level accuracy, Physical Review Physics Education Research 21 (2025) 010126. URL: https://link.aps.org/doi/10.1103/PhysRevPhysEducRes.21.010126. doi:10. 1103/PhysRevPhysEducRes.21.010126

    doi:10.1103/physrevphyseducres.21.010126 2025

Show all 21 references
  1. [9]

    McGinness, Overview of AI Grading of Physics Olympiad Exams, in: A

    L. McGinness, Overview of AI Grading of Physics Olympiad Exams, in: A. I. Cristea, E. Walker, Y. Lu, O. C. Santos, S. Isotani (Eds.), Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners...

    2025

  2. [10]

    Baumgartner, L

    P. Baumgartner, L. McGinness, Combining LLMs and Term Rewriting for Marking Algebraic Expressions in Physics Exams, in: KR Workshop: LLMs and KRR for Trustworthy AI, 2025. URL: https://openreview.net/forum?id=lir85TBz4V

    2025

  3. [11]

    C. D. Jaldi, A. Saini, S. Zhang, N. Schroeder, C. Shimizu, E. Ilkou, Small, Private Language Models as Teammates for Educational Assessment Design, 2026. URL: http://arxiv.org/abs/2605.15015. doi:10.48550/arXiv.2605.15015, arXiv:2605.15015 [cs.AI]

    2026 doi arXiv

  4. [12]

    McGinness, P

    L. McGinness, P. Baumgartner, Can large language models correctly interpret equations with errors?, Physical Review Physics Education Research 21 (2025) 020155. URL: https://link.aps.org/ doi/10.1103/v8f8-s11v. doi:10.1103/v8f8-s11v

    2025 doi

  5. [13]

    Z. Reza, A. Mazur, M. T. Dugdale, R. Ray-Chaudhuri, Small Models, Big Support: A Local LLM Framework for Educator-Centric Content Creation and Assessment with RAG and CAG, 2025. URL: http://arxiv.org/abs/2506.05925. doi:10.48550/arXiv.2506.05925, arXiv:2506.05925 [cs.CY]

    2025 doi

  6. [14]

    Kamath, J

    A. Kamath, J. Ferret, R. Dadashi, L. Hussenot, Gemma 3 Technical Report, 2025. URL: http://arxiv. org/abs/2503.19786. doi:10.48550/arXiv.2503.19786, arXiv:2503.19786 [cs.CL]

    2025 doi arXiv

  7. [15]

    A. Yang, B. Yang, Z. Qiu, Qwen2.5 Technical Report, 2025. URL: http://arxiv.org/abs/2412.15115. doi:10.48550/arXiv.2412.15115, arXiv:2412.15115 [cs.CL]

    2025 doi arXiv

  8. [16]

    P. K. A. Vasu, F. Faghri, C.-L. Li, O. Tuzel, H. Pouransari, FastVLM: Efficient Vi- sion Encoding for Vision Language Models, 2025, pp. 19769–19780. URL: https: //openaccess.thecvf.com/content/CVPR2025/html/Vasu_FastVLM_Efficient_Vision_Encoding_ for_Vision_Language_Models_CVP...

    2025

  9. [17]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling Laws for Neural Language Models, 2020. URL: http://arxiv.org/abs/2001.08361. doi: 10. 48550/arXiv.2001.08361

    2020 doi arXiv

  10. [18]

    S. C. Eimler, L. Erle, D. Flood, A. Haiman, L. Hackert, A. Helgert, L. McGinness, B. Yapici, The Environmental Cost of LLMs in AIED: Reporting and Practices, in: Lecture Notes in Computer Science (LNCS), Springer Nature Switzerland, 2026

    2026

  11. [19]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, Chain-of- Thought Prompting Elicits Reasoning in Large Language Models, in: 36th Conference on Neural Information Processing Systems, NeurIPS, 2022

    2022

  12. [20]

    McGinness, P

    L. McGinness, P. Baumgartner, Large Language Models Imitate Logical Reasoning, but at what Cost?, 2025. doi:10.48550/arXiv.2509.12645

    2025 doi

  13. [21]

    Bytez.com, R. Liu, J. Geng, A. J. Wu, I. Sucholutsky, T. Lombrozo, T. L. Griffiths, Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse, 2025. URL: https://bytez.com/docs/icml/45714/paper

    2025

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.