Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision

Anhong Guo; Rada Mihalcea; Rosiana Natalie; Ruei-Che Chang; Wenqian Xu

REVIEW 2 major objections 4 minor 18 references

Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision

T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that GPT-4o, even when primed with a participant's vision profile and one example response, agrees with that low-vision participant's image answers only 70% of the time, so VLM simulation of low-vision perception is not ye

desk verdict Useful benchmark and a plausible negative result, but the headline agreement metric is too coarse to separate perception simulation from response-style mimicry, so the central claim needs a sharper evaluation. read the letter →

arxiv 2508.10972 v1 pith:I4XLKT22 submitted 2025-08-14 cs.CV cs.AIcs.HC

Rosiana Natalie , Wenqian Xu , Ruei-Che Chang , Rada Mihalcea , Anhong Guo This is my paper

classification cs.CVcs.AIcs.HC

keywords lowvisionvision-languagemodelsvisualperceptionsimulationaccessibilityfew-shotpromptingGPT-4oimagebenchmarkdisability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a vision-language model (GPT-4o) can be prompted to see images the way a particular person with low vision sees them, and whether such simulated agents could stand in for user research when direct participation is impractical. It builds a benchmark from 40 low-vision participants: brief and detailed vision descriptions plus open-ended and multiple-choice responses to up to 25 images. The central finding is that the model does not get there yet: with no or minimal prompting it behaves like a sighted viewer (0.94 accuracy against ground truth) and agrees with participants only 0.59 of the time. Adding either the participant's vision information alone or example responses alone leaves agreement at 0.59, but combining both—a single example containing open-ended and MCQ answers—raises agreement to 0.70 (p < 0.0001). The paper concludes that such simulations are not yet reliable enough for standalone use and should be treated as complementary, human-in-the-loop tools.

What carries the argument

The central object is the prompt configuration space for constructing a simulated agent of a participant: a system prompt carrying vision information (diagnosis-only, brief, or detailed) and a user prompt carrying example image responses (open-ended, MCQ, or both; single or multiple). The work it does is isolating which information actually constrains GPT-4o's answers toward a participant's perception; the key comparison is the agreement score between model and participant multiple-choice answers, computed across the six content questions (object recognition, color, counting) on the ten mandatory images.

What would settle it

Score the open-ended descriptions produced by the best 0.70-agreement agent: if human raters judge them as detailed and confident as sighted descriptions, or if they agree with the participant's own descriptions at near-zero rates, the central claim loses support. A second check: recompute agreement after removing or collapsing the 'I can't tell' answers from both sides; if the gain from combining vision information with an example disappears, the result reflects matching uncertainty rather than perceptual simulation.

Watch

Extended reading notes

Core claim

The paper's central claim is that VLM-based agents, built from a participant's actual survey responses, can approximate but not faithfully reproduce that person's visual perception of images. On the benchmark of ten mandatory images, a GPT-4o agent prompted only as a sighted or minimally specified assistant answers the content questions correctly 0.94 and 0.92 of the time, exactly the failure mode to be expected from a model that infers beyond the stated vision ability; agreement with the low-vision participants is 0.59. Vision information alone (diagnosis, brief, or detailed) does not move agreement, and examples alone do not either. The best configuration—vision information plus a single e

Load-bearing premise

The load-bearing premise is that agreement on six multiple-choice answers, over ten mandatory images, with an 'I can't tell' option, is a faithful measure of whether the model actually simulates a participant's visual perception; open-ended descriptions, where simulation fidelity would be most observable, are never scored.

Editorial extensions

If this is right

VLM agents cannot yet replace low-vision participants in accessibility evaluation; they should be used only as an early-stage complement with human-in-the-loop validation.
Adding one participant-grounded example that mixes open-ended and MCQ responses is the most effective prompt improvement tested, raising agreement to 0.70, and the minimal data needed takes about 15 minutes to collect.
Adding more examples (up to nine) does not significantly improve agreement, so prompt designers gain little from larger few-shot sets.
Naively or minimally prompted agents systematically overestimate what a low-vision user can see, so such agents should not be used without explicit perceptual constraints.
If these agents are used to tailor assistive output, the 0.70 ceiling means generated descriptions may misstate what a user can perceive and need verification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper scores only multiple-choice answers; the open-ended descriptions that participants wrote are used as prompt inputs but never evaluated for fidelity. A version of the study that scored open-ended descriptions would likely find the simulation ceiling is lower than 0.70.
Because every MCQ includes an 'I can't tell' option, a model could raise agreement by matching a participant's tendency to say 'I can't tell' without reproducing how they perceive images; recomputing agreement after removing that option would test this.
The same benchmark and 15-minute minimal data collection could be applied to newer reasoning models or other VLM families, and could support personalized accessibility profiles for adaptive assistive tools.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

My take: this is a solid empirical contribution that would benefit from a more careful evaluation frame. The dataset—40 low vision participants with vision profiles, open-ended descriptions, and MCQ responses to 25 images—is genuinely useful, and the prompt-ablation design is systematic. The finding that a single combined example plus vision information beats either alone (0.70 vs 0.59) is actionable and likely to be cited. The leave-one-out setup is appropriate, and the GLMM with participant and question random effects is a reasonable way to handle the nested data. I also appreciate the explicit ethical discussion and the honest tone in Section 6.1.

The soft spots are real but not fatal. First, the paper evaluates agreement only on MCQs; open-ended descriptions are collected and used as examples but never scored. The abstract and discussion overstate the scope, implying the simulation quality was assessed more broadly. That mismatch needs fixing. Second—and this is the stress-test concern, which I think holds up—every MCQ includes an “I can’t tell” option, and participants choose it roughly 35% of the time (that’s the blind-agent agreement). The best prompt gives the model one of the participant’s own MCQ responses as an example, which can teach the model the participant’s tendency to say “I can’t tell” without requiring any visual perception. The paper never reports agreement conditioned on substantive answers, nor the agent’s “I can’t tell” rate per condition. So part of the 0.70 vs 0.59 advantage could be response-style mimicry rather than perception simulation. This weakens the positive claims but does not undercut the main negative result: VLMs are not yet good at this task.

Also worth flagging: no data or code are released, which is a concrete limitation for a benchmark paper, and the multiple pairwise comparisons are reported without explicit correction, though the p-values are so small that this likely doesn’t change the conclusions.

Who should read this: accessibility researchers, HCI people working on simulation or VLM evaluation, and anyone considering LLM role-play as a proxy for user research. It deserves a serious referee, but the revision should include a conditional agreement analysis, some scoring of open-ended outputs (even qualitative), and a commitment to share artifacts. If those get addressed, the paper will be a useful reference.

I’d send it to peer review, with a clear request to tighten the evaluation scope before publication.

Referee Report

2 major / 4 minor

Summary. The paper builds a benchmark from 40 low-vision participants, containing brief/detailed vision information and open-ended plus multiple-choice responses to up to 25 images. It then creates GPT-4o 'agents' of each participant under 16 prompt designs (varying vision information, examples, and example format) and measures agreement between agent and participant MCQ answers. Main findings: no/minimal prompting and vision-info-only prompts yield mean agreement 0.59; combining vision information with a single example containing both open-ended and MCQ responses raises it to 0.70 (p<0.0001); additional examples give no significant gain. The paper concludes that VLMs are 'not there yet' for simulating low-vision visual perception.

Significance. If the reported agreement metric measures what the paper claims, this is a timely and useful result: it provides the first benchmark for VLM simulation of low-vision perception, uses real participant data, evaluates held-out responses with a GLMM including participant and question random effects, and documents prompt engineering transparently. The ethical limitations are discussed seriously. However, the metric is limited to multiple-choice agreement, and the 'I can't tell' option creates a response-style confound; these issues must be addressed before the central claim is settled. With those analyses, the benchmark and the prompt-design comparisons would be a solid contribution to accessibility and human-AI simulation research.

major comments (2)

[§4.3, §5.1, Appendix C] The headline 0.70 agreement is open to a response-style explanation that the paper does not rule out. Every MCQ includes an 'I can't tell' option, and the blind-agent baseline (which always answers 'I can't tell') has agreement 0.35, so participants choose 'I can't tell' on roughly 35% of items. The best prompt (§5.3) includes a same-participant example containing the participant's MCQ responses, which can teach the model the participant's 'I can't tell' rate and question-answering style. A model that mimics all 'I can't tell' answers and matches only 54% of substantive answers would reach 0.70. The paper reports neither (a) how often the agent chooses 'I can't tell' in each condition nor (b) agreement restricted to items where the participant gave a substantive answer. Please add these analyses; if the 0.70-vs-0.59 difference disappears on substantive items, the conclusion should be ref
[§4.3 vs. abstract/§6.2] The evaluation is narrower than the claims. Agreement is defined only for MCQ responses; open-ended descriptions are used as prompt inputs and quoted qualitatively (§5.2) but never scored. Yet the abstract says the benchmark collects both types, and the proposed applications (§6.2) are largely about generating descriptions (e.g., personalized image feedback). Since open-ended output is where perceptual simulation fidelity would be most visible, the paper should either add an open-ended evaluation (e.g., human ratings or content-unit matching) or explicitly limit the central claim to multiple-choice response agreement and adjust the title/abstract accordingly.

minor comments (4)

[§5.2-5.3] Many pairwise z-tests are reported without any multiple-comparison adjustment. The p<0.0001 headline is likely robust, but p<0.05 comparisons should be interpreted cautiously and should ideally be reported with adjusted p-values or a planned contrast structure.
[Abstract] Typo: 'agent’ and participants' should be 'agent's and participants'.
[Appendix B/C] Minor typos: 'Stangle et. al.' should be 'Stangl et al.'; 'squating' appears in Appendix C; 'then image' should be 'the image'.
[§5.1] The sentence 'we did not pass them any visual image' is confusing because the agent is asked to answer image questions. Specify that the blind-agent prompt omits the image and instructs the model to answer 'I can't tell'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is a held-out agreement benchmark with no fitted parameter or self-citation chain determining the outcome.

full rationale

The paper's derivation chain is empirical rather than formal: it collects survey responses from 40 low-vision participants, constructs prompts for GPT-4o with varied vision information and example responses, and measures agreement against held-out participant answers. The headline result (0.70 agreement for vision information plus a single combined open-ended/MCQ example) is computed from predictions on images whose participant answers are not included as in-context examples; the multiple-example condition is explicitly leave-one-out ("nine example answers corresponding to all mandatory images except the one being used for prediction"). No parameter is fitted to the target answers and then renamed as a prediction; temperature is fixed at 0, and the model weights are not updated. The same participant's answers serve both as prompt examples and as ground truth, but for disjoint items, which is standard few-shot evaluation rather than circularity. The paper's self-citations (e.g., Liu et al. 2025, Huang et al. 2024) appear only as motivational related work and are not load-bearing for the agreement scores. The skeptic concern about "I can't tell" response-style mimicry is a construct-validity limitation of the MCQ agreement metric, not a case where the predicted quantity is defined in terms of the prompt inputs; the paper's own Discussion also acknowledges that hallucinations, bias, and fidelity gaps mean simulations are "not yet ready for standalone deployment." Therefore, under the required standard of exhibiting a specific reduction or fitted-parameter-as-prediction, no circular step is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the self-reported nature of the vision profiles, the choice of MCQ proxy for perception, and the unadjusted statistical model. No scientific free parameters are fitted by the paper; the GLMM coefficients are estimation machinery, not claimed results.

assumptions (5)

domain assumption Participants' self-reported vision information accurately describes their visual perception.
The benchmark and agent prompts are built entirely from survey self-reports (Section 3.1); no clinical measurements are taken.
domain assumption The 25 images and the six MCQ types (object, color, counting) capture the visual skills relevant to low vision perception.
Section 3.1 says images were selected following Stangl et al.'s classification and MCQs follow the three visual skills from Zeng et al. 2020.
domain assumption Agreement on multiple-choice responses is a valid measure of perception simulation fidelity.
Section 4.3 defines agreement only over MCQ responses; open-ended outputs are not scored, so the headline 0.70 depends on this proxy.
domain assumption GPT-4o with temperature 0 represents current VLM simulation capability.
Section 4.2 evaluates only gpt-4o-2024-11-20; the paper notes the limitation of not testing reasoning models in Section 6.4.
standard math GLMM binomial/logit specification and EMM pairwise comparisons provide valid inference.
Section 4.3 describes the model; the paper does not report multiple-comparison adjustment.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision." pith.science (2026). https://pith.science/paper/I4XLKT22

@misc{pith2026250810972,
  author       = {Pith},
  title        = {Pith review of: Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I4XLKT22}},
  note         = {Machine review of arXiv:2508.10972}
}

read the original abstract

Advances in vision language models (VLMs) have enabled the simulation of general human behavior through their reasoning and problem solving capabilities. However, prior research has not investigated such simulation capabilities in the accessibility domain. In this paper, we evaluate the extent to which VLMs can simulate the vision perception of low vision individuals when interpreting images. We first compile a benchmark dataset through a survey study with 40 low vision participants, collecting their brief and detailed vision information and both open-ended and multiple-choice image perception and recognition responses to up to 25 images. Using these responses, we construct prompts for VLMs (GPT-4o) to create simulated agents of each participant, varying the included information on vision information and example image responses. We evaluate the agreement between VLM-generated responses and participants' original answers. Our results indicate that VLMs tend to infer beyond the specified vision ability when given minimal prompts, resulting in low agreement (0.59). The agreement between the agent' and participants' responses remains low when only either the vision information (0.59) or example image responses (0.59) are provided, whereas a combination of both significantly increase the agreement (0.70, p < 0.0001). Notably, a single example combining both open-ended and multiple-choice responses, offers significant performance improvements over either alone (p < 0.0001), while additional examples provided minimal benefits (p > 0.05).

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 14 canonical work pages

[1]

What is your level of vision? e.g., visual acuity, visual and color perception

work page
[2]

When did you lose your vision? (a) Congenital (or since birth) (b) Acquired (or later in life)

work page
[3]

Can you provide more details about your vision onset and how it has progressed over time?

work page
[4]

What device are you using currently to complete the survey? (a) Laptop/PC (b) Mobile Phone (c) Tablet

work page
[5]

C.2 Section 1 - Brief Vision Information

What assistive technologies are you currently using to complete the survey? e.g., Screen reader, Magnifier, etc. C.2 Section 1 - Brief Vision Information

work page
[6]

How much difficulty do you have finding something on a crowded shelf? (a) No difficulty at all (b) A little difficulty (c) Moderate difficulty (d) Extreme difficulty (e) Stopped doing this because of your eyesight (f) Stopped doing this for other reasons or not interested in doing this

work page
[7]

How much difficulty do you have seeing how people react to things you say? (a) No difficulty at all (b) A little difficulty (c) Moderate difficulty (d) Extreme difficulty (e) Stopped doing this because of your eyesight (f) Stopped doing this for other reasons or not interested in doing this C.4 Section 3 - Image Perception and Recognition Questions Note:T...

work page
[11]

What is the cause of your visual impairment? e.g., the medical diagnosis C.3 Section 2 - Detailed Vision Information

work page

Show all 18 references

[12]

Can you describe in detail how you perceive your surroundings?
[13]

How would you describe what you see in different lighting conditions (e.g., bright sunlight, dim indoor lighting, nighttime)?
[14]

Which of the following best describes your visual perception? (Select all that apply) (a) I can see shapes but not details (b) I can see colors but not fine details (c) I rely more on contrast and movement (d) I have light perception but no form vision (e) I have no visual perception
[15]

In your experience, do you know if your vision or way of experiencing the world is unique compared to others with the same diagnosis? If so, in what ways?
[16]

How much difficulty do you have doing work or hobbies that require you to see well up close, such as cooking, sewing, fixing things around the house, or using hand tools? (a) No difficulty at all (b) A little difficulty (c) Moderate difficulty (d) Extreme difficulty (e) Stoppe...
[2017]

Nelson, E

Crip for a Day: The Unintended Negative Conse- quences of Disability Simulations.Rehabilitation Psychol- ogy, 62: 324–333. Nelson, E. E. C.; Spence, A. D.; and Gormley, G. J. 2023. Stepping into the shoes of older people: a scoping review of simulating ageing experiences for h...

2023
[2018]

Hali, M.; Diagne, B.; and Walker, J

Introducing Virtual Reality Technologies to Design Education.Seminar.net. Hali, M.; Diagne, B.; and Walker, J. 2022. Measuring Rep- resentational Harms in Image Captioning. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT), 943–954....

2022 arXiv
[2022]

Ehibhatiomhan, R.; Foreman, E.; Barrott, L.; Shek, J.; and Nabhani-Gebara, S

Language models show human-like content effects on reasoning tasks.arXiv preprint arXiv:2207.07051. Ehibhatiomhan, R.; Foreman, E.; Barrott, L.; Shek, J.; and Nabhani-Gebara, S. 2022. ‘A life in a day’ simulation expe- rience: perceptions of oncology nurses and pharmacy staff....

2022 arXiv
[2023]

Human-like problem-solving abilities in large lan- guage models using ChatGPT.Frontiers in artificial intelli- gence, 6: 1199350. Park, J. S.; O’Brien, J.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative agents: Interactive simulacra of human behavi...

2023 arXiv
[2024]

Colwell, C

Proxona: Leveraging LLM-Driven Personas to En- hance Creators’ Understanding of Their Audience.arXiv preprint arXiv:2408.10937. Colwell, C. M. 2013. Simulating disabilities as a tool for al- tering individual perceptions of working with children with special needs.Internationa...

2013 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

What is your level of vision? e.g., visual acuity, visual and color perception

work page

[2] [2]

When did you lose your vision? (a) Congenital (or since birth) (b) Acquired (or later in life)

work page

[3] [3]

Can you provide more details about your vision onset and how it has progressed over time?

work page

[4] [4]

What device are you using currently to complete the survey? (a) Laptop/PC (b) Mobile Phone (c) Tablet

work page

[5] [5]

C.2 Section 1 - Brief Vision Information

What assistive technologies are you currently using to complete the survey? e.g., Screen reader, Magnifier, etc. C.2 Section 1 - Brief Vision Information

work page

[6] [6]

How much difficulty do you have finding something on a crowded shelf? (a) No difficulty at all (b) A little difficulty (c) Moderate difficulty (d) Extreme difficulty (e) Stopped doing this because of your eyesight (f) Stopped doing this for other reasons or not interested in doing this

work page

[7] [7]

How much difficulty do you have seeing how people react to things you say? (a) No difficulty at all (b) A little difficulty (c) Moderate difficulty (d) Extreme difficulty (e) Stopped doing this because of your eyesight (f) Stopped doing this for other reasons or not interested in doing this C.4 Section 3 - Image Perception and Recognition Questions Note:T...

work page

[8] [11]

What is the cause of your visual impairment? e.g., the medical diagnosis C.3 Section 2 - Detailed Vision Information

work page