{"id":"1f970ef2-49c0-471f-b050-c2bf2fb15f77","arxiv_id":"2607.21760","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Five blind and low-vision co-designers, using the agentic programming tool ProgramAT, created 37 custom camera-based assistive tools, revealing motivations, iterative strategies, and challenges like model limits and specification conflicts.","lead":"This paper reports a two-month study in which five tech-savvy blind and low-vision co-designers used an agentic programming tool (ProgramAT) to create 37 bespoke camera-based assistive tools, from Uber identifiers to hand-gesture readers. It documents what they chose to build, why, and the creation strategies and failures they encountered, offering design guidance for making AI-assisted programming useful for accessibility.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No objective ground-truth evaluation: '37 tools created' may not mean '37 tools worked,' so the viability claim is underdetermined.","rationale":"The reader's weakest assumption identified author-participant bias and the lack of sighted ground-truth assessment. I agree that this is a real threat, but the more precise and load-bearing issue is that 'successful creation' is operationalized through self-report and spec-compliance checks, not through measuring whether the generated tools actually perform their intended visual task. Section 4.4's code review is explicitly spec-based, and Section 5.2.2's success criteria are subjective. The paper also reports at least one phase with 0/5 successful tools, yet still leads with 'created over 37 tools' in the abstract, making it easy for readers to infer that most or all of these tools worked. This is an internal evidentiary gap, not just a concern about participant impartiality. The proposed ground-truth evaluation would settle whether the tools are genuinely functional and would directly test the central claim. Since the reader already assigned CONDITIONAL, my concern does not change the verdict; it strengthens the condition that must be met before the claim is fully accepted.","tokens_in":39141,"tokens_out":3150,"duration_ms":37907,"concrete_test":"Run a functional ground-truth evaluation of all final tool versions (or a stratified sample of tools described as successful, including the Uber car identifier, hand-gesture interpreter, business-card reader, and braille translator) on a standardized corpus of images/videos with sighted-generated expected outputs. For each tool, compute task-level accuracy (e.g., precision/recall for car detection, exact/partial match for OCR fields, gesture-classification accuracy). Compare these objective scores against the co-designers' self-reported success ratings and the spec-based code-review verdicts. If a substantial fraction of 'successful' tools fall below a task-appropriate accuracy threshold (e.g., <80% on held-out inputs), the central viability claim is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that agentic programming via ProgramAT enables BLV users to successfully create bespoke camera-based AT (Abstract; §1). But the evidence for 'successful creation' conflates artifact generation with functional correctness. Section 4.4 describes a human-AI code review that 'focuses on whether generated code satisfies the provided specs'—i.e., it checks code against user-written specifications, not whether the tool produces accurate outputs under real-world camera conditions. There is no sighted ground-truth verification of tool behavior. Section 5.2.2 defines success pragmatically through co-designer self-report (e.g., 'reliably good enough to support a real-world goal'), and the supporting quotes emphasize perceived uniqueness or friction reduction rather than measured accuracy. Meanwhile, §5.2.3 documents model failures and spec conflicts, and the paper never reports how many of the 37 tools actually reached a working threshold. The text even notes D2's Phase I success rate was 0/5 (0/5 successful tools), which shows the headline count of 37 includes failed or abandoned artifacts. Without an objective functional evaluation, the load-bearing assumption that creation implies usable, reliable tools is unsupported. If many created tools did not function correctly in situ, the claim that agentic programming is a viable creation paradigm—and the design recommendations built on that claim—would be substantially weakened.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ProgramAT, a mobile agentic programming instrument (natural-language requests mediated by GitHub Copilot) that enables blind and low-vision (BLV) users to author, iterate, and test bespoke camera-based assistive technologies. Through a two-phase longitudinal co-design study with five tech-savvy BLV co-designers (all also paper authors), the authors report 37 created tools across 68 iterations, 24+ hours of interviews, and development-log analysis. The paper addresses three research questions: what tools BLV users would create, why they create them, and what supports are needed. Findings characterize a broad design space (from Uber finders and hand-gesture interpreters to Braille readers and chess-move validators), distinct prompting/iteration strategies, model-capability and specification-conflict failures, and design recommendations for conversational scaffolding, tool sharing, and bring-your-own-data support.","tokens_in":39422,"tokens_out":3057,"duration_ms":36960,"significance":"If its central claim is accepted, the paper provides one of the first empirical demonstrations that agentic programming can lower barriers for BLV users to create personalized camera-based AT, with direct implications for future tool design. The open-source ProgramAT instrument, the two-month longitudinal design, the rich qualitative corpus, and the explicit disclosure of the co-designers-as-authors relationship are notable strengths. However, the study’s validity rests on treating self-reported and code-review-verified tool generation as evidence of functional, in-situ success, and the paper does not report how many of the 37 tools actually reached a working threshold. The design recommendations are plausibly useful, but they would be more convincing if the claim of 'viability' were grounded in a transparent per-tool success/failure accounting and at least some independent functional verification.","major_comments":[{"comment":"The headline claim that co-designers 'created over 37 tools' conflates artifact generation with functional success. The paper itself notes in §5.2.1 that D2's Phase I tools had a success rate of 0/5, and §5.2.3 documents multiple failures. No per-tool outcome table (working, partially working, abandoned, failed) is provided. This is load-bearing for the central viability claim: before 'viable' can be assessed, the paper should report the number of tools that met the pragmatic success threshold defined in §5.2.2, and how many were abandoned or never functioned adequately.","section":"Abstract; §5.2.1"},{"comment":"There is no objective or sighted ground-truth verification of tool behavior. The code-review procedure in §4.4 checks whether generated code satisfies user-written specifications, not whether the tool produces correct outputs under real-world camera conditions. Success in §5.2.2 is defined by co-designer self-report ('reliably good enough to support a real-world goal'). The supporting quotes emphasize perceived value (novelty, reduced friction) rather than measured accuracy, leaving open the possibility that many tools were 'created' but not actually usable. At minimum, the authors should report the outcome status of each of the 37 tools and, if possible, independently validate a subset (e.g., a sighted evaluator comparing tool outputs on known stimuli). Alternatively, the paper should reframe its claims from 'viable creation paradigm' to 'users can produce prototypes they perceive as us","section":"§4.4; §5.2.2"},{"comment":"All five co-designers are also authors of the paper. While this is disclosed in §4.1, the paper does not discuss how this dual role may shape the qualitative analysis. For example, the claim that the Uber-identifier and hand-gesture interpreter address needs 'unmet by any existing commercial AT' rests on author-participant self-reports and could be influenced by investment in the system's success. The manuscript should include a reflexivity statement addressing researcher-participant positionality, and should distinguish between 'participants found the tool valuable' and 'the tool objectively met an unmet need.' A few sentences of mitigation would materially strengthen trustworthiness.","section":"§4.1; §5.1.1"},{"comment":"The examples cited in the abstract—'identifying Uber rides or interpreting hand gestures'—are presented as success stories, but the table does not indicate whether these tools were ultimately merged, used repeatedly, or abandoned. D2's car finder is described elsewhere as 'stood out' but no functional accuracy data are given; D1's hand-gesture interpreter is described as 'working well' but again without any objective test. Please provide the creation/iteration/outcome trajectory for these two flagship tools, including how many iterations were needed and whether they were still in use at study end.","section":"§5.1.1, Table 2"}],"minor_comments":[{"comment":"There are duplicate figure numbers: two figures are labeled 'Figure 1' (the teaser and the workflow overview) and two are labeled 'Figure 2' (the walkthrough and the timeline). Please renumber.","section":"General"},{"comment":"The phrase 'cripepistemologies' appears to be an intentional portmanteau, but it is not clearly defined; if intended, please define it on first use and consider using the more standard 'crip epistemologies'.","section":"§5.2.1"},{"comment":"Minor typos in the instruction file: 'runable' should be 'runnable' and 'priroritize' in Appendix C.1.1 should be 'prioritize.'","section":"Appendix A"},{"comment":"The description of the thematic-analysis process is useful but does not specify how many diary entries were collected per co-designer or how many were analyzed. Including a small table of data volumes (interviews, diary entries, PRs reviewed) would help readers calibrate the qualitative evidence.","section":"§4.4"},{"comment":"The three failure categories (unsupported technical approach, model failure, spec conflict) are clear and well illustrated. Consider providing a summary table of which of the 37 tools fell into each category—this would complement the proposed per-tool outcome reporting and strengthen the paper's analytical contribution.","section":"§5.2.3"}],"recommendation":"major_revision","confidential_remarks":"This paper makes a genuinely useful empirical contribution to accessibility and end-user programming, and the open-source artifact is a plus. My main concern is that the central 'viability' claim is not yet supported by the evidence as presented: the paper needs a transparent success/failure accounting and either independent functional verification or a more modest framing. The author-participant relationship is disclosed but not methodologically addressed; a reflexivity statement would help. These are fixable with additional analysis and reframing rather than new experiments, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. This is the first longitudinal, multi-user account of BLV people building camera-based assistive tools with an agentic coding system, and it is worth your time. The authors open-source ProgramAT, ran a two-month co-design with five tech-savvy BLV participants, and documented 37 tools, 68 iterations, and 24+ hours of interviews. The qualitative findings—motivations, prompt strategies, spec conflicts, the shift from pragmatic to exploratory tools—are concrete and often vivid. The paper is also unusually honest about failures: D2's Phase I success rate was 0/5, the straight-line walker died for lack of persistent memory, and the Braille reader hallucinated. That honesty earns real credit.\n\nThe soft spots are real but mostly disclosed. All five co-designers are also authors, which is a genuine conflict that the paper mentions but doesn't deeply analyze. And the stress-test note is right that 'created' does not mean 'works': the 37 count includes abandoned and failed tools, and the code review checks whether generated code follows the user's written specs, not whether the tool actually recognizes an Uber in a dark street. So the abstract's phrase about tools 'addressing needs unmet by any existing commercial AT' rests on self-report, not independent verification.\n\nThat said, I don't think this sinks the paper. The central claims are about what BLV people choose to build, how they iterate, and where they struggle—not a controlled evaluation of tool accuracy. The failure analysis actually strengthens the qualitative contribution. What's missing is a clearer accounting of the denominator: of the 37 tools, how many did the co-designer themselves consider good enough for real use, and how many were abandoned? A simple per-tool success classification would make the 'viability' claim much easier to assess.\n\nWho is this for? HCI and accessibility researchers working on end-user programming, DIY assistive tech, and conversational coding. It would also inform anyone designing agentic programming environments for non-traditional programmers. It deserves a serious peer review—the novelty and the open-source instrument earn that. My recommendation: send it out, but ask the authors to add a reflexivity section on the dual role of co-designers, report per-tool status, and ideally have a sighted evaluator spot-check a subset of tools against ground truth. That would turn a good empirical study into a rigorous one.","headline":"First longitudinal study of BLV users building camera-based AT with agentic programming—rich qualitative findings, but '37 tools created' should not be read as '37 tools worked.'","tokens_in":39931,"tokens_out":3352,"would_cite":true,"duration_ms":39515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Blind and low-vision users create 37 bespoke vision tools with natural-language coding agents.","keywords":["agentic programming","assistive technology","blind and low vision","co-design","end-user programming","generative AI","camera-based tools","conversational programming"],"falsifier":"Run the same study with BLV participants who are not co-authors and who have no development experience; if most cannot create working tools even with the proposed supports (conversational guidance, tool sharing, personal data), the paradigmatic claim fails. Additionally, have independent sighted evaluators verify tool behavior (e.g., does the Uber identifier actually match a car, does the hand-gesture interpreter recognize correctly) against ground truth; if tools are largely non-functional, the '37 tools' success count collapses.","tokens_in":39036,"feed_emoji":"🛠️","tokens_out":2155,"duration_ms":26421,"temperature":0.7,"pith_summary":"The paper claims that agentic programming—describing a tool in plain language and letting an AI coding agent write the code—can let blind and low-vision (BLV) people author their own camera-based assistive technology. Over two months, five tech-savvy BLV co-designers used a purpose-built instrument called ProgramAT to create 37 tools, including an Uber ride identifier and a hand-gesture interpreter that no commercial assistive app offers. The authors argue this validates a new creation paradigm for accessibility: instead of waiting for general-purpose AT, users can prototype, test, and refine personalized tools in the moment. But the paper also shows that success rarely comes from a single prompt—it emerges through iterative testing, debugging reports, and adjusting to what the underlying AI models can actually do.","feed_headline":"BLV users built 37 custom vision tools by chatting with a coding agent","feed_subtitle":"A two-month study shows natural-language programming can fill gaps no commercial assistive app covers.","key_machinery":"ProgramAT, the study instrument: a mobile app plus a GitHub-based agentic coding environment (using a coding agent under a structured instruction file) that turns natural-language requests into working camera tools. Its load-bearing components are the issue-template structure that converts free-form prompts into grounded specifications, modular 'building-block' code that lets tools reuse and remix prior components, automatic model selection (object detection, OCR, vision-language), an iterative pull-request workflow for refinement, and an audio-first output schema so generated tools speak their results.","core_discovery":"Agentic programming can meaningfully lower the barrier to creating bespoke camera-based assistive tools for technically experienced BLV users, who in this study produced 37 working prototypes across 68 iterations—several addressing needs unmet by any commercial assistive technology, such as identifying a specific Uber car or interpreting hand gestures in video. The paper documents that creators adopt varied prompt strategies (long structured prompts, short action-oriented prompts, progressive specification), that failures stem from three identifiable sources (unsupported technical requirements, model capability limits, and specification conflicts), and that community sharing of tools and use","pith_inferences":["The paradigm likely generalizes beyond camera-based tasks and beyond the BLV community: any disability community with identifiable access needs could use the same conversational-creation loop to build bespoke tools, provided the agent scaffolding and privacy controls are adapted.","Because all five co-designers were tech-savvy and authored the paper, the success rate may be an upper bound; less technical users would probably need the proposed conversational scaffolding just to reach the starting line, making the recommendations more critical than the headline '37 tools' number suggests.","The privacy concerns raised (e.g., D5's test-with-known-examples strategy) point toward a broader design requirement: AI-authored tools are hard to trust, so observability and user control over data flow must be first-class features of any such system."],"forward_implications":["If agentic programming works for BLV users, general-purpose commercial assistive apps could adopt plugin-style architectures that let users build custom tools on top of them, rather than waiting for one-size-fits-all features.","Design recommendations from the study—conversational scaffolding during creation, community tool-sharing, and bring-your-own-data support—become concrete requirements for future accessibility-oriented coding agents.","The finding that tool success depends on iteration and 'experience reports' suggests that agentic AT creation should be designed as a feedback loop, not a one-shot generation step.","The prevalence of tools addressing unmet niche needs (Uber identification, hand gestures, food placement) indicates a real long-tail demand that commercial AT leaves unserved and that end-user creation can partially fill."],"fun_headline_variants":["BLV users build 37 custom tools by chatting with AI coder","Agentic programming unlocks bespoke vision aids for BLV users","Blind users create 37 personalized camera tools via agentic AI","Chat with coding agent yields 37 unique assistive vision tools","Study: conversational coding lets BLV users make their own AT"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The findings rest on five co-designers who are also the paper's authors and all technically experienced; if their success reflects their expertise and motivation rather than the agentic-programming paradigm itself, the central claims about feasibility would be overstated.","fun_headline_variants_meta":{"raw":{"variants":["BLV users build 37 custom tools by chatting with AI coder","Agentic programming unlocks bespoke vision aids for BLV users","Blind users create 37 personalized camera tools via agentic AI","Chat with coding agent yields 37 unique assistive vision tools","Study: conversational coding lets BLV users make their own AT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000661,"raw_usage":{"total_tokens":2846,"prompt_tokens":719,"completion_tokens":2127,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":2038}},"tokens_in":463,"tokens_out":2127,"duration_ms":13109,"temperature":1.0,"reasoning_tokens":2038,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:45:10.175090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same study with BLV participants who are not co-authors and who have no development experience; if most cannot create working tools even with the proposed supports (conversational guidance, tool sharing, personal data), the paradigmatic claim fails. Additionally, have independent sighted evaluators verify tool behavior (e.g., does the Uber identifier actually match a car, does the hand-gesture interpreter recognize correctly) against ground truth; if tools are largely non-functional, the '37 tools' success count collapses.","supporting_citations":[],"review_version":1}