Pith. sign in

REVIEW 4 major objections 5 minor 80 references

AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multi-modal Information Between Reality and Videos

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AROMA lets blind and low-vision cooks follow video recipes by pairing their own senses with a live camera feed.

desk verdict A serious system-building paper for accessibility HCI with a genuinely new mixed-initiative concept, but the effectiveness claim outruns the evidence because the video-grounding pipeline is never validated. read the letter →

arxiv 2507.10963 v1 pith:AP55UGXC submitted 2025-07-15 cs.HC

Zheng Ning , Leyang Li , Daniel Killough , JooYoung Seo , Patrick Carrington , Yapeng Tian , Yuhang Zhao , Franklin Mingzhe Li
show 1 more author
Toby Jia-Jun Li
This is my paper
classification cs.HC
keywords accessiblecookingblindandlow-visionmixed-initiativeinteractionmultimodalgroundingvideorecipeswearablecameravoiceassistiveAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a mixed-initiative AI system, AROMA, can help blind and low-vision (BLV) cooks follow instructional cooking videos by combining three information streams: the cook's own verbalized non-visual perceptions (texture, smell, taste, sound), a real-time wearable camera feed, and the video recipe's extracted content. The motivation is that video recipes are rich but largely inaccessible to BLV people, who rely on non-visual cues. The paper reports a study with eight BLV participants in which AROMA maps requests to the right interaction event with 0.82 accuracy and gives factually correct immediate responses 0.67 of the time, with high usability ratings for conversational features. If correct, this is a working proof of concept for real-time, collaborative, video-grounded cooking assistance that treats the user's embodied knowledge as a first-class input.

What carries the argument

The load-bearing mechanism is the coupling of a state-machine interaction framework with a multimodal grounding pipeline. The state machine (nine events, seven states) decides, from the user's voice query and the current visual scene, which kind of response to generate and which prompt template the backend LLM should use. The grounding pipeline converts each recipe video into a structured JSON file containing a sentence-level transcript, per-sentence visual descriptions from key frames, and audio descriptions from an audio-language model; this JSON, together with the session history and the periodic visual analysis results, is what every grounded answer and proactive alert is retrieved from.

What would settle it

Take the three recipe videos used in the study, have human annotators write a step list for each, and compare AROMA's extracted JSON steps against it; if the JSON mislabels or omits a meaningful fraction of steps, the grounding pipeline is a likely source of the 0.67 factual accuracy, and the study could be repeated with corrected JSON to see whether responses improve.

Watch

Extended reading notes

Core claim

On the paper's own terms, AROMA's central discovery is that grounding audio-visual video content in the user's physical cooking state works best as a two-way collaboration: the user contributes non-visual perceptions that the system cannot get from a camera, and the system contributes real-time visual analysis and recipe knowledge that the user cannot get by touch or sound alone. The system answers on-demand questions (food state, step clarification, problem solving, and general visual questions) through a state machine with nine events and seven states, while a visual analysis agent polls the first-person camera every two seconds to detect missed, incorrect, or newly started steps and proactively alerts the user. All answers are conditioned on a structured JSON representation of the recipe, built offline by transcribing the video's speech, extracting key frames with scene detection, describing them with a vision-language model, and describing sounds with an audio-language model. The user study with eight BLV participants is offered as evidence of effectiveness, with the accuracy figures and usability ratings reported as the quantitative support.

Load-bearing premise

The system grounds every answer in a JSON summary of the recipe that is built automatically by transcribing speech, extracting key frames, and describing images and sounds with AI models, and this summary is never checked against the original video or a human-authored step list, so any error or omission in that summary propagates into every response and alert.

Editorial extensions

If this is right

  • If AROMA's accuracy holds, BLV users could follow an unfamiliar video recipe in their own kitchen with only a phone camera and voice interaction, without a sighted assistant.
  • The hybrid perception loop suggests assistive systems should treat users' tactile, olfactory, and auditory judgments as reliable inputs rather than as substitutes to be replaced.
  • Proactive alerts that catch missed steps (e.g., forgotten salt, misordered ingredients) could reduce cognitive load and error recovery time during cooking.
  • The same grounding pattern could be applied to other procedural video tasks, such as makeup, repair, crafts, or gardening, as the paper itself suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 0.67 factual-answer accuracy is the system-side ceiling in this small study; the paper does not break down which errors come from the video-to-JSON extraction versus the LLM's response generation, so a validation of the JSON against human step annotations would separate those two failure sources.
  • If the JSON representation were validated, a natural next test is whether proactive alerts improve task completion time or error rate compared with on-demand assistance alone, which the current study does not measure.
  • The offline preprocessing requirement is a practical barrier: until the pipeline runs on arbitrary new videos in near real time, the system cannot be a plug-and-play assistant for everyday YouTube recipes, a limitation the paper acknowledges.
  • The paper's emphasis on user agency suggests a configurable proactivity level (e.g., alert sensitivity) would be worth testing, since participants were frustrated by alerts during intentional deviations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents AROMA, a mixed-initiative multimodal AI system that assists blind and low-vision (BLV) users in cooking by grounding real-time user verbalized non-visual cues and a wearable camera feed in knowledge extracted from instructional cooking videos. The system contains an offline video-to-JSON extraction pipeline (ASR transcript, PySceneDetect keyframes, GPT-4o visual descriptions, GAMA audio descriptions), a real-time visual analysis agent, a conversational LLM agent, and a state-machine interaction model with nine events and seven states. It offers three features: contextualized Q&A grounded in visual and non-visual perception, proactive monitoring and alerts, and video segment replay plus memory retrieval. The evaluation is a user study with eight BLV participants cooking in their own or chosen kitchens using one of three recipes. Reported quantitative results are 0.82 event-mapping accuracy and 0.67 factual response accuracy, with high Likert ratings for conversational features; qualitative analysis contributes design insights on non-visual perception, spatial awareness, and proactive support. The paper claims to 'demonstrate the effectiveness of AROMA' and to contribute the system, the user study, and design insights.

Significance. The system addresses a genuinely important and underserved problem: making instructional cooking videos accessible and actionable for BLV users through a mixed-initiative collaboration that combines the user's embodied non-visual skills with AI visual grounding. The design is well motivated by prior work, the implementation is described with enough specificity to be largely reproducible, and the qualitative findings—especially around confirmation of touch-based inferences, procedural fluidity, and proactive-alert timing—are valuable for the assistive technology community. The paper's strengths include a realistic in-the-wild study setting, a clear articulation of design goals, and honest reporting of latency and safety limitations. However, the central empirical claim of 'effectiveness' currently rests on researcher-labeled accuracy metrics and self-report ratings, and the video-grounding representation that is the core of the system is never independently validated. If the missing validation is supplied or the claims are appropriately softened, the work would be a solid proof-of-concept contribution.

major comments (4)
  1. [Sec. 3.5 and Sec. 4.3 / Table 3] The offline extraction pipeline (Sec. 3.5) is the sole source of video-recipe grounding for all three features, and the real-time visual agent compares live observations against this JSON structure. The paper never validates this representation against the original video or against an independent human-authored step list: there is no key-frame alignment check, no step-boundary agreement, no fidelity or completeness measure. Consequently, the 0.82 event-mapping and 0.67 response-accuracy figures in Table 3 label final responses, not the grounding source; a correct answer could be produced from LLM commonsense or the live camera even if the JSON is wrong. I ask the authors to add a validation study of the extracted JSON (e.g., annotator agreement with a gold step list, timestamp alignment, and an ablation that removes the JSON source) or to substantially soften the claim that responses are grounded in video content.
  2. [Sec. 4.3] The two quantitative metrics rely entirely on two researchers' manual labels, with non-agreement cases 'discussed to reconcile differences,' but no inter-rater reliability statistic is reported for the event-category mapping or for the factual-correctness boolean. Without Cohen's kappa or an equivalent measure, the reader cannot tell whether the 0.82/0.67 numbers reflect system performance or labeling protocol. The authors should report per-labeler agreement, kappa for both annotation tasks, and a precise procedure for resolving disagreements.
  3. [Sec. 5.1 and Sec. 8] The conclusion that the study 'demonstrate[s] the effectiveness of AROMA' is not supported by the reported evidence. There is no baseline or comparison condition, the quantitative accuracy metrics are computed by the researchers rather than measuring task outcomes, and the usability ratings are self-reports after a single session. I recommend either adding a comparison condition (e.g., conventional video access or a voice-only assistant) with task-completion measures, or reframing the contribution as a usability proof-of-concept with clearly scoped claims.
  4. [Sec. 4.2 and Sec. 7] The study excluded heat-generating appliances and sharp knives, and participants were instructed to skip heat-related steps. Since the proactive monitoring feature and many food-state responses (e.g., 'internal temperature has not reached 165°F') are most safety-critical in precisely these omitted scenarios, the evaluation does not test the system's value in the situations where it would matter most. The limitations section acknowledges this, but the framing should be stronger: the current data support usability of the interaction model for assembly and ingredient-identification tasks, not effectiveness for the full range of cooking activities.
minor comments (5)
  1. [Sec. 5.4.1] The phrase 'blind chief' should read 'blind chef.'
  2. [Sec. 4.3] The accuracy calculation excludes user follow-ups (events 7 and 8), but the paper does not report how many queries were excluded per participant; please include these counts in Table 3 or its caption.
  3. [Sec. 3.4 / Fig. 3] The nine events and seven states are not fully defined in the text; please add a table listing each event's trigger condition and each state's response template so the state machine is reproducible.
  4. [Sec. 3.5] The implementation names specific APIs but omits all prompt templates and decoding parameters; adding the prompts (in an appendix) would substantially improve reproducibility.
  5. [Sec. 5.2] The sentence reporting proactive-assistance ratings is missing the word 'useful' before the second parenthetical; the values should be presented consistently with the conversational-feature ratings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims are anchored in external human labels and participant self-report, and no metric is defined in terms of AROMA's own outputs.

full rationale

The paper's derivation chain is not circular. The central quantitative results (Sec 4.3, Table 3) are obtained by having two researchers annotate each user request's event category as ground truth and label whether each immediate response is correct and on-topic; participants' ratings are 7-point Likert self-reports (Sec 5.1, Fig. 4). None of these measures is defined in terms of AROMA's own generated outputs, and no parameter is fitted to force specific outcomes. The offline video-to-JSON extraction pipeline in Sec 3.5 is indeed never validated against the original videos, which means the claimed 'video grounding' may be weaker than stated, but that is a validity or reproducibility concern, not circularity: a response can be judged correct or incorrect independently of where its content came from, so the evaluation does not reduce to the system's inputs by construction. Prior work by the same author group is cited as design rationale for the state machine and interaction preferences (e.g., [33, 34, 41] in Sec 3.3), but those citations are motivational, not load-bearing: the effectiveness claim rests on the new user study data, not on accepting any self-cited theorem. Accordingly, no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

This is a system paper with no fitted parameters: the reported numbers (0.82 mapping, 0.67 response accuracy) are measurements, not fits. The load-bearing premises are domain assumptions about the fidelity of automated video extraction, the expressiveness of users' verbalized perception, the adequacy of the camera view, and the coverage of the event model. None of these is benchmarked independently.

free parameters (2)
  • Visual monitoring interval = 2 seconds
    Chosen by hand (Sec 3.4.3, Sec 3.5). Sets the staleness bound of the visual state; a timing constant, not a fitted parameter, and the central claim does not hinge on this exact value.
  • Idle return timeout = 5 seconds
    Chosen by hand for the state machine to return to the initial state (Fig 3, Sec 3.5). Minor interaction constant with no effect on the central claim.
assumptions (4)
  • domain assumption The video-to-JSON preprocessing pipeline yields a faithful structured representation of the recipe's instructional content.
    Invoked in Sec 3.3 and Sec 3.5. All grounded answers and alerts refer to this representation; its quality is never measured against the original video or a human step list.
  • domain assumption BLV users can verbalize their non-visual perceptions (touch, smell, taste, sound) in ways an LLM can ground into useful answers.
    Core interaction channel for Ft.1 (Sec 3.4.1). The study shows successful examples, but follow-up events 7 and 8 exist precisely because initial interpretations often fail, so the assumption holds only partially.
  • domain assumption The chest-mounted phone camera provides a sufficiently informative view of the cooking state for the vision agent's periodic judgments.
    Adopted from prior findings [34] in Sec 4.3. Latency and action-misalignment issues are acknowledged in Sec 7, so the view is not always current.
  • domain assumption The nine-event, seven-state interaction model covers the range of BLV users' cooking requests.
    The state machine (Fig 3) is drawn from prior studies [33, 34, 41] per Sec 3.3. Eighteen percent of requests were mapped to the wrong event (Sec 5.1), so coverage is incomplete.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multi-modal Information Between Reality and Videos." pith.science (2026). https://pith.science/paper/AP55UGXC

@misc{pith2026250710963,
  author       = {Pith},
  title        = {Pith review of: AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multi-modal Information Between Reality and Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AP55UGXC}},
  note         = {Machine review of arXiv:2507.10963}
}
read the original abstract

Videos offer rich audiovisual information that can support people in performing activities of daily living (ADLs), but they remain largely inaccessible to blind or low-vision (BLV) individuals. In cooking, BLV people often rely on non-visual cues, such as touch, taste, and smell, to navigate their environment, making it difficult to follow the predominantly audiovisual instructions found in video recipes. To address this problem, we introduce AROMA, an AI system that provides timely responses to the user based on real-time, context-aware assistance by integrating non-visual cues perceived by the user, a wearable camera feed, and video recipe content. AROMA uses a mixed-initiative approach: it responds to user requests while also proactively monitoring the video stream to offer timely alerts and guidance. This collaborative design leverages the complementary strengths of the user and AI system to align the physical environment with the video recipe, helping the user interpret their current cooking state and make sense of the steps. We evaluated AROMA through a study with eight BLV participants and offered insights for designing interactive AI systems to support BLV individuals in performing ADLs.

Figures

Figures reproduced from arXiv: 2507.10963 by the authors.

Figure 1
Figure 1. An illustration of how a blind or low-vision (BLV) user uses the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the architecture of the system, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The state-machine-like framework in Aroma to handle various user queries and reality visual scene changes. The system starts at the initial state (state 0), and transitions between different states are determined by various events. Each event is decided by the current visual scene and the user’s query (if available) by an LLM. When transitioning to a new state, an LLM generates a response using a predefined prompt t… view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: The accuracy is calculated by dividing the correctly mapped [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 4
Figure 4. Figure 4: Participants’ ratings of the usability and usefulness statements of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 38 canonical work pages

  1. [1]

    Nayyer Aafaq, Ajmal Mian, Wei Liu, Syed Zulqarnain Gilani, and Mubarak Shah

    work page

  2. [2]

    Tousif Ahmed, Roberto Hoyle, Kay Connelly, David Crandall, and Apu Kapadia

  3. [4]

    Ava Bartolome and Shuo Niu. 2023. A Literature Review of Video-Sharing Platform Research in HCI. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–20. doi:10.1145/3544548.3581107

    arXiv 2023

  4. [5]

    Virginia Braun, , and Victoria Clarke. 2006. Using thematic analy- sis in psychology.Qualitative Research in Psychology3, 2 (Jan. 2006), 77–101. doi:10.1191/1478088706qp063oa Publisher: Routledge _eprint: https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa

    doi:10.1191/1478088706qp063oa 2006

  5. [6]

    Cai, Jonas Jongejan, and Jess Holbrook

    Carrie J. Cai, Jonas Jongejan, and Jess Holbrook. 2019. The effects of example- based explanations in a machine learning interface. InProceedings of the 24th In- ternational Conference on Intelligent User Interfaces (IUI ’19). Association for Com- puting Machinery, New York, NY, USA, 258–262. doi:10.1145/3301275.3302289

    arXiv 2019

  6. [7]

    Xinyun Cao and Dhruv Jain. 2024. Supporting Sound Accessibility by Ex- ploring Sound Augmentations in Virtual Reality. InProceedings of the 26th In- ternational ACM SIGACCESS Conference on Computers and Accessibility (AS- SETS ’24). Association for Computing Machinery, New York, NY, USA, 1–5. doi:10.1145/3663548.3688525

    arXiv 2024

  7. [8]

    Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, and Rada Mihalcea. 2020. LifeQA: A Real-life Dataset for Video Question Answering. InProceedings of the Twelfth Language Resources and Eval- uation Conference, Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, ...

    work page 2020

  8. [9]

    Paul Chandler, , and John Sweller. 1991. Cognitive Load Theory and the Format of Instruction.Cognition and Instruction8, 4 (Dec. 1991), 293–332. doi:10.1207/s1532690xci0804_2 Publisher: Routledge _eprint: https://doi.org/10.1207/s1532690xci0804_2

    doi:10.1207/s1532690xci0804_2 1991

Show all 80 references
  1. [10]

    Guillain, Hyeungshik Jung, Vivian M

    Minsuk Chang, Leonore V. Guillain, Hyeungshik Jung, Vivian M. Hare, Juho Kim, and Maneesh Agrawala. 2018. RecipeScape: An Interactive Tool for Analyzing Cooking Instructions at Scale. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). Ass...

    2018

  2. [11]

    Corbin and Anselm Strauss

    Juliet M. Corbin and Anselm Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria.Qualitative Sociology13, 1 (March 1990), 3–21. doi:10.1007/BF00988593

    1990 doi

  3. [12]

    Yinpei Dai, Run Peng, Sikai Li, and Joyce Chai. 2024. Think, Act, and Ask: Open- World Interactive Personalized Robot Navigation. http://arxiv.org/abs/2310.07968 arXiv:2310.07968

    2024 arXiv

  4. [13]

    2001.Where the Action Is: The Foundations of Embodied Interaction

    Paul Dourish. 2001.Where the Action Is: The Foundations of Embodied Interaction. The MIT Press. doi:10.7551/mitpress/7221.001.0001

    2001 doi

  5. [14]

    Hudson, Mahya Beheshti, Mau- rizio Porfiri, and John-Ross Rizzo

    Junchi Feng, Giles Hamilton-Fletcher, Todd E. Hudson, Mahya Beheshti, Mau- rizio Porfiri, and John-Ross Rizzo. 2025. Haptics-based, higher-order sensory substitution designed for object negotiation in blindness and low vision: Virtual Whiskers.Disability and Rehabilitation. As...

    2025

  6. [15]

    2016.An Introduction to Audio Description: A practical guide

    Louise Fryer. 2016.An Introduction to Audio Description: A practical guide. Rout- ledge, London. doi:10.4324/9781315707228

    2016 doi

  7. [16]

    Lishuai Gao, Yujie Zhong, Yingsen Zeng, Haoxian Tan, Dengjie Li, and Zheng Zhao. 2024. LinVT: Empower Your Image-level Large Language Model to Un- derstand Videos. doi:10.48550/arXiv.2412.05185 arXiv:2412.05185 [cs] version: 2

    2024 doi arXiv

  8. [17]

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities. InProceedings of the 2024...

    2024

  9. [18]

    Google Cloud. 2024. Vertex AI Video Understanding - Generative AI on Google Cloud. https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/video- understanding Accessed: 2025-04-08

    2024

  10. [19]

    Xu, and Jeffrey P

    Anhong Guo, Junhan Kong, Michael Rivera, Frank F. Xu, and Jeffrey P. Bigham

  11. [20]

    Haggard and Kenneth S

    Ernest A. Haggard and Kenneth S. Isaacs. 1966. Micromomentary facial expres- sions as indicators of ego mechanisms in psychotherapy. InMethods of Research in Psychotherapy, Louis A. Gottschalk and Arthur H. Auerbach (Eds.). Springer US, Boston, MA, 154–165. doi:10.1007/978-1-4...

    1966 doi

  12. [21]

    Keita Higuch, Ryo Yonetani, and Yoichi Sato. 2016. Can Eye Help You? Effects of Visualizing Eye Fixations on Remote Collaboration Scenarios for Physical Tasks. InProceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing M...

    2016

  13. [23]

    Felix Huppert, Gerold Hoelzl, and Matthias Kranz. 2021. GuideCopter - A Precise Drone-Based Haptic Guidance Interface for Blind or Visually Impaired People. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–14. doi:10.1145/...

    2021

  14. [24]

    Gaurav Jain, Basel Hindi, Connor Courtien, Xin Yi Therese Xu, Conrad Wyrick, Michael Malcolm, and Brian A. Smith. 2023. Front Row: Automatically Generating Immersive Audio Representations of Tennis Broadcasts for Blind Viewers. In Proceedings of the 36th Annual ACM Symposium o...

    2023

  15. [25]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. InProceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI ’99. ACM Press, Pittsburgh, Pennsylvania, United States, 159–166. doi:10.1145/302979.303030 UIST ’25, Sep...

    1999

  16. [26]

    Mohammad Kianpisheh, Franklin Mingzhe Li, and Khai N Truong. 2019. Face recognition assistant for people with visual impairments.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies3, 3 (2019), 1–24

    2019

  17. [27]

    Benjamin Lafreniere, Tovi Grossman, and George Fitzmaurice. 2013. Commu- nity enhanced tutorials: improving tutorials with multiple demonstrations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New...

    2013

  18. [28]

    Lucy Jiang, Crescentia Jung, Mahika Phutane, Abigale Stangl, and Shiri Azenkot

  19. [29]

    Tjahjadi, Jiho Kim, Junpu Yu, Minji Park, Jiawen Zhang, Jon E

    Jaewook Lee, Andrew D. Tjahjadi, Jiho Kim, Junpu Yu, Minji Park, Jiawen Zhang, Jon E. Froehlich, Yapeng Tian, and Yuhang Zhao. 2024. CookAR: Affordance Aug- mentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision. InProceedings of the 37th Ann...

    2024

  20. [30]

    Berg, Mohit Bansal, and Jingjing Liu

    Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. 2021. Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling. 7331–7341. https://openaccess.thecvf. com/content/CVPR2021/html/Lei_Less_Is_More_ClipBERT_for_Video-and- ...

    2021

  21. [31]

    Barbara Leporini, Michele Rosellini, and Nicola Forgione. 2020. Designing assis- tive technology for getting more independence for blind people when performing everyday tasks: an auditory-based tool as a case study.Journal of Ambient Intel- ligence and Humanized Computing11 (2...

    2020

  22. [32]

    Loretto Lambe. 1995. Gardening: A multisensory experience. InMaking leisure provision for people with profound learning and multiple disabilities. Springer, 113–130

    1995

  23. [33]

    Franklin Mingzhe Li, Jamie Dorst, Peter Cederberg, and Patrick Carrington. 2021. Non-Visual Cooking: Exploring Practices and Challenges of Meal Preparation by People with Visual Impairments. InProceedings of the 23rd International ACM SIGACCESS Conference on Computers and Acce...

    2021 doi

  24. [34]

    Kane, and Patrick Carrington

    Franklin Mingzhe Li, Michael Xieyang Liu, Shaun K. Kane, and Patrick Carrington

  25. [36]

    i choose assistive devices that save my face

    Franklin Mingzhe Li, Di Laura Chen, Mingming Fan, and Khai N Truong. 2021. “i choose assistive devices that save my face” a study on perceptions of accessibility and assistive technology use conducted in china. InProceedings of the 2021 CHI Conference on Human Factors in Compu...

    2021

  26. [37]

    Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, and Patrick Carrington. 2025. OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking. InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25). As...

    2025

  27. [38]

    Franklin Mingzhe Li, Akihiko Oharazawa, Chloe Qingyu Zhu, Misty Fan, Daisuke Sato, Chieko Asakawa, and Patrick Carrington. 2025. More than One Step at a Time: Designing Procedural Feedback for Non-visual Makeup Routines.arXiv preprint arXiv:2507.03942(2025)

    2025 arXiv

  28. [39]

    In Proceedings of the CHI Conference on Human Factors in Computing Systems

    A Contextual Inquiry of People with Vision Impairments in Cooking. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–14. doi:10.1145/3613904.3642233

  29. [40]

    It feels like taking a gamble

    Franklin Mingzhe Li, Franchesca Spektor, Meng Xia, Mina Huh, Peter Cederberg, Yuqi Gong, Kristen Shinohara, and Patrick Carrington. 2022. “It feels like taking a gamble”: Exploring perceptions, practices, and challenges of using makeup and cosmetics for people with visual impa...

    2022

  30. [41]

    Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, and Patrick Carrington. 2025. Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking. arXiv preprint arXiv:2507.03330(2025)

    2025 arXiv

  31. [42]

    Franklin Mingzhe Li, Lotus Zhang, Maryam Bandukda, Abigale Stangl, Kristen Shinohara, Leah Findlater, and Patrick Carrington. 2023. Understanding visual arts experiences of blind people. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21

    2023

  32. [43]

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu

  33. [44]

    It Feels Like Taking a Gamble

    Franklin Mingzhe Li, Franchesca Spektor, Meng Xia, Mina Huh, Peter Cederberg, Yuqi Gong, Kristen Shinohara, and Patrick Carrington. 2022. “It Feels Like Taking a Gamble”: Exploring Perceptions, Practices, and Challenges of Using Makeup and Cosmetics for People with Visual Impa...

    2022

  34. [45]

    Inescapable

    Yunzhi Li, Franklin Mingzhe Li, and Patrick Carrington. 2023. Breaking the “Inescapable” Cycle of Pain: Supporting Wheelchair Users’ Upper Extremity Health Awareness and Management with Tracking Technologies. InProceedings of the 2023 CHI Conference on Human Factors in Computi...

    2023

  35. [46]

    Franklin Mingzhe Li, Ashley Wang, Patrick Carrington, and Shaun K. Kane. 2024. A Recipe for Success? Exploring Strategies for Improving Non-Visual Access to Cooking Instructions. InThe 26th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, St. John’s ...

    2024

  36. [47]

    Georgianna Lin, Jin Yi Li, Afsaneh Fazly, Vladimir Pavlovic, and Khai Truong

  37. [48]

    Kate Lister, Tim Coughlan, Francisco Iniesto, Nick Freear, and Peter Devine

  38. [49]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts.Transactions of the Association for Computational Linguistics 12 (2024), 157–173. doi:10.1162/tacl_a_00638

    2024 doi

  39. [50]

    Mingzhe Li, Mingming Fan, and Khai N Truong. 2017. BrailleSketch: A gesture- based text input method for people with visual impairments. InProceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. 12–21

    2017

  40. [51]

    Xingyu "Bruce" Liu, Ruolin Wang, Dingzeyu Li, Xiang Anthony Chen, and Amy Pavel. 2022. CrossA11y: Identifying Video Accessibility Issues via Cross-modal Grounding. InProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST ’22). Association ...

    2022

  41. [52]

    Ziming Li, Shannon Connell, Wendy Dannels, and Roshan Peiris. 2022. Sound- VizVR: Sound Indicators for Accessible Sounds in Virtual Reality for Deaf or Hard-of-Hearing Users. InProceedings of the 24th International ACM SIGAC- CESS Conference on Computers and Accessibility (ASS...

    2022

  42. [53]

    Yiming Liu, Chun Ki Yiu, Zhao Zhao, Wooyoung Park, Rui Shi, Xingcan Huang, Yuyang Zeng, Kuan Wang, Tsz Hung Wong, Shengxin Jia, Jingkun Zhou, Zhan Gao, Ling Zhao, Kuanming Yao, Jian Li, Chuanlu Sha, Yuyu Gao, Guangyao Zhao, Ya Huang, Dengfeng Li, Qinglei Guo, Yuhang Li, and Xi...

    2023 doi

  43. [54]

    Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M

    Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M. Rush. 2024. A Controlled Study on Long Context Extension and Generalization in LLMs.CoRRabs/2409.12181 (2024). arXiv:2409.12181 doi:10.48550/ARXIV.2409.12181

    2024 doi arXiv

  44. [55]

    Lucas, Zack Arambula, Alexandra M

    Jacob C. Lucas, Zack Arambula, Alexandra M. Arambula, Katherine Yu, Nathan Farrokhian, Linda D’Silva, Hinrich Staecker, and Jennifer A. Villwock. 2022. Olfactory, Auditory, and Vestibular Performance: Multisensory Impairment Is Significantly Associated With Incident Cognitive ...

    2022

  45. [56]

    Martin Merkt, Sonja Weigand, Anke Heier, and Stephan Schwan. 2011. Learning with videos vs. learning with print: The role of interactive features.Learning and Instruction21, 6 (Dec. 2011), 687–704. doi:10.1016/j.learninstruc.2011.03.004

    2011 doi

  46. [57]

    Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. 2019. A Survey on Food Computing.ACM Comput. Surv.52, 5 (Sept. 2019), 92:1–92:36. doi:10.1145/3329168

    2019 doi

  47. [58]

    Xingyu Liu, Patrick Carrington, Xiang ’Anthony’ Chen, and Amy Pavel. 2021. What Makes Videos Accessible to Blind and Visually Impaired People?. InPro- ceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New ...

    2021

  48. [59]

    Mohammadreza Mirzaei, Peter Kán, and Hannes Kaufmann. 2020. EarVR: Us- ing Ear Haptics in Virtual Reality for Deaf and Hard-of-Hearing People.IEEE Transactions on Visualization and Computer Graphics26, 5 (May 2020), 2084–2093. doi:10.1109/TVCG.2020.2973441

    2020

  49. [60]

    Yiming Liu, Shengxin Jia, Chun Ki Yiu, Wooyoung Park, Zhenlin Chen, Jin Nan, Xingcan Huang, Hongting Chen, Wenyang Li, Yuyu Gao, Weike Song, Tomoyuki Yokota, Takao Someya, Zhao Zhao, Yuhang Li, and Xinge Yu. 2024. Intelligent wearable olfactory interface for latency-free mixed...

    2024 doi

  50. [61]

    OpenAI. 2023. OpenAI Text-to-Speech Model. https://platform.openai.com/ docs/guides/text-to-speech

    2023

  51. [62]

    Sharon Oviatt. 1999. Mutual disambiguation of recognition errors in a multimodel architecture. InProceedings of the SIGCHI conference on Human factors in comput- ing systems the CHI is the limit - CHI ’99. ACM Press, Pittsburgh, Pennsylvania, United States, 576–583. doi:10.114...

    1999

  52. [63]

    Patil, Don Kurian Dennis, Chirag Pabbaraju, Nadeem Shaheer, Har- sha Vardhan Simhadri, Vivek Seshadri, Manik Varma, and Prateek Jain

    Shishir G. Patil, Don Kurian Dennis, Chirag Pabbaraju, Nadeem Shaheer, Har- sha Vardhan Simhadri, Vivek Seshadri, Manik Varma, and Prateek Jain. 2019. GesturePod: Enabling On-device Gesture-based Interaction for White Cane Users. InProceedings of the 32nd Annual ACM Symposium ...

    2019

  53. [64]

    Yash Prakash, Akshay Kolgar Nayak, Shoaib Mohammed Alyaan, Pathan Aseef Khan, Hae-Na Lee, and Vikas Ashok. 2024. Improving Usability of Data Charts in Multimodal Documents for Low Vision Users. InInternational Conference on Multimodel Interaction. ACM, San Jose Costa Rica, 498...

    2024 doi

  54. [65]

    Oliver Schneider, Jotaro Shigeyama, Robert Kovacs, Thijs Jan Roumen, Sebastian Marwecki, Nico Boeckhoff, Daniel Amadeus Gloeckner, Jonas Bounama, and Patrick Baudisch. 2018. DualPanto: A Haptic Device that Enables Blind Users to Continuously Interact with Virtual Worlds. InPro...

    2018 doi

  55. [67]

    Joel Snyder. 2005. Audio description: The visual made verbal.International Congress Series1282 (Sept. 2005), 935–939. doi:10.1016/j.ics.2005.05.215

    2005 doi

  56. [68]

    Zheng Ning, Brianna L Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, and Toby Jia-Jun Li. 2024. SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers. InProceedings of the 2024 CHI Conference o...

    2024

  57. [69]

    Turkstra, Tanya Bhatia, Alexa Van Os, and Michael Beyeler

    Lily M. Turkstra, Tanya Bhatia, Alexa Van Os, and Michael Beyeler. 2025. Assistive technology use in domestic activities by people who are blind.Scientific Reports 15, 1 (March 2025), 7486. doi:10.1038/s41598-025-91755-w Publisher: Nature Publishing Group

    2025 doi

  58. [70]

    Stephanie Valencia, Mark Steidl, Michael Rivera, Cynthia Bennett, Jeffrey Bigham, and Henny Admoni. 2021. Aided Nonverbal Communication through Physical Expressive Objects. InProceedings of the 23rd International ACM SIGACCESS Con- ference on Computers and Accessibility (ASSET...

    2021

  59. [71]

    Tess Van Daele, Akhil Iyer, Yuning Zhang, Jalyn C Derry, Mina Huh, and Amy Pavel. 2024. Making Short-Form Videos Accessible with Hierarchical Video Sum- maries. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing ...

    2024

  60. [72]

    Yujia Wang, Wei Liang, Haikun Huang, Yongqi Zhang, Dingzeyu Li, and Lap- Fai Yu. 2021. Toward Automatic Audio Description Generation for Accessible Videos. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machi...

    2021

  61. [73]

    R. S. Woodworth and E. L. Thorndike. 1901. The influence of improvement in one mental function upon the efficiency of other functions. (I).Psychological Review 8, 3 (1901), 247–261. doi:10.1037/h0074898 Place: US Publisher: The Macmillan Company

    1901 doi

  62. [74]

    Amarjot Singh, Ketan Bacchuwar, and Akshay Bhasin. 2012. A Survey of OCR Applications.International Journal of Machine Learning and Computing(2012), 314–318. doi:10.7763/IJMLC.2012.V2.137

    2012 doi

  63. [75]

    YouTube. [n.d.]. YouTube for Press. https://blog.youtube/press/ Accessed: 2025-04-07

    2025

  64. [76]

    Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Ur- tasun, and Sanja Fidler. 2016. MovieQA: Understanding Stories in Movies Through Question-Answering. 4631–4640. https://openaccess.thecvf.com/content_cvpr_ 2016/html/Tapaswi_MovieQA_Understanding_Sto...

    2016

  65. [82]

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid

  66. [85]

    Yuhang Zhao, Edward Cutrell, Christian Holz, Meredith Ringel Morris, Eyal Ofek, and Andrew D. Wilson. 2019. SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision. InProceedings of the 2019 CHI Conference on Human Factors in Computing System...

    2019

  67. [2015]

    In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

    Privacy concerns and behaviors of people with visual impairments. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3523–3532

  68. [2019]

    ACM Computing Surveys (CSUR)52, 6 (2019), 1–37

    Video description: A survey of methods, datasets, and evaluation metrics. ACM Computing Surveys (CSUR)52, 6 (2019), 1–37

    2019

  69. [2020]

    In Proceedings of the 17th International Web for All Conference

    Accessible conversational user interfaces: considerations for design. In Proceedings of the 17th International Web for All Conference. ACM, Taipei Taiwan, 1–11. doi:10.1145/3371300.3383343

  70. [2021]

    1686–1697

    Just Ask: Learning To Answer Questions From Millions of Narrated Videos. 1686–1697. https://openaccess.thecvf.com/content/ICCV2021/html/Yang_Just_ Ask_Learning_To_Answer_Questions_From_Millions_of_Narrated_ICCV_ 2021_paper.html?ref=https://githubhelp.com

  71. [2022]

    In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Learning to Answer Questions in Dynamic Audio-Visual Scenarios. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 19086–19096. doi:10.1109/CVPR52688.2022.01852 ISSN: 2575-7075

    2022

  72. [2023]

    InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems

    Identifying Multimodal Context Awareness Requirements for Supporting User Interaction with Procedural Videos. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–17. doi:10. 1145/3544548.3581006

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.