REVIEW 4 major objections 5 minor 42 references
EmoAssist: Emotional Assistant for Visual Impairment Community
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that fine-tuning a 7B vision-language model with DPO on emotionally rewritten answers yields an assistive assistant that outscores GPT-4o on Empathy and Suggestion in a new benchmark that makes emotional intelligence a…
desk verdict First VI-assistive emotional-intelligence benchmark, but the headline gains over GPT-4o are likely rubric imitation until the evaluation is validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is Direct Preference Optimization (DPO), a training objective that increases the probability of a chosen response and decreases that of a rejected response while regularizing against a frozen copy of the model. Here the chosen responses are the EmoAssist Dataset's human-reviewed, emotionally rewritten answers, and the rejected responses are the untuned LLaVA model's own outputs; LoRA (low-rank adaptation) keeps the fine-tuning parameter-efficient. The evaluation machinery is the EmoAssist Benchmark's human-scoring rubric, five 0-2 scales authored for this paper, with Empathy and Suggestion as the dimensions meant to measure emotional intelligence.
What would settle it
Re-score the 200 held-out responses with at least three independent, identity-blinded annotators who include VI users; if the EmoAssist Model's Empathy and Suggestion advantages over GPT-4o shrink to within measurement error, or if annotator agreement on the 0-2 scales is low, the claimed gains would not reproduce.
Extended reading notes
Core claim
The central claim is that an assistive VQA model for the visual impairment community can and should be trained and measured with emotional intelligence as a first-class requirement. The EmoAssist Benchmark defines five 0-2 human-scored dimensions—Relevance, Empathy, Suggestion, Coherence, and Fluency—and the paper reports that the EmoAssist Model, a LLaVA-1.6-7B fine-tuned with LoRA and DPO, achieves 1.14 on Empathy and 1.10 on Suggestion on a 200-sample held-out set. These numbers represent improvements of 147.8% and 89.7% over the untuned base model and outperform GPT-4o by +0.68 and +0.19 on the two emotion-carrying dimensions. The paper interprets this as evidence that contrastive preference tuning against emotionally rich reference answers, rather than larger scale or stronger visual reasoning, is what closes the empathy gap in VI assistance.
Load-bearing premise
The load-bearing premise is that the paper's human scoring on the EmoAssist Benchmark is a valid, reliable, and unbiased measure of emotional intelligence for VI assistance, even though it reports no inter-annotator agreement, no blinded scoring, no annotator demographics, and no validation against actual VI users' preferences.
Editorial extensions
If this is right
- If the reported scores hold, a 7B open-weights model can deliver emotional support on par with or beyond a frontier API model in VI VQA, so emotional intelligence does not require the largest available model.
- The EmoAssist Benchmark gives subsequent assistive-LMM work a shared 200-sample evaluation set with explicit Empathy and Suggestion dimensions, making emotion a standard axis of comparison.
- The DPO recipe—chosen responses from an emotionally rewritten reference set, rejected responses from the untuned model—offers a practical template for injecting empathy into other small multimodal assistants.
- Because 30% of the benchmark is unanswerable, the model must combine emotional support with correct abstention, so the benchmark ties empathy to honesty about image content.
Reading between the lines
- The reported empathy gain may partly reflect stylistic mimicry: the references are GPT-4o outputs rewritten to be warm and actionable, so a DPO-tuned model could be learning recognizable emotional phrasing rather than deeper understanding of a user's state. That distinction is ours, not the paper's.
- Because the 200 test samples come from the same dataset and rewriting pipeline as the 800 training samples, an independent collection of VI users' real queries would be a stronger test of whether the advantage generalizes.
- The same benchmark-and-DPO recipe could transfer to other assistive settings such as navigation, shopping, or telehealth where emotional support matters, but the paper does not test that.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the EmoAssist Benchmark, a 1,000-sample dataset derived from VizWiz-VQA with GPT-4o-generated, human-reviewed ground-truth answers, and a five-dimension human scoring rubric (Relevance, Empathy, Suggestion, Coherence, Fluency). It also proposes EmoAssist-7B, a LLaVA-1.6-7B model fine-tuned with LoRA and DPO on 800 samples, with 200 samples reserved for evaluation. The central quantitative claim is that the model attains Empathy 1.14 and Suggestion 1.10, improving over the base model by 147.8% and 89.7% and over GPT-4o by +0.68 and +0.19, leading to the conclusion that a small fine-tuned model can surpass GPT-4o in emotional intelligence for VI assistance.
Significance. If the human scoring rubric were shown to be a valid and reliable measure of emotional intelligence for VI assistance, the work would fill a real gap: prior VI assistive benchmarks omit emotional dimensions, and EmoAssist integrates them with automatic metrics in a unified evaluation. The technical contribution of a LoRA/DPO-tuned 7B model that is more empathetic than its base model is also plausible and potentially useful. However, the current evidence does not establish the rubric's validity, and the benchmark construction makes the headline superiority over GPT-4o an expected consequence of style imitation rather than a demonstrated capability. The method sections are clearly written and Table II is informative, but the missing reliability, validity, and statistical support are load-bearing issues.
major comments (4)
- [Section III-B, Table I; Section V-A] The entire Empathy/Suggestion superiority claim rests on a human rubric authored by the authors, with no reported inter-annotator agreement, no blinding to model identity, no annotator count or demographics, and no validation against the preferences of VI users. Because the rubric rewards warmth, compassion, and actionable suggestions, scores from unblinded annotators using this rubric are not established as a measure of emotional intelligence. Please report the annotation procedure, agreement statistics such as Krippendorff's alpha, and a validation study with VI users before treating the human scores in Table II as evidence.
- [Section III-C and IV-B; Section V-A] The DPO training uses the same GPT-4o-generated, human-reviewed ground-truth responses as chosen responses and untuned LLaVA outputs as rejected responses, and the evaluation set is a random 200-sample subset of the same 1,000-sample pool. The EmoAssist model is therefore trained to reproduce the exact response style that the human rubric rewards, while GPT-4o is evaluated zero-shot. This makes the reported +0.68 Empathy and +0.19 Suggestion margins over GPT-4o largely a measure of style matching to the training distribution. Please evaluate on an independently constructed set, and compare against a GPT-4o-based DPO baseline or a control model trained on non-emotional instruction data.
- [Table II] The results in Table II show that EmoAssist-7B is worse than GPT-4o on LA VE (2.45 vs 2.70), Abs-Acc (0.85 vs 0.91), Relevance (1.56 vs 1.79), and Coherence (1.62 vs 1.75), and worse than many baselines on these metrics. The abstract's statement that the model 'even outperforms state-of-the-art LLMs such as GPT-4o' is therefore selective; the model only exceeds GPT-4o on the two human-scored dimensions it was explicitly trained to optimize. Please report all metrics in the abstract and conclusion and discuss the trade-off between emotional style and answer relevance, coherence, and accuracy.
- [Section V-A and V-D-2] No confidence intervals, standard errors, significance tests, or per-annotator score distributions are provided for the 200-sample human evaluation. Given that the absolute human scores are on a 0-2 scale, the reported differences such as +0.19 in Suggestion may be within scoring noise. Please provide error bars, the number of annotators, and statistical tests, and clarify how the aggregate scores in Table II were computed from individual annotations.
minor comments (5)
- [Table II heading] The table header reads 'LMMs Proformance' and should be 'LMMs Performance'; the metric name 'LA VE' is also written inconsistently as 'LAVE' in the text.
- [Section I, paragraph 2] The phrase 'with the the development' contains a duplicated article and should be corrected.
- [Section III-C] The description of 'manual review' does not state how many reviewers participated or how disagreements were resolved; the filtering criteria are described but the resulting sample counts are not reported, so readers cannot assess dataset composition.
- [General] The manuscript does not state whether the EmoAssist Dataset, annotation code, or model weights will be released; please include a reproducibility statement with a data/code availability link.
- [Section VI] The conclusion repeats the strong claim that results 'demonstrate' high emotional intelligence without acknowledging the limitations of the evaluation protocol or the trade-offs visible in Table II.
Circularity Check
The claimed Empathy/Suggestion gains are partly circular: the DPO training signal and the human scoring rubric come from the same author-constructed, GPT-4o-seeded emotional ground truths, so the benchmark margin over zero-shot GPT-4o largely measures style imitation rather than independently validated emotional intelligence.
-
fitted input called prediction
[Section III-B/Table I; Section III-C; Section IV-B; Section V-A; Section V-D-2]
"we prompted GPT-4o to generate complete sentences from the extracted words or phrases, ensuring that the responses expressed emotions towards VI individuals. This was followed by a manual review ... we use the output of the untuned LMM as the rejected response ... In contrast, the Ground Truth responses from the EmoAssist Dataset are employed as the chosen responses."
The DPO chosen responses are the paper's own GPT-4o-generated, manually edited ground truths, explicitly constructed to express emotion. The human Empathy and Suggestion rubric in Table I rewards precisely that emotional style. Training happens on 800 samples from the EmoAssist Dataset and evaluation on 200 held-out samples from the same constructed dataset, so the reported 1.14/1.10 Empathy/Suggestion scores are the expected outcome of fitting the model to the very construct the benchmark measures. GPT-4o, evaluated zero-shot, was never exposed to this rubric or to the training distribution, so the +0.68 Empathy and +0.19 Suggestion margins are a style-imitation effect of the authors' data-generation choices rather than independent evidence of higher emotional intelligence.
full rationale
The paper's central empirical claim is that DPO fine-tuning of LLaVA-7B on 800 EmoAssist samples improves Empathy/Suggestion scores on 200 held-out samples. That train/eval split is real and disjoint, and the automatic metrics (BERTScore, ROUGE) are external overlap measures, so not everything in the paper is circular. However, the load-bearing emotional-intelligence claim is circular in a partial but important way: the training labels, the evaluation rubric, and the ground-truth construction procedure are all authored by the same paper and share a single operationalization of emotional intelligence. The chosen responses were generated by GPT-4o and manually edited to express emotion, and the DPO objective explicitly makes those responses the preferred outputs. The human rubric then rewards that same style. Consequently, the large gains over zero-shot GPT-4o are a self-fulfilling consequence of training and evaluating on the same construct, not a validated measure of empathy for the VI community. The paper does not report annotator blinding, inter-annotator agreement, or validation against actual VI-user preferences; those are validity threats rather than circularity itself, but they compound the self-referential design. There are no load-bearing self-citations or imported uniqueness theorems, so the circularity is confined to the benchmark-and-fine-tuning loop. Score 5 reflects partial circularity: the central comparison is confounded by construction, but the held-out split and external automatic metrics retain some independent content.
Assumptions & free parameters
free parameters (4)
- DPO beta =
0.1
- Learning rate =
5e-6
- Unanswerable question cap =
30%
- Held-out split =
140 answerable / 60 unanswerable (200 total)
assumptions (4)
- domain assumption The five human-scoring dimensions and 0-2 scales are a valid measure of emotional intelligence for VI assistance.
- domain assumption GPT-4o-generated answers, after human review, are adequate ground truths for empathetic and actionable responses.
- domain assumption The 200-sample held-out set is representative of VI VQA queries.
- domain assumption Fine-tuning on 800 samples does not degrade general VQA capability while improving emotional metrics.
Cite this review
Pith. "Pith review of EmoAssist: Emotional Assistant for Visual Impairment Community." pith.science (2026). https://pith.science/paper/UKAKW67Q
@misc{pith2026250209285,
author = {Pith},
title = {Pith review of: EmoAssist: Emotional Assistant for Visual Impairment Community},
year = {2026},
howpublished = {\url{https://pith.science/paper/UKAKW67Q}},
note = {Machine review of arXiv:2502.09285}
}
read the original abstract
The rapid advancement of large multi-modality models (LMMs) has significantly propelled the integration of artificial intelligence into practical applications. Visual Question Answering (VQA) systems, which can process multi-modal data including vision, text, and audio, hold great potential for assisting the Visual Impairment (VI) community in navigating complex and dynamic real-world environments. However, existing VI assistive LMMs overlook the emotional needs of VI individuals, and current benchmarks lack emotional evaluation of these LMMs. To address these gaps, this paper introduces the EmoAssist Benchmark, a comprehensive benchmark designed to evaluate the assistive performance of LMMs for the VI community. To the best of our knowledge, this is the first benchmark that incorporates emotional intelligence as a key consideration. Furthermore, we propose the EmoAssist Model, an Emotion-Assistive LMM specifically designed for the VI community. The EmoAssist Model utilizes Direct Preference Optimization (DPO) to align outputs with human emotional preferences. Experiment results demonstrate that the EmoAssist Model significantly enhances the recognition of implicit emotions and intentions of VI users, delivers empathetic responses, and provides actionable guidance. Specifically, it shows respective improvements of 147.8% and 89.7% in the Empathy and Suggestion metrics on the EmoAssist Benchmark, compared to the pre-tuning LMM, and even outperforms state-of-the-art LLMs such as GPT-4o.
Figures
Reference graph
Works this paper leans on
-
[1]
Blindness and vision impairment
WHO. Blindness and vision impairment . 2024. URL: https://www.who.int/news- room/fact- sheets/detail/ blindness-and-visual-impairment
work page 2024
-
[2]
Timothy R Fricke et al. “Global prevalence of visual im- pairment associated with myopic macular degeneration and temporal trends from 2000 through 2050: system- atic review, meta-analysis and modelling”. In: British Journal of Ophthalmology 102.7 (2018), pp. 855–862
work page 2018
-
[3]
Bineeth Kuriakose, Raju Shrestha, and Frode Eika Sandnes. “DeepNA VI: A deep learning based smart- phone navigation assistant for people with visual im- pairments”. In: Expert Systems with Applications 212 (2023), p. 118720. ISSN : 0957-4174. DOI: https : / / doi . org / 10 . 1016 / j . eswa . 2022 . 118720. URL: https : / / www. sciencedirect . com / sci...
work page 2023
-
[4]
V-eye: A vision-based navigation system for the visually impaired
Ping-Jung Duh et al. “V-eye: A vision-based navigation system for the visually impaired”. In: IEEE Transac- tions on Multimedia 23 (2020), pp. 1567–1580
work page 2020
-
[5]
Deep learning based wearable assistive system for visually impaired people
Yimin Lin et al. “Deep learning based wearable assistive system for visually impaired people”. In: Proceedings of the IEEE/CVF international conference on computer vision workshops. 2019, pp. 0–0
work page 2019
-
[6]
A multi-world approach to question answering about real-world scenes based on uncertain input
Mateusz Malinowski and Mario Fritz. “A multi-world approach to question answering about real-world scenes based on uncertain input”. In: Advances in neural information processing systems 27 (2014)
work page 2014
-
[7]
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Yash Goyal et al. “Making the v in vqa matter: Elevating the role of image understanding in visual question answering”. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2017, pp. 6904–6913
work page 2017
-
[8]
Be My Eyes - See the world together — bemyeyes.com . https://www.bemyeyes.com/. [Accessed 14-01-2025]
work page 2025
Show all 42 references
-
[9]
VQAsk: a multimodal Android GPT-based application to help blind users visualize pictures
Maria De Marsico et al. “VQAsk: a multimodal Android GPT-based application to help blind users visualize pictures”. In: Proceedings of the 2024 International Conference on Advanced Visual Interfaces. 2024, pp. 1– 5
2024
-
[10]
Open scene understanding: Grounded situation recognition meets segment anything for helping people with visual impairments
Ruiping Liu et al. “Open scene understanding: Grounded situation recognition meets segment anything for helping people with visual impairments”. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. 2023, pp. 1857–1867
2023
-
[11]
Vialm: A survey and benchmark of visually impaired assistance with large models
Yi Zhao et al. “Vialm: A survey and benchmark of visually impaired assistance with large models”. In: arXiv preprint arXiv:2402.01735 (2024)
2024 arXiv
-
[12]
VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impair- ments
Bufang Yang et al. “VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impair- ments”. In: arXiv preprint arXiv:2404.02508 (2024)
2024 arXiv
-
[13]
Long-Form Answers to Visual Ques- tions from Blind and Low Vision People
Mina Huh et al. “Long-Form Answers to Visual Ques- tions from Blind and Low Vision People”. In: arXiv preprint arXiv:2408.06303 (2024)
2024 arXiv
-
[14]
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin. “ROUGE: A Package for Automatic Evaluation of Summaries”. In: Text Summarization Branches Out. Barcelona, Spain: Association for Com- putational Linguistics, July 2004, pp. 74–81. URL: https: //aclanthology.org/W04-1013/
2004
-
[15]
METEOR: An Au- tomatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments
Alon Lavie and Abhaya Agarwal. “METEOR: An Au- tomatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments”. In: Proceedings of the Second Workshop on Statistical Machine Trans- lation. Ed. by Chris Callison-Burch et al. Prague, Czech Republic: Associ...
2007
-
[16]
Bertscore: Evaluating text gener- ation with bert
Tianyi Zhang et al. “Bertscore: Evaluating text gener- ation with bert”. In: arXiv preprint arXiv:1904.09675 (2019)
2019 arXiv
-
[17]
Improving automatic vqa evaluation using large lan- guage models
Oscar Ma ˜nas, Benno Krojer, and Aishwarya Agrawal. “Improving automatic vqa evaluation using large lan- guage models”. In: Proceedings of the AAAI Conference on Artificial Intelligence . V ol. 38. 5. 2024, pp. 4171– 4179
2024
-
[18]
Gpt-4 technical report
Josh Achiam et al. “Gpt-4 technical report”. In: arXiv preprint arXiv:2303.08774 (2023)
2023 arXiv
-
[19]
Emotional and social conse- quences of age-related low vision
David Burmedi et al. “Emotional and social conse- quences of age-related low vision”. In: Visual Impair- ment Research 4.1 (2002), pp. 47–71. DOI: 10.1076/ vimr.4.1.47.15634. eprint: https://doi.org/10.1076/vimr. 4.1.47.15634. URL: https://doi.org/10.1076/vimr.4.1.47. 15634
2002 doi
-
[20]
Loneliness, adaptation to vision impairment, social support and depression among visu- ally impaired elderly
P.F.J. Verstraten et al. “Loneliness, adaptation to vision impairment, social support and depression among visu- ally impaired elderly”. In: International Congress Series 1282 (2005). Vision 2005, pp. 317–321. ISSN : 0531-
2005
-
[21]
Vqa: Visual question answer- ing
Stanislaw Antol et al. “Vqa: Visual question answer- ing”. In: Proceedings of the IEEE international confer- ence on computer vision . 2015, pp. 2425–2433
2015
-
[22]
Gpt-4o system card
Aaron Hurst et al. “Gpt-4o system card”. In: arXiv preprint arXiv:2410.21276 (2024)
2024 arXiv
-
[23]
LoRA: Low-Rank Adaptation of Large Language Models
J. Edward Hu et al. “LoRA: Low-Rank Adaptation of Large Language Models”. In: ArXiv abs/2106.09685 (2021). URL: https://api.semanticscholar.org/CorpusID: 235458009
2021 arXiv
-
[24]
Direct preference optimization: Your language model is secretly a reward model
Rafael Rafailov et al. “Direct preference optimization: Your language model is secretly a reward model”. In: Advances in Neural Information Processing Systems 36 (2024)
2024
-
[25]
Visual instruction tuning
Haotian Liu et al. “Visual instruction tuning”. In: Ad- vances in neural information processing systems 36 (2024)
2024
-
[26]
Are you talking to a machine? dataset and methods for multilingual image question
Haoyuan Gao et al. “Are you talking to a machine? dataset and methods for multilingual image question”. In: Advances in neural information processing systems 28 (2015)
2015
-
[27]
Clevr: A diagnostic dataset for compositional language and elementary visual reason- ing
Justin Johnson et al. “Clevr: A diagnostic dataset for compositional language and elementary visual reason- ing”. In: Proceedings of the IEEE conference on com- puter vision and pattern recognition . 2017, pp. 2901– 2910
2017
-
[28]
Vizwiz: nearly real-time an- swers to visual questions
Jeffrey P Bigham et al. “Vizwiz: nearly real-time an- swers to visual questions”. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology. 2010, pp. 333–342
2010
-
[29]
Emotional intelligence of large lan- guage models
Xuena Wang et al. “Emotional intelligence of large lan- guage models”. In: Journal of Pacific Rim Psychology 17 (2023), p. 18344909231213958
2023
-
[30]
EmoBench: Evaluating the Emo- tional Intelligence of Large Language Models
Sahand Sabour et al. “EmoBench: Evaluating the Emo- tional Intelligence of Large Language Models”. In: arXiv preprint arXiv:2402.12071 (2024)
2024 arXiv
-
[31]
Emotionqueen: A benchmark for evaluating empathy of large language models
Yuyan Chen et al. “Emotionqueen: A benchmark for evaluating empathy of large language models”. In:arXiv preprint arXiv:2409.13359 (2024)
2024 arXiv
-
[32]
Segment anything
Alexander Kirillov et al. “Segment anything”. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. 2023, pp. 4015–4026
2023
-
[33]
http://mousi.org/
mousi — mousi.org . http://mousi.org/. [Accessed 14- 01-2025]
2025
-
[34]
Emphi: Generating empathetic responses with human-like in- tents
Mao Yan Chen, Siheng Li, and Yujiu Yang. “Emphi: Generating empathetic responses with human-like in- tents”. In: arXiv preprint arXiv:2204.12191 (2022)
2022 arXiv
-
[35]
Improving multi-turn emotional sup- port dialogue generation with lookahead strategy plan- ning
Yi Cheng et al. “Improving multi-turn emotional sup- port dialogue generation with lookahead strategy plan- ning”. In: arXiv preprint arXiv:2210.04242 (2022)
2022 arXiv
-
[36]
Exploring the role of an emo- tional support and counselling service for people with visual impairments
Suzanne Hodge et al. “Exploring the role of an emo- tional support and counselling service for people with visual impairments”. In: British Journal of Visual Im- pairment 31.1 (2013), pp. 5–19
2013
-
[37]
CoMAE: A multi-factor hierarchi- cal framework for empathetic response generation
Chujie Zheng et al. “CoMAE: A multi-factor hierarchi- cal framework for empathetic response generation”. In: arXiv preprint arXiv:2105.08316 (2021)
2021 arXiv
-
[38]
ESC-Eval: Evaluating Emotion Support Conversations in Large Language Models
Haiquan Zhao et al. “ESC-Eval: Evaluating Emotion Support Conversations in Large Language Models”. In: arXiv preprint arXiv:2406.14952 (2024)
2024 arXiv
-
[39]
Towards empathetic open-domain conversation models: A new benchmark and dataset
Hannah Rashkin. “Towards empathetic open-domain conversation models: A new benchmark and dataset”. In: arXiv preprint arXiv:1811.00207 (2018)
2018 arXiv
-
[40]
Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling
Zhe Chen et al. “Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling”. In: arXiv preprint arXiv:2412.05271 (2024)
2024 arXiv
-
[41]
Qwen2-vl: Enhancing vision- language model’s perception of the world at any res- olution
Peng Wang et al. “Qwen2-vl: Enhancing vision- language model’s perception of the world at any res- olution”. In: arXiv preprint arXiv:2409.12191 (2024)
2024 arXiv
-
[5131]
URL: https://www.sciencedirect.com/science/article/pii/ S0531513105007375
DOI: https://doi.org/10.1016/j.ics.2005.04.017. URL: https://www.sciencedirect.com/science/article/pii/ S0531513105007375
2005 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.