{"id":"f6a00f24-19cd-4332-9204-acf5c5158885","arxiv_id":"2505.02859","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A fine-tuned Llama-2 chatbot that converts SHAP values into conversational explanations was rated as more interpretable than SHAP plots by survey users, with the largest gains for XAI novices.","lead":"Researchers built and tested a chatbot that translates SHAP explanation values into plain-language answers about why a machine-learning model made a prediction. In a survey, users, especially those with limited XAI experience, rated the fine-tuned chatbot's explanations as clearer and more interpretable than standard SHAP plots.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Survey evidence for 'enhanced interpretability' rests on subjective preference ratings, not on measured comprehension; the claim is not yet supported.","rationale":"Read in good faith, this is a design science artifact paper: the contribution is a reference architecture and prototype, and the authors provide a public GitHub repository, a detailed three-step fine-tuning pipeline, expert interviews, demonstrations, and a survey. Those elements support feasibility and design-value claims. The headline claim, however, is about human interpretability, and the only direct evidence is the Section 5.3 survey. For that claim to hold, the survey would need to show that users genuinely understand model behavior better with the fine-tuned chatbot, not merely that they prefer its text. The Likert items measure subjective experience; without a comprehension-grounded outcome, the central claim is underdetermined. The fixed presentation order compounds this: participants always saw the SHAP plot first, so practice or anchoring effects favor later conditions. Additionally, the fine-tuning results are described as a 'substantial improvement across all evaluated metrics,' but Table 1 shows negligible perplexity gains for the global-explanation step and a loss improvement on a self-generated evaluation document. These issues do not show the approach is wrong; they show the evidence does not yet establish the stated claim. The reader's CONDITIONAL verdict is appropriate, and the requested additions—statistical tests, randomized order, and a comprehension task—are exactly what would resolve the concern, so no change to the verdict is needed.","tokens_in":15601,"tokens_out":2890,"duration_ms":27734,"concrete_test":"Re-run the Section 5.3 evaluation with a counterbalanced within-subjects design (randomize or Latin-square the order of SHAP plot, non-fine-tuned, and fine-tuned conditions) and add objective comprehension items after each condition, e.g., 'Which feature contributed most to this prediction?' and 'If feature X increased, would predicted SoH increase or decrease?' Compare answer accuracy and response time across conditions and proficiency levels, with paired significance tests (Wilcoxon signed-rank). If the fine-tuned chatbot does not yield significantly higher comprehension accuracy than the SHAP plot, the headline interpretability claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the fine-tuned chatbot enhances human interpretability, especially for XAI novices—is supported primarily by the ex-post survey in Section 5.3. That survey asked participants to rate clarity, interpretability, and cognitive effort after seeing a SHAP waterfall plot, a non-fine-tuned chatbot explanation, and the fine-tuned chatbot explanation, always in that order. Self-reported ratings are not a measure of understanding: an explanation can sound fluent and be rated clearer while leaving the user unable to make correct inferences about the model. No objective comprehension task appears in the survey, and the fixed presentation order allows familiarity or learning effects to inflate later conditions regardless of content. The paper also reports no significance tests, so 'surpassed ... across all three evaluation metrics' is an informal claim. This matters because the 'substantial improvement across all evaluated metrics' statement in Section 1 is broader than Table 1 supports: global explanation fine-tuning improves perplexity by only 1.3%, and the 51.5% loss improvement from human-alignment fine-tuning is measured on a Q&A document generated by the same automatic pipeline used to create the training data, so it is not independent evidence of explanation quality. If interpretability is defined as actual human understanding, the prototype's advantage is currently unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reference architecture for an interactive chatbot that translates SHAP-based explanations of a machine learning model into natural language, powered by a fine-tuned LLM (Llama-2-13b-chat). The architecture is instantiated for a battery State-of-Health prediction use case. The authors follow a design science research process: they derive design objectives from literature and expert interviews, implement a prototype, and evaluate it through an ex-ante expert interview (n=16), an intermediate demonstration and competing-artifact analysis, and an ex-post online survey (n=61). The survey compares a SHAP waterfall plot, a non-fine-tuned chatbot, and the fine-tuned chatbot on self-reported clarity, interpretability, and cognitive effort. The authors report that the fine-tuned chatbot outperforms both baselines on all three metrics, especially for participants with low XAI proficiency. They also present a three-step fine-tuning ablation in Table 1, with perplexity and loss improvements, including a 51.5% loss reduction for the human-alignment step. The central claim is that the prototype enhances human interpretability of ML models, particularly for users less experienced with XAI.","tokens_in":15781,"tokens_out":4449,"duration_ms":42466,"significance":"If the central claim were convincingly established, the contribution would be valuable: a concrete, reproducible pipeline for making SHAP outputs accessible to non-experts through fine-tuned LLMs, with potential implications for XAI practice in domains like battery health monitoring. The paper's strengths include a clearly described reference architecture, a publicly available prototype (GitHub repository), and a use case that is both relevant and non-trivial. The design objectives are grounded in the literature and expert input, and the three-step fine-tuning procedure is a useful engineering contribution. However, the significance is conditional on the evaluation actually supporting the claim that the chatbot improves human understanding, and the current evidence is not sufficient: the ex-post survey measures subjective preference rather than comprehension, and the key fine-tuning result is evaluated on data generated by the same procedure used to create the training data. These issues are load-bearing for the central claim.","major_comments":[{"comment":"The abstract's and conclusion's central claim that the fine-tuned chatbot 'enhances human interpretability' rests on self-reported Likert ratings of clarity, interpretability, and cognitive effort. These are subjective preference measures, not measures of whether participants actually understand the model's behavior. The survey includes no comprehension-grounded task, so an explanation that sounds fluent but does not improve the user's ability to make correct inferences about the prediction would still appear to perform well. Furthermore, the presentation order is fixed: all participants evaluate the SHAP plot first, then the non-fine-tuned chatbot, then the fine-tuned chatbot. The sentence 'To minimize order effects, all participants first evaluated the SHAP plot, followed by the chatbot-generated explanations' is contradictory; a fixed order does not minimize order effects and confounds condition with practice and familiarity. No confidence intervals, significance tests, or effect sizes are reported for the comparisons in Figures 5 and 6. Because this survey is the only evidence for the central claim, the claim is not supported as it stands.","section":"Section 5.3, Figures 5 and 6"},{"comment":"The 51.5% loss improvement from human-alignment fine-tuning is measured on a 'Context Q&A Dataset' generated with the same automated procedure used to create the training data. This evaluation therefore measures how well the model reproduces the authors' own answer style in the generated format, not the correctness, quality, or usefulness of the explanations. It is a circular evaluation that cannot, by itself, demonstrate that fine-tuning improves explanation quality. In addition, the global explanation fine-tuning step yields only a 1.3% perplexity improvement (Table 1), so the Introduction's assertion of 'a substantial improvement across all evaluated metrics' is not supported by the quantitative results; at most one step shows a large movement, and that movement is on a non-independent metric.","section":"Section 4.3.3, Step 3 and Table 1"},{"comment":"The claim that low-proficiency users benefit 'especially' from the fine-tuned chatbot rests on the disparity analysis in Figure 6, but no statistical tests, confidence intervals, or effect sizes are provided for the differences between proficiency groups. The paper reports only the observed pattern of decreasing disparities with increasing proficiency. Without inference statistics, it is impossible to determine whether this pattern is robust or due to chance, and the absence of a control for the fixed presentation order further weakens the interpretation. This is a load-bearing claim because the abstract explicitly singles out 'users with less experience with XAI' as the main beneficiaries.","section":"Section 5.3, proficiency-level analysis"}],"minor_comments":[{"comment":"The paper uses 'LoRa' in the text but 'LoRA' in the fine-tuning footnotes; please standardize to 'LoRA'.","section":"Section 2.1"},{"comment":"The reference 'Vaswani, A. (2017). Attention is all you need' is incomplete; it should include the full author list and venue, e.g., Advances in Neural Information Processing Systems 30.","section":"References"},{"comment":"The citation of Doshi-Velez and Kim (2017) to justify the fixed presentation order is inappropriate, since that reference concerns interpretability evaluation, not survey order effects; a methods reference on experimental design would be more suitable.","section":"Section 5.3"},{"comment":"The column headers and row labels in Table 1 are ambiguous; for example, it is not clear which evaluation document is used for each row and why some perplexity values remain unchanged across fine-tuning steps. Please clarify the layout and the meaning of each value.","section":"Table 1"},{"comment":"The caption 'Survey results for overall interpretability and clarity' mentions only two of the three metrics; the figure appears to include cognitive effort as well, so the caption should list all three.","section":"Figure 5"},{"comment":"The text refers to 'Oobabooga text-generation-webui' without a version or a proper citation; please provide a version number or repository reference.","section":"Section 4.3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is an accepted ECIS 2025 conference paper, and the evaluation standards appropriate for a conference may differ from those of a journal. The two main weaknesses for a journal version are the circular fine-tuning evaluation and the lack of any comprehension-based measure in the survey. If the authors can provide a non-circular evaluation of the human-alignment fine-tuning (e.g., against an independently created gold standard) and add a comprehension task with proper statistical analysis to the ex-post evaluation, the central claim would be substantially better supported. There is also a scope question: the paper's contribution is framed as a reference architecture and design-science evaluation, which may be a better fit for an IS journal than for an ML-focused venue; the current version does not make the novelty claim precise enough for a claims-oriented audience."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Johannes, quick take on arXiv:2505.02859. The genuinely new thing here is the three-step fine-tuning pipeline — in-domain, global explanation, human alignment — that adapts Llama-2 to turn SHAP values into readable text, and the code is public. That is a concrete, reproducible contribution that goes beyond the template-and-rules chatbots of Gao, Slack, and Nguyen. The reference architecture is clean, and the competing artifact analysis is fair.\n\nWhat the paper does not do is rigorously show that this improves human understanding. The ex-post survey (n=61) asks participants to rate clarity, interpretability, and cognitive effort after seeing a SHAP waterfall plot, then a non-fine-tuned chatbot, then the fine-tuned one, always in that order. Self-reported ratings are not comprehension, and the fixed order invites learning effects. There are no confidence intervals, significance tests, or effect sizes. On top of that, the human-alignment step is evaluated on a context Q&A document generated by the same automated procedure used to build the training set, so the 51.5% loss reduction largely measures stylistic reproduction, not better explanations. And the abstract's 'substantial improvement across all evaluated metrics' overstates what Table 1 shows: the global explanation step moves perplexity by just 1.3%.\n\nNone of this kills the paper. The direction is plausible and the prototype is real. But the central claim should be read as 'users prefer the fine-tuned chatbot' rather than 'users understand the model better.' For a design-science paper at ECIS that is probably acceptable, but a serious referee would ask for a comprehension-grounded task, statistical testing, and release of the evaluation prompts.\n\nWho is this for? People building XAI conversational interfaces and IS researchers doing DSR. It's a decent example of the genre. I'd bring it to a reading group if the topic is on your list, and I'd cite it for the fine-tuning approach, not for the evaluation. Recommendation: it deserves peer review, but with heavy revision expectations.","headline":"Useful fine-tuning recipe and open-source prototype for a SHAP-explaining chatbot, but the human-interpretability claim rests on weak subjective survey evidence.","tokens_in":16388,"tokens_out":3704,"would_cite":true,"duration_ms":30809,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned large language model that translates SHAP values into conversational text can make machine-learning explanations usable by non-experts.","keywords":["eXplainable AI","large language models","fine-tuning","SHAP","chatbot","battery State of Health","model interpretability","design science research"],"falsifier":"A randomized experiment in which users answer objective questions about a battery prediction after seeing either the standard SHAP chart or the fine-tuned chatbot's explanation would settle the claim: if chatbot users rate the text as clearer but answer follow-up and prediction questions no better than chart readers, the claimed interpretability gain would not be supported. Counterbalancing the presentation order would also rule out learning effects.","tokens_in":15344,"feed_emoji":"🤖","tokens_out":10592,"duration_ms":92024,"temperature":0.7,"pith_summary":"The paper sets out to show that a chatbot built on a fine-tuned large language model can make machine-learning explanations accessible to people unfamiliar with explainable-AI tools. It proposes a reference architecture in which SHAP values, the numeric contributions of each feature to a prediction, are fed into a fine-tuned LLM that answers user questions in natural language. The design is instantiated for battery State-of-Health prediction and evaluated through an online survey with 61 participants. The reported results indicate that the fine-tuned chatbot was rated higher than both a SHAP waterfall plot and a non-fine-tuned chatbot on clarity, interpretability, and cognitive effort, with the largest advantage among users with the least XAI experience.","feed_headline":"Fine-tuned chatbot makes AI explanations clearer for novices","feed_subtitle":"In a 61-person survey, less-experienced XAI users preferred the chatbot's plain text over standard SHAP charts.","key_machinery":"The carrying mechanism is a three-stage fine-tuning pipeline applied to a general-purpose LLM with low-rank adaptation, so only a small set of parameters is updated. Stage one enriches the model's background knowledge of SHAP and battery health from unstructured texts; stage two exposes the model to a summary of global SHAP findings; stage three trains it in a supervised way on structured examples that mirror real user questions about the most important features and their SHAP values. After each stage, the model is evaluated by perplexity or loss. The final model is wired into a reference architecture that computes a prediction from a pre-trained CatBoost model, derives SHAP values, and inserts them into an information prompt so the chatbot can answer inferential questions while referring to the XAI visualization.","core_discovery":"On the paper's own terms, the central claim is that domain-specific fine-tuning of an LLM substantially improves its ability to turn SHAP outputs into understandable textual explanations, and that this translation is what makes ML interpretable to non-experts. The authors report a three-step fine-tuning process: unsupervised in-domain training on SHAP and battery material, unsupervised training on a written summary of global SHAP findings, and supervised human-alignment training on structured instruction-context-output pairs. Each step is measured by perplexity or loss, with the final alignment step producing the largest reported improvement. In the battery State-of-Health use case, a CatBoost model produces the prediction, SHAP computes feature contributions, and the fine-tuned chatbot converts those values into conversational answers. The survey evidence is interpreted as showing that the prototype enhances human interpretability of ML, especially for users with less experience with XAI.","pith_inferences":["If the reported preference is driven by the fluent tone of generated text rather than by comprehension, a task-based study asking users to predict the model's behavior on new inputs would separate those explanations; the current survey only measured self-report.","The three-step fine-tuning recipe could plausibly transfer to other XAI outputs such as LIME or counterfactual explanations, but that transfer is not tested in the paper.","A prompt-only condition would clarify how much of the gain comes from fine-tuning versus from the information prompt that already contains the SHAP values; the paper's ablation compares fine-tuned against non-fine-tuned but does not isolate prompt engineering."],"forward_implications":["A user without statistical training can query a model's prediction in natural language and receive a domain-aware explanation instead of having to read a SHAP plot.","Fine-tuning an LLM on in-domain text plus task-format examples can be monitored step by step with perplexity and loss, and the supervised human-alignment stage gives the largest reported drop in loss.","The reference architecture is designed to be domain-agnostic, so other use cases could substitute a different ML model, XAI technique, or LLM while keeping the same interaction pattern.","Unlike template-based conversational XAI systems, a fine-tuned LLM can answer inferential and follow-up questions rather than being restricted to a fixed question bank."],"supporting_citations":[{"why":"Defines SHAP values and the SHAP library that provide the XAI input the chatbot translates.","marker":"Lundberg & Lee, 2017"},{"why":"Supplies the Llama-2-13b-chat base model that the prototype fine-tunes.","marker":"Touvron et al., 2023"},{"why":"Provides the low-rank adaptation method used for parameter-efficient fine-tuning.","marker":"Hu et al., 2022"},{"why":"Provides the CatBoost model whose battery State-of-Health predictions are explained.","marker":"Prokhorenkova et al., 2018"},{"why":"A template-based chatbot-XAI system that serves as a related baseline in the comparison.","marker":"Gao et al., 2022"},{"why":"TalkToModel, an interactive dialogue system that relies on templates and is contrasted with the proposed artifact.","marker":"Slack et al., 2023"},{"why":"A question-phrase-bank XAI conversation system that the paper contrasts with its fine-tuned approach.","marker":"Nguyen et al., 2023"},{"why":"Establishes that interpreting SHAP outputs requires domain expertise, motivating the chatbot's translation goal.","marker":"Kaur et al., 2020"}],"fun_headline_variants":["Fine-tuned LLM chatbot translates SHAP into plain text for novices","Chatbot uses fine-tuned LLM to make AI explanations beginner-friendly","LLM fine-tuning improves XAI clarity for less-experienced users","61-person survey: fine-tuned LLM chatbot aids XAI novices"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that participants' self-reported impressions that an explanation is clearer and easier to follow reflect genuine understanding, since the survey contained no comprehension task and always showed the standard chart before the chatbot text.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLM chatbot translates SHAP into plain text for novices","Chatbot uses fine-tuned LLM to make AI explanations beginner-friendly","LLM fine-tuning improves XAI clarity for less-experienced users","61-person survey: fine-tuned LLM chatbot aids XAI novices"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2910,"prompt_tokens":845,"completion_tokens":2065,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":1997}},"tokens_in":461,"tokens_out":2065,"duration_ms":16189,"temperature":1.0,"reasoning_tokens":1997,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:15:15.286969+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A randomized experiment in which users answer objective questions about a battery prediction after seeing either the standard SHAP chart or the fine-tuned chatbot's explanation would settle the claim: if chatbot users rate the text as clearer but answer follow-up and prediction questions no better than chart readers, the claimed interpretability gain would not be supported. Counterbalancing the presentation order would also rule out learning effects.","supporting_citations":[],"review_version":1}