{"id":"36880d28-8945-4423-8a36-733514af5efe","arxiv_id":"2412.12318","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"G-Tex injects attention-based highlight tokens into a language model through a graph neural network layer, improving faithfulness of generated explanations.","lead":"This paper introduces G-Tex, a framework that finds the most important words in a model's input and uses them, via a graph structure, to guide the model when writing a natural language explanation of its answer. The method reports faithfulness improvements of up to 12.18% on three reasoning tasks, making AI explanations more consistent with the model's internal reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The counterfactual faithfulness metric is one-sided, and G-Tex's highlight-injection mechanism may inflate scores by encouraging input copying rather than genuine faithfulness.","rationale":"The reader's weakest assumption (attention reliability) is a real limitation, but it is partly mitigated by the paper's citation of Sun et al. (2024) for attention-based highlights and by the fact that the method could still work even if attention is imperfect. A more direct threat to the central claim is the one-sidedness of the counterfactual faithfulness metric, which is the sole quantitative support for the headline 12.18% improvement. G-Tex's design—injecting highlighted input tokens as a graph into the encoder—naturally increases the model's exposure to input tokens during generation, making it plausible that the metric gains come from copying rather than reasoning. This can be tested with a false-positive analysis; if the concern lands, the faithfulness claim is not established. I also note an internal inconsistency that strengthens the need for caution: Table 1 and Table 5 report different values for the same baselines (e.g., ComVE BART Fine-tuningbase Total Unfaith is 70.50 in Table 1 but 72.82 in Table 5), which further undermines the precision of the 12.18% claim and should be reconciled. Given these concerns, the appropriate verdict remains CONDITIONAL: the paper should not be accepted until the authors provide the false-positive analysis and reconcile the tables.","tokens_in":26032,"tokens_out":9814,"duration_ms":85816,"concrete_test":"Compute the false-positive rate on the existing perturbation data: for each model (Fine-tuningbase, Prompt, and each G-Tex variant), on perturbed instances where the predicted label does NOT change, measure the percentage of NLEs that contain the inserted adjective. If G-Tex's false-positive rate is significantly higher than baselines (e.g., >5 percentage points), the counterfactual improvement is at least partly an artifact of copying. Also re-run the faithfulness comparison after adding a symmetric penalty that counts both directions (missing the inserted word on label change and mentioning it on label stability); if the improvement shrinks or reverses, the central claim is not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Abstract; §5.1) is that G-Tex improves NLE faithfulness by up to 12.18 percentage points on the counterfactual test of Atanasova et al. (2023). That test, as implemented in §H, is one-sided: when a random adjective inserted before a noun changes the predicted label, an NLE is counted as unfaithful only if the inserted word is absent; when the label does not change, the NLE is never checked for whether it incorrectly includes the inserted word (the paper explicitly states 'the unchanged label provides no relevant information about the faithfulness of the NLE'). G-Tex, by inserting a graph built from highlighted input tokens into the encoder (§3.3–§3.5) and fine-tuning the model to generate NLEs conditioned on those highlights, may increase the model's general tendency to copy input spans into the NLE. A model that indiscriminately copies input tokens would trivially pass the counterfactual test on label-changing perturbations while also mentioning inserted adjectives when the label is unchanged (unpenalized). Thus the reported 12.18% reduction in Total Unfaith may reflect increased lexical copying rather than better alignment with the model's reasoning. This is a validity threat to the central claim, distinct from but related to the reader's concern about attention reliability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes G-Tex, a framework for improving the faithfulness of natural language explanations (NLEs) generated by encoder-decoder models. G-Tex first extracts post-hoc highlight explanations (highlight tokens, token interactions, span interactions) from a fine-tuned base model using attention-based attribution, then converts these highlights into a graph that is injected into the model via a GNN layer placed after the 3/4-th encoder layer. The model is fine-tuned to jointly produce the task label and the NLE, with the graph providing explicit cues about important input fragments. Experiments on e-SNLI, ComVE, and ECQA with T5-large and BART-large compare G-Tex against a fine-tuning baseline and a prompt-based highlight baseline. The paper reports reductions in counterfactual unfaithfulness (up to 12.18 percentage points in Total Unfaith), higher lexical/semantic similarity to human explanations, and favorable human evaluation scores on non-redundancy and overall quality.","tokens_in":26342,"tokens_out":4307,"duration_ms":37747,"significance":"If the reported effects are genuine, G-Tex would be a useful contribution: it provides an explicit architectural mechanism for steering NLE generation toward input features that the model demonstrably relies on, going beyond prompt-based or loss-based extrinsic signals. The paper is technically solid in scope: it includes three datasets, two base models, three GNN variants, a prompt baseline, five random seeds with p-values, and human evaluations. The authors also acknowledge limitations regarding decoder-only models, GNN internal mechanisms, and attribution-method choices. The main concern is not the novelty but whether the central faithfulness claim is supported by the reported numbers and whether the evaluation metric captures true alignment with model reasoning rather than a copying artifact.","major_comments":[{"comment":"The statement that \"our method outperforms all baselines in counterfactual unfaithfulness and total faithfulness\" is contradicted by the paper's own tables. In Table 5, BART-based Tex-GCN with span interactions on ComVE gives Total Unfaith 96.44% versus 72.82% for Fine-tuningbase, and T5-based Tex-GCN with token interactions gives 77.03% versus 73.73% for Fine-tuningbase. Table 1 similarly shows T5-based Tex-SAGE with token interactions on ComVE at 76.94% Total Unfaith versus 68.96% for Fine-tuningbase. These are not small or negligible differences, and they contradict the blanket claim in the abstract and in §5.1. The paper should either temper the claim to the specific configurations that actually improve (e.g., highlight tokens on ComVE, span interactions on e-SNLI) or report a summary statistic over all configurations that is consistent with the data.","section":"§5.1 and Table 5"},{"comment":"The counterfactual faithfulness test is one-sided: when an inserted adjective does not change the predicted label, the generated NLE is never checked for whether it incorrectly mentions the inserted word, as the paper itself states in Appendix H (\"the unchanged label provides no relevant information about the faithfulness of the NLE\"). Because G-Tex injects the top-30% highlighted tokens into the encoder during fine-tuning (§3.3, §F), the model may learn to copy input spans into the NLE. Such copying would trivially pass the label-changing half of the test (the inserted word appears because it is copied) while remaining unpenalized on the label-unchanged half. The reported 12.18% reduction in Total Unfaith could therefore reflect increased lexical copying rather than better alignment with the model's reasoning. The authors should report the rate at which G-Tex and baselines include inserted adjectives when the label does not change, or use a two-sided faithfulness test, to rule out this artifact.","section":"§H and §3.5"},{"comment":"The importance scores that define all three highlight-explanation types are derived exclusively from attention weights: a single head is selected and self-attention scores are averaged to produce token and interaction importances (§3.2). This choice is known to be contested (Jain and Wallace 2019; Serrano and Smith 2019), and the paper's own Limitations section acknowledges that other attribution methods (Shapley, Integrated Gradients, saliency maps) are not explored. Since the graph construction and the GNN injection both depend on these scores, the claim that G-Tex aligns NLEs with the model's \"underlying reasoning\" would be considerably stronger if at least one alternative attribution method were used to show that the faithfulness gains are not an artifact of the specific attention-based signal. As it stands, the central mechanism rests on a single, heavily debated attribution choice.","section":"§3.2 and §3.3"}],"minor_comments":[{"comment":"The sentence \"While G-Tex with T5 slightly underperforms the prompt baseline on ComVE with token interactive explanations\" is inaccurate: the T5 Tex-SAGE token-interaction configuration in Table 1 (Total Unfaith 76.94) also underperforms Fine-tuningbase (68.96), and the gap is not slight.","section":"§5.1"},{"comment":"The text contains a typo: \"against comment sense\" should be \"against common sense\" in the paragraph discussing ComVE dataset characteristics.","section":"Appendix I"},{"comment":"The p-values are computed only for Tex-SAGE versus Fine-tuningbase. Since the paper's claim is \"outperforms all baselines,\" significance tests against the Prompt baseline are also needed, especially for configurations where the numerical gap is small.","section":"Appendix J"},{"comment":"The pairwise annotator agreement is low in several dimensions (e.g., mean agreement of 0.18 for Overall Quality on ECQA). The paper should state what level of agreement is considered acceptable and how disagreements were resolved.","section":"Appendix M.3"},{"comment":"Checkpoint selection is performed using BLEU on the validation set. Since the paper's main outcome is faithfulness, the authors should clarify whether selecting checkpoints by BLEU could bias faithfulness results, or report faithfulness on the validation set as an alternative selection criterion.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a compelling central idea and an extensive experimental setup, but the central claim as written is not supported by the full set of reported results, and the one-sided counterfactual metric raises a nontrivial validity concern given the input-copying mechanism. The authors should be asked to revise the claims and to provide a two-sided analysis or a copying control before the paper is publishable. I do not suspect any misconduct; the issue is a mismatch between the reported numbers and the strength of the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, G-Tex is a concrete, testable recipe: extract attention-based highlights, build a graph, inject through a GNN layer into a T5/BART encoder, and fine-tune for joint label and explanation generation. That combination is new relative to the cited work, and the authors report it carefully: five seeds, standard deviations, p-values for the main Tex-SAGE comparisons, and a human evaluation. Second, the central faithfulness result is not yet trustworthy, because the evaluation metric is one-sided in a way that lines up with the method's inductive bias.\n\nThe counterfactual test from Atanasova et al. (2023), as implemented in Appendix H, only checks whether the inserted adjective appears in the new NLE when the predicted label changes. When the label does not change, the NLE is never penalized for including the inserted word. G-Tex conditions generation on the top-30% highlighted tokens, so it plausibly increases the model's tendency to copy input spans into the explanation. A model that copies aggressively would sail through the label-changing cases and never be caught on the label-unchanged cases. The reported 12.18-point drop in Total Unfaith on ComVE may therefore reflect lexical copying rather than alignment with the model's reasoning. I would want to see the symmetric test run, or at least a copying-rate analysis, before taking the headline gain at face value.\n\nSecond soft spot: the appendix contains a result that contradicts the main claim. Table 5 shows BART Tex-GCN with span interactions on ComVE at 96.44% Total Unfaith versus 72.82% for the fine-tuning baseline; Appendix I even says 'almost all' models outperform baselines, which concedes exceptions that Section 5.1's blanket statement ignores. That inconsistency needs fixing.\n\nThe circularity concern is real but not damning. The highlights come from a framework developed by the same group, and the counterfactual test is from a co-author's earlier work; the method then injects those highlights directly into the encoder. This overlapping design does not invalidate the approach, but it increases the burden on the authors to show that the improvement is not an artifact of the metric rewarding the very signal they feed in.\n\nThe paper is worth a serious referee. The recipe is novel, the setup is clean, and the flaw is fixable: run a symmetric faithfulness evaluation, report the Table 5 contradiction, and ideally add a control that measures copying. With those changes the empirical claim would be much more solid. Without them, treat the 12.18% as an upper bound on the true effect.","headline":"G-Tex is a concrete recipe with a real validity threat: the one-sided counterfactual test may reward input copying, so the headline faithfulness gain is not yet established.","tokens_in":26855,"tokens_out":3094,"would_cite":true,"duration_ms":27360,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"G-Tex claims that injecting a model's own attention-derived highlight explanations into its encoder as a graph reduces counterfactual unfaithfulness in generated natural language explanations by up to 12.18 percentage points.","keywords":["natural language explanations","faithfulness","highlight explanations","graph neural networks","self-rationalization","counterfactual faithfulness test","attention attribution","T5 and BART"],"falsifier":"Re-run G-Tex with the same architecture and data, but replace the attention-derived highlight edges with equally sized edges chosen at random or from a perturbation-based attribution method such as leave-one-out. If random or perturbation-based highlights produce the same or larger faithfulness improvements, the claim that attention-based highlights are the faithful cue carrying the effect would be falsified.","tokens_in":25877,"feed_emoji":"🧠","tokens_out":10620,"duration_ms":83545,"temperature":0.7,"pith_summary":"G-Tex is a method for making the free-text explanations that a model generates for its own predictions actually reflect the model's internal reasoning. The paper's central claim is that if one first extracts faithful highlight explanations from the model—the input tokens and token/span pairs its attention marks as important for the predicted label—and injects those highlights into the encoder as a graph, then the generated natural language explanations pass counterfactual faithfulness tests far more often. The reported gain is up to 12.18 percentage points in total unfaithfulness on ComVE, with consistent reductions on e-SNLI and ECQA and answer accuracy that stays comparable or better. G-Tex explanations are also closer to human-written explanations in lexical and semantic similarity and are judged by humans to be less redundant and higher in overall quality. If this is right, faithfulness can be steered directly by the model's own high-faithfulness cues instead of by external knowledge, prompts, or modified training losses.","feed_headline":"Highlight graphs cut unfaithful explanations by up to 12.18%","feed_subtitle":"Injecting the model's own attention highlights as a graph steers free-text explanations toward its real reasoning.","key_machinery":"The central mechanism is the highlight-explanation graph. Each input token is a node; edges are drawn according to the three highlight types—top-$k\\%$ single tokens, token pairs, and span pairs—with extra edges connecting subword tokens after tokenization. This graph is encoded by a GNN layer, implemented as GraphSAGE, GAT, or GCN, which is inserted after the $3/4$-th encoder layer of T5 or BART. The GNN updates token representations by aggregating information from highlighted neighbours, and the decoder then produces the label and the natural language explanation from these augmented encoder states. The graph edges are initialized equally and are updated during fine-tuning, so the highlights act as a soft structural prior on the information flow that enters the explanation rather than as a hard constraint.","core_discovery":"The discovery that G-Tex argues for is that a model's own high-faithfulness highlight explanations can be turned into a graph and used as a structural guide for generating its natural language explanations. The framework first trains a base encoder-decoder model for label prediction, extracts three types of highlights from its decoder attention—single important tokens, interacting token pairs across the two input parts, and interacting spans—and then builds a graph in which input tokens are nodes and edges encode the selected highlights. A graph neural network layer placed after the three-quarter point of the encoder aggregates these node representations, and the decoder generates the label and the explanation from the augmented encoder states. Across T5 and BART on e-SNLI, ComVE, and ECQA, the authors report that this lowers counterfactual unfaithfulness below both plain fine-tuning and prompt-injection baselines, by up to 12.18 percentage points, while preserving or improving label accuracy. The authors also find that interactive highlights help most for two-part inputs where relations between parts matter, while single-token highlights help most when the first input part is a fixed instruction.","pith_inferences":["The counterfactual faithfulness test rewards explanations that mention the inserted adjective after a label change, so part of G-Tex's improvement may be a sharper ability to echo injected tokens rather than a deeper alignment with reasoning; a deletion-based or consistency-based faithfulness test would separate these.","Because the graph controls which input tokens the decoder can see, the same wiring could steer other generation properties—factual grounding, style, or coverage—by substituting different edge sets for different criteria.","The paper's dataset-dependent finding suggests a practical selection rule: use interactive highlights for multi-part inputs and token highlights for fixed-template inputs, which could be tested as an automatic per-instance or per-dataset choice.","Extending the graph injection to decoder-only models would require a different way of merging graph embeddings into next-token prediction, since the decoder attends only to preceding tokens; the authors leave that adaptation open."],"forward_implications":["Faithfulness of natural language explanations can be improved by guiding generation with the model's own high-faithfulness cues, without sacrificing answer accuracy: G-Tex reports label accuracy comparable to or better than the baselines.","The best choice of highlight type follows the input structure: token and span interactive explanations give the largest faithfulness gains on tasks with meaningful interaction between two input parts, while token explanations work best when one input part is a fixed instruction.","G-Tex explanations are more similar to human-written explanations in both lexical and semantic measures, and human judges rate them as less redundant and higher in overall quality.","The method adds very few parameters (about 0.28% over T5 and 0.24% over BART) and nearly identical training time, so the faithfulness gain does not come from a large capacity increase."],"supporting_citations":[{"why":"Supplies the counterfactual faithfulness test used to measure the paper's central outcome, counterfactual and total unfaithfulness.","marker":"Atanasova et al. (2023)"},{"why":"Defines the three highlight explanation types and shows attention-based highlights have measurable faithfulness; G-Tex builds on these as faithful cues.","marker":"Sun et al. (2024)"},{"why":"Provides the most-important-head selection and Louvain span-extraction methods used to generate token and span interactive explanations.","marker":"Ray Choudhury et al. (2023)"},{"why":"Supplies the GNN-layer placement at the three-quarter encoder point and the information-flow rationale that G-Tex adapts.","marker":"Yuan et al. (2024)"},{"why":"Provides graph construction conventions, including subword edges, and GNN encoding of token-level structure.","marker":"Yuan and Färber (2024)"},{"why":"Establishes the self-rationalization setup and the faithfulness problem for free-text rationales that G-Tex targets.","marker":"Wiegreffe et al. (2021)"},{"why":"Provides e-SNLI, the primary dataset with human-annotated explanations used in the faithfulness evaluations.","marker":"Camburu et al. (2018)"},{"why":"Represents the prior highlight-aware NLE generation approach that relies on external knowledge, which G-Tex contrasts with direct graph injection.","marker":"Majumder et al. (2021)"},{"why":"Supplies the attention-averaging procedure used to score individual token importance for highlight extraction.","marker":"Jain and Wallace (2019)"}],"fun_headline_variants":["Graph-guided explanations cut unfaithfulness by 12.18%","Model highlights as graphs boost explanation faithfulness","G-Tex: graph-guided NLE generation improves faithfulness","Turning attention highlights into graphs steers NLEs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the attention weights of the base model, averaged over tokens, reliably identify the input fragments the model truly uses when it makes its prediction, and that injecting those fragments as graph edges is what makes the generated explanation track the model's real reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Graph-guided explanations cut unfaithfulness by 12.18%","Model highlights as graphs boost explanation faithfulness","G-Tex: graph-guided NLE generation improves faithfulness","Turning attention highlights into graphs steers NLEs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000622,"raw_usage":{"total_tokens":2907,"prompt_tokens":996,"completion_tokens":1911,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":1847}},"tokens_in":612,"tokens_out":1911,"duration_ms":12345,"temperature":1.0,"reasoning_tokens":1847,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:12:39.958612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run G-Tex with the same architecture and data, but replace the attention-derived highlight edges with equally sized edges chosen at random or from a perturbation-based attribution method such as leave-one-out. If random or perturbation-based highlights produce the same or larger faithfulness improvements, the claim that attention-based highlights are the faithful cue carrying the effect would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the self-rationalization setup and the faithfulness problem for free-text rationales that G-Tex targets."}],"review_version":1}