{"id":"45b380d9-1322-4238-be7a-f22e311f2a0e","arxiv_id":"2411.19203","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A broad evaluation of twelve LLMs on five data-to-text datasets shows Llama 2 leads on factual consistency, larger size usually helps, and reference divergence hurts scores.","lead":"This paper evaluates how factually consistent outputs from twelve language models are when generating text from structured data across five datasets. It reports that Llama 2 models generally produce the most factually consistent text, while larger model size helps but does not guarantee improvement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference-based metrics undermine the source-reference divergence finding; a source-only recomputation is needed before that headline claim can stand.","rationale":"This concern is load-bearing because it targets the construct validity of one of the paper's three central claims. The first and second findings are better supported: the human evaluation in Section 7 is source-based and QAFactEval is source-based, so a model that matches the reference more closely is not automatically judged faithful to the source. The third finding, however, rests on automatic metrics whose context is either explicitly the reference (NEOverlap) or unspecified (SummaC-Conv, AlignScore) in the divergence analysis. If those metrics compare against the reference, the observed decrease with source-reference divergence is partly mechanical, because the reference itself contains information absent from the source and the model cannot be expected to reproduce facts it never saw. The paper does not report a source-only version of these metrics or a human evaluation stratified by divergence, so the claim is not yet supported at the level of the paper's own definition. The reader's weakest assumption identifies the same issue, and there is no basis to move beyond the existing conditional verdict: the analysis should be repeated with source as the context and with human judgments on the same divergence bins.","tokens_in":992,"tokens_out":704,"duration_ms":56041,"concrete_test":"Re-run the Section 6 divergence-bin analysis replacing the reference with the linearized source as the context for SummaC-Conv, NEOverlap, and AlignScore (matching QAFactEval's source-based setup), and additionally have annotators judge the same divergence-stratified samples for source faithfulness. If the monotone decrease across low-to-high divergence bins persists in all four source-based metrics and in the human judgments, the finding is confirmed. If the decrease attenuates, flips, or disappears for the three metrics, the current result is an artifact of reference-based scoring.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The third headline finding is the least secure, because Section 6's operationalization may not measure what the paper says it measures. Section 1 defines factual consistency as whether generated text 'accurately reflects factual information from the input/source data,' and Section 3.3 says the reference text (or sometimes the source) is used as context. But Section 5.2 explicitly states that for NEOverlap the reference text is treated as the context and the generated text as the claim; the paper never states which context is used for SummaC-Conv and AlignScore in Section 6. If the reference is the context, then a high source-reference divergence bin means the reference contains facts absent from the source. A model that faithfully follows the source will 'miss' those extra facts, and any metric comparing to the reference will decline mechanically. Thus the observed drops in Figures 11-14 for three of the four metrics would be a scoring artifact, not evidence of reduced factual consistency. QAFactEval uses the source text for answer checking, so it provides one source-based signal, but no human evaluation or source-based version of the other metrics is reported for the divergence split. The paper therefore does not establish that source-reference divergence reduces factual consistency as defined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an evaluation of factual consistency in data-to-text generation (DTG) across five datasets (E2E, ViGGo, WikiTableText, DART, WebNLG) and twelve models from five LLM families (BART, T5, OPT, BLOOM, Llama 2), using four automatic metrics (SummaC-Conv, NEOverlap, AlignScore, QAFactEval) plus a small human annotation study. Three findings are claimed: Llama 2 generally produces more factually consistent text; smaller T5/BART models can be competitive on large, lexically less diverse datasets; increasing model size generally improves factual consistency as measured by AROC; and source-reference divergence reduces factual consistency.","tokens_in":16462,"tokens_out":5256,"duration_ms":45910,"significance":"If the findings hold, the paper would provide a useful comparative benchmark and practical guidance for model selection in DTG. The study has genuine breadth: twelve models, five datasets, three DTG task types, four complementary automatic metrics, and a human evaluation. The explicit investigation of source-reference divergence is a valuable direction. However, the third headline finding is threatened by a reference-based metric artifact, and the statistical support for the other findings is incomplete. The first two findings are plausible but need stronger uncertainty quantification before they can be considered established.","major_comments":[{"comment":"The third headline finding is not established by the reported experiment. Section 3.3 defines the context for factual-consistency evaluation as the reference text 'or sometimes source data,' and Section 5.2 explicitly states that NEOverlap treats the reference text as the context. Section 6 reuses NEOverlap, SummaC-Conv, AlignScore, and QAFactEval but does not state which context is used for SummaC-Conv and AlignScore in that section. When the reference is the context, the high-divergence group contains references with content absent from the source; a model that faithfully reproduces the source will necessarily score lower on any reference-context metric. The downward slopes in Figures 11–14 are therefore at least partly mechanical for three of the four metrics. QAFactEval uses the source text for answer checking, but it is a single metric, and no source-based recomputation or human evaluation binned by divergence is reported. To support the claim that source-reference divergence reduces factual consistency of LLMs, the authors should either recompute the analysis with the source as context for all metrics, or report human annotations split by divergence level, or explicitly narrow the claim to reference-based agreement.","section":"§6, Figs. 11–14; §5.2; §3.3"},{"comment":"The statistical significance reporting is incomplete and cannot be checked from the manuscript. The text states only that Welch's t-test was used 'with a significance level of p < 0.05 and a sample size of 6'; it does not define what the six observations are, how many generated outputs underlie each reported score, or whether the t-test is applied to per-instance metric scores, bootstraps, or model runs. No variance, confidence interval, or effect size is reported in Tables 2–5, and some adjacent differences are extremely small (e.g., Table 2, OPT on E2E: 0.3637 vs. 0.3621). The blanket statement that 'all results statistically significant' is not verifiable. The authors should provide the test definition, standard deviations or confidence intervals, and either exact p-values or a correction for multiple comparisons.","section":"§4.3, Tables 2–5"},{"comment":"The AROC analysis uses only the smallest and largest model in each family, so it does not actually test monotonicity of the size effect. Equations (3)–(5) define AROC as the chord slope between f(M1) and f(M2), and Figures 7–10 plot only those endpoint slopes. This is particularly problematic for OPT and BLOOM, where three sizes were evaluated and the intermediate point is available; for example, Table 2 shows OPT on E2E does not improve monotonically (0.3637, 0.3597, 0.3621), yet the two-endpoint AROC compresses this into a single number. The claim that larger models 'generally enhance' factual consistency should be supported by a regression over all available sizes, pairwise comparisons, or at least a discussion of the intermediate points.","section":"§5.5, Figs. 7–10; Eqs. (3)–(5)"}],"minor_comments":[{"comment":"The model name/size is inconsistent: Figure 5 and the Section 6 figure captions use 'Llama2-6B,' while Tables 2–5 use 'Llama2-7B'; these should be unified.","section":"Figure 5 and §6 captions"},{"comment":"The caption says 'A lower value indicates higher factual consistency,' but Section 7 describes yes/no annotations aggregated as percentages; clarify whether Table 6 reports inconsistency rates rather than consistency rates.","section":"Table 6"},{"comment":"Equation (1) uses argmax over the loss L(Mθ(s), r); maximizing a loss is nonstandard and should be argmin, or L should be defined explicitly as a negative log-likelihood.","section":"Eq. (1)"},{"comment":"The table caption says it shows 'average length (in tokens),' but the columns list TTR, unique tokens, and total tokens; update the caption or add the missing average-length column.","section":"Table 1"},{"comment":"The paper says source and generated text lengths are fixed at 256 tokens but does not describe the truncation policy for longer sources or references.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid evaluation study with a useful scope, but the source-reference divergence finding needs re-analysis before it can stand as a headline result. The statistical reporting is also weak for a paper whose main contributions are empirical rankings. The issues are fixable with additional experiments and reporting, so I do not see grounds for rejection if the authors are willing to address them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the two things worth knowing. This is a genuinely broad evaluation of factual consistency in data-to-text generation: 12 models across 5 families, 5 datasets, 4 metrics, and a small human study. The specific numbers in Tables 2-5 and Figures 7-10 are new, and the first two headline results -- Llama 2 is the most consistently factual family, and larger models within a family generally help -- are credible enough to inform model choice. The third headline, that source-reference divergence reduces factual consistency, is not well supported by the evidence as reported.\n\nWhat the paper does well: the experimental design covers the three main DTG task types (MR-to-text, table-to-text, graph-to-text), the metrics span NLI, entity overlap, alignment, and QA-based methods, and the human evaluation, though small, lines up with the automatic metrics on the family ranking. The paper is clearly written and would be a useful reference point for people working on DTG evaluation. The related work is appropriately cited -- Li et al. 2022 and Kasner & Dusek 2024 are the right touchstones.\n\nSoft spots, in order of severity. First, the statistical reporting is thin: every result is a point estimate, with no variance, no decoding seeds, and no confidence intervals. The Welch t-test is cited with a 'sample size of 6' but the paper never says what those six units are, and without standard deviations the significance claims cannot be checked. This needs to be fixed, but it's an additive change.\n\nSecond, the AROC analysis uses only the smallest and largest model in each family. For OPT and BLOOM, that ignores the middle point, which can hide non-monotonic behavior. As a crude trend summary it's fine, but the paper leans on it as if it were a rigorous scaling law.\n\nThird, and most importantly, the source-reference divergence finding is at least partly an artifact of metric design. NEOverlap explicitly treats the reference as the context and the generated text as the claim (Section 5.2). SummaC-Conv and AlignScore are not specified for the divergence analysis, and Section 3.3's general rule is 'reference text (or sometimes source data).' When the reference contains facts the source does not, a model that faithfully follows the source will miss those extra facts, and overlap-based metrics like NEOverlap will drop mechanically. QAFactEval is source-based and does show a decline, but there is no human evaluation split by divergence and no source-based recomputation of the other metrics. So the third finding, as worded in the abstract, overstates what the paper shows.\n\nWho it's for: practitioners who want a broad, current ranking of LLM families for data-to-text, and researchers who need a convenient benchmark table. It deserves serious peer review, but a referee should push for variance reporting, a clarified metric-context specification, and a source-based reanalysis of the divergence claim.\n\nRecommendation: engage with it, with major revision expected before acceptance.","headline":"Useful breadth, credible size/ranking findings, but the divergence claim outruns the evidence because the metrics lean on references.","tokens_in":16978,"tokens_out":6126,"would_cite":false,"duration_ms":53585,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that factual consistency in data-to-text generation generally improves with model size, that Llama 2 models are the most consistent overall, and that source-reference divergence reliably lowers consistency scores.","keywords":["data-to-text generation","factual consistency","large language models","hallucination","source-reference divergence","average rate of change","automatic evaluation metrics","human evaluation"],"falsifier":"Take the high-divergence test groups from the paper's Section 6 and have human annotators score generated text against the source data instead of against the reference; if the drop in consistency with divergence weakens or reverses, the third finding is an artifact of reference-based scoring rather than a property of model behavior.","tokens_in":16071,"feed_emoji":"📊","tokens_out":8884,"duration_ms":68127,"temperature":0.7,"pith_summary":"This paper attempts to establish three empirical regularities about how large language models generate factual text from structured data such as tables, graphs, and meaning representations. Across five datasets, twelve models from five families, four automatic metrics, and human annotations, it finds that Llama 2 models usually produce the most factually consistent text, that increasing model size within a family generally improves consistency as measured by the average rate of change (AROC), and that source-reference divergence—when the reference text carries extra or different information than the source—lowers consistency scores for every model. Smaller models such as T5-large and BART-large remain competitive on large, lexically repetitive datasets such as E2E. Because scoring is reference-based, \"factual consistency\" here means agreement with the reference text rather than with the raw source, which shapes the third finding.","feed_headline":"Bigger LLMs generally write more factually consistent data-to-text","feed_subtitle":"Twelve models, five datasets, four metrics: Llama 2 leads, but small models win on simple, repetitive data.","key_machinery":"The load-bearing object is the evaluation setup rather than a new algorithm. Four reference-based metrics—SummaC-Conv (natural-language-inference scoring with document-level aggregation), NEOverlap (named-entity overlap), AlignScore (multitask information alignment), and QAFactEval (question generation and answering)—score each generated text against the reference. The paper's summary statistic is AROC, the log-log rate of change in factual consistency per model-size change, $AROC = \\frac{\\log f(M_2) - \\log f(M_1)}{\\log m_2 - \\log m_1}$, where $f(M)$ is a metric score and $m$ is trainable parameter count. For the divergence analysis, source-reference divergence is computed as $div(s,r) = 1 - \\frac{LCS(s,r)}{\\max(|s|,|r|)}$, where $LCS$ is longest common subsequence length. These two quantities carry the argument: positive AROC supports the scaling claim, and the divergence bins support the claim that reference drift lowers consistency.","core_discovery":"The paper's central claim is that factual consistency in data-to-text generation is not a single model property but a joint effect of model size, model family, and dataset characteristics. Concretely, the Llama 2 family outperforms BART, T5, OPT, and BLOOM on almost every dataset under SummaC-Conv, NEOverlap, AlignScore, and QAFactEval, with human annotations agreeing on the ordering. At the same time, T5-large and BART-large beat larger OPT and BLOOM models on E2E and DART, which are large and lexically homogeneous, indicating that small models can win when datasets are big and repetitive. AROC values—the log-scaled change in consistency per log-scaled change in parameter count—are positive for most families and datasets, which the authors read as evidence that growing model size generally improves factual consistency. Finally, when the reference text contains information not in the source, consistency scores fall for every family, and larger models fall less, suggesting that they are more robust to source-reference divergence.","pith_inferences":["One implication the paper leaves implicit is that its size trend is a statement about matching references, not about faithfulness to source data; ordering could shift under source-grounded evaluation.","Because AROC is log-scaled on both axes, the steep gains of T5 and BLOOM relative to Llama 2 may partly reflect the small size of those families rather than faster learning.","A testable extension would be to filter or down-weight high-divergence training instances, which the paper's divergence finding suggests should improve reference-based consistency.","In safety-critical deployments such as medical or financial report generation, the paper's results argue for preferring a larger model family even when computational budget favors a small one."],"forward_implications":["Within a model family, increasing trainable parameters—especially in T5 and BLOOM—tends to raise factual consistency scores across the five datasets.","Models from the Llama 2 family are the safest default for reference-level factual consistency, with the largest margins on DART and WebNLG.","On large, lexically repetitive datasets like E2E, smaller encoder-decoder models can match or beat much larger decoder-only models.","When source and reference diverge, every family's consistency score drops, so dataset cleaning or divergence-aware evaluation is needed before trusting reported scores.","Human evaluation agrees with the automatic metrics on the ordering of model families, supporting the use of reference-based automatic metrics in similar settings."],"supporting_citations":[{"why":"Provides the E2E MR-to-text dataset used across all automatic-metric and human evaluations.","marker":"Dusek et al. [2020]"},{"why":"Provides the ViGGo dialogue MR-to-text dataset.","marker":"Juraska et al. [2019]"},{"why":"Provides the WikiTableText table-to-text dataset with high lexical diversity.","marker":"Bao et al. [2018]"},{"why":"Provides the DART graph-to-text dataset and earlier DTG evaluation context.","marker":"Nan et al. [2021]"},{"why":"Provides the WebNLG RDF-to-text dataset.","marker":"Gardent et al. [2017]"},{"why":"Introduces the divergent-reference problem and the LCS-based divergence measure used in Section 6.","marker":"Dhingra et al. [2019]"},{"why":"Supplies the SummaC-Conv metric and the NEOverlap implementation used for scoring.","marker":"Laban et al. [2022]"},{"why":"Supplies the QAFactEval question-generation/answering metric used for scoring.","marker":"Fabbri et al. [2022]"},{"why":"Supplies the AlignScore information-alignment metric used for scoring.","marker":"Zha et al. [2023]"}],"fun_headline_variants":["Scaling LLMs improves data-to-text factual consistency","Small models win on simple data-to-text; Llama 2 leads overall","LLM consistency in data-to-text: size matters, data also matters","Source-reference gaps reduce factual consistency in LLM text","Five datasets, four metrics: how LLMs stay factually consistent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the reference text as the factual context for scoring, so a model that follows the source data when the reference has drifted away from it is scored as less factually consistent.","fun_headline_variants_meta":{"raw":{"variants":["Scaling LLMs improves data-to-text factual consistency","Small models win on simple data-to-text; Llama 2 leads overall","LLM consistency in data-to-text: size matters, data also matters","Source-reference gaps reduce factual consistency in LLM text","Five datasets, four metrics: how LLMs stay factually consistent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1452,"prompt_tokens":1030,"completion_tokens":422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":334}},"tokens_in":646,"tokens_out":422,"duration_ms":4348,"temperature":1.0,"reasoning_tokens":334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:25:04.567425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the high-divergence test groups from the paper's Section 6 and have human annotators score generated text against the source data instead of against the reference; if the drop in consistency with divergence weakens or reverses, the third finding is an artifact of reference-based scoring rather than a property of model behavior.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ViGGo dialogue MR-to-text dataset."},{"cited_title":"Table-to-text: Describing table region with natural language","cited_arxiv_id":null,"evidence_quote":"Provides the WikiTableText table-to-text dataset with high lexical diversity."}],"review_version":1}