{"id":"c2e11b24-156c-414d-908d-8066abc96642","arxiv_id":"1908.06605","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A planning-based hierarchical variational model that generates sentence-level content plans and then realizes each sentence with hierarchical latent variables beats prior data-to-text systems on coverage, coherence, and diversity.","lead":"This paper presents a neural system for writing long product descriptions and recipes from structured data: it first decides which features to mention in each sentence, then writes each sentence with random variations at two levels. On e-commerce advertising text and recipes, it reports better coverage, coherence, and diversity than four prior systems, though with lower BLEU on the advertising task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage gains may be an artifact of using the same string-match heuristic for plan supervision and for the coverage metric.","rationale":"I read the paper as claiming a specific causal story: a hierarchical planning mechanism improves coverage and diversity on long data-to-text tasks. The evidence for that story is consistency across two datasets plus manual pairwise evaluation, but the coverage numbers are computed with the same string matching used to construct the supervised planning signal. That makes the main quantitative support, especially Tables 3 and 5, vulnerable to a real but addressable confound. I do not see a reason to reject outright: the architecture is clearly described, ablations attribute distinct-4 and repetition-4 mainly to the latent variables, and the manual evaluation in Table 4 is an independent signal for overall quality. However, the manual evaluation does not directly validate coverage, so the coverage claim remains conditional. The reader's weakest assumption identified plan supervision noise; I agree partially, but the more specific issue is the shared heuristic on both sides of the training and evaluation loop, which makes the coverage metric partially align with what the model was trained to optimize. The proposed concrete test, using human gold plans and gold coverage judgments, would settle whether the concern actually lands. If the coverage advantage persists under semantic annotation, the central claim is strengthened; if it does not, the paper's headline quantitative result would need qualification.","tokens_in":17993,"tokens_out":6016,"duration_ms":63982,"concrete_test":"Take a random sample of 200 advertising test instances. Have two annotators (with disagreement adjudication) produce gold plans for the reference text, marking which input attribute-value pairs are semantically expressed in each sentence and allowing paraphrase, and gold coverage judgments for the corresponding PHVM and Pointer-S2S outputs. Then (i) compare the gold plans with the string-match plans used to train Equation (6), reporting per-sentence group F1; and (ii) recompute Table 3 coverage from the gold judgments. If heuristic plan F1 is low (e.g., below 0.8) or PHVM's coverage advantage over Pointer-S2S disappears under gold annotation, the central planning-driven coverage claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that planning is responsible for PHVM's coverage and diversity gains rests on a supervision-and-evaluation loop built entirely on string-match heuristics. In Section 5.3 the reference plans used to train the plan decoder are obtained by \"recognizing attribute values (or ingredients) in each sentence with string match heuristics,\" and in Section 5.4 coverage is measured the same way: \"We recognized attribute values (ingredients) with string match heuristics.\" Equation (6) optimizes the plan decoder against these extracted groups, so the model is directly trained to reproduce the exact surface strings that the coverage metric later rewards. This is not an internal contradiction, but it makes the automatic coverage numbers in Tables 3 and 5 unable to distinguish genuine content planning from matching the heuristic: a generated sentence that mentions a value token in any context counts as covered, and the plan decoder is supervised to place that token in the predicted group. The manual pairwise evaluation (Table 4) supports grammaticality and coherence but does not assess coverage independently, so the headline \"coverage improves to 87.05%\" is not yet backed by a metric that is independent of the training signal. The paper acknowledges in Section 5.3 that extraction \"can introduce errors\" but only asserts, without evidence, that plans are \"sufficient to train a good plan decoder\" (Section 4.3). That assertion is exactly the load-bearing point to be tested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Planning-based Hierarchical Variational Model (PHVM) for long and diverse data-to-text generation. The model first generates a sequence of groups of input items via a planning decoder, then generates each sentence conditioned on its group, previous context, a global planning latent variable, and local sentence-level latent variables. The method is evaluated on a new Chinese advertising dataset and on a recipe generation dataset. The reported results claim that PHVM outperforms four baselines in coverage, diversity (distinct-4, repetition-4), and human-judged grammaticality and coherence. The paper also includes ablations, a planning-diversity analysis, and a case study.","tokens_in":18239,"tokens_out":3692,"duration_ms":38501,"significance":"If the claims hold, PHVM addresses two recognised problems in neural data-to-text generation: long-text coherence and expression diversity. The model is clearly specified, the code and corpus are released, and the experimental section is extensive, including automatic metrics, manual pairwise evaluation with inter-annotator agreement, ablation studies, and a new dataset for long diverse text generation. The diversity gains (distinct-4 and repetition-4) are not tied to the string-match heuristic and are large and internally consistent across both tasks. However, the central coverage claim is weakened by the fact that the coverage metric uses the same string-match heuristic that provides the supervision signal for the plan decoder, and by the dataset construction which filters out uncovered attribute-value pairs. These issues undermine the evidence that planning is responsible for the coverage improvements, although they do not invalidate the diversity or coherence findings. The paper merits revision to provide an independent evaluation of coverage and to justify or correct the planning-supervision loop.","major_comments":[{"comment":"The reference plans used to train the plan decoder are extracted by 'string match heuristics' (Sec. 5.3), and the coverage metric is computed by the same 'string match heuristics' (Sec. 5.4). Since Eq. (6) directly optimizes the plan decoder to reproduce these extracted groups, the coverage metric is not independent of the training objective. The reported 87.05% coverage in Table 3 may partly reflect PHVM's ability to match surface tokens rather than to produce semantically complete content. Please add a human-annotated coverage evaluation on a sample of generated texts, or an alternative metric that does not rely on exact string matching (e.g., semantic similarity or textual entailment), to validate the coverage claim.","section":"5.3, 5.4, Eq. (6), Table 3"},{"comment":"The advertising dataset is constructed by removing any attribute-value pair that does not occur in the corresponding text and adding pairs that occur in the text but not in the table. This guarantees that every reference text covers all input pairs by construction, which simplifies the coverage task and makes the coverage score trivially bounded at 100% for the training data. This construction may favor a model explicitly supervised to place input tokens in specific groups (as PHVM is) over models that generate content without token-level plan supervision. Please report coverage on a held-out set constructed without such filtering, or otherwise justify that the filtering does not bias the comparison in favor of PHVM.","section":"5.1, Table 1"},{"comment":"The paper states that 'the extracted plans are sufficient to train a good plan decoder' but provides no supporting evidence. Since the plan decoder is a core component of the model, the assertion is load-bearing for the claimed causal link between planning and the reported gains. Please provide either (a) an estimate of plan-extraction accuracy on a sample, (b) a comparison of PHVM trained with heuristic plans versus plans corrected by human annotators, or (c) an analysis of how extraction errors propagate to final generation quality. Without one of these, the planning supervision remains an uncontrolled source of noise that limits the interpretability of the experimental results.","section":"4.3"}],"minor_comments":[{"comment":"The caption of Figure 2 says 'group' but the figure's internal labels (e.g., the small subscripts and the 'Probability of discarding/selecting' arrows) are not defined in the caption. Please clarify the notation so that the planning and generation flow is understandable without referring back to the text.","section":"Figure 2"},{"comment":"The y-axis labels 'Avg. Score' and 'Avg. # Distinct Planning' are not defined precisely. Please specify how 'distinct planning' is counted (e.g., exact group-sequence equality, or some distance measure) and how the average score is computed.","section":"5.5.3, Figure 3"},{"comment":"The Fleiss' kappa values appear as column headings ('Grammaticality κ' and 'Coherence κ'). It would be clearer to present the kappa values in a separate row or explicitly state in the caption that they denote inter-annotator agreement coefficients.","section":"Table 4"},{"comment":"The paper states that the best model is selected by 'L1 + L2' on the validation set, but L3 (bag-of-word loss) is part of the training objective. Please clarify whether L3 was intentionally excluded from model selection and explain the rationale.","section":"5.3"},{"comment":"The BLEU scores in Tables 3 and 5 are all very low (around 2-5). A brief comment on why BLEU is a weak metric for this task (e.g., due to the high diversity of the reference texts) would help readers interpret those numbers.","section":"5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is from a strong group and the code/data release is a positive sign. The main risk is the circularity between the plan-supervision signal and the coverage metric; this is fixable with an independent evaluation. I would not reject the paper on this basis, but the coverage claim should be re-supported before publication. The diversity and coherence results are the more robust contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on PHVM. The architecture is genuinely new: a planning decoder that produces sentence-level groups of input items, coupled with a global planning latent and sequentially dependent local latents. That combination is not in VHCR or ml-VAE-D. The paper also does solid empirical work: two tasks, ablations, manual evaluation, and code/data are public. The diversity gains (distinct-4 up from 41.7 to 72.9 on ads, repetition-4 down from 36.6 to 3.9) are large and credible, and the manual pairwise wins on coherence are consistent with the planning story.\n\nThe soft spot is the coverage metric. Plans for training are extracted by string-matching attribute values, and coverage is measured the same way. The model is literally trained to place those exact strings in predicted groups, then rewarded for finding them. So the 87.05% coverage number is not independent evidence of content planning. The dataset construction also removes input pairs that don't appear in the reference text, which makes the coverage task easier. The paper acknowledges extraction errors but asserts the plans are sufficient without supporting analysis. The planning-diversity section lacks baseline comparison, which weakens it.\n\nThat said, the central claim is not dead. The diversity and coherence improvements do not depend on the string-match loop, and the ablations show both latent levels matter. The fix is straightforward: an independent coverage evaluation (e.g., human judgments of whether each attribute is addressed, or a learned semantic match) and a small study of plan extraction quality. Those are addressable in revision.\n\nThis paper deserves a serious referee. It is a substantive contribution to data-to-text and hierarchical variational generation, and the issues are about evidence quality, not about the architecture being incoherent. I'd accept for review with a request for stronger coverage evaluation.","headline":"A real architectural contribution to data-to-text with credible diversity gains, but the coverage headline is entangled with the same string-match heuristic used for training.","tokens_in":18766,"tokens_out":2503,"would_cite":true,"duration_ms":24071,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Planning first, then sentence-by-sentence writing, makes long text generation both more complete and far more varied.","keywords":["data-to-text generation","content planning","hierarchical variational autoencoder","long text generation","text diversity","coverage","repetition","conditional VAE"],"falsifier":"Have annotators mark the true input items expressed by each sentence on a random sample of the advertising and recipe training data, then compare those human groups with the string-match groups used for training; if agreement is low, or if retraining PHVM with human-annotated plans changes coverage and repetition outcomes on a fixed test set, the planning-supervision premise is decided.","tokens_in":17811,"feed_emoji":"🧩","tokens_out":6839,"duration_ms":64294,"temperature":0.7,"pith_summary":"Long data-to-text generation usually asks one decoder to do three jobs at once: choose which input facts to mention, keep successive sentences coherent, and vary the wording. The paper's claim is that these failures are best attacked by separating a high-level plan from low-level realization. The proposed PHVM first partitions the input into a sequence of groups, each group being the subset of items one sentence should cover, and then generates each sentence conditioned on its group, the previous context, and two tiers of latent variables: a global planning latent that captures the variety of reasonable orderings and per-sentence local latents that control expression and carry dependencies across sentences. On a new advertising-text dataset and on recipe generation, this design raises coverage and distinct-n diversity while cutting 4-gram repetition from 36.58% to 3.90% on advertising text. A sympathetic reader should take away that planning plus hierarchical variability, rather than a more powerful single decoder, is the load-bearing idea.","feed_headline":"A plan-first generator cuts repetition and lifts diversity","feed_subtitle":"By splitting inputs into sentence-sized groups and adding latent variation at plan and word levels, it tops baselines on ads and recipes.","key_machinery":"The central object is the group plan combined with a hierarchical variational latent structure. A plan is a sequence $g = g_1 \\cdots g_T$ in which each $g_t$ is a subset of input items meant to be covered by sentence $s_t$; a GRU plan decoder selects items with a per-item sigmoid conditioned on the global latent $z_p$ and the bag-of-words of previous groups, and stops when its stop probability exceeds 0.5. Generation then proceeds sentence by sentence: a sentence decoder produces a representation $h_t^s$ and samples a local latent $z_t^s$, with $z_t^s$ depending on previous local latents through the recurrent state, and a word decoder realizes $s_t$ from $\\{h_t^s, z_t^s, g_t\\}$. The training objective is an evidence lower bound over planning and sentence generation, plus a stop-signal loss and per-sentence bag-of-word losses, with KL annealing to counter collapse. This machinery is what lets the model first decide what to say and only then decide how to say it.","core_discovery":"The paper establishes that a planning-based hierarchical variational model outperforms state-of-the-art data-to-text baselines on long, diverse generation. On advertising text, PHVM covers 87.05% of input attribute-value pairs (best baseline 85.97%), reaches distinct-4 of 72.87% against 41.69% for the best baseline, and reduces repetition-4 to 3.90% from 36.58%; manual pairwise comparisons favor it on both grammaticality and coherence. On recipes, it attains 73.2% coverage and 67.86% distinct-4, again best among the compared models, with repetition-4 at 17.3%. Ablations attribute part of the gain to the global planning latent, whose removal lowers coverage and distinct-4, and part to the local sentence latents, whose removal raises repetition-4 to 20.88% and shortens sentences. The intended message is that decomposing generation into planned, dependent sentence sub-tasks with variational control at both levels is what produces complete, coherent, and varied long texts.","pith_inferences":["Beyond the paper: a testable extension is replacing the string-match reference plans with human-annotated or learned alignments, which would show how much of the coverage gain depends on the quality of the planning supervision, since paraphrase-heavy sentences are exactly where string matching misassigns groups.","Beyond the paper: the same plan-and-realize decomposition could apply to long-form machine translation or story generation, where no explicit input table exists; in those settings one could treat the latent plan as an unsupervised intermediate and evaluate whether coherence and diversity improve over flat hierarchical decoders.","Beyond the paper: the large reduction in repetition-4 suggests the planning mechanism may be acting as a coverage regularizer; one could test this by attaching a group-planning head to an existing attention-based generator and comparing repetition and coverage without variational latents."],"forward_implications":["If the central claim holds, long structured-to-text tasks beyond advertising copy and recipes—such as game reports, biographies, or event summaries—can be improved by inserting a similar group-planning stage before sentence realization.","Repeatedly sampling the latent variables yields multiple high-quality texts for the same input: with ten samples per input, 79.0% of the 1,000 generated advertising texts scored above 4 on a 1–5 scale, and the average self-BLEU across texts was only 16.87%.","The ablation results imply that the global planning latent is specifically responsible for planning completeness and diversity, while local latents are responsible for expression variation and cross-sentence dependency; removing local latents raises repetition-4 to 20.88%.","The recipe results suggest the approach generalizes to other domains: PHVM uses more given ingredients, generates more varied cooking steps, and produces the generic phrase 'all ingredients' less often than the baselines."],"supporting_citations":[{"why":"Supplies the checklist baseline and the recipe-generation task, including its train/validation/test split.","marker":"Kiddon et al. (2016)"},{"why":"Supplies the CVAE baseline, the conditional variational formulation, and the bag-of-word loss used to combat KL collapse.","marker":"Zhao et al. (2017)"},{"why":"Defines the two-stage Pointer-S2S baseline that plans by reordering input items before generation.","marker":"Puduppully et al. (2019)"},{"why":"Defines the Link-S2S baseline whose static link matrix models ordering between input item types.","marker":"Sha et al. (2018)"},{"why":"Provides the closest hierarchical latent model (VHCR), which the paper contrasts with its own planning-conditioned latents.","marker":"Park et al. (2018)"},{"why":"Provides the multi-level latent-variable comparison (ml-VAE-D) that motivates the paper's task-specific hierarchical structure.","marker":"Shen et al. (2019)"},{"why":"Supplies the distinct-n metric used to measure expression diversity.","marker":"Li et al. (2016)"},{"why":"Documents the long-text incoherence challenge in data-to-document generation that PHVM is designed to address.","marker":"Wiseman et al. (2017)"}],"fun_headline_variants":["Plan-first text model boosts diversity and length","Hierarchical planning yields more varied long text","Group-then-generate model cuts repetition, adds variety","Planning-based model outperforms in diverse text generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire planning supervision rests on string-match heuristics that approximate which input items each sentence covers, so if paraphrases or split mentions make those extracted groups unreliable, the plan decoder learns from corrupted targets and the reported coverage and coherence gains weaken.","fun_headline_variants_meta":{"raw":{"variants":["Plan-first text model boosts diversity and length","Hierarchical planning yields more varied long text","Group-then-generate model cuts repetition, adds variety","Planning-based model outperforms in diverse text generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1181,"prompt_tokens":905,"completion_tokens":276,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":217}},"tokens_in":521,"tokens_out":276,"duration_ms":3525,"temperature":1.0,"reasoning_tokens":217,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:39:37.246852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have annotators mark the true input items expressed by each sentence on a random sample of the advertising and recipe training data, then compare those human groups with the string-match groups used for training; if agreement is low, or if retraining PHVM with human-annotated plans changes coverage and repetition outcomes on a fixed test set, the planning-supervision premise is decided.","supporting_citations":[],"review_version":1}