{"id":"2d613b96-8313-4ece-9eaf-62a852365b2a","arxiv_id":"2507.04884","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A pipeline that extracts propositions from documents, generates synthetic grounded dialogs, and uses them to train lightweight question rewriters and retrievers for open-retrieval conversational QA.","lead":"The authors describe a pipeline that turns plain product and government documents into synthetic conversational question-answer data, including rewritten 'decontextualized' user questions and the document propositions each answer rests on. The result is a way to train compact question rewriters and retrievers for conversational QA without manually labeled dialogs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transfer claim rests on self-confirming annotations: Prompt 2.3 has the same Dialog-LLM that wrote each dialog label its own grounding propositions, so the synthetic training signal and the synthetic-retrieval evidence for proposition quality may reflect generator style, not real relevance; the…","rationale":"The paper's main contribution is a domain-transfer recipe: documents to propositions, propositions to synthetic dialogs, synthetic dialogs to a fine-tuned rewriter/retriever, and finally real-world OR-CONVQA. For the central claim to hold, the synthetic dialogs and especially the ground-truth propositions must be a valid proxy for real user grounding. The most fragile element is the annotation loop. In Step 1, Dialog-LLM creates propositions from documents. In Step 2 Prompt 2.1, the same model writes decontextualized questions and answers grounded in sampled propositions. In Prompt 2.3, the same model is told to act as a human annotator and list the propositions used for each QA pair, with the odd instruction that first and last pairs should always be accepted. The ground truth used to train the rewriter and retriever is therefore the output of the same distribution that generated the training examples. If the model has a consistent stylistic bias, such as generating answers whose words overlap its own propositions or choosing propositions it had in mind rather than the minimal set a human would need, the retriever can learn to exploit that bias. The synthetic evaluation in Tables 2-4 is not independent of this bias: retrieval quality is measured against Prompt 2.3 labels and against Dialog-LLM answers, so the conclusion that propositions are superior to sentences could partly reflect which representation is easier for the same LLM to self-annotate, rather than which representation produces more realistic dialogs. This would not necessarily weaken the rewriter's ability to learn decontextualization in general, but it would weaken the specific claim that the trained retriever and rewriter improve real-world retrieval by learning document-grounded relevance. Table 5's real-world gains are modest and are reported without variance or significance testing, so they cannot by themselves distinguish generalizable decontextualization from style-consistency artifacts. The paper's own Section 4.8 human checks cover propositions on a small sample and do not validate Prompt 2.3 labels, and Section 6 acknowledges that synthetic data may contain hallucinations; neither resolves the loop. The proposed human-label ablation directly tests whether replacing self-labels with independent labels changes transfer; if it does not, the concern is retired. Because the reader already issued a CONDITIONAL verdict based on the same transfer weakness, no verdict adjustment is needed.","tokens_in":21724,"tokens_out":5929,"duration_ms":66512,"concrete_test":"Retrain the pipeline on 100-200 sampled synthetic dialogs whose Prompt 2.3 'ground truth' proposition sets are replaced by independently produced labels: have three human annotators (or a different LLM, e.g., GPT-4o) select, from each dialog's proposition sublist, the propositions actually needed to answer each user question, seeing only the dialog and the sublist. First measure agreement (e.g., Fleiss kappa or token-level overlap) with the Prompt 2.3 labels. Then retrain the T5 rewriter and MiniLM retriever exactly as in Section 4.3 on the human-labeled subset, and for control on a same-size subset with the original self-generated labels, and rerun the real-world DOC2DIAL and MultiDoc2Dial retrieval evaluation of Table 5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that synthetic dialogs and annotations from the pipeline can train rewriters/retrievers that improve real-world OR-CONVQA. For that to hold, the annotations in Step 2 must be faithful to real user information needs and document grounding. The weakest link is Prompt 2.3 (Section 3.2): the Dialog-LLM both generates each question-answer pair and then 'act[s] as a human annotator' to select the propositions each pair is grounded in, with the additional instruction that the first and last pairs 'should always be accepted.' The same model also produced the propositions in Step 1, so the labels can be self-confirming: the model may mark propositions that lexically match its own generated answer rather than propositions a human user would actually need. This is not only an evaluation nuisance; these labels are the supervision for the fine-tuned T5 rewriter and MiniLM retriever, and the synthetic retrieval results in Tables 2-3 are used to conclude that proposition-based dialogs are superior. The real-world transfer evidence in Table 5 is modest (MAP 0.21 and 0.24 vs 0.17 for Queryco), so a meaningful share of the measured gain could be a style-consistency artifact between the annotator/propositionizer and the training-signal generator. The paper's own Section 4.8 reports only a small manual check of propositions and no independent human check of Prompt 2.3 labels, so this assumption is asserted rather than tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-step LLM-prompting pipeline that converts plain documents into propositions and then generates OR-CONVQA dialogs containing contextualized and decontextualized user questions, system responses, and grounding propositions. The authors use the resulting synthetic dialogs to fine-tune lightweight question rewriters (Mamba, GPT-2, T5) and a MiniLM retriever, and evaluate them on synthetic test dialogs and on the human-annotated DOC2DIAL and MultiDoc2Dial test sets. The central claims are that proposition-based dialogs are higher quality than sentence-chunk-based dialogs, and that models trained only on synthetic data improve real-world retrieval and response generation relative to using the original conversational questions. The paper also introduces a conditional rewriting mechanism that reduces inference latency and releases source code and a synthetic dataset.","tokens_in":22108,"tokens_out":8111,"duration_ms":85823,"significance":"If the central claims hold, the pipeline would be a practical answer to the scarcity of domain-specific OR-CONVQA training data: it needs only plain documents, avoids the one-to-one sentence/question mapping assumption, and produces training signal for lightweight rewriters and retrievers. The public release of code and a synthetic dataset is a concrete strength, as is the use of several complementary automatic metrics (MAP, R@k, SBLEU, METEOR, BERTScore, perplexity) and the comparison against a strong prompted-LLM rewriter. However, the evidence is currently weakened by self-annotated grounding labels, the absence of statistical significance testing, and a transfer evaluation that uses the same document collection for both synthetic training and real-world testing. These issues affect the two main claims, so the contribution is promising but not yet fully demonstrated.","major_comments":[{"comment":"The supervision and the synthetic-retrieval ground truth are produced by the same Dialog-LLM that generates the dialogs. Prompt 2.3 explicitly instructs the model to “act as a human annotator,” to select the propositions each QA pair is grounded in, and to always accept the first and last pairs. Because the answers, the propositions, and the grounding labels all come from the same model, the labels may reflect generator consistency (for example, lexical overlap between an answer and the propositions it was generated from) rather than the document information a real user would need. These labels are load-bearing: they are used both as the evaluation targets in Tables 2–4 and as the training signal for the T5 rewriter and the MiniLM retriever, so the measured gains on synthetic data, and to some extent the transfer results in Table 5, inherit this bias. Section 4.8 reports a manual check of the Step 1 propositions but no independent human check of the Prompt 2.3 grounding labels. The paper should add a human annotation study on a sample of the Prompt 2.3 outputs, or otherwise validate the grounding labels against independent annotations, before the synthetic-retrieval numbers can be read as evidence of dialog quality.","section":"§3.2, Prompt 2.3; Tables 2–4"},{"comment":"The paper reports three-seed averages for the synthetic test sets but no standard deviations, confidence intervals, or significance tests, and the real-world DOC2DIAL/MultiDoc2Dial experiments are run once. The headline retrieval gains are modest in absolute terms (for example, T5 MAP 0.21 vs. 0.17 for Queryco in Table 5), and the response-generation improvements are uneven (T5 SBLEU 6.54 vs. Claude 6.52 on MultiDoc2Dial in Table 6), with only MiniLM showing a larger gain (8.89 vs. 6.16). Without variance estimates or paired significance tests, the claim that fine-tuning on synthetic data “substantially improves” real-world performance is not established at the reported precision. Please report per-seed results, standard deviations, and appropriate significance tests for the synthetic experiments, and at least bootstrap confidence intervals or paired tests for the real-world comparisons.","section":"§4.1, §4.5, Tables 5–6"},{"comment":"The real-world transfer evaluation uses the same 488 DOC2DIAL/MultiDoc2Dial documents both to generate the synthetic training dialogs and as the proposition repository for the human test dialogs. This setup tests transfer from synthetic to human dialogs within a single document collection, but it does not test the stated goal of applying the pipeline to a genuinely new application domain with different documents. The proprietary-document experiments are evaluated only on synthetic test dialogs, so no real-world test exists for a held-out document set. To support the “new application domain” claim, the authors should either hold out a document collection for real-world evaluation or explicitly soften the claim to same-corpus transfer.","section":"§3.1, §4.5"}],"minor_comments":[{"comment":"The stated BM25 parameters k1 = 0.05 and b = 5 are unusual: b is normally restricted to [0, 1] and k1 is typically near 1.2. Please clarify whether this is a typo and report the actual values used, since retrieval results can be sensitive to these settings.","section":"§4.2, footnote"},{"comment":"The conditional rewriting section states that “we find no difference in performance” without reporting the corresponding retrieval numbers in a table or in the text; please provide the quantitative comparison alongside the latency reduction.","section":"§4.7"},{"comment":"The manual evaluation discussion reports qualitative findings (“little to no hallucinated information,” “cases of propositions that convey no information”) without counts, example frequencies, or agreement statistics; please report the size of the sample and the proportion of cases in each category.","section":"§4.8"},{"comment":"The reference list contains Lin et al. 2020a and Lin et al. 2020b with the same title and venue; please consolidate the duplicate or clarify the distinction between the two entries.","section":"References"},{"comment":"The abstract says that “the retrieved information and the decontextualized question” are passed to the response LLM, but for the fine-tuned MiniLM retriever the paper instead passes the dialog history along with the last user question; please make the abstract consistent with the experimental setup.","section":"Abstract and §4.6"}],"recommendation":"major_revision","confidential_remarks":"The core idea is useful and the paper is clearly written, but the self-annotation loop and the same-corpus transfer evaluation are central to the claims. I would like to see the revision add independent validation of the Prompt 2.3 grounding labels, statistical significance testing, and either a held-out-document transfer experiment or a more modest framing of the domain-transfer claim. A direct comparison with an existing synthetic-data pipeline such as Dialog Inpainting would also strengthen the positioning against prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful engineering paper with a clear pipeline and public artifacts, but the strongest claims—that proposition-based synthetic dialogs are superior and that the synthetic annotations are trustworthy enough to train rewriters and retrievers—rest on a self-confirming loop that the paper never independently validates.\n\nWhat's new is the combination: proposition extraction, LLM-generated dialogs with paired contextualized and decontextualized questions, and fine-tuning lightweight rewriters and retrievers on the synthetic data, with no manual dialogs for the target domain. The real-world transfer numbers are positive and worth taking seriously: the T5 rewriter lifts MAP from 0.17 to 0.21 on both Doc2Dial and MultiDoc2Dial, and the MiniLM retriever lifts response SBLEU from 6.16 to 8.89 on MultiDoc2Dial. Those are not huge gains, but they are in the right direction and suggest the synthetic data carries real signal.\n\nThe soft spots, in order of actual weight. First, Prompt 2.3 asks the same Dialog-LLM that generated each question-answer pair to act as a human annotator and select the grounding propositions, with an instruction that the first and last pairs are always accepted. That makes the synthetic retrieval scores in Tables 2–3 partly measures of the generator's internal consistency, not of how well the propositions match real user needs. The paper's own Section 4.8 manual check is a small sample and does not audit the Prompt 2.3 labels. Second, the three seeds are averaged without reporting variance or significance tests, so the modest real-world differences in Tables 5–6 could be noise. Third, the coherence metric actually slightly favors sentence-based dialogs, so the proposition-superiority conclusion leans heavily on the same circular retrieval metric.\n\nNone of this kills the paper. The real-world transfer is positive, the method is reproducible, and the flaws are addressable. The reader's take and the stress-test note both land where I land. This deserves a serious referee: the pipeline is useful, the artifacts are public, and the issues are fixable with significance testing, an independent human sample of Prompt 2.3 labels, and an ablation that drops the 'always accept' instruction.","headline":"A useful pipeline with public artifacts, but the synthetic annotation loop is self-confirming and the real-world transfer gains, while positive, are smaller and less statistically grounded than the prose claims.","tokens_in":22608,"tokens_out":2180,"would_cite":true,"duration_ms":23353,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a fully synthetic pipeline can generate realistic conversational QA dialogs and annotations from plain documents, and that fine-tuning light rewriters or retrievers on this synthetic data improves retrieval and…","keywords":["conversational question answering","open-retrieval question answering","synthetic data generation","question rewriting","decontextualization","propositions","retrieval-augmented generation","document-grounded dialog"],"falsifier":"Run the full pipeline on a held-out collection of plain documents from a fresh domain for which a small human-annotated OR-CONVQA test set exists, then measure retrieval MAP using the synthetic-trained rewriter against the raw contextualized questions as baseline; if the rewriter's MAP is not clearly above the baseline in repeated seeded runs, the transfer assumption is falsified. A cheaper check is to repeat the DOC2DIAL and MultiDoc2Dial real-world experiments while swapping the Dialog-LLM for a different model and seeing whether the synthetic-trained models still beat using the raw contextualized question.","tokens_in":21559,"feed_emoji":"💬","tokens_out":7894,"duration_ms":74680,"temperature":0.7,"pith_summary":"This paper claims that a purely synthetic pipeline can generate realistic, document-grounded conversational question-answering (OR-CONVQA) dialogs from plain domain documents, with no manually annotated training data and no assumption that each document sentence corresponds to one user question. The key move is to have a large language model first convert documents into stand-alone 'propositions' and then use those propositions to generate full dialogs, including both contextualized and decontextualized versions of each user question and the propositions that ground each answer. The authors show that fine-tuning small question rewriters or retrievers on this synthetic data substantially improves retrieval and response generation on real-world DOC2DIAL and MultiDoc2Dial test dialogs compared with using the original conversational questions. The practical stake is that an organization with a repository of plain product or policy documents could bootstrap a domain-specific conversational QA system without the expensive human annotation that normally blocks such deployments.","feed_headline":"Synthetic dialogs from plain documents beat raw questions","feed_subtitle":"LLM-generated dialogs and rewritten questions lift retrieval on real-world conversational QA benchmarks without manual annotation.","key_machinery":"The load-bearing mechanism is the proposition-first pipeline. A 'proposition' is defined as a standalone simple sentence that conveys information from a document which a user might actually ask about; Step 1 has an LLM split compound sentences, remove anaphora, and discard content no one would query. Step 2 then samples sublists of these propositions and prompts the same LLM to generate a dialog, first writing self-contained (decontextualized) user questions, then re-introducing context to create the contextualized versions, and finally asking the model to annotate which propositions ground each question-answer pair, marking pairs 'accepted' or 'not_accepted'. The accepted pairs become the training signal, and BM25 is used to replace the regenerated ground-truth propositions with their closest Step-1 originals, preventing vocabulary drift. This machinery matters because it removes the one-sentence-to-one-question assumption of earlier dialog-inpainting methods and lets retrieved propositions, rather than whole documents, carry the answer.","core_discovery":"The central claim is that the quality of synthetic dialog generation depends less on the LLM's ability to write questions than on the representation of the source material: documents first distilled into propositions—short, self-contained sentences carrying information a user would plausibly ask about—yield dialogs whose retrieval performance is substantially higher than dialogs generated from raw document sentences. On top of that, the self-annotated decontextualized questions and ground-truth propositions in the synthetic dialogs provide enough supervision to fine-tune lightweight models: a small question rewriter and a small retriever trained only on synthetic data outperform using the raw contextualized user question by a clear margin on DOC2DIAL and MultiDoc2Dial, and the retriever's performance is comparable to prompting a larger LLM for rewrite generation. The paper also shows that fine-tuning a small retriever on the synthetic data is at least as effective as question rewriting, and that a joint 'rewrite/no_rewrite' classifier can halve average rewriter latency without hurting accuracy.","pith_inferences":["Because the pipeline needs only plain text, the same prompts could be run through a multilingual LLM to generate OR-CONVQA dialogs for low-resource languages, an extension the authors name but do not test; the main risk is that proposition quality degrades in lower-resource language models.","If the synthetic dialogs were scaled up, for example by resampling propositions or using multiple LLM seeds, the authors' own preliminary result that small response generators underperform prompted LLMs might reverse, since they attribute that gap to insufficient synthetic data.","The proposition granularity likely interacts with the retriever: the largest gains appear when propositions are short enough for a sparse-dense fusion to match them exactly, so domains with longer or more technical passages may need adjusted proposition lengths."],"forward_implications":["Organizations with only plain text documents, such as product documentation or policy manuals, can build domain-specific conversational QA systems without manually annotated dialogs.","Lightweight question rewriters trained on synthetic data can replace expensive LLM prompting for decontextualization, cutting inference cost while preserving retrieval quality.","Existing dialog-unaware retrievers can be reused as-is, because the rewriter turns each user turn into a self-contained query.","A small retriever fine-tuned on synthetic data can match or exceed the usefulness of gold decontextualized questions, contradicting the common assumption that retriever fine-tuning needs large amounts of human-annotated data.","The 'no_rewrite' decision mechanism roughly halves rewriter latency with no measured performance drop, which matters for real-time assistants."],"supporting_citations":[{"why":"Supplies the proposition-extraction design principle and the finding that large LLMs are required for it.","marker":"Chen et al. (2024b)"},{"why":"Provides the DOC2DIAL real-world test dialogs used to measure transfer.","marker":"Feng et al. (2020)"},{"why":"Provides the MultiDoc2Dial real-world test dialogs with multi-document grounding.","marker":"Feng et al. (2021)"},{"why":"Gives the BM25 sparse retriever used both for retrieval and for replacing ground-truth propositions.","marker":"Robertson and Zaragoza (2009)"},{"why":"Supplies the reciprocal rank fusion method used in the main retrieval experiments.","marker":"Cormack et al. (2009)"},{"why":"Is the dialog-inpainting baseline whose one-sentence-to-one-question assumption this pipeline removes.","marker":"Dai et al. (2022)"},{"why":"Provides the manual question-rewriting dataset that the synthetic-trained rewriter is compared against and outperforms.","marker":"Anantha et al. (2021)"}],"fun_headline_variants":["Propositions, not raw text, fuel synthetic dialog QA","Distilling docs into propositions boosts synthetic dialogs","Synthetic dialogs via propositions beat raw question baselines","Decontextualizing questions with small models, not big LLMs","Better conversational QA from decontextualized synthetic data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training signal rests on the assumption that the LLM's self-generated dialogs and self-annotated grounding propositions approximate real user questions and document grounding closely enough that models trained on them transfer to genuine human dialogs.","fun_headline_variants_meta":{"raw":{"variants":["Propositions, not raw text, fuel synthetic dialog QA","Distilling docs into propositions boosts synthetic dialogs","Synthetic dialogs via propositions beat raw question baselines","Decontextualizing questions with small models, not big LLMs","Better conversational QA from decontextualized synthetic data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2609,"prompt_tokens":961,"completion_tokens":1648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1568}},"tokens_in":577,"tokens_out":1648,"duration_ms":12642,"temperature":1.0,"reasoning_tokens":1568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:37:09.133363+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on a held-out collection of plain documents from a fresh domain for which a small human-annotated OR-CONVQA test set exists, then measure retrieval MAP using the synthetic-trained rewriter against the raw contextualized questions as baseline; if the rewriter's MAP is not clearly above the baseline in repeated seeded runs, the transfer assumption is falsified. A cheaper check is to repeat the DOC2DIAL and MultiDoc2Dial real-world experiments while swapping the Dialog-LLM for a different model and seeing whether the synthetic-trained models still beat using the raw contextualized question.","supporting_citations":[],"review_version":1}