{"id":"43f490ab-7f23-4123-9f4c-d0343e1508e8","arxiv_id":"2608.00712","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Jointly training multi-hop question generation with question answering through alignment and contrastive losses improves generated question quality on HotpotQA and MuSiQue over vanilla backbones.","lead":"This paper trains one AI model to do two jobs at once: write multi-step questions from documents and answers, and answer questions from documents and questions. It reports small to moderate gains in question-quality metrics over plain models on two benchmarks, mostly by making generated questions easier to answer.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The alignment losses in Eqs.5-8 appear to pair teacher-forced gold question/answer states rather than decoded model outputs; unless clarified, the claimed mechanism of aligning MQG-generated questions with QA-generated answers is not actually implemented.","rationale":"The reader's weakest assumption is close: the positive-pair signal in Eqs.5-8 is defined by dataset correspondence rather than semantic verification. My concern sharpens this. The paper says it aligns 'MQG-generated question states' and 'QA-generated answer states,' but the training objective in Eqs.3-4 is standard teacher-forced NLL, so the hidden states available during training are states of the gold Q and gold A. Unless the authors specify a sampling or scheduled-decoding step, the alignment loss never sees the model's actual generated question tokens. This is not an external-consensus dispute; it is an internal mismatch between the abstract's causal claim and the described objective. It is load-bearing because if the loss is only reference-state regularization, the observed gains could be a multi-task or regularization effect rather than evidence for the duality mechanism. I still would not reject: the method is coherent, ablations and human evaluation are reported, and the same-backbone controls help. The conditional verdict stands, with the condition being a clarifying experiment that uses decoded states and reports uncertainty.","tokens_in":17635,"tokens_out":10776,"duration_ms":103812,"concrete_test":"Re-run the LLaMA(3B) and LLaMA(8B) HotpotQA Full-setting experiments with hq and ha in Eqs.5-10 computed from free-run decoded outputs (detached greedy decoding of the MQG question and QA answer for each batch example) instead of teacher-forced gold tokens, with 5 seeds and paired bootstrap confidence intervals on Table 2. If the gains vanish or reverse, the current improvements come from aligning gold reference states, not from aligning generated questions with generated answers, and the central claim must be restated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on L_Q↔A and L_CL enforcing 'strict mutual correspondence' between MQG-generated questions and QA-generated answers. As described, Eqs.5-8 use hq and ha from the same training triple (D,A,Q), and the standard NLL training in Eqs.3-4 is teacher-forced. The positive pair (zq,ha+) is therefore the hidden state of the gold question Q paired with the hidden state of the gold answer A, not a state obtained by decoding a question from (D,A) and then answering that question. The objective aligns reference representations, so the causal story that better alignment of generated outputs drives the Table 2 gains is unsupported. The paper's conclusion concedes the positive-pair signal is 'broader' than standard contrastive learning and its 'theoretical basis remains to be further refined.' Many reported gains are under one BLEU-4 point and several cells decrease; without significance testing or released code/data, 'significantly improves' is also not established. A test with free-run decoded states for hq/ha would settle whether the mechanism is output-level or reference-level.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QQ, a framework that jointly trains multi-hop question generation (MQG) and question answering (QA) in a unified architecture, using a bidirectional alignment loss (Eqs. 5-8) and a contrastive loss (Eqs. 9-10) to pull paired question-answer representations together and push unpaired ones apart. The authors evaluate QQ on HotpotQA and MuSiQue across GPT and LLaMA backbones with several automatic metrics, answerability (EM/F1), and human evaluation, reporting consistent gains relative to vanilla backbones and better performance than several published baselines.","tokens_in":17907,"tokens_out":5941,"duration_ms":53228,"significance":"The paper identifies a plausible and under-explored direction: exploiting the task-level duality between MQG and QA rather than using QA only as an auxiliary signal. If the proposed mechanism genuinely aligns the representations of generated questions with generated answers, the framework would be a useful contribution to multi-hop question generation. The evaluation is broad, covering multiple backbone sizes, two datasets, ablation studies, answerability checks, and human judgments, and the paper is honest about some limitations. However, the core mechanism is underspecified in the current write-up, and the empirical evidence, while mostly positive, is not statistically grounded. With clarification and additional validation, the idea has real potential.","major_comments":[{"comment":"The variables hq and ha are never defined; Eqs. (1)-(2) only introduce token-level hidden states h_q_t and h_a_t. As written, the positive pairs (zq, ha_+) and (za, hq_+) appear to be formed from the hidden states of the gold question and gold answer in the teacher-forced training pass, not from states produced by decoding a question from (D,A) and then answering that question. Since the NLL objectives in Eqs. (3)-(4) are teacher-forced, the alignment losses as specified align reference representations. The abstract's claim that the framework enforces \"strict mutual correspondence between the questions generated by the MQG model and the answers produced by the QA model\" is therefore not supported by the described implementation. Please define hq and ha explicitly, and either describe a free-run or sampling procedure that produces output-level states (with a clear gradient path) or revise the claims to describe reference-level alignment and discuss why that yields the observed transfer.","section":"Bidirectional Alignment and Contrastive Objectives, Eqs. (5)-(8)"},{"comment":"The claim that QQ \"consistently improves\" and \"significantly improves\" question generation is not supported by the reported numbers. Several cells decrease, including LLaMA(8B) SF ROUGE-L by 0.28, MuSiQue LLaMA(1B) 2-hop BLEU-4 by 0.76 and ROUGE-L by 0.62, LLaMA(3B) 3-hop METEOR by 0.55, and LLaMA(8B) 4-hop BLEU-4 by 0.73. No confidence intervals, significance tests, or multiple-seed variance are reported. Please provide paired significance tests or variance estimates, or temper the wording from \"significantly\" to a more hedged claim with an explicit discussion of the negative cells.","section":"Table 2"},{"comment":"The construction of the negative sets V_a, V_q, and S_a is not specified. If these are in-batch negatives, then with the reported batch sizes (e.g., 2 for LLaMA-8B on MuSiQue), the effective number of negatives is very small and the behavior of the alignment and contrastive losses changes qualitatively. The paper should state the negative sampling strategy, the batch size used for the contrastive/alignment heads, and how negatives are shared across the MQG and QA directions.","section":"Eqs. (5)-(9) and Implementation Details"}],"minor_comments":[{"comment":"The caption says \"Bold indicates L_Q↔A dominance, while underlined indicates LCL dominance,\" but the table appears to use bold and underlining to mark improvement magnitudes over the vanilla backbone; please clarify the intended meaning.","section":"Table 3 caption"},{"comment":"The case study in the main text refers to \"Table 8,\" but the first case study table is numbered Table 5; the cross-reference should be fixed.","section":"Case Study, main text"},{"comment":"The legend in Figures 4-6 uses the label \"LLaMA-3-8B w/ QQ,\" which is inconsistent with \"LLaMA(8B) w/ QQ\" used in the text.","section":"Figures 4-6"},{"comment":"Table 9 uses \"LLaMA-3(8B)\" while the rest of the paper uses \"LLaMA(8B)\"; please standardize the notation.","section":"Table 9"},{"comment":"The section introduces Qwen-plus but does not provide a citation; please add a reference for the model.","section":"Prompting Large Language Models (supplementary)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The paper's actual contribution is a unified MQG/QA training framework: one decoder trained with NLL on both tasks plus an alignment loss (Eqs. 5-8) and a contrastive loss (Eq. 9). That combination is not in the baselines; QA4QG uses a QA model as a constraint, not joint training. The experiments are broad: six backbones, two datasets, both HotpotQA settings, and MuSiQue split by hop count. The ablation suggests both losses add value, with alignment contributing more, and the human eval is small but reasonable.\n\nNow the soft spots. The stress-test note lands. The text repeatedly says the losses align 'MQG-generated questions' and 'QA-generated answers,' and Figure 1 says 'during training, we enforce bidirectional alignment between MQG-generated questions and QA-generated answers.' But Eqs. 5-8 use hq and ha, which are never defined, and the standard NLL losses (Eqs. 3-4) are teacher-forced. As written, the positive pair is the gold question state with the gold answer state, not a state obtained from decoded outputs. So the causal claim in the abstract—that aligning generated outputs drives the gains—is not supported. It's possible the losses work fine as reference-level regularization; the paper would be just as good or better if it said that. But then 'strict mutual correspondence' is an overstatement. The conclusion itself concedes the positive-pair signal is 'broader' and its theoretical basis 'remains to be further refined.' That is honest, but it should be in the abstract too.\n\nOther concerns are proportionally minor: single runs without significance tests, a few negative cells (e.g., LLaMA-8B SF ROUGE-L -0.28, MuSiQue 2-hop BLEU-4 -0.12) mean 'significantly improves' is not established. No code or data released; the DPKG split comes from the authors' prior paper, which is fine but slows independent verification. hq and ha need a one-sentence definition.\n\nNet: the central idea is plausible and the results are mostly consistent, but the mechanism as described is not exactly what is implemented. That is fixable. A revision should either clarify that alignment is applied to reference hidden states and soften the claims, or run a version with free-run decoded states to test the output-level mechanism.\n\nI'd send this to serious peer review. The missing definitions and the reference-vs-output gap are exactly what referees are for. It's not a desk reject.","headline":"Solid empirical paper with a plausible joint-training idea; the central mechanism is under-specified and may be aligning gold states, not generated outputs.","tokens_in":18403,"tokens_out":3190,"would_cite":false,"duration_ms":26981,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-hop question generation improves when the generator is trained jointly with a question-answering head, using bidirectional alignment and contrastive losses that keep generated questions and answers mutually predictable.","keywords":["multi-hop question generation","question answering","task duality","bidirectional alignment","contrastive learning","HotpotQA","MuSiQue","answerability"],"falsifier":"Train the same unified backbone with $L_{Q \\leftrightarrow A}$ and $L_{\\mathrm{CL}}$ but permute the positive pairing so that each question state is aligned with a random answer state from another example; if BLEU-4, METEOR, ROUGE-L, and answerability Exact Match and F1 stay at the same level or improve on the dev sets, the specific question–answer pairing is not the source of the reported gains.","tokens_in":17462,"feed_emoji":"🤝","tokens_out":9120,"duration_ms":70824,"temperature":0.7,"pith_summary":"This paper sets out to show that multi-hop question generation (MQG) gets better when the generator is trained inside the same model as a question-answering (QA) head, rather than treating answer information as a fixed input. The proposed QQ framework runs both tasks on one architecture: the same parameters produce a multi-hop question from documents plus a target answer, and produce an answer from documents plus a question. Two auxiliary objectives enforce the claimed duality: a bidirectional alignment loss that makes question states and answer states mutually predictable in a shared latent space, and a contrastive loss that pulls matched question–answer representations together while pushing unmatched pairs apart. On HotpotQA and MuSiQue, across GPT and LLaMA backbones, the paper reports consistent gains in BLEU-4, METEOR, ROUGE-L, and BERTScore over the same backbones trained on MQG alone, and higher Exact Match and F1 when generated questions are answered by the unified model. If the claim holds, it offers a practical way to exploit the interdependence of asking and answering instead of keeping the two tasks separate.","feed_headline":"One model, two tasks yields better multi-hop questions","feed_subtitle":"Unified generation and answering with alignment losses lifts quality on HotpotQA and MuSiQue.","key_machinery":"The load-bearing mechanism is a unified encoder–decoder with two output heads, one for MQG and one for QA, trained jointly with four losses. The alignment loss maps the hidden states of generated questions and answers through a projection module followed by a predictor (both two-layer MLPs with ReLU) into a shared predictive space, then maximizes cosine similarity between positive question–answer pairs and minimizes it against negatives in the batch (Eqs. 5–8). The contrastive loss maps the same hidden states through the projection module into a separate contrastive space, pulling paired representations together and pushing unpaired ones apart (Eqs. 9–10). Both losses operate on the outputs the model itself generated, so the ‘intrinsic duality’ is operationalized as a consistency constraint between the two task directions rather than as an external QA check.","core_discovery":"The central claim is that MQG and QA are intrinsically dual, and that jointly training them with mutual-predictability constraints transfers utility from the answer-direction task into the question-generation task. Formally, the authors define alignment losses $L_{Q \\to A}$ and $L_{A \\to Q}$ (Eqs. 5–6), averaged into $L_{Q \\leftrightarrow A}$ (Eq. 8), which treat the question state and the QA-generated answer state from the same training example as a positive pair in a learned predictive space; a contrastive loss $L_{\\mathrm{CL}}$ (Eq. 9) does the same in a second contrastive space with temperature $\\tau_2$. The full objective is $L = L_{\\mathrm{MQG}} + L_{\\mathrm{QA}} + L_{Q \\leftrightarrow A} + L_{\\mathrm{CL}}$, and at inference only the MQG head runs. The authors report that this improves question quality on both datasets and across model scales, and that it improves the answerability of generated questions as measured by Exact Match and F1. They also state a limitation: because one answer can correspond to many valid questions, the positive-pair signal is broader than in standard contrastive learning, and its theoretical basis ‘remains to be further refined.’","pith_inferences":["Beyond the paper: a natural tightening is to define positive pairs by answer verification rather than by training-instance identity, for example sampling several candidate questions, answering each, and keeping the one whose answer matches the target answer, which would address the paper's own caveat that one answer admits many valid questions.","Beyond the paper: because the alignment loss acts on generated output states rather than on ground-truth text, the same objective could serve in semi-supervised settings where reference questions are scarce but documents and answers are available, with the QA head acting as a consistency teacher.","Beyond the paper: the bidirectional-consistency idea should port to other dual generation tasks, such as summarization or information extraction, where the reverse direction supplies a cheap check on whether the generated output preserves the content it was meant to capture."],"forward_implications":["Because the QA head is trained jointly and not bolted on, the framework yields a single model that both generates and answers questions, so answerability checks on generated questions can be run without a separate QA system.","The reported gains appear across GPT and LLaMA backbones of different sizes and on both HotpotQA and MuSiQue, which suggests the alignment signal transfers across architectures and reasoning depths rather than fitting one configuration.","Ablations in the paper show that the bidirectional alignment loss contributes more consistently than the contrastive loss, with the contrastive loss occasionally hurting individual metrics, so the alignment term is the primary driver of the reported improvement.","On MuSiQue, the largest relative gains in several metrics appear in the 2-hop and 4-hop subsets, which the authors attribute to higher reasoning complexity, indicating the method is aimed at harder multi-hop cases."],"supporting_citations":[{"why":"Supplies the HotpotQA benchmark, including its supporting-facts and full-context evaluation settings.","marker":"(Yang et al. 2018)"},{"why":"Supplies the MuSiQue dataset with 2-, 3-, and 4-hop reasoning instances.","marker":"(Trivedi et al. 2022)"},{"why":"Represents the QA4QG baseline that uses a QA model only as external input, the contrast that motivates joint training.","marker":"(Su, Xu, and Fung 2022)"},{"why":"Supports the asymmetric alignment loss that predicts one state from the other without a symmetry constraint.","marker":"(Grill et al. 2020)"},{"why":"Supplies the contrastive learning formulation and the temperature value used in the contrastive loss.","marker":"(He et al. 2020)"},{"why":"Provides the DPKG baselines and the HotpotQA data split used for evaluation.","marker":"(Li, Zhang, and Kong 2025a)"},{"why":"Provides the GPT backbone family used in experiments.","marker":"(Radford et al. 2019)"},{"why":"Provides the LLaMA backbone family used in experiments.","marker":"(Dubey et al. 2024)"}],"fun_headline_variants":["Dual-task model turns answering into better questioning","Unified QA-generation boosts multi-hop question quality","Mutual training of QA and question generation lifts quality","Exploiting question-answer duality for sharper multi-hop queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a generated question and the QA head's answer to it, taken from the same training example, form a genuinely aligned positive pair, so forcing their latent states to match improves the question rather than reinforcing whatever superficial agreement the two heads already have.","fun_headline_variants_meta":{"raw":{"variants":["Dual-task model turns answering into better questioning","Unified QA-generation boosts multi-hop question quality","Mutual training of QA and question generation lifts quality","Exploiting question-answer duality for sharper multi-hop queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1310,"prompt_tokens":980,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":267}},"tokens_in":596,"tokens_out":330,"duration_ms":3594,"temperature":1.0,"reasoning_tokens":267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:17:44.253663+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same unified backbone with $L_{Q \\leftrightarrow A}$ and $L_{\\mathrm{CL}}$ but permute the positive pairing so that each question state is aligned with a random answer state from another example; if BLEU-4, METEOR, ROUGE-L, and answerability Exact Match and F1 stay at the same level or improve on the dev sets, the specific question–answer pairing is not the source of the reported gains.","supporting_citations":[],"review_version":2}