{"id":"bd1dc104-03e5-4761-a1dd-95dd47ea6ea8","arxiv_id":"1909.00564","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A query-guided capsule network that clusters preceding-sentence context into perspectives and adds a source-target regularization loss gives small BLEU and Meteor gains on TED and Europarl En-De translation, but not on News.","lead":"This paper adds a query-guided capsule network to document-level neural machine translation, letting the current sentence steer how context words are clustered. The authors report BLEU and Meteor gains on TED and Europarl English-German translation, but the same model falls behind strong baselines on the News dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of significant gains on multiple datasets lacks statistical support: Europarl improves by only +0.07 BLEU, News is 2.66 BLEU below the best baseline, and no significance tests are reported.","rationale":"I read the paper as claiming a practical, measurable improvement, not merely a plausible mechanism. The load-bearing condition is therefore that Table 2's gains are real and consistent. Table 2 does not support 'multiple datasets': News is strongly negative, Europarl is within noise, and TED is positive but its size depends on unexplained hyperparameter selection. I did not select the PCC-semantics concern from the reader's weakest_assumption as primary because the architecture could still produce gains even if the correlation interpretation is imperfect; the empirical claim is the claim that must bear weight. The paper deserves credit for releasing code, which makes the proposed test feasible. The self-reported limitation about long-sentence filtering on News is important and should be flagged. The reader's CONDITIONAL verdict is appropriate; my test would either firm it up or expose the claim as overstated, but does not require rejection without the numbers.","tokens_in":943,"tokens_out":1132,"duration_ms":312130,"concrete_test":"Run the released code on TED and Europarl with at least three random seeds for each of Transformer+Reg, Transformer+QCN, Transformer+QCN+Reg, and, under identical sentence-filtering, the SAN/HAN baselines (or use their released outputs). Bootstrap 95% confidence intervals for BLEU and Meteor differences on the fixed test sets. Separately, document the dev-set scores used in Figures 4 and 5 and freeze hyperparameters before touching the test set. If the Europarl difference vs SAN includes 0, or if the TED full-model gain over Transformer+Reg is not significant, the claim of consistent significant gains on multiple datasets should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 4.2 claim significant gains over strong baselines on multiple datasets, but Table 2 does not establish this. On Europarl, the full model beats the best context-aware baseline (SAN, Maruf et al. 2019) by only +0.07 BLEU (29.82 vs 29.75), a difference almost certainly within run-to-run and evaluation noise; no confidence intervals, paired tests, or multi-seed results are reported anywhere. On News, the same model is 2.66 BLEU below HAN (22.37 vs 25.03); Section 4.2 attributes this to GPU-memory-driven filtering of long sentences, but that is a self-admitted confound that prevents the 'multiple data sets of different domains' claim from holding on one of the three domains. Additionally, the selection of hyperparameters (three historical sentences, four capsules) is based on Figures 4 and 5, which plot TED scores; the paper does not state whether these figures use the development set or the test set, so the TED advantage of +0.61 BLEU over HAN may partly reflect test-set selection. Finally, the ablation isolates the QCN poorly: 'Transformer + Regularization Term' alone reaches 24.55 TED BLEU and 29.42 Europarl BLEU, while 'Transformer + QCN' alone reaches only 24.41 and 29.48; the full model's improvement over the regularization-only variant is +0.64 TED BLEU and +0.40 Europarl BLEU, so the contribution of the paper's new routing mechanism to the headline result is not cleanly demonstrated. The central claim therefore rests on a fragile empirical base even before the semantic validity of the PCC routing signal is considered.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Query-guided Capsule Network (QCN) for document-level neural machine translation, together with a regularization layer based on Pearson correlation coefficients that ties source- and target-side capsule representations during training. The method is integrated into a Transformer by adding a context-aware attention sub-layer in the encoder. The authors report experiments on English-German TED, News, and Europarl corpora and claim that their model significantly outperforms strong context-agnostic and context-aware baselines on multiple datasets of different domains. The central empirical claim is that the full model, Transformer + QCN + Regularization Term, beats hierarchical attention and selective-attention baselines on TED and Europarl, while the News result is explained as being hurt by GPU-memory-driven filtering of long sentences.","tokens_in":11386,"tokens_out":2286,"duration_ms":21272,"significance":"If the claimed gains were statistically solid, the paper would make a useful contribution: it introduces a novel routing mechanism for document-level context modeling, releases code, and applies capsule networks to document-level NMT for the first time. The proposed mechanism is mechanistically plausible, and the ablation design is a reasonable attempt to separate the contributions of the routing module and the regularization term. However, the evidence presented in Table 2 does not support the abstract's 'significantly outperform on multiple datasets' claim: the News result is below every context-aware baseline, the Europarl BLEU gain over the best baseline is only +0.07, and no significance tests, confidence intervals, or multi-seed runs are reported. The contribution is therefore interesting but currently under-validated.","major_comments":[{"comment":"The central claim that the method 'can significantly outperform strong baselines on multiple data sets of different domains' is contradicted by the paper's own News results. In Table 2, the full model scores 22.37 BLEU on News, which is 2.66 BLEU below HAN (25.03) and 2.47 below SAN (24.84). Section 4.2 attributes this to long-sentence filtering for GPU memory, but that is a self-admitted confound that prevents the 'multiple datasets' claim from holding on one of the three domains. The abstract and Section 4.2 should be revised to report state-of-the-art on two of three datasets, or the News experiment should be repeated without the filtering confound so that the claimed cross-domain generalization can be assessed.","section":"Abstract and Section 4.2, Table 2"},{"comment":"No statistical support is provided for the claimed gains. On Europarl, the full model improves over the best context-aware baseline (SAN) by only +0.07 BLEU (29.82 vs. 29.75), a difference that is likely within run-to-run and evaluation noise. No confidence intervals, paired significance tests, or multiple random-seed results are reported anywhere, despite the abstract's use of the word 'significantly'. The authors should provide multi-seed runs with standard deviations and paired tests (or at minimum bootstrap confidence intervals) for the BLEU and Meteor differences that underlie the headline claims.","section":"Section 4.2, Table 2"},{"comment":"The hyperparameters 'number of historical sentences' and 'number of feature capsules' are selected using Figures 4 and 5, which plot TED scores, but the paper does not state whether these plots use the development set or the test set. If the TED test set was used for hyperparameter selection, the reported +0.61 BLEU gain over HAN on TED may partly reflect test-set overfitting. The authors should clarify which split is used in these figures and, if it is the test set, either re-select hyperparameters on development data or report the selection procedure transparently.","section":"Section 4.2, Figures 4 and 5"},{"comment":"The ablation does not cleanly isolate the contribution of the QCN routing mechanism. The 'Transformer + Regularization Term' row alone reaches 24.55 TED BLEU and 29.42 Europarl BLEU, while 'Transformer + QCN' alone reaches 24.41 and 29.48. The full model's improvement over the regularization-only variant is only +0.64 TED BLEU and +0.40 Europarl BLEU, while the regularization term alone already accounts for most of the gain over the Transformer baseline (24.55 vs. 23.28 on TED). The paper should either provide a finer ablation that holds the regularization term fixed while adding QCN, or temper the claim that the new routing mechanism is the primary source of the improvement.","section":"Table 2, ablation rows"}],"minor_comments":[{"comment":"The notation in Eq. (7) is ambiguous: the sums run over dimensions of the vectors A and B, but the formula as written does not make this explicit, and the indexed symbols a_i and b_i are not defined in the surrounding text. Please define the dimension index clearly.","section":"Algorithm 1 and Eq. (7)"},{"comment":"The update q_j <- (q_j + v_j)/2 is stated without discussion of whether this averaging preserves the query signal across iterations. A short justification for this specific update rule would improve readability and help the reader assess the mechanism described in Section 3.2.","section":"Algorithm 1, lines 29-31"},{"comment":"There are several typographical errors, including 'recieves' (Section 3.2), 'firslty' (Section 5), and the author name 'V oita' (references). These should be corrected in a final revision.","section":"Throughout"},{"comment":"The phrase 'we use the default Adam optimizer with a fixed learning rate of 0.0001' is slightly unclear: 'default' typically refers to the hyperparameters beta1, beta2, and epsilon, but the learning rate is explicitly set. Please clarify the optimizer configuration.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The novelty of applying capsule routing to document-level NMT is real, and the code release is a positive sign. The main barrier is the gap between the abstract's 'significant gains on multiple datasets' claim and the evidence in Table 2, especially the News result and the lack of statistical significance testing. This is fixable within the scope of the manuscript: revise the claims to match the evidence, add multi-seed or bootstrap significance results, and clarify the hyperparameter selection split. I would not reject on the basis of the mechanism, but the empirical validation must be strengthened before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The query-guided capsule network is the real contribution here. As far as I can tell, applying capsules to document-level context and using a PCC-weighted coupling coefficient to guide routing is new, and the code release makes the work reproducible. The overall architecture is coherent, and the regularization idea—tying higher-level capsules from the source and target into the same semantic space—is a reasonable design choice.\n\nBut the headline claim does not survive contact with the paper's own tables. On Europarl, the full model beats the best context-aware baseline by +0.07 BLEU, which is a rounding error. On News, it is 2.66 BLEU below HAN, and the authors admit they filtered long sentences for GPU memory reasons. That is a self-confounded result, so the \"multiple data sets of different domains\" claim only really holds for one dataset, TED. There are no significance tests, no multi-seed runs, and no confidence intervals anywhere. The hyperparameter curves in Figures 4 and 5 also do not say whether they plot dev or test scores, so the TED advantage might partly reflect test-set selection.\n\nThe ablation deepens the problem. Transformer + Regularization alone nearly matches the full model on TED (24.55 vs 25.19 BLEU) and on Europarl (29.42 vs 29.82). Transformer + QCN alone is actually worse than the regularization-only variant on TED. So the new routing mechanism is not cleanly responsible for the final gain; the interaction with the regularization term seems to be doing most of the work. That is not a fatal flaw, but it means the paper's central story needs restructuring.\n\nWho should read this? Researchers working on context aggregation for document-level NMT, especially those interested in alternatives to hierarchical attention. The idea is worth thinking about, even if the empirical evidence is currently too thin to change practice. I would not cite it as a state-of-the-art result, but I might mention it as a novel mechanism.\n\nAs a referee, I would send this to peer review with a clear request for major revision: add proper significance testing, clarify dev/test usage, fix the News confound, and run a cleaner ablation that isolates QCN from the regularization term. The technical novelty justifies reader time, but the claims need to be scaled back.","headline":"A genuinely new capsule routing mechanism for document-level NMT, but the reported evidence for its advantage is far weaker than the abstract suggests.","tokens_in":709,"tokens_out":1264,"would_cite":false,"duration_ms":37907,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Query-guided capsule clustering of context words lifts document translation over attention baselines.","keywords":["document-level neural machine translation","capsule networks","dynamic routing","query-guided attention","context modeling","Pearson correlation","Transformer","English-German translation"],"falsifier":"Train the same QCN model on the TED English-German set with the query-guided terms $p_{ij}$ replaced by zeros or by random signs while keeping every other component fixed. If BLEU stays within noise of the reported 25.19, the Pearson-correlation routing is not the active ingredient; if it drops toward the context-agnostic Transformer's 23.28, the query guidance is load-bearing.","tokens_in":10776,"feed_emoji":"🌐","tokens_out":9717,"duration_ms":87251,"temperature":0.7,"pith_summary":"The paper argues that document-level translation improves when historical context is not compressed into a single attention vector but clustered into several perspective capsules under the guidance of the sentence currently being translated. To do this it introduces a query-guided capsule network whose iterative routing uses the Pearson correlation between each context word and the query to raise or lower that word's vote, together with a regularization term that pulls source-side and target-side capsule representations closer in training. On English-German experiments over TED talks, news, and Europarl, the model reports the best BLEU and Meteor scores on TED and Europarl, beating both the context-agnostic Transformer and context-aware hierarchical and selective attention baselines. If the claim holds, translation systems can decide which parts of prior sentences matter for the current sentence rather than treating all context alike.","feed_headline":"Capsule routing lifts document translation on TED and Europarl","feed_subtitle":"Clustering prior context into query-guided capsules beats hierarchical and selective attention baselines in English-German tests.","key_machinery":"The carrying object is the query-guided capsule network (QCN), an improved dynamic routing procedure over lower-level capsules $u_i$ (context words from previous sentences, each concatenated with a distance one-hot vector) and higher-level capsules $v_j$ (perspective features). The query $q$ is a linear projection of the summed embeddings of the current source sentence. In each routing iteration the coupling coefficients $c_{ij}=\\mathrm{softmax}(\\alpha_i)$ are augmented by $p_{ij}=\\tanh(\\mathrm{PCCs}(u_i,q_j))$, the higher-level capsule is formed as $s_j=\\sum_i (c_{ij}+p_{ij})\\hat{u}_{j|i}$ and then squashed, and the query is updated to $q_j\\leftarrow(q_j+v_j)/2$. Pearson correlation, defined in Eq. (7), is the linear correlation between two vectors; positive correlation raises a context word's contribution and negative correlation lowers it. This routing mechanism is what lets the network select context relevant to the current sentence, and the regularization layer's PCC loss is what ties the source and target capsule spaces together during training.","core_discovery":"On the paper's own terms, the discovery is that capsule-style clustering—each context word voting for one of several higher-level feature capsules—can replace a single compressed context representation, and the vote can be steered by the current source sentence. The improved routing computes a Pearson correlation between each context-word capsule and the query vector, adds a tanh-squashed version of that correlation to the ordinary coupling coefficient, and during each iteration updates the query toward the higher-level capsule it has just produced. These query-guided context capsules are then fed into the Transformer encoder through an additional attention sub-layer. A separate regularization layer runs ordinary capsule networks over encoder and decoder inputs and adds a Pearson-correlation loss that keeps the two sides in a shared semantic space. The reported result is that this combination surpasses hierarchical and selective attention baselines on two of the three English-German test sets, with the largest margin on TED.","pith_inferences":["Beyond the paper's experiments, the same query-guided routing could be tested on dialogue response generation and long-document summarization, where the current utterance or query sentence must select relevant history from a long context.","Because the relevance gate is a fixed linear correlation, replacing it with a learned similarity function is a direct extension; if a learned gate improves on TED, the linear-correlation assumption is the limiting factor.","The source-target capsule regularization suggests a general recipe of aligning intermediate representations across languages at the capsule level, which could be probed on low-resource or multilingual pairs.","The paper's scope experiments show BLEU and Meteor peak at different numbers of historical sentences, so learning the context-window size adaptively is a plausible next step beyond the fixed three-sentence setting."],"forward_implications":["Document-level translation can improve by routing context words into multiple perspective capsules rather than folding all context into one attention vector.","The current source sentence can serve as the query that decides which historical context features are extracted, making context selection sentence-specific.","Adding the Pearson-correlation regularization between source and target capsule representations yields further gains on TED and Europarl, suggesting that aligning source and target at capsule level helps coherent translation.","With only the previous three sentences, the model matches or beats a whole-document selective attention baseline on Europarl, so long context windows are not always necessary.","The claimed gains are not uniform across domains: on the News set, long-sentence filtering hurts performance and the model does not surpass the context-aware baselines."],"supporting_citations":[{"why":"Supplies the original dynamic-routing algorithm, capsule voting, and squash function that the query-guided routing modifies.","marker":"(Sabour et al., 2017)"},{"why":"Defines the hierarchical attention document-level baseline that QCN is designed to outperform.","marker":"(Miculicich et al., 2018)"},{"why":"Provides the selective attention whole-document baseline and the extracted TED, News, and Europarl corpora used in the experiments.","marker":"(Maruf et al., 2019)"},{"why":"Defines the Transformer-DocNMT baseline that incorporates document context into the Transformer, a key comparison point.","marker":"(Zhang et al., 2018a)"},{"why":"Provides the Transformer architecture whose encoder is extended with the context-aware attention sub-layer.","marker":"(Vaswani et al., 2017)"},{"why":"Supplies the memory-network document-context approach and the Europarl extraction procedure used to build one of the test corpora.","marker":"(Maruf and Haffari, 2018)"},{"why":"Introduces the capsule notion of parts-wholes vector representations that QCN adapts to context words.","marker":"(Hinton et al., 2011)"},{"why":"Supplies the IWSLT TED talk corpus used as the primary evaluation domain.","marker":"(Cettolo et al., 2012)"},{"why":"Supplies the Europarl corpus, the source of the third evaluation domain.","marker":"(Koehn, 2005)"}],"fun_headline_variants":["Query-guided capsules beat attention on TED and Europarl","Capsule clustering steered by queries improves doc-level NMT","Doc translation: query-guided capsule context beats hierarchical attention","Query-steered capsule routing lifts document translation scores","Capsule context with query guidance outperforms attention baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the linear correlation between a context word's vector and the current-sentence query is a dependable sign of whether that context word matters for the translation, and that repeatedly averaging the query with the extracted perspective capsules does not wash out the query's own meaning.","fun_headline_variants_meta":{"raw":{"variants":["Query-guided capsules beat attention on TED and Europarl","Capsule clustering steered by queries improves doc-level NMT","Doc translation: query-guided capsule context beats hierarchical attention","Query-steered capsule routing lifts document translation scores","Capsule context with query guidance outperforms attention baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1586,"prompt_tokens":800,"completion_tokens":786,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":705}},"tokens_in":416,"tokens_out":786,"duration_ms":7686,"temperature":1.0,"reasoning_tokens":705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:44:52.761498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same QCN model on the TED English-German set with the query-guided terms $p_{ij}$ replaced by zeros or by random signs while keeping every other component fixed. If BLEU stays within noise of the reported 25.19, the Pearson-correlation routing is not the active ingredient; if it drops toward the context-agnostic Transformer's 23.28, the query guidance is load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the original dynamic-routing algorithm, capsule voting, and squash function that the query-guided routing modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the hierarchical attention document-level baseline that QCN is designed to outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the selective attention whole-document baseline and the extracted TED, News, and Europarl corpora used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the memory-network document-context approach and the Europarl extraction procedure used to build one of the test corpora."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the capsule notion of parts-wholes vector representations that QCN adapts to context words."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the IWSLT TED talk corpus used as the primary evaluation domain."}],"review_version":1}