Attending to Future Tokens For Bidirectional Sequence Generation

Bhushan Kotnis; Carolin Lawrence; Mathias Niepert

REVIEW 4 major objections 5 minor 19 references

Attending to Future Tokens For Bidirectional Sequence Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A bidirectional transformer encoder can be fine-tuned for sequence generation by replacing output tokens with placeholders.

desk verdict BISON is a genuinely new encoder-only bidirectional generation method with large reported gains, but the missing coverage analysis for the all-placeholder inference start and the ambiguous loss specification make the empirical claim less secure than the abstract suggests. read the letter →

arxiv 1908.05915 v2 pith:7LV2POYQ submitted 2019-08-16 stat.ML cs.CLcs.LG

Carolin Lawrence , Bhushan Kotnis , Mathias Niepert This is my paper · ORCID

classification stat.MLcs.CLcs.LG

keywords bidirectionalsequencegenerationplaceholdertokenstransformerencoderBERTfine-tuningGPT-2baselinedialoguemaskedlanguagemodeling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes BISON, a way to turn a bidirectional transformer encoder into a sequence generator by inserting placeholder tokens for all as-yet-unknown output positions. Because every placeholder attends to every other token in the concatenated input and output sequence, the model can condition each generated token on both past and future output tokens. The authors show that initializing this encoder with pretrained BERT and fine-tuning on dialogue tasks yields substantial gains over left-to-right baselines: BISON outperforms GPT-2 by 12.3 BLEU-4 points on SHARC and by 6.2 points on DailyDialog, and its future-attention ablations show that most of the gain comes from looking ahead. If true, the result means a single bidirectional encoder, rather than a causal decoder, can be fine-tuned directly for generation.

What carries the argument

The central mechanism is the placeholder token, inserted for every not-yet-generated output position and treated as a node in the fully connected self-attention graph. Each placeholder's hidden representation is computed by attending to all other tokens in the concatenated input and output sequence, and a language-model classification head maps that representation to a distribution over the output vocabulary. At training time, a random subset of gold output tokens is replaced by placeholders using either Bernoulli or Gaussian random variables to control how many positions are masked; at inference time, all output positions start as placeholders and are replaced iteratively. The comparison of a past-only variant, which suppresses attention from future placeholders, isolates how much of the gain comes from bidirectionality.

What would settle it

Run a trained BISON model while varying the fraction of output positions initialized as placeholders from 0 to 1. If BLEU-4 or accuracy degrades sharply as the fraction approaches 1, the reported advantage over GPT-2 would not hold as evidence for true bidirectional planning.

Watch

Extended reading notes

Core claim

BISON's central claim is that sequence generation can be cast as iterative replacement of placeholder tokens inside a fully connected attention graph, so that at every decoding step the hidden representation of a placeholder is built from tokens on both sides. At inference, a sequence of placeholders is progressively uncovered, usually from left to right, but with each decision informed by the still-masked positions. The paper reports that this bidirectional setup, initialized from BERT-BASE-UNCASED, achieves BLEU-4 of 46.2 versus GPT-2's 33.9 on SHARC and 25.6 versus 19.4 on DailyDialog, and that suppressing attention to future placeholders drops BLEU-4 by roughly 25 points on SHARC and roughly 10 points on DailyDialog. The authors also show that a randomly initialized BISON beats a standard encoder-decoder transformer, so bidirectionality itself is doing work beyond pretraining.

Load-bearing premise

The load-bearing premise is that training on partially masked sequences transfers to fully masked inference starts; the paper gives no proof or systematic study that the masking distribution covers the all-placeholder configuration.

Editorial extensions

If this is right

A pretrained bidirectional encoder can be used directly for generation, so a separate decoder stack and causal masking are not strictly necessary.
Generation order becomes a free choice; the best strategy on SHARC was highest-probability decoding rather than left-to-right, showing the order can be optimized per task.
Cutting off attention to future placeholders costs roughly 25 BLEU-4 points on SHARC and 10 points on DailyDialog, so future context is a major source of the reported gain.
Even with randomly initialized weights, the placeholder mechanism beats a transformer encoder-decoder, isolating the mechanism from pretraining.
The success of high-variance Gaussian masking suggests that the diversity of placeholder configurations at training time is important for learning to generate from fully masked starts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct extension, not tested in the paper, would be to change BERT's pretraining corruption from masked tokens to placeholder tokens, potentially closing the train/inference gap and improving fine-tuned generation.
Because inference order is arbitrary, the approach suggests a route to constrained decoding: fix known output slots such as required entities and let placeholders fill the rest, which autoregressive decoders cannot do as naturally.
The strong dependence on high-variance masking raises the question whether a curriculum from partially masked to fully masked training sequences would make all-placeholder starts easier for the model to learn; the paper does not explore this.
Whether the gains transfer to tasks with longer or more structured outputs, such as summarization or translation, is unknown because both benchmark tasks are dialogue.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is real and worth taking seriously. BISON treats the output sequence as a set of placeholder tokens from the start, lets a Transformer encoder attend to all of them, and replaces placeholders iteratively at inference. That is a genuine departure from decoder-only generation, and it is not the same as Mask-Predict: encoder-only, direct BERT fine-tuning, and a variance-controlled placeholder strategy are all new. The paper also does the right thing by citing Ghazvininejad et al. as concurrent work rather than hiding it. The strongest empirical evidence is the ablation: removing future attention drops BLEU-4 by 25 points on SHARC and 10 on DailyDialog, which directly supports the bidirectionality claim. The no-pretraining version still beating a 270M-parameter encoder-decoder transformer is also a fair demonstration that the mechanism itself carries weight.

The soft spots are real but addressable. First, the training distribution does not provably cover the all-placeholder inference start. With Gaussian mu=0.5, sigma=0.6 the probability of a length-50 sequence being all placeholders is tiny, and the paper gives no measurement of how often the training procedure actually approaches that configuration. It could be that left-to-right inference only works because the model sees a mix after the first step, not because it handles the fully masked start gracefully. That assumption is testable and should have been tested. Second, the loss in Section 3 sums over all output positions, but the text says only placeholder tokens are mapped to output distributions. If the implementation follows the equation, the model is trained to reconstruct unmasked tokens, which can copy their own embeddings and may dilute the learning signal. The paper needs to state explicitly which positions contribute to the loss. Third, the hidden-test comparison uses the best of three runs with no error bars or significance testing. The margins are large, so I would not bet against the method, but the reporting is weaker than it should be for a SOTA claim.

Overall, the central idea is sound and the paper is a legitimate contribution to non-autoregressive and bidirectional generation. The issues are not fatal; they are missing details and missing analysis that a careful revision could supply. I would cite this work and would bring it to a reading group. A serious editor should send it to review, and the review should focus on requiring code, clarifying the loss, and asking for a direct test of the all-placeholder coverage assumption.

Referee Report

4 major / 5 minor

Summary. The paper proposes BISON (Bidirectional Sequence generation), a method for sequence generation in which a Transformer encoder is applied to the concatenation of the input sequence and an output sequence whose tokens are initially replaced by a special placeholder token. Because the placeholder positions are present from the start, each output position can attend to both past and future tokens. At inference, the model starts from a fully placeholded sequence and iteratively uncovers tokens, using strategies such as left-to-right or confidence-based ordering. At training time, the output sequence is corrupted with either Bernoulli or Gaussian placeholder replacement strategies. The method is initialized with BERT and evaluated on SHARC and DailyDialog, where it reportedly outperforms GPT2 baselines by large margins in BLEU and accuracy. The paper also reports ablations showing that suppressing attention to future tokens substantially degrades performance, and that BISON with random initialization outperforms an encoder-decoder Transformer without pretraining.

Significance. If the empirical claims hold, this is a valuable and influential idea: it is one of the first demonstrations that a bidirectional Transformer encoder can be fine-tuned directly for sequence generation, and it provides a clean way to evaluate the benefit of attending to future tokens. The 'past only' ablation is a particularly sound experimental design, and the random-initialization ablation helps separate the contribution of placeholders from the contribution of BERT pretraining. The main source of doubt is not the conceptual proposal but the reproducibility and statistical rigor of the reported results: the loss specification is ambiguous, the train/inference coverage of placeholder distributions is not quantified, and the headline margins come from three runs without variance or significance testing.

major comments (4)

[Section 3, Eq. (1)] The training objective in Eq. (1) sums the cross-entropy over every output position j, but Section 3 states that only placeholder tokens are mapped to an output distribution. If the implementation follows Eq. (1), then for positions where p_j = y_j the model is trained to reconstruct a token from its own input embedding via the residual connection, which introduces a trivial copying path and dilutes the learning signal for placeholder prediction. Please state which positions enter the loss and, if the loss is currently over all positions, report results with the loss restricted to placeholder positions (and ideally to masked positions as in BERT pretraining).
[Section 3.1] The paper selects the Gaussian strategy with μ=0.5, σ=0.6 on SHARC because low-variance Bernoulli strategies generate too few clarification questions, but it never reports the induced distribution over the number of placeholders. In particular, the inference-time input is a sequence of all placeholders; for Bernoulli μ=0.7 the probability of an all-placeholder sequence of length 50 is essentially zero, while for the chosen Gaussian it may be substantial only if P is not clipped and can exceed 1. Please report the actual distribution of P (including any clipping/truncation), the fraction of training examples that contain all placeholders or close to all placeholders, and a sensitivity analysis of the final BLEU to this coverage. Without this, the success of the method could be an artifact of an unverified train/inference coverage assumption.
[Section 4.1 and Table 1] On SHARC, BLEU-1/BLEU-4 are computed only on instances for which the model generated a clarification question (the 'More' category). Different models may generate very different numbers of clarification questions, so the BLEU scores in Table 1 are not computed over a common subset; the paper only gives an anecdote that Bernoulli μ=0.5 produced 9 clarification questions versus 846 in the gold data. Please report the number of clarification questions generated by each model in every comparison, and consider reporting a combined metric or BLEU over a fixed subset so that the comparison is not confounded by the classifier's tendency to emit 'More'.
[Section 4.4, Tables 1 and 3] The main claims that BISON outperforms GPT2 by 12.3 BLEU-4 points on SHARC and 6.2 points on DailyDialog rest on averages over three runs with no standard deviations or significance tests. Moreover, Table 2 is obtained by submitting 'the best BISON model out of the random three', which is only a valid comparison if the selection was made on the development set; the text does not say so. Please report per-run numbers, standard deviations, and a significance test, and clarify the model-selection procedure for the hidden test submission.

minor comments (5)

[Section 3.1] Please specify how sampled P values outside [0,1] are handled when computing the number of placeholders, and how the rounding to the nearest integer is performed for non-integer counts.
[Section 3, Eq. (1)] The symbol p_j is used both for the placeholder token and as the random variable representing the output position; please clarify the notation in the equation and in the surrounding text.
[Section 5, Table 7] The decomposition α2 is described as attention on 'the current word and already generated words', while the text says α2 is attention on 'the already produced sequence'; please align these descriptions.
[Section 4.4, Table 2] Replace 'the best BISON model out of the random three' with an explicit statement of the selection criterion (development BLEU-4 or something else).
[Throughout] There are several formatting issues, such as 'state-of-art' (should be 'state-of-the-art'), 'B I-SON' and 'D AILY DIALOG' spacing artifacts, and inconsistent capitalization of 'BERT' in some references.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical claims are benchmarked on held-out data, hyperparameters are tuned on a dev split, and the bidirectionality effect is tested by a future-attention ablation.

full rationale

The paper's load-bearing claims are experimental: BISON initialized from BERT-BASE-UNCASED outperforms GPT2 and transformer baselines on SHARC and DailyDialog. These results are evaluated on held-out test sets, including the hidden SHARC test set, while the placeholder-strategy hyperparameters (Bernoulli and Gaussian means/variances, learning rates, and generation strategy) are selected on a development set constructed from training data, not on the test labels. The central bidirectionality claim is probed directly by the “past only” ablation in Table 4, which suppresses attention to future placeholders and shows a large BLEU drop; this is a test of the mechanism, not an assumption that makes the result true by construction. No equation in the paper defines placeholders in terms of the target output, no fitted parameter is renamed as a prediction, and no load-bearing premise depends on a self-citation: the authors cite BERT, GPT2, and other external systems, and the paper contains no citation to the authors' own prior work. The concern about whether the training-time placeholder distribution covers the all-placeholder inference start is a robustness or generalization question about train/inference distribution mismatch, not a circularity, because the empirical result would not be forced by the paper's definitions or fitting procedure. No specific reduction of a prediction to an input can be quoted, so no circular step is identified and the score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central empirical claim depends on several tuned hyperparameters, of which the placeholder distribution parameters are the most specific to the method. They were selected on the SHARC development set and then applied to DailyDialog. The model also rests on domain assumptions about BERT transfer, the adequacy of partial masking as a proxy for full masking, and interpretability of attention weights. The placeholder token is an invented construct without independent evidence, but it is a device, not a physical entity.

free parameters (5)

Bernoulli placeholder mean µ = 0.7 (best on SHARC dev)
Controls expected fraction of output tokens replaced by placeholders during training; grid searched over [0.2, 0.8] in 0.1 increments.
Gaussian placeholder mean µ = 0.5
Grid searched over {0.4, 0.5, 0.6}; best on SHARC dev when combined with σ=0.6.
Gaussian placeholder standard deviation σ = 0.6
Grid searched over {0.3, 0.6, 0.9}; high variance was needed because Bernoulli setups, with variance 0.25 or less, generated too few clarification questions on SHARC dev.
Maximum generated sequence length = 50 (SHARC), 100 (DailyDialog)
Chosen based on lengths observed in the training data; affects the number of placeholder tokens at inference.
Fine-tuning learning rate = 3e-5 (SHARC), 1e-4 (DailyDialog)
Selected via preliminary experiments; the paper reports the best value for each dataset.

assumptions (5)

domain assumption Pretrained BERT weights transfer to the placeholder-based generation objective after fine-tuning.
BISON initializes from BERT-BASE-UNCASED and the main results depend on this transfer (Section 4.2). The random-initialization experiment shows the architecture alone is weaker on SHARC, so BERT initialization is load-bearing for the SOTA claims.
domain assumption Training with partial placeholder replacement is a sufficient proxy for the all-placeholder inference condition.
Section 3.1 introduces Bernoulli and Gaussian replacement strategies to expose the model to diverse configurations, but the paper does not prove or systematically verify that this training distribution covers the fully masked start used at test time.
standard math The number of placeholders can be modeled as a rounded draw from a Normal distribution N(µ, σ²).
Section 3.1 approximates the Binomial distribution of placeholder counts with a Normal and rounds |y|·P to the nearest integer; this is a standard approximation, though it introduces rounding error.
domain assumption Attention weights can be interpreted as evidence of which tokens the model uses.
Section 5 computes average attention to future placeholders and treats high attention as proof of planning ahead; this is a common but debated interpretability assumption.
domain assumption Applying the training loss to all output positions, including known tokens, is a valid objective for learning placeholder prediction.
The loss in Section 3 sums over all positions j, not only placeholders, which could let the model copy known tokens through self-attention; the paper does not justify why this does not interfere with learning to predict placeholders.

invented entities (1)

Placeholder token (p̄)
purpose: Serves as a stand-in for unknown output tokens during training and inference, so a Transformer encoder can attend to future positions before they are generated.
Introduced in Section 3. It is a modeling device with no external falsifiable handle; its role is internal to the mechanism, so it does not make an independently testable prediction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Attending to Future Tokens For Bidirectional Sequence Generation." pith.science (2026). https://pith.science/paper/7LV2POYQ

@misc{pith2026190805915,
  author       = {Pith},
  title        = {Pith review of: Attending to Future Tokens For Bidirectional Sequence Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7LV2POYQ}},
  note         = {Machine review of arXiv:1908.05915}
}

read the original abstract

Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, has been shown to perform much better. We propose to make the sequence generation process bidirectional by employing special placeholder tokens. Treated as a node in a fully connected graph, a placeholder token can take past and future tokens into consideration when generating the actual output token. We verify the effectiveness of our approach experimentally on two conversational tasks where the proposed bidirectional model outperforms competitive baselines by a large margin.

Figures

Figures reproduced from arXiv: 1908.05915 by the authors.

**Figure 2.** Heat map (darker hues indicate higher atten [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 4.** α¯ 2 (Backward Attention) and α¯ 3 (Forward Attention) across the 12 attention heads for the DAILY DIALOG dataset. B Heat Maps The heat maps (darker hues indicate higher attention) of Figures 5 and 6 show examples of where an attention head strongly looks into the future while generating from left to right. Each row shows the attention over the output sequence for this row’s placeholder token at that point in time.… view at source ↗

Figures from the paper (1 more)

**Figure 3.** Figure 3: α¯ 2 (Backward Attention) and α¯ 3 (Forward Attention) across the 12 attention heads for the SHARC dataset [PITH_FULL_IMAGE:figures/full_fig_p010_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 8 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. http://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . ArXiv e-prints, 1810.04805

arXiv 2018
[4]

Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. http://arxiv.org/abs/1904.09324 Constant- Time Machine Translation with Conditional Masked Language Models . arXiv:1904.09324 [cs, stat]. ArXiv: 1904.09324

arXiv 2019
[5]

Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019. http://arxiv.org/abs/1902.01370 Insertion-based decoding with automatically inferred generation order . CoRR, abs/1902.01370

arXiv 2019
[6]

Diederick P Kingma and Jimmy Ba. 2015. https://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In International Conference on Learning Representations (ICLR), San Diego, CA, USA

arXiv 2015
[7]

Klein , Y

G. Klein , Y. Kim , Y. Deng , J. Senellart , and A. M. Rush . 2017. https://arxiv.org/abs/1701.02810 OpenNMT: Open-Source Toolkit for Neural Machine Translation . ArXiv e-prints, 1701.02810

arXiv 2017
[8]

Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. http://aclweb.org/anthology/I17-1099 Dailydialog: A manually labelled multi-turn dialogue dataset . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJCNLP), Taipei, Taiwan

work page 2017

Show all 19 references

[9]

Liangchen Luo, Jingjing Xu, Junyang Lin, Qi Zeng, and Xu Sun. 2018. http://aclweb.org/anthology/D18-1075 An auto-encoder matching model for learning utterance-level semantic dependency in dialogue generation . In Proceedings of the 2018 Conference on Empirical Methods in Natur...

2018
[10]

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf Improving Language Understanding by Generative Pre-Training . Technical Report Tech...

2018
[11]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, and Dario Amodei. 2019. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language Models are Unsupervised Multitask Learners . Technical report, OpenAI

2019
[12]

Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rockt \"a schel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. http://aclweb.org/anthology/D18-1233 Interpretation of natural language rules in conversational machine reading . In Proceedings of the ...

2018
[13]

Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. http://arxiv.org/abs/1902.03249 Insertion transformer: Flexible sequence generation via insertion operations . CoRR, abs/1902.03249

2019 arXiv
[14]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is All you Need . In Advances in Neural Information Processing Systems 30 (NIPS)

2017
[15]

Sean Welleck, Kiant \' e Brantley, Hal Daum \' e III, and Kyunghyun Cho. 2019. http://arxiv.org/abs/1902.02192 Non-monotonic sequential text generation . CoRR, abs/1902.02192

2019 arXiv
[16]

Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. http://arxiv.org/abs/1901.08149 TransferTransfo : A Transfer Learning Approach for Neural Network Based Conversational Agents . ArXiv e-prints, 1901.08149

2019 arXiv
[17]

Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. https://www.aclweb.org/anthology/P18-1205 Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computati...

2018
[18]

Victor Zhong and Luke Zettlemoyer. 2019. https://www.aclweb.org/anthology/P19-1223 E 3: Entailment-driven extracting and editing for conversational machine reading . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy

2019
[19]

Long Zhou, Jiajun Zhang, and Chengqing Zong. 2019. https://doi.org/10.1162/tacl\_a\_00256 Synchronous bidirectional neural machine translation . Transactions of the Association for Computational Linguistics, 7:91--105

2019 doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. http://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . ArXiv e-prints, 1810.04805

arXiv 2018

[4] [4]

Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. http://arxiv.org/abs/1904.09324 Constant- Time Machine Translation with Conditional Masked Language Models . arXiv:1904.09324 [cs, stat]. ArXiv: 1904.09324

arXiv 2019

[5] [5]

Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019. http://arxiv.org/abs/1902.01370 Insertion-based decoding with automatically inferred generation order . CoRR, abs/1902.01370

arXiv 2019

[6] [6]

Diederick P Kingma and Jimmy Ba. 2015. https://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In International Conference on Learning Representations (ICLR), San Diego, CA, USA

arXiv 2015

[7] [7]

Klein , Y

G. Klein , Y. Kim , Y. Deng , J. Senellart , and A. M. Rush . 2017. https://arxiv.org/abs/1701.02810 OpenNMT: Open-Source Toolkit for Neural Machine Translation . ArXiv e-prints, 1701.02810

arXiv 2017

[8] [8]

Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. http://aclweb.org/anthology/I17-1099 Dailydialog: A manually labelled multi-turn dialogue dataset . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJCNLP), Taipei, Taiwan

work page 2017