Pith. sign in

REVIEW 3 major objections 5 minor 26 references

TransSent: Towards Generation of Structured Sentences with Discourse Marker

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that generating a structured sentence—two clauses joined by a discourse marker—becomes easier when the model separates meaning from structure.

desk verdict The paper contributes a genuinely new task and three datasets for discourse transfer, but the training objective never supervises the decoded translated tail, so the central generation mechanism is unverified and the reported numbers likely outrun the model description. read the letter →

arxiv 1909.05364 v3 pith:G6GANQOS submitted 2019-09-05 cs.CL cs.AI

Xing Wu , Dongjun Wei , Liangjun Zang , Jizhong Han , Songlin Hu This is my paper · ORCID
classification cs.CLcs.AI
keywords discoursetransfermarkersstructuredsentencegenerationtranslation-basedrelationmodelingBERTvariationalautoencoderdialogue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that generating a structured sentence—two clauses joined by a discourse marker—becomes easier if the model never mixes meaning and structure in one hidden vector. It defines a 'discourse transfer' task: given a head clause and a marker such as 'because' or 'but,' the model must produce a grammatical tail clause that preserves the stated relation. The proposed TransSent system encodes the head clause, translates its representation into a relation-specific space, and decodes the tail clause. Evaluations on three corpora and on free-text and dialogue tasks report higher relation accuracy and better human ratings than sequence-to-sequence, VAE, and planning baselines.

What carries the argument

The load-bearing object is the relation translation network together with the two distance losses. In discourse space, head and tail clauses have vectors z_h and z_t; a learned matrix M_r projects them into a relation space, and the translated tail is W'[z_h M_r ⊕ z_r]. The L_dis loss minimizes the squared distance to the projected real tail, while L_ratio minimizes the ratio of that distance to the distance from the head, forcing the translation to point toward the tail rather than stay near the head. These losses, added to a reconstruction loss, form the training objective in Equation 10.

What would settle it

Train an independent discourse-marker classifier on the same DMP data but with a different architecture and no exposure to TransSent training, then run TransSent's generated tails through it. If its accuracy stays near the reported 50–80% on open-domain data, the metric is robust; if it falls toward baseline levels, the shared BERT scorer inflated the result.

Watch

Extended reading notes

Core claim

The central claim is that a discourse marker between two clauses can be modeled as a translation operation on sentence embeddings, analogous to knowledge-graph translation models. TransSent's encoder-decoder reconstructs both head and tail clauses while a relation network projects clause representations into a relation-specific space and requires the translated tail representation to be close to the actual tail (L_dis) and farther from the head than from the tail (L_ratio). The training objective couples these losses with reconstruction. The paper claims that this explicit separation lets the model generate grammatical tails that preserve the intended relation, and that the same module transfers to free text and dialogue generation when paired with VAE or CVAE.

Load-bearing premise

The automatic relation-accuracy metric is computed by the same fine-tuned BERT that encodes the discourses, so if that BERT's relation space favors TransSent's own outputs, the reported accuracy gains could be inflated.

Editorial extensions

If this is right

  • Structured sentences can be built compositionally: generate a head clause, pick a discourse marker, transfer to a tail clause, and concatenate the three parts.
  • The model can be combined with existing generation models such as VAE and CVAE, so discourse structure can be added to free-text and dialogue systems without retraining them from scratch.
  • Relation accuracy is notably lower on open-domain corpora than on a single-domain review corpus, suggesting that relation translation is harder when discourse content is diverse and pointing to larger or more targeted datasets as the next step.
  • The fine-tuned BERT used for DMP doubles as an automatic relation-accuracy scorer, so the same representation that guides generation also judges it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shared BERT scorer is biased toward TransSent's own outputs, the reported accuracy gains would shrink; an independent relation classifier trained without TransSent in the loop would settle this.
  • The translation analogy suggests that discourse markers could be composed recursively: chaining multiple markers would yield multi-clause sentences, which the paper itself lists as future work.
  • The ratio loss is a margin-like constraint that may generalize to other structured prediction tasks where a transformation should move a source representation toward a target rather than merely away from the source.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript introduces a new task, discourse transfer, in which a model receives a head discourse and an explicit discourse marker (and, but, because, if, when) and must generate the tail discourse of a structured sentence. The authors propose TransSent, an encoder-decoder model with a frozen BERT encoder, a unidirectional LSTM decoder, and a relation translation network inspired by TransR. The relation network projects head and tail discourse representations into a separate relation space, performs a translation-like operation using the discourse-marker embedding, and maps the result back to discourse space for decoding. The training loss in Eq. (10) combines a reconstruction loss on genuine head/tail encodings with two relation-space losses. The paper reports experiments on three datasets (Yelp-dm, Wiki-dm, Book-dm) for discourse transfer, free text generation, and dialogue generation, and claims that TransSent produces structured sentences with high quality, based on automatic metrics (ACC, NLL, PPL) and human ratings.

Significance. If the proposed mechanism worked as described, explicit discourse-marker-conditioned generation would be a useful step toward controllable and structured text generation, and the analogy to knowledge-graph translation would be an interesting architectural idea. The three constructed datasets and the task formulation are potentially reusable resources. However, the central training objective does not supervise the actual generation path (decoding from translated representations), and the automatic evaluation metric uses the same fine-tuned BERT as the model encoder. These are load-bearing issues in the current manuscript: the strong quantitative claims are not yet supported by the described method. The significance would be substantially improved by adding a loss that trains the decoder on translated representations, reporting an independent automatic metric, and providing ablations that separate the contribution of the relation translation from the benefit of the pretrained BERT encoder.

major comments (3)
  1. [§4.2, Eq. (10)] The training objective never supervises the decoder on the translated representation. In Eq. (4), Lrec only reconstructs sh and st from their genuine encodings zh and zt; Ldis in Eq. (8) and Lratio in Eq. (9) are computed entirely in relation space (between zt*r and ztr, and between zt*r and zhr). The feed-forward inverse mapping that produces zt* and the decoder Gdec that produces st* receive no gradient from these terms. Therefore, at inference the decoder is asked to decode vectors zt* that are drawn from a distribution on which it was never trained. This is the load-bearing path for the paper's central claim that TransSent generates coherent, relation-correct tails. A loss such as log p(st | zt*) or an explicit distributional alignment between zt* and zt is needed; alternatively, the authors should provide an experiment or analysis showing that the untrained decoding path nevertheless works.
  2. [§6.3, Table 3] The automatic ACC metric uses the same fine-tuned BERT that serves as the frozen encoder in §4.1. Because that BERT was fine-tuned on the DMP task and then used as the discriminator, ACC is not an independent measure of discourse correctness; it measures agreement with the model's own representation learner and can therefore overstate the relation accuracy of TransSent outputs. The human evaluation is too small to compensate: five annotators rate 200 examples per model/dataset, with no inter-annotator agreement and no confidence intervals, and the average ratings (3.0-3.4 out of 5.0) are modest. The authors should report ACC with a separately trained classifier and provide human-evaluation statistics, e.g., confidence intervals or agreement measures.
  3. [§6.1, §6.4] All baselines use random-initialized RNN/LSTM encoder-decoder architectures, while TransSent uses a frozen BERT encoder. The large gains in ACC, PPL, and NLL may therefore be attributable to the pretrained encoder rather than to the proposed relation translation network. No ablation is reported that isolates the contribution of the relation translation, such as a version of TransSent without the relation path or with a simple additive translation in discourse space. An ablation is needed to support the claim that translation in relation space is the source of improvement.
minor comments (5)
  1. [§4.1, Eq. (3)] The notation p(st) in Eq. (3) is ambiguous: the decoder should condition on the encoded representation, so p(st | zh) or similar would be clearer. Also, Eq. (3) appears to define the decoder for zh only, while the same decoder is used for zt; this should be stated explicitly.
  2. [Throughout] There are several typos: 'nolinear' should be 'nonlinear' in §1 and §4; 'comparision' should be 'comparison' in §6.3; 'CA VE' should be 'CVAE' in §2 and §6.1.
  3. [§6.2] The paper states that beam search is not used, but it does not specify the decoding procedure (e.g., maximum-likelihood decoding or sampling). This detail is needed for reproducibility.
  4. [Table 3 and Table 4] The tables report mean scores without error bars or significance tests. Given the modest differences on some cells, statistical significance should be reported.
  5. [Algorithm 1] In Algorithm 1, line 8 says 'Calculate Ldis and Lratio' and line 9 'Calculate L', but Eq. (10) is the total objective. It would be clearer to separate the network parameters that are updated by each loss term, since the decoder and inverse mapping are not updated by the relation-space losses.

Circularity Check

1 steps flagged · score 4.0 of 10

Main automatic metric is partially self-referential (same BERT as encoder and ACC scorer), but human evaluation and other metrics keep the central claim from reducing to a fit.

  1. fitted input called prediction [Section 4.1 (Encoder-Decoder Network), Section 6.2 (Experiment Details), Section 6.3 (Accuracy), Eq. 8-10]
    "The fine-tune process can also be considered as teaching BERT to understand the structural relations between discourse pairs. One extra benefit is that the fine-tuned BERT is also used for automatic evaluation, to judge whether the discourse marker within a structured sentences is correct or not. ... We fix the weights of the fine-tuned BERT and use the fine-tuned Bert as our encoder and discriminator. ... Accuracy We use the fine-tuned BERT as the discriminator to assess whether relation between discourses in generated sentences holds."

    The ACC metric is not an independent yardstick: the same DMP-fine-tuned BERT supplies both the fixed discourse representations used by the relation-translation objective (Eq. 8-10) and the discriminator that scores the generated tails. The model is trained to minimize Ldis and Lratio in the relation space defined by this BERT, i.e., to make the translated representation look like a true tail representation in exactly the space the DMP classifier reads. Therefore a high ACC partly reports success on the model's own training objective, not an external property of the generated text. The paper introduces this double use as a 'benefit,' so the metric is coupled by construction to the model's representation space.

full rationale

TransSent's generation path is not supervised: Eq. 10 sums Lrec (Eq. 4), which only evaluates log p(sh|zh) and log p(st|zt), with Ldis and Lratio (Eqs. 8-9), which act in the relation space of the fixed BERT. No term supervises the feed-forward inverse mapping or the decoder on the translated representation zt*, so the claim that s*t = Gdec(zt*) is well generated rests on an untested closeness assumption. This is a training-objective gap, not itself a circularity. The circular component is in the ACC evaluation: the same fine-tuned BERT is both the encoder that defines the relation space for Ldis/Lratio and the discriminator that scores generated pairs. The reported ACC is therefore partly a self-consistency score against the model's own representation space, not an independent text-quality measurement. That said, the paper also reports PPL/NLL from a language model and an independent, though small (five annotators, 200 samples), human evaluation with ratings around 3/5, so not every load-bearing claim reduces to the self-referential metric. No load-bearing self-citations or imported uniqueness theorems were found. Overall, the central mechanism has independent content, but the headline automatic evidence is partially circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central mechanism depends on a geometric translation hypothesis and on a representation space learned by fine-tuned BERT; both are assumed rather than derived. The only tunable scalars reported are dimension 768, dropout 0.1, and 20 training epochs, but the critical loss weights λd and λr are missing. No new physical or linguistic entities are postulated; the relation space is a latent construction, not an entity with independent falsifiable handles.

free parameters (2)
  • loss weights λd and λr = not reported
    They balance reconstruction, L2 distance, and ratio losses in Eq. 10; the paper does not state their values, yet the final generation quality depends on them.
  • relation embeddings zr for five discourse markers = not reported
    Each of the five markers (and, but, because, if, when) gets a learned vector used in Eq. 7; these are fitted to the data and are central to the translation mechanism.
assumptions (4)
  • domain assumption There exists a mapping f(zh, zr) ≈ zt (Eq. 5), so a discourse relation behaves like a translation in embedding space.
    This is the core modeling hypothesis borrowed from TransE/TransR. If sentence-level semantics do not support additive translation, the relation network cannot learn the intended mapping.
  • domain assumption The BERT model fine-tuned on the DMP task produces a representation space that is good enough for both encoding discourses and judging whether a generated tail has the correct relation.
    Section 4.1 freezes this BERT as the encoder, and Section 6.3 uses the same BERT as the ACC discriminator. The validity of the main automatic metric rests on this assumption.
  • domain assumption Discourse pairs extracted from Yelp, WikiText-103, and BookCorpus using dependency parsing and explicit markers are accurate and representative.
    Section 5 relies on CoreNLP parses and filters lengths to 5 to 15 words. Parsing errors or domain bias would propagate into training and evaluation.
  • domain assumption Closeness in L2 distance in relation space corresponds to semantic relation correctness.
    The losses in Eqs. 8 and 9 push the translated representation toward ztr and away from zhr, but no theoretical guarantee connects L2 closeness to human judgments of discourse relations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TransSent: Towards Generation of Structured Sentences with Discourse Marker." pith.science (2026). https://pith.science/paper/G6GANQOS

@misc{pith2026190905364,
  author       = {Pith},
  title        = {Pith review of: TransSent: Towards Generation of Structured Sentences with Discourse Marker},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6GANQOS}},
  note         = {Machine review of arXiv:1909.05364}
}
read the original abstract

Structured sentences are important expressions in human writings and dialogues. Previous works on neural text generation fused semantic and structural information by encoding the entire sentence into a mixed hidden representation. However, when a generated sentence becomes complicated, the structure is difficult to be properly maintained. To alleviate this problem, we explicitly separate the modeling process of semantic and structural information. Intuitively, humans generate structured sentences by directly connecting discourses with discourse markers (such as and, but, etc.). Therefore, we propose a task that mimics this process, called discourse transfer. This task represents a structured sentence as (head discourse, discourse marker, tail discourse), and aims at tail discourse generation based on head discourse and discourse marker. We also propose a corresponding model called TransSent, which interprets the relationship between two discourses as a translation1 from the head discourse to the tail discourse in the embedding space. We experiment TransSent not only in discourse transfer task but also in free text generation and dialogue generation tasks. Automatic and human evaluation results show that TransSent can generate structured sentences with high quality, and has certain scalability in different tasks.

Figures

Figures reproduced from arXiv: 1909.05364 by the authors.

Figure 1
Figure 1. (a) An example of discourse transfer. (b) An application [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The overall model architecture of TransSent. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 2
Figure 2. Simple illustration of TransSent. Inspired by the success of these translation models, we per￾form the similar translation between two discourses in the embedding space, in order to simulate the semantic relation￾ship indicated by a discourse marker, as shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 17 canonical work pages

  1. [1]

    Translating embeddings for modeling multi-relational data

    [Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787–2795,

    work page 2013

  2. [5]

    Long text generation via adversarial training with leaked information

    [Guo et al., 2018] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Thirty- Second AAAI Conference on Artificial Intelligence,

    work page 2018

  3. [6]

    Literature and cognition

    [Hobbs, 1990] Jerry R Hobbs. Literature and cognition. (21),

    work page 1990

  4. [12]

    Learning entity and relation embeddings for knowledge graph completion

    [Lin et al., 2015] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Twenty- ninth AAAI conference on artificial intelligence,

    work page 2015

  5. [13]

    Pointer sentinel mixture models

    [Merity et al., 2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843,

    arXiv 2016

  6. [14]

    Sequence to Sequence Learning for Event Prediction

    [Nguyen et al., 2017] Dai Quoc Nguyen, Dat Quoc Nguyen, Cuong Xuan Chu, Stefan Thater, and Manfred Pinkal. Se- quence to sequence learning for event prediction. arXiv preprint arXiv:1709.06033,

    work page Pith review arXiv 2017

  7. [15]

    Bennett, and Noah D

    [Nie et al., 2017] Allen Nie, Erin D. Bennett, and Noah D. Goodman. Dissent: Sentence representation learning from explicit discourse relations. CoRR, abs/1710.04334,

    arXiv 2017

  8. [16]

    Ad- versarial generation of natural language

    [Rajeswar et al., 2017] Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, and Aaron Courville. Ad- versarial generation of natural language. arXiv preprint arXiv:1705.10929,

    arXiv 2017

Show all 26 references
  1. [17]

    Enhanced english universal dependencies: An improved representation for natural language understanding tasks

    [Schuster and Manning, 2016] Sebastian Schuster and Christopher D Manning. Enhanced english universal dependencies: An improved representation for natural language understanding tasks. In LREC, pages 23–28. Portoroˇz, Slovenia,

    2016

  2. [18]

    Long and diverse text generation with planning-based hierarchical varia- tional model

    [Shao et al., 2019] Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Xu, and Xiaoyan Zhu. Long and diverse text generation with planning-based hierarchical varia- tional model. arXiv preprint arXiv:1908.06605,

    2019 arXiv

  3. [20]

    Policy gradient methods for reinforcement learning with function approxi- mation

    [Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxi- mation. In Advances in neural information processing sys- tems, pages 1057–1063,

    2000

  4. [22]

    Modeling coherence for discourse neu- ral machine translation

    [Xiong et al., 2018] Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. Modeling coherence for discourse neu- ral machine translation. arXiv preprint arXiv:1811.05683,

    2018 arXiv

  5. [23]

    Unsupervised text style transfer using language models as discrimina- tors

    [Yang et al., 2018] Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. Unsupervised text style transfer using language models as discrimina- tors. In Advances in Neural Information Processing Sys- tems, pages 7287–7298,

    2018

  6. [24]

    Seqgan: Sequence generative adversarial nets with policy gradient

    [Yu et al., 2017] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence,

    2017

  7. [25]

    Learning discourse-level diversity for neural dialog models using conditional variational autoencoders

    [Zhao et al., 2017] Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv:1703.10960,

    2017 arXiv

  8. [26]

    Aligning books and movies: Towards story- like visual explanations by watching movies and reading books

    [Zhu et al., 2015] Yukun Zhu, Ryan Kiros, Rich Zemel, Rus- lan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story- like visual explanations by watching movies and reading books. In Proceedings of the IEEE international c...

    2015

  9. [1990]

    Long short-term memory

    [Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780,

    1997

  10. [1997]

    Toward controlled generation of text

    [Hu et al., 2017] Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. InProceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1587–

    2017

  11. [2000]

    Knowledge graph embedding by translating on hyperplanes

    [Wang et al., 2014] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Twenty-Eighth AAAI con- ference on artificial intelligence,

    2014

  12. [2013]

    Discourse-aware neural rewards for coherent text generation

    [Bosselut et al., 2018] Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. Discourse-aware neural rewards for coherent text generation. arXiv preprint arXiv:1805.03766,

    2018 arXiv

  13. [2014]

    Controlling output length in neural encoder-decoders

    [Kikuchi et al., 2016] Yuta Kikuchi, Graham Neubig, Ry- ohei Sasano, Hiroya Takamura, and Manabu Okumura. Controlling output length in neural encoder-decoders. arXiv preprint arXiv:1609.09552,

    2016 arXiv

  14. [2015]

    BERT: pre-training of deep bidirectional transformers for language understand- ing

    [Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. CoRR, abs/1810.04805,

    2018 arXiv

  15. [2016]

    Adam: A method for stochastic optimization

    [Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    2014 arXiv

  16. [2017]

    Sequence to sequence learning with neu- ral net- works

    [Ilya Sutskever and Le, 2014] Oriol Vinyals Ilya Sutskever and Quoc V Le. Sequence to sequence learning with neu- ral net- works. In Advances in neural information process- ing systems,

    2014

  17. [2018]

    Generating sentences from a continuous space

    [Bowman et al., 2015] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349,

    2015 arXiv

  18. [2019]

    Towards generating long and coherent text with multi-level latent variable models.arXiv preprint arXiv:1902.00154,

    [Shen et al., 2019] Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, and Lawrence Carin. Towards generating long and coherent text with multi-level latent variable models.arXiv preprint arXiv:1902.00154,

    2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.