{"id":"b8de740a-139e-4e80-ba79-a260c0425aec","arxiv_id":"1909.00142","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DiscoEval is a new benchmark for measuring discourse awareness in sentence embeddings, and Wikipedia-structure training losses modestly change, but do not beat, pretrained encoders.","lead":"The paper introduces DiscoEval, a suite of seven tasks that test whether sentence embeddings capture discourse-level information such as sentence position, ordering, and coherence. It also shows that training sentence encoders to predict Wikipedia document structure can shift what the embeddings encode, though pretrained models like BERT still score highest overall.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DiscoEval's construct validity is unestablished: several tasks may be solvable by sentence-internal lexical cues rather than broader context, and the paper reports no surface baselines to rule this out.","rationale":"The reader's weakest assumption concerns the linear probe: logistic regression may not reveal discourse information encoded non-linearly. This is a real limitation, but the paper partly addresses it by using a hidden layer for DC and by explicitly discussing the linear-probe choice, and it is a known limitation of probing benchmarks generally. The more fundamental and less addressed issue is discriminant validity: whether the tasks measure discourse context at all, or whether they can be solved by sentence-internal or shallow lexical cues. The paper's own control experiment (Table 6) shows that context contributes only about 4 accuracy points in SP, suggesting a potential construct-validity problem. No simple lexical baselines are reported, so the alternative explanation that high scores come from surface cues has not been ruled out. If the proposed TF-IDF baseline matches the neural embeddings on several tasks, the central claim that DiscoEval measures 'broader context information' would be substantially weakened; if it does not, the concern is resolved and the benchmark is more credible. Either way, the appropriate outcome is the reader's conditional acceptance: the benchmark is a useful contribution, but it should be strengthened by adding surface baselines and, if needed, redesigning tasks like SSP and SP to be more context-dependent. I do not see grounds to reject the paper outright, since the datasets and training objectives may still be valuable, and the authors are transparent about limitations. Hence I leave the verdict unchanged while flagging a load-bearing concern that differs in emphasis from the reader's linear-probe worry.","tokens_in":17480,"tokens_out":6152,"duration_ms":136128,"concrete_test":"Run a unigram TF-IDF logistic regression baseline on every DiscoEval task, using the same train/dev/test splits and the same input construction (for SSP, the sentence's bag of words; for SP/BSO/DC, concatenated bags of the relevant sentences; for PDTB/RST-DT, bags of the two inputs). Compare to BERT-Large and the proposed models. If the lexical baseline matches or exceeds the best neural embeddings on SSP, SP, and PDTB-E, the benchmark primarily measures surface cue accessibility rather than discourse awareness. Additionally, for SP, compute the lexical baseline with and without surrounding sentences; if the context gain is as small as the 4.1-point gain observed for the neural baseline, the task is not strongly discourse-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that DiscoEval evaluates whether sentence representations 'include broader context information' (Abstract). The load-bearing assumption is that the tasks genuinely require discourse context, not just lexical or stylistic cues available within a single sentence. This assumption is not established. The SSP task (Sec. 3.5) feeds only the sentence embedding and asks whether it comes from the Abstract or a middle section; the authors themselves note cues like 'Empirically is usually used in the abstract.' Such a task can be solved from sentence-internal register/style, with no broader context. In the SP task, the 'w/o context' baseline (target sentence only) already reaches 43.2% versus 20% random, and adding context improves to only 47.3% (Table 6). Thus most of the SP signal is sentence-internal, weakening the claim that the task probes broader context. For PDTB-E, the explicit connective is removed, but other discourse-marking words may remain. The paper reports no bag-of-words, TF-IDF, or other feature-based baselines on any DiscoEval task, so it is unknown whether high scores reflect discourse knowledge or shallow surface statistics. This is a construct-validity gap: if the tasks are solvable by simple lexical cues, the benchmark does not measure what it claims, and the empirical conclusions about BERT/ELMo and the proposed training objectives are underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiscoEval, a suite of seven probing tasks for evaluating whether sentence representations encode discourse-related information: discourse relations (PDTB-E, PDTB-I, RST-DT), sentence position (SP), binary sentence ordering (BSO), discourse coherence (DC), and sentence section prediction (SSP). It also proposes auxiliary training objectives for sentence encoders trained on Wikipedia: NSP, nesting level (NL), sentence/paragraph position (SPP), and section/document title prediction (SDT). The authors benchmark their models against Skip-thought, InferSent, DisSent, ELMo, and BERT on DiscoEval and SentEval. The main empirical findings are that the NL+SPP combination gives a small average improvement over the NSP baseline, SDT tends to hurt discourse-task performance, and BERT/ELMo perform strongly, with deeper layers better on DiscoEval than on SentEval.","tokens_in":17793,"tokens_out":8935,"duration_ms":82835,"significance":"If the benchmark's validity were established, DiscoEval would fill a real gap: existing sentence-embedding benchmarks mostly test stand-alone sentence meaning, and a discourse-focused suite with public code and data would be widely used. The paper is commendably transparent about negative results, makes data and scripts available, and provides a per-layer analysis that is a useful reference. The main risk is that the tasks may partly measure sentence-internal lexical or register cues rather than broader-context information, and the training-objective comparisons lack variance estimates. The contribution is promising as a resource and an empirical study, but the central claims need additional support.","major_comments":[{"comment":"The benchmark's central claim is that DiscoEval evaluates whether sentence representations include broader context information (Abstract). This is not yet established. In SSP (Section 3.5), the classifier receives only the sentence embedding, and the authors themselves note that a within-sentence cue such as 'Empirically' is predictive of the Abstract section; the task can therefore be solved as sentence-internal register classification. In SP (Table 6), the 'w/o context' baseline reaches 43.2% versus 20% random, and adding the four surrounding sentences yields only 47.3%, so most of the signal is sentence-internal. I ask for non-neural surface baselines (bag-of-words, TF-IDF, n-grams, and randomized embeddings) on every DiscoEval task so the context-dependent component is quantified; without them, the high scores of BERT/ELMo and the training-objective effects are underdetermined.","section":"Section 3.2, Section 3.5, Table 6"},{"comment":"The conclusions about the proposed training objectives rest on small differences with no error bars or significance tests. The NL+SPP model beats the NSP baseline by 0.7% on the DiscoEval average, and the claim that SDT hurts performance rests on similarly small margins. Moreover, the clearest gains for NL+SPP are on SP and BSO derived from Wikipedia, the same corpus used for the proposed training objectives, while the external human-annotated tasks show mixed or negligible effects (e.g., PDTB-I 39.1 to 40.5; RST-DT 56.7 to 56.4 for the NSP baseline vs. NL+SPP). Please report standard deviations across classifier seeds and a breakdown separating Wikipedia-derived tasks from external tasks, and temper the generalization claims accordingly.","section":"Section 5.2, Table 2"}],"minor_comments":[{"comment":"The citations for GloVe and Adam appear as unresolved placeholders ('?'); these need to be filled in, and the typo 'intialize' should be corrected.","section":"Appendix A"},{"comment":"The human evaluation is based on a single native-English-speaking annotator with 50 examples per domain and no inter-annotator reliability information; the human accuracies should be presented as indicative rather than definitive.","section":"Human Evaluation, Table 5"},{"comment":"The DC task uses a 2000-unit hidden layer while all other tasks use logistic regression; this difference in classifier capacity should be flagged in the main results table so readers do not compare the DC column directly with the other columns.","section":"Section 3.4, Table 2"},{"comment":"For the main BERT and ELMo results the paper averages across layers, but the per-layer analysis uses [CLS] for BERT and layer-specific averages for ELMo; please clarify whether the pooling is otherwise identical so the layer comparison is clean.","section":"Section 5.1"},{"comment":"Because each column is standardized separately, cross-column differences in raw accuracy are not visually comparable; adding raw values or a shared color scale would help.","section":"Figure 8"},{"comment":"The 'w/o context' condition is not defined precisely (input x1 only, or the full input with the x1 - xi terms removed?); please state the exact input representation.","section":"Section 3.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The resource is potentially useful and the honest reporting is appreciated. The main risk is that the benchmark may be adopted without its validity being established; I therefore see major revision as the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth knowing about. The new thing is DiscoEval, a suite of seven probing tasks for whether sentence embeddings encode discourse context, plus a set of training objectives (nesting level, sentence/paragraph position, section/document title) derived from Wikipedia's structure. It fills a real gap: SentEval mostly probes stand-alone sentences. They ship code and data, which matters. The experiments are solid for 2019 practice: frozen encoders, simple classifiers, and they report mixed results honestly—SDT hurts, NL+SPP helps by 0.7%, BERT and ELMo come out strong. That honesty makes the empirical claims credible.\n\nThe soft spots are real but not fatal. The biggest is construct validity. Several tasks can be solved from sentence-internal cues. SSP is essentially a register/style task—the paper itself notes 'Empirically is usually used in the abstract.' SP's w/o-context baseline is already 43.2% versus 20% random, and adding context only gets 47.3%. That means most of the SP signal is inside the target sentence, not broader context. There are no bag-of-words or TF-IDF baselines anywhere in DiscoEval, so we cannot tell how much of a model's score comes from simple lexical statistics. That weakens the interpretation that high DiscoEval scores mean discourse awareness.\n\nOther issues are minor. There are no error bars or significance tests—typical for the era, but it makes the 0.7% NL+SPP gain hard to interpret. The human evaluation uses one annotator, so the human-vs-model gaps are suggestive only. There is also mild circularity: SP-Wiki, BSO-Wiki, and DC-Wiki come from the same Wikipedia structure the training objectives use, so part of the measured benefit may be domain overlap. The paper mitigates this by including external datasets (PDTB, RST-DT, Ubuntu), and the PDTB-I improvement appears on data the model never saw.\n\nWho is this for? Anyone building or evaluating sentence encoders. The suite is useful as a secondary evaluation even if the construct-validity questions are unresolved. I would send it to peer review—the contributions are concrete, the writing is clear, and the weaknesses are addressable in revision rather than load-bearing. The main thing I would ask the authors to add is lexical baselines and an SP context-importance analysis; that would settle most of my concerns.","headline":"DiscoEval is a genuinely useful discourse-focused probing suite plus Wikipedia-derived training objectives, honestly evaluated; its main weakness is that several tasks may reward lexical/register cues rather than broader context, and the paper provides no surface baselines to rule that out.","tokens_in":18276,"tokens_out":1678,"would_cite":true,"duration_ms":16443,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes DiscoEval, a seven-task benchmark for whether sentence representations encode discourse context, and shows that Wikipedia-derived training objectives and deep BERT/ELMo layers capture different aspects of it.","keywords":["DiscoEval","discourse-aware sentence representations","sentence embedding evaluation","probing tasks","discourse relations","sentence ordering","discourse coherence","training objectives"],"falsifier":"On the same frozen embeddings, replace the logistic-regression probe with a single-hidden-layer probe (as the paper itself already does for the Discourse Coherence task) on PDTB-I and RST-DT. If accuracy rises substantially above the Table 2 numbers, DiscoEval's ranking reflects linear accessibility, not whether discourse information is present in the representation.","tokens_in":17289,"feed_emoji":"🧠","tokens_out":7393,"duration_ms":63657,"temperature":0.7,"pith_summary":"Most sentence-embedding benchmarks judge a representation on standalone semantics; this paper argues that a sentence's role in its surrounding discourse is a separate, measurable capacity. It introduces DiscoEval, a suite of seven probing tasks—discourse relations from two hand-annotated treebanks, sentence position, binary sentence ordering, discourse coherence, and section prediction—scored by training simple classifiers on frozen sentence vectors. It also proposes training objectives that exploit Wikipedia's natural structure: predicting a sentence's nesting level, its position in paragraph and document, and its section and document titles. The empirical claim is that these objectives push encoders to capture different, complementary aspects of document structure, and that on DiscoEval deep layers of BERT and ELMo consistently outperform shallow layers, the opposite of what existing sentence-level evaluation suites tend to show.","feed_headline":"DiscoEval: seven tasks probe discourse in sentence vectors","feed_subtitle":"A seven-task suite checks whether sentence embeddings capture relations, order, and coherence, not just meaning.","key_machinery":"The central object is DiscoEval itself: a collection of seven probing task groups in which a frozen sentence encoder produces vectors and a logistic-regression classifier (one task uses a 2000-unit hidden layer) must predict discourse-related targets—RST and PDTB relations, a shuffled sentence's original position among five, the correct order of two sentences, whether six sentences form a coherent passage, and whether a sentence comes from an abstract. The companion machinery is a multi-task training setup with one Bi-GRU encoder and decoders predicting neighboring sentences, nesting level, sentence and paragraph position, and section and document titles. DiscoEval supplies the measurement; the losses supply the learning signal.","core_discovery":"DiscoEval is presented as a valid probe of whether sentence representations include discourse information. Across seven task groups built from Wikipedia, stories, dialogues, and scientific papers, the paper benchmarks fixed sentence encoders and finds that BERT-Large has the highest average DiscoEval accuracy, that ELMo is competitive, and that models trained with neighboring-sentence prediction plus two structure-aware losses—nesting level combined with sentence and paragraph position—improve over the neighboring-sentence baseline, while the section-and-document-title loss tends to hurt position-related tasks. Per-layer analysis shows that for BERT and ELMo, deeper layers perform best on DiscoEval, in contrast to sentence-level evaluation where shallower layers lead.","pith_inferences":["If the linear-probe assumption is relaxed, some encoders now ranked low on DiscoEval might turn out to encode discourse information nonlinearly; reporting both linear and nonlinear probe scores would separate representation content from probe accessibility.","The layer-depth result suggests discourse structure is a higher-level abstraction in contextual encoders; one testable consequence is that freezing deeper BERT layers should help downstream discourse-heavy tasks such as summarization and coherence ranking.","The same Wikipedia-derived objectives could transfer to other structured corpora with section headers or positions, such as legal or scientific documents, and the DiscoEval tasks could be re-run there to test generality.","A simple extension would be to anneal or gate the section-title loss during training so the model gets topical signal early without erasing the positional distinctions that sentence-position tasks require."],"forward_implications":["Sentence-embedding evaluation that stops at standalone semantics misses a whole axis: the same representation that looks weak on semantic similarity can be strong on discourse tasks, so future benchmarks should include both.","Discourse-aware training does not require expensive annotation: Wikipedia's headings, positions, and nesting levels give free labels, and combining position and nesting losses is better than either alone.","For BERT and ELMo, the best layer for discourse probing is consistently deep, so applications that need discourse structure should select layers accordingly rather than defaulting to the pooled output.","The section-and-document-title objective's negative effect on position and ordering tasks suggests a real trade-off between topical invariance and positional sensitivity in learned sentence vectors.","DiscoEval gives a shared measurement that future discourse-aware encoders can be compared against, alongside the released preprocessing and evaluation scripts."],"supporting_citations":[{"why":"Provides the SentEval evaluation framework that DiscoEval builds on and the sentence-level benchmark that lacks discourse tasks.","marker":"Conneau and Kiela, 2018"},{"why":"Supplies the Skip-thought neighboring-sentence prediction objective that the paper's baseline NSP model is built upon.","marker":"Kiros et al., 2015"},{"why":"Provides BERT, the best-performing pretrained encoder on DiscoEval and the subject of the per-layer analysis.","marker":"Devlin et al., 2019"},{"why":"Provides ELMo, a strong DiscoEval performer and the other model analyzed per layer.","marker":"Peters et al., 2018a"},{"why":"Supplies the RST Discourse Treebank with human-annotated discourse trees used for the RST-DT discourse relation task.","marker":"Carlson et al., 2001"},{"why":"Supplies the Penn Discourse Treebank with explicit and implicit relation annotations used for the PDTB tasks.","marker":"Prasad et al., 2008"},{"why":"Supplies the ROC Stories corpus used to construct the Sentence Position and Binary Sentence Ordering tasks.","marker":"Mostafazadeh et al., 2016"},{"why":"Supplies the human-annotated Ubuntu conversation thread structure used for the Discourse Coherence task.","marker":"Kummerfeld et al., 2019"},{"why":"Supplies PeerRead, the scientific-paper corpus used to construct the Sentence Section Prediction task.","marker":"Kang et al., 2018"}],"fun_headline_variants":["DiscoEval: seven tasks probe discourse in sentence embeddings","BERT and ELMo lead benchmark for discourse-aware sentence vectors","New test suite checks if sentence representations capture context","Training objectives boost discourse modeling in sentence embeddings","Deep layers shine for discourse, shallow for syntax in BERT and ELMo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a logistic-regression classifier trained on frozen sentence vectors tells us what a representation 'includes': if discourse information is present but not linearly separable, DiscoEval will score the encoder low even though the information is encoded.","fun_headline_variants_meta":{"raw":{"variants":["DiscoEval: seven tasks probe discourse in sentence embeddings","BERT and ELMo lead benchmark for discourse-aware sentence vectors","New test suite checks if sentence representations capture context","Training objectives boost discourse modeling in sentence embeddings","Deep layers shine for discourse, shallow for syntax in BERT and ELMo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3002,"prompt_tokens":799,"completion_tokens":2203,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":415,"completion_tokens_details":{"reasoning_tokens":2122}},"tokens_in":415,"tokens_out":2203,"duration_ms":20374,"temperature":1.0,"reasoning_tokens":2122,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T06:00:09.966192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same frozen embeddings, replace the logistic-regression probe with a single-hidden-layer probe (as the paper itself already does for the Discourse Coherence task) on PDTB-I and RST-DT. If accuracy rises substantially above the Table 2 numbers, DiscoEval's ranking reflects linear accessibility, not whether discourse information is present in the representation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RST Discourse Treebank with human-annotated discourse trees used for the RST-DT discourse relation task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Penn Discourse Treebank with explicit and implicit relation annotations used for the PDTB tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ROC Stories corpus used to construct the Sentence Position and Binary Sentence Ordering tasks."}],"review_version":1}