Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

MDC-R: The Minecraft Dialogue Corpus with Reference

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces MDC-R, a Minecraft dialogue corpus in which every anaphoric and deictic reference is annotated and linked through block IDs and bounding boxes to the exact blocks it denotes.

desk verdict A genuinely new situated-reference resource with block-level grounding, but the reliability evidence for its central annotation layer is thinner than the paper's own lower-bound framing suggests. read the letter →

arxiv 2506.22062 v2 pith:KWH5YGFO submitted 2025-06-27 cs.CL

Chris Madge , Maris Camilleri , Paloma Carretero Garcia , Vanja Karan , Juexi Shao , Prashant Jayannavar , Julian Hough , Benjamin Roth
show 1 more author
Massimo Poesio
This is my paper
classification cs.CL
keywords MinecraftDialogueCorpusreferenceannotationanaphoricdeicticreferringexpressioncomprehensionsituatedmultimodalbenchmarkvoxelgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MDC-R, a new language resource built on the Minecraft Dialogue Corpus: 101 dialogues in which an Architect instructs a Builder to construct a structure, with expert annotations of anaphoric and deictic reference. Each referring expression is linked through unique block IDs and 2D bounding boxes to the specific blocks it denotes, making this, in the authors' account, the first dataset that provides information about the constituent parts of referenced objects. The paper argues this resource enables studying reference in a dynamic, multi-turn situated setting, where the visual scene changes as the dialogue unfolds, and it backs this with a referring-expression comprehension experiment in which two strong baselines perform markedly worse than on static-image benchmarks, indicating a challenging new testbed for multimodal reasoning.

What carries the argument

The load-bearing mechanism is the annotation infrastructure: each voxel block in the world is assigned a unique alphanumeric tuple label, tied to its 3D Cartesian coordinates, and annotators using a modified MMAX2 interface see the object-labelled image of the world before each utterance and enter the labels of the blocks a noun phrase refers to in an 'Object' slot. The coordinates then yield 2D bounding boxes in the perspective-based screenshot, so a referring expression is grounded as a set of blocks. This lets the corpus support both classic REC with a merged single bounding box and generalized REC with multiple bounding boxes, and it makes reference to abstract shapes and composites, such as towers, beams, and bells, transparently decomposable into their constituent voxels.

What would settle it

Double-annotate a random sample of the singly annotated dialogues and measure agreement on block-set assignment; if Cohen's kappa on which blocks a referring expression denotes falls well below 0.43, or if disagreements systematically concern whether an expression refers to one block versus a multi-block structure, the claim that MDC-R provides reliable gold labels for constituent-part reference is undercut.

Watch

Extended reading notes

Core claim

MDC-R is an expert-annotated extension of the Minecraft Dialogue Corpus in which every referring expression in 101 dialogues (7,600 markables) is annotated for anaphoric and deictic reference and grounded to the set of voxel blocks it denotes. Each block carries a unique alphanumeric label tied to its 3D coordinates, and each reference is associated with the object-labelled image of the world state at that utterance, from which a 2D bounding box is derived; references thus reduce to references to sets of blocks, which the authors align with generalized referring expression comprehension. The authors claim MDC-R is the first dataset to provide information about the constituent parts of referenced objects, and they position it as a resource for studying reference in a dynamically changing visual world, where grounding, perspective mismatches, and conceptual pacts produce complex referential phenomena.

Load-bearing premise

The load-bearing premise is that the expert annotations are reliable enough to be gold labels, even though only 5 of 101 dialogues were double-coded, with a kappa of 0.43, and the rest were annotated by a single annotator.

Editorial extensions

If this is right

  • MDC-R provides a benchmark for referring expression comprehension in a dynamic, multi-turn situated setting, where the scene changes as the dialogue progresses and the two interlocutors hold different perspectives.
  • Because references are grounded as block sets, the corpus supports generalized REC with multiple targets, going beyond the single-object assumption of classic datasets.
  • The low baseline scores (mIoU 30.4 for Qwen2-VL in classic REC; F1@0.5 of 9 for MDETR in GREC) indicate that state-of-the-art multimodal models still struggle with embodied, dialogue-grounded reference in voxel worlds.
  • The annotations capture rare linguistic phenomena, including intensional references to objects not yet built, bridging references, and negotiated conceptual pacts, that static-image REC datasets do not exhibit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If annotation reliability can be raised beyond the reported kappa of 0.43 on the double-coded subset, MDC-R could also serve as a testbed for models that track object identity across state changes, since block IDs persist while the scene is edited.
  • The object-set granularity invites a move beyond bounding-box evaluation: the corpus could support voxel-level grounding metrics that penalize partial-set predictions more precisely than IoU on a merged box.
  • The apparent fuzziness of set boundaries, where 'the green block' can mean one block or a whole tower, suggests referent sets might be better modelled as distributions, and the corpus's disagreement data could be used to study that ambiguity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces MDC-R, an annotation layer on top of the Minecraft Dialogue Corpus (MDC). The authors extend the CODI/CRAC-ARRAU annotation scheme with block-level grounding: each referring expression is linked to unique block IDs and to bounding boxes derived from screenshots of the dynamic Minecraft world. The paper reports corpus statistics, qualitative observations on dynamically changing scenes, perspective, grounding, and intensional objects, and a small referring expression comprehension (REC) experiment using Qwen2-VL and MDETR. The central claim is that MDC-R is a new resource for studying anaphoric and deictic reference in a situated, dynamic setting and is, in the authors' words, 'the first dataset which provides information about the constituent parts of the referenced objects' (Section 8).

Significance. If the reliability of the block-ID annotations can be established, MDC-R is a potentially valuable resource for situated reference, anaphora, and REC in dynamic visual environments. The paper openly releases the corpus and is transparent about its pilot reliability study and limitations. The annotation scheme is a documented extension of existing guidelines, and the baseline experiments provide a first probe of task difficulty. The low baseline performance itself signals that the corpus is challenging and likely useful for future work. However, the reliability evidence is currently too thin to treat the gold labels as authoritative, and the REC experiment is underspecified in several ways. The resource is significant in potential, but the paper needs additional annotation-quality analysis before the central claims can be fully supported.

major comments (3)
  1. [Section 5.1, Table 1, Section 8] The reliability evidence for the central annotation decision is insufficient. The only inter-annotator agreement reported is a preliminary kappa of 0.43 on 5 dialogues containing about 400 markables, while Table 1 reports 7,600 markables across 101 documents and Section 5.3 states that the data were annotated by two professional linguists without indicating that the remaining 96 dialogues were double-coded. The 'lower bound' interpretation is not convincing: disagreements about slightly different sets of blocks are not a side issue but concern exactly the constituent-part information that the corpus claims to provide. The reported kappa also conflates mention detection with object-set assignment. The authors should report agreement separately for markable detection, for block-set membership, and for exact set equality, ideally with a larger double-coded sample and post-adjudication agreement. Without this, the gold labels, the bounding boxes, and the REC baselines built on them inherit an unquantified level of noise.
  2. [Section 7.1 and Section 7.5] The REC subset is underspecified, which weakens the claim that the experiment demonstrates the corpus's usefulness. The paper states that the dataset comprises 1,150 bounding boxes, 423 referring expressions, and 101 distinct scenarios, but it does not say how the 423 expressions were sampled from the 7,600 markables, whether they are all deictic referents with associated screenshots, or how the ground-truth merged bounding box is derived from the block coordinates. No train/test split, few-shot setup, or evaluation protocol is described. The results in Tables 2 and 3 are reported without confidence intervals or significance tests, and Table 4 contains several NP categories with only one or two examples, making those rows uninformative. In addition, the Qwen2-VL evaluation uses the merge-box protocol while the MDETR evaluation uses per-block boxes, so the mIoU values in Tables 2 and 3 are not directly comparable. The authors should provide the full dataset-construction details, report variance, and clearly separate the two evaluation protocols.
  3. [Section 8] The claim that MDC-R is 'the first dataset which provides information about the constituent parts of the referenced objects' is too strong without a systematic comparison to existing resources. Several simulated environments (e.g., CLEVR, CAESAR, REVERIE) provide object-level or part-level annotations, and the novelty of MDC-R appears to be the combination of dynamic dialogue, block-level grounding, and bounding boxes, not the mere existence of constituent-part information. The claim should be scoped to the specific combination of features or supported with a comparison table. This is not a fatal issue, but it affects how the contribution is positioned.
minor comments (7)
  1. [Section 5.3 and Table 1] There is a numerical inconsistency: Section 5.3 says a random subset of 100 dialogues was selected, while Table 1 reports 101 documents and Section 7.1 reports 101 distinct scenarios. This should be reconciled.
  2. [Section 5.1] The reliability statistic is reported only as 'kappa = 0.43' without specifying whether it is Cohen's kappa or Krippendorff's alpha, or what the annotation unit and chance-correction model were. This information is needed to interpret the value.
  3. [Section 7.5] The footnote says 'random bounding box limits (mIoU = 5.1) and limits over the entire image (mIoU = 10.8)'; this appears to contain a typo, and the intended meaning of 'limits' is unclear. The sentence should be rewritten to describe the random baseline procedures.
  4. [Table 4] The NP category table omits several values (NP11, NP17, NP21-NP27) without explanation; the authors should state that these categories have zero quantity, or otherwise explain the gaps. Rows with one or two instances should not be interpreted as meaningful performance estimates.
  5. [Section 6.1] The sentence 'we discuss in this Section how a selection of interesting linguistic phenomena in this Section end up being captured' contains a redundancy and should be rephrased.
  6. [Figure 6 caption] The caption contains a typo: 'thee bell' should be 'the bell'.
  7. [References] There are several small reference errors, including 'V on Ahn' (should be 'von Ahn') and duplicate entries for He et al. 2023a/2023b; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MDC-R is a resource paper whose central claim is the annotation layer itself, and the REC experiment uses external pretrained models plus a random sanity check rather than reusing fitted values as predictions.

full rationale

MDC-R is a resource paper, not a derivation chain. The central claim is that the corpus adds expert anaphoric and deictic reference annotations with block-ID sets and bounding boxes, and this is attested by the released annotation layer: annotators entered block labels in MMAX2, and those labels are tied to 3D coordinates from which bounding boxes are derived. No parameter is fitted to one subset of the data and then renamed as a prediction. The REC experiment in Section 7 uses two externally pretrained models, Qwen2-VL and MDETR, evaluated against the annotated bounding boxes, and the comparison against random bounding-box limits is a sanity check, not a circular reuse of the corpus labels. The annotation scheme's lineage from ARRAU and CODI-CRAC guidelines, including work by the authors, is intellectual background rather than load-bearing evidence: the scheme is described concretely and a double-annotated kappa of 0.43 is reported. That low kappa raises a legitimate reliability concern about the gold labels, but low inter-annotator agreement is a correctness and quality risk, not circularity. There is no self-definitional step, no fitted input called a prediction, no uniqueness theorem imported from prior author work, and no ansatz smuggled in via citation. The paper is therefore self-contained with respect to the circularity criteria.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central resource claim rests on representational and procedural assumptions rather than fitted parameters or invented entities. The most sensitive assumptions are the block-set representation of reference, the use of builder-perspective images for annotating speaker intent, and the transferability of a 5-dialogue reliability pilot to the rest of the corpus.

assumptions (4)
  • domain assumption A referent in the Minecraft world is adequately represented as a set of block IDs, and reference to an object is reference to the set of blocks composing it.
    Section 5.1 instructs annotators to enter block labels in the Object slot and Section 5.2 derives bounding boxes from block coordinates; the entire corpus and the REC task depend on this block-set representation.
  • domain assumption The visual context for annotation is the object-labelled image of the world state before each utterance, as seen by the Builder, and this supports determining the speaker's intended referent.
    Section 5.1 describes this image; Section 6.2 notes Architect/Builder perspective mismatches and Section 6.3 says annotators mark the speaker's intended reference, so the annotation relies on this mapping without a separate speaker-view image.
  • domain assumption Single-annotator labels for the 96 non-pilot dialogues are reliable enough to act as gold standard, despite the small pilot agreement.
    Section 5.1 reports only 5 double-coded dialogues with kappa 0.43; the rest of the corpus is singly annotated, yet downstream evaluation treats labels as ground truth.
  • domain assumption The 423 expressions selected for the REC benchmark are a valid and representative sample of the 7,600 markables.
    Section 7.1 states the REC dataset composition but does not describe markable filtering criteria, so the representativeness of the subset is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MDC-R: The Minecraft Dialogue Corpus with Reference." pith.science (2026). https://pith.science/paper/KWH5YGFO

@misc{pith2026250622062,
  author       = {Pith},
  title        = {Pith review of: MDC-R: The Minecraft Dialogue Corpus with Reference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWH5YGFO}},
  note         = {Machine review of arXiv:2506.22062}
}
read the original abstract

We introduce the Minecraft Dialogue Corpus with Reference (MDC-R). MDC-R is a new language resource that supplements the original Minecraft Dialogue Corpus (MDC) with expert annotations of anaphoric and deictic reference. MDC's task-orientated, multi-turn, situated dialogue in a dynamic environment has motivated multiple annotation efforts, owing to the interesting linguistic phenomena that this setting gives rise to. We believe it can serve as a valuable resource when annotated with reference, too. Here, we discuss our method of annotation and the resulting corpus, and provide both a quantitative and a qualitative analysis of the data. Furthermore, we carry out a short experiment demonstrating the usefulness of our corpus for referring expression comprehension.

Figures

Figures reproduced from arXiv: 2506.22062 by the authors.

Figure 1
Figure 1. Example of Minecraft Collaborative Builder [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. References in a MDC dialogue (Narayan-Chen et al., 2019) incorporates spatial relations. Bonial et al. (2021) apply a separate extension of AMR on MDC, with the Dialogue-AMR (Bonial et al., 2020) representa￾tion, a form of AMR that captures the illocutionary force of dialogue acts. Thompson et al. (2024) produced the Minecraft Structured Dialogue Corpus (MSDC), that, through Segmented Discourse Representation Theory… view at source ↗
Figure 5
Figure 5. Annotating references in MMAX2 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Reference annotation and grounding in MMAX2 of the referring mention. A reliability test with this revised scheme and guidelines (2 experienced annotators double-coded 5 dialogues containing a total of about 400 markables), resulting in a prelim￾inary κ = 0.43. We expe…
Figure 6
Figure 6. Figure 6: Introducing the part of the construction to be [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Referring to the bell after it has been built. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Example case of recognizing merged blocks [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Example case of recognizing independent blocks by MDETR under the GREC setting. 9.3 Example model outputs This section provides images of the model out￾puts for particularly easy ( [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Example case where there are only a few blocks in the scene, making the task less challenging [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 17
Figure 17. Figure 17: Example case where the perspective makes [PITH_FULL_IMAGE:figures/full_fig_p013_17.png]
Figure 15
Figure 15. Figure 15: Example case where there is a large number [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: Example case where the object is partially [PITH_FULL_IMAGE:figures/full_fig_p013_16.png]
Figure 14
Figure 14. Figure 14: Example case where the object is clearly [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Making Dialogue Grounding Data Rich: A Three-Tier Data Synthesis Framework for Generalized Referring Expression Comprehension

    cs.CL 2025-12 unverdicted novelty 5.0 of 10

    A three-tier data synthesis method produces realistic and controllable dialogue grounding data, enabling fine-tuned models to achieve consistent improvements on GREC tasks under distribution shift.

  2. Referential ambiguity and clarification requests: comparing human and LLM behaviour

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Humans seldom ask clarification questions for referential ambiguity, while LLMs ask them more often, and reasoning prompts increase LLM question frequency and relevance.

Reference graph

Works this paper leans on

63 extracted references · 43 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The hcrc map task corpus. Language and speech, 34(4):351--366

    work page 1991

  4. [4]

    Nicholas Asher and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press

    2003

  5. [5]

    Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178--186

    work page 2013

  6. [6]

    Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.85 Mindcraft: Theory of mind modeling for situated dialogue in collaborative tasks . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, page 1112–1125. Association for Computational Linguistics

    work page doi:10.18653/v1/2021.emnlp-main.85 2021

  7. [7]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    arXiv 2020

  8. [8]

    Claire Bonial, Mitchell Abrams, David Traum, and Clare Voss. 2021. Builder, we have done it: evaluating & extending dialogue-amr nlu pipeline for two collaborative domains. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 173--183

    2021

Show all 63 references
  1. [9]

    Claire Bonial, Lucia Donatelli, Mitchell Abrams, Stephanie Lukin, Stephen Tratz, Matthew Marge, Ron Artstein, David Traum, and Clare Voss. 2020. Dialogue-amr: abstract meaning representation for dialogue. In Proceedings of the Twelfth Language Resources and Evaluation Conferen...

    2020

  2. [10]

    Julia Bonn, Martha Palmer, Jon Cai, and Kristin Wright-Bettner. 2020. Spatial AMR : Expanded spatial annotation in the context of a grounded minecraft corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020),

    2020

  3. [11]

    Susan E Brennan and Herbert H Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of experimental psychology: Learning, memory, and cognition, 22(6):1482

    1996

  4. [12]

    Jon Chamberlain, Massimo Poesio, Udo Kruschwitz, et al. 2008. Phrase detectives: A web-based collaborative annotation game. In Proceedings of the international conference on semantic systems (I-Semantics’ 08), pages 42--49

    2008

  5. [13]

    Zhihong Chen, Ruifei Zhang, Yibing Song, Xiang Wan, and Guanbin Li. 2023. Advancing visual grounding with scene knowledge: Benchmark and method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15039--15049

    2023

  6. [14]

    Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1--39

    1986

  7. [15]

    Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5503--5512

    2017

  8. [16]

    Simon Dobnik, John D Kelleher, and Christine Howes. 2020. Local alignment of frame of reference assignment in english and swedish dialogue. In German Conference on Spatial Cognition, pages 251--267. Springer

    2020

  9. [17]

    Sebastiano Gigliobianco, Dimosthenis Kontogiorgos, and David Schlangen. 2024. Learning task-oriented dialogues through various degrees of interactivity. In Proceedings of the 28th Workshop on the Semantics and Pragmatics of Dialogue

    2024

  10. [18]

    Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. 2023 a . Semi-supervised multimodal coreference resolution in image narrations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4567--4578

    2023

  11. [19]

    Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. 2023 b . Who are you referring to? coreference resolution in image narrations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5247--5257

    2023

  12. [21]

    Shuting He, Henghui Ding, Chang Liu, and Xudong Jiang. 2023 b . https://arxiv.org/abs/2308.16182 Grec: Generalized referring expression comprehension . Preprint, arXiv:2308.16182

    2023 arXiv

  13. [22]

    Bhathiya Hemanthage, Hakan Bilen, Phil Bartie, Christian Dondrup, and Oliver Lemon. 2024. Recantformer: Referring expression comprehension with varying numbers of targets. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21784--21798

    2024

  14. [23]

    Md Mofijul Islam, Reza Mirzaiee, Alexi Gladstone, Haley Green, and Tariq Iqbal. 2022. Caesar: An embodied simulator for generating multimodal referring expression datasets. Advances in Neural Information Processing Systems, 35:21001--21015

    2022

  15. [24]

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern reco...

    2017

  16. [25]

    Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation. In IJCAI , volume 16, pages 4246--4247

    2016

  17. [26]

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1780--1790

    2021

  18. [27]

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. https://doi.org/10.3115/v1/D14-1086 R efer I t G ame: Referring to objects in photographs of natural scenes . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNL...

    2014 doi

  19. [28]

    Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail Burtsev, Alexey Skrynnik, Artem Zholus, Aleksandr Panov, Kavya Srinet, et al. 2022. Interactive grounded language understanding in a collaborative environment: IGLU 2021. In NeurIPS 2...

    2022

  20. [29]

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. 2017. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474

    2017 arXiv

  21. [30]

    Alex Lascarides and Nicholas Asher. 2007. Segmented discourse representation theory: Dynamic semantics with discourse structure. In Computing meaning, pages 87--124. Springer

    2007

  22. [31]

    Chang Liu, Henghui Ding, and Xudong Jiang. 2023 a . GRES : Generalized referring expression segmentation. In CVPR

    2023

  23. [32]

    Chang Liu, Henghui Ding, and Xudong Jiang. 2023 b . Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23592--23601

    2023

  24. [33]

    Sharid Lo \'a iciga, Simon Dobnik, and David Schlangen. 2021. Annotating anaphoric phenomena in situated dialogue. In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pages 78--88

    2021

  25. [34]

    Sharid Lo \'a iciga, Simon Dobnik, and David Schlangen. 2022. Anaphoric phenomena in situated dialog: A first round of annotations. In Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 31--37

    2022

  26. [35]

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11--20

    2016

  27. [36]

    Christoph M \"u ller and Michael Strube. 2006. Multi-level annotation of linguistic data with mmax2. Corpus technology and language pedagogy: New resources, new tools, new methods, 3:197--214

    2006

  28. [37]

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. 2016. Modeling context between objects for referring expression understanding. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, pages 792--80...

    2016

  29. [38]

    Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019. Collaborative dialogue in minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5405--5415

    2019

  30. [39]

    Haruna Ogawa, Hitoshi Nishikawa, Takenobu Tokunaga, and Hikaru Yokono. 2020. Gamification platform for collecting task-oriented dialogue data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7084--7093

    2020

  31. [40]

    Poesio, R

    M. Poesio, R. Delmonte, A. Bristot, L. Chiran, and S. Tonelli. 2004. The VENEX corpus of anaphoric information in spoken and written I talian. Unpublished. Available online at http://cswww.essex.ac.uk/staff/poesio/publications/VENEX04.pdf

    2004

  32. [41]

    Massimo Poesio, Maris Camilleri, Paloma Carretero Garcia, and Ron Artstein. 2024 a . https://sites.google.com/view/arrau/corpus The ARRAU 3 Annotation Manual , v. 1.1 edition. Queen Mary University of London

    2024

  33. [42]

    Massimo Poesio, Maris Camilleri, Paloma Carretero Garcia, Juntao Yu, and Mark-Christoph M \" u ller. 2024 b . https://aclanthology.org/2024.codi-1.12 The arrau 3.0 corpus . In Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI ), pages 127--138. Ass...

    2024

  34. [43]

    Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Silviu Paun, Alexandra Uma, and Juntao Yu. 2019. https://doi.org/10.18653/v1/N19-1176 A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation . In Proc. of NAACL, page 1778–1789, Minneapolis. Ass...

    2019 doi

  35. [44]

    Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. https://doi.org/10.1145/2448116.2448119 Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation . ACM Transactions on Intelligent Interactive Sy...

    2013

  36. [45]

    Massimo Poesio, Juntao Yu, Silviu Paun, Abdulrahman Aloraini, Pengcheng Lu, Janosch Haber, and Derya Cokal. 2023. https://www.annualreviews.org/doi/full/10.1146/annurev-linguistics-031120-111653 Computational models of anaphora . Annual Review of Linguistics, 9:561--587

    2023 doi

  37. [46]

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494--8502

    2018

  38. [47]

    Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. 2020. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2020

  39. [48]

    David Schlangen. 2023. Dialogue games for benchmarking language understanding: Motivation, taxonomy, strategy. arXiv preprint arXiv:2304.07007

    2023 arXiv

  40. [49]

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pa...

    2020

  41. [50]

    Philipp Spanger, Masaaki Yasuhara, Ryu Iida, Takenobu Tokunaga, Asuka Terai, and Naoko Kuriyama. 2012. Rex-j: Japanese referring expression corpus of situated dialogs. Language Resources and Evaluation, 46:461--491

    2012

  42. [51]

    Alane Suhr, Claudia Yan, Charlotte Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situated collaborative interactions. arXiv preprint arXiv:1910.03655

    2019 arXiv

  43. [52]

    Arthur Szlam, Jonathan Gray, Kavya Srinet, Yacine Jernite, Armand Joulin, Gabriel Synnaeve, Douwe Kiela, Haonan Yu, Zhuoyuan Chen, Siddharth Goyal, et al. 2019. Why build an assistant in minecraft? arXiv preprint arXiv:1907.09273

    2019 arXiv

  44. [53]

    Kate Thompson, Julie Hunter, and Nicholas Asher. 2024. Discourse structure for the minecraft corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation ( LREC-COLING 2024) , pages 4957--4967

    2024

  45. [54]

    Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rockt \"a schel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. arXiv preprint arXiv:1903.03094

    2019 arXiv

  46. [55]

    Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J Rodriguez, and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in a variety of genres: the arrau corpus. Natural Language Engineering, 26(1):95--128

    2020

  47. [56]

    Kees van Deemter. 2016. Computational Models of Referring. The MIT Press

    2016

  48. [57]

    Luis Von Ahn. 2006. Games with a purpose. Computer, 39(6):92--94

    2006

  49. [58]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language mode...

    2024 arXiv

  50. [59]

    Peng Wang, Dongyang Liu, Hui Li, and Qi Wu. 2020. Give me something to eat: Referring expression comprehension with commonsense knowledge. In Proceedings of the 28th ACM International Conference on Multimedia, pages 28--36

    2020

  51. [60]

    Haoxuan You, Rui Sun, Zhecan Wang, Kai-Wei Chang, and Shih-Fu Chang. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.399 Find someone who: Visual commonsense understanding in human-centric grounding . In Findings of the Association for Computational Linguistics: EMNLP 20...

    2022 doi

  52. [61]

    Juntao Yu, Sopan Khosla, Ramesh Manuvinakurike, Lori Levin, Vincent Ng, Massimo Poesio, Michael Strube, and Carolyn Ros \'e . 2022 a . The CODI-CRAC 2022 shared task on anaphora, bridging, and discourse deixis in dialogue. In Proceedings of the CODI-CRAC 2022 Shared Task on An...

    2022

  53. [62]

    Xintong Yu, Hongming Zhang, Ruixin Hong, Yangqiu Song, and Changshui Zhang. 2022 b . Vd-pcr: Improving visual dialog with pronoun coreference resolution. arXiv preprint arXiv:2205.14693

    2022 arXiv

  54. [63]

    Xintong Yu, Hongming Zhang, Yangqiu Song, Yan Song, and Changshui Zhang. 2019. What you see is what you get: Visual pronoun coreference resolution in dialogues. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International ...

    2019

  55. [64]

    Sina Zarrie , Julian Hough, Casey Kennington, Ramesh Manuvinakurike, David DeVault, Raquel Fern \'a ndez, and David Schlangen. 2016. https://aclanthology.org/L16-1019/ P ento R ef: A corpus of spoken references in task-oriented dialogues . In Proceedings of the Tenth Internati...

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.