Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Coreference resolution learned on text transfers to visual grounding, giving a single similarity-based model that resolves pronouns and omitted referents in Japanese dialogue.

desk verdict The coreference-helps-pronoun-grounding claim is plausible but not yet cleanly established, because the pretrained model also sees extra text; the paper is still worth a serious referee. read the letter →

arxiv 2505.11726 v2 pith:SGO3BLJ2 submitted 2025-05-16 cs.CL

Shun Inadumi , Nobuhiro Ueda , Koichiro Yoshino This is my paper · ORCID
classification cs.CL
keywords multimodalreferenceresolutionphrasegroundingcoreferencepredicate-argumentstructureanalysisbridginganaphoravisuallygroundeddialoguezeroJapanese
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that resolving references inside the text—coreference, predicate-argument structure, and bridging anaphora—makes a model better at resolving references between text and images in visually grounded dialogue, especially for pronouns and omitted subjects or objects. To test this, the authors build one framework in which textual and multimodal reference resolution are the same operation: mentions and detected objects are embedded, and a mention is linked to another mention or to an object by dot-product similarity. The headline evidence is on Japanese two-party dialogues, where initializing the shared text encoder with coreference resolution raises pronoun phrase-grounding Recall@1 from 0.277 to 0.361, above MDETR and GLIP. The same gain does not appear on ordinary caption grounding, which the paper reads as evidence that textual reference structure matters specifically for dialogue-style ambiguity.

What carries the argument

The load-bearing object is a relation-specific similarity matrix. For each of six reference labels $\mathcal{L}$—direct reference plus the Japanese case relations 'ga', 'wo', 'ni', 'de/no', and bridging anaphora—the model projects mention embeddings and object embeddings into a shared space and computes $S_l = \hat{T}_l \hat{T}_l^{\top}$ for text-to-text links and $U_l = \hat{X}_l \hat{T}_l^{\top}$ for mention-to-object links. Both textual and multimodal reference resolution are trained with cross-entropy losses on these matrices, and the weights of the text encoder are shared between the two training stages. That shared encoder is the transfer channel: learning to score textual references first reorganizes the mention embeddings so that pronouns and zero references become easier to ground in images.

What would settle it

Train the same text encoder on the same Japanese pretraining corpus—J-CRe3 plus the 6,542 web, Wikipedia, and blog documents—with a non-reference objective such as masked language modeling, then fine-tune the multimodal model identically; if pronoun phrase-grounding Recall@1 stays at about 0.361 rather than falling back toward 0.277, the reference structure is not the cause of the gain.

Watch

Extended reading notes

Core claim

The central discovery is that textual reference resolution is not just a separate annotation layer but a transferable training signal for multimodal reference resolution. The authors claim that a model trained first to recognize coreference, predicate-argument structure, and bridging anaphora—using a similarity matrix between mention embeddings—then fine-tuned to link mentions to image regions, grounds ambiguous expressions better than models trained only on image-text pairs. On J-CRe3, the model with coreference initialization reaches pronoun phrase-grounding Recall@1 of 0.361, compared with 0.277 for the same architecture without that initialization, 0.241 for GLIP, and 0.133 for MDETR. The model also scores indirect references, including zero references, directly through the same text-object similarity matrix, which pipeline methods cannot do. The authors are careful to report that direct-reference performance did not improve and that on Flickr30k-Ent-JP no gain from coreference was observed.

Load-bearing premise

The experiment assumes that the pronoun-grounding improvement comes from the reference labels used in textual pretraining, rather than from the extra Japanese text data or the larger text encoder used to learn those labels.

Editorial extensions

If this is right

  • Pronoun-heavy dialogue is where textual reference supervision pays off: on J-CRe3, pronoun phrase-grounding Recall@1 rises from 0.277 to 0.361, above GLIP (0.241) and MDETR (0.133), while noun grounding is roughly unchanged.
  • Zero references—objects implied by predicates but never named—can be scored directly by the same similarity model, something a pipeline that links separate phrase-grounding and text-analysis outputs cannot do.
  • Longer dialogue context improves pronoun grounding in proportion to the number of coreference relations available, and the model's noun performance stays stable where GLIP degrades.
  • On standard caption grounding (Flickr30k-Ent-JP), the same coreference initialization produces no measurable gain, and GLIP remains the stronger model, so the effect appears tied to dialogue-style ambiguity, not to image-text alignment in general.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to isolate the cause of the pronoun gain: pretraining the same text encoder on the same Japanese corpora without reference labels would settle whether coreference structure, extra text data, or the larger encoder drives the improvement.
  • If the similarity framework generalizes, languages with frequent ellipsis and case marking, such as Korean and Chinese, should show similar pronoun and zero-reference gains, while English may show less because omitted arguments are rarer.
  • The paper's confidence-score analysis implies a calibration effect, not just an accuracy effect; label smoothing or confidence regularization would be a natural follow-up before deploying the model in assistive-robot settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a unified framework for textual and multimodal reference resolution in visually grounded Japanese dialogues. A text encoder is first trained on textual reference resolution (TRR) tasks — coreference resolution, predicate-argument structure (PAS) analysis, and bridging anaphora (BA) resolution — and then fine-tuned for multimodal reference resolution (MRR) on J-CRe3 and Flickr30k-Ent-JP. The MRR model computes dot-product similarity between mention embeddings and object embeddings across six relation types, and the authors report that TRR pretraining, especially coreference resolution, improves pronoun phrase grounding over baselines and over MDETR and GLIP. The paper includes three-seed experiments, an ablation over TRR relation types, utterance-length analyses, and qualitative confidence-score comparisons.

Significance. If the central claim holds, the work would demonstrate a transferable benefit of textual reference supervision to multimodal reference resolution in a low-resource Japanese dialogue setting, and the similarity-based unified framework with explicit zero-reference handling would be a meaningful contribution. The paper is commendable for shipping public code, using public datasets, reporting three seed runs with standard deviations, and providing extensive ablations (Tables 5–7, Figures 4, 6–8). The treatment of zero references and indirect case relations is a genuine step beyond standard phrase-grounding models. However, the experimental design does not currently isolate the effect of reference semantics from the effect of additional Japanese text pretraining, and the headline pronoun result rests on a small sample without significance testing. These issues directly affect the main attribution and need to be resolved before the central claim can be accepted.

major comments (4)
  1. [Section 4.1, Table 3] The main comparison between 'Baseline' and 'Baseline w/ Ours' conflates TRR pretraining with additional Japanese text exposure. Baseline w/ Ours is initialized from a TRR model trained on J-CRe3 plus 6,542 web documents (KWDLC, Wikipedia, blogs), whereas Baseline is initialized from vanilla Japanese DeBERTa-v2-large and receives no such text pretraining. There is no control model pretrained on the same corpora with the same training schedule but without reference labels (e.g., masked language modeling only). Consequently, the pronoun R@1 gain (0.361 vs 0.277) cannot be attributed specifically to learning coreference or predicate-argument structure; it may partly reflect extra text pretraining, different initialization, or optimizer trajectory. This is load-bearing for the paper's central claim that textual reference resolution positively affects multimodal reference resolution.
  2. [Section 4.2, Figure 4 and Table 5] The ablation over Coref, PAS/BA, and TRR does not remove the confound identified above, because every ablation variant still receives the same additional text pretraining and the same DeBERTa-v2-large architecture; the variants differ only in which textual relation labels are used in the TRR objective. Since there is no label-free control, the ablation cannot separate the benefit of reference semantics from the benefit of additional text pretraining. A control trained on the same corpus with an equivalent objective and shuffled or masked reference labels is needed to support the claim that the observed gains come from reference structure rather than from more text.
  3. [Table 3, pronoun row] The headline pronoun phrase-grounding result is based on only 120 positive instances, and the reported standard deviations overlap: Baseline w/ Ours is 0.361 ± 0.070 while Baseline is 0.277 ± 0.020. No significance test (bootstrap, permutation, or paired test) is reported, so the difference may be within random seed variation. The abstract's stronger claim — that the coreference model performs better than MDETR and GLIP on pronoun grounding — needs statistical support or explicit confidence intervals, and per-seed results should be reported for the pronoun subset.
  4. [Limitations, Experiments paragraph] The Limitations section concedes that 'investigating the correlation between TRR and MRR performance could provide valuable insights, though this would require multiple training and evaluation iterations of TRR and MRR models,' an experiment that is not reported. This concession is directly relevant to the paper's main attribution: without such iterations or a control experiment, the causal wording in the abstract ('learning textual reference resolution ... positively affects performance') is stronger than the evidence supports. The authors should either perform the recommended analysis or temper the causal claim accordingly.
minor comments (5)
  1. [Equations (1)–(5)] The notation for bT is inconsistent between Eq. (1), where bT ∈ R^{p×dT×|L|}, and Eq. (3), where bT ∈ R^{p×dS×|L|}; please clarify the relationship between dT and dS and whether the same symbol denotes different tensors.
  2. [Section 3.2.3] The loss notation 'loss{S_l;S^{(l,ground)}}' is undefined; please specify the exact softmax cross-entropy formulation, including how the similarity matrix is normalized over the mention or object axis.
  3. [Section 4.1, implementation details] Table 2 shows that Ours uses a 339M-parameter DeBERTa-v2-large text encoder, while MDETR and GLIP use mDeBERTa-v3-base. The comparison to these models is therefore not parameter-matched; this should at least be discussed as a potential confound for the claim of outperforming representative models.
  4. [Tables 3 and 5] GLIP and MDETR results appear to be single runs, while the MRR models are averaged over three seeds; please state whether the comparison numbers are averaged and whether the 120-instance pronoun subset is stable across the three seeds.
  5. [Appendix D] The confidence-score analysis suggests the TRR-initialized models may be overconfident, and the authors mention label smoothing as a potential remedy, but no calibration experiment is conducted. Please clarify whether calibration is left for future work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claim is an empirically tested transfer-learning comparison rather than a derivation from fitted inputs.

full rationale

The paper's central claim is that learning textual reference resolution (TRR) positively affects multimodal reference resolution (MRR). This is established by comparing an MRR baseline with an MRR model initialized from a TRR model on the J-CRe3 benchmark, and by comparing against the external models MDETR and GLIP. The framework's equations (1)-(7) define similarity-based losses from ground-truth reference matrices; the MRR predictions are evaluated by Recall@k against held-out object bounding boxes, so no predicted quantity is defined as the training input. The TRR pretraining uses textual annotations (coreference, predicate-argument structure, bridging anaphora) from J-CRe3 and additional Japanese corpora, while MRR is fine-tuned on multimodal annotations; the transfer is empirical and could fail, as it indeed does for several noun and indirect-reference settings in Tables 4-7. The paper's self-citations, such as J-CRe3 (Ueda et al., 2024) and KWJA (Ueda et al., 2023), supply the dataset and an alternative TRR analyzer rather than the conclusion itself, and the decisive comparisons to MDETR and GLIP are external. The strongest concern raised by the reader's skeptical take, namely the absence of a control model pretrained on the same text corpus without reference labels, is a confound in experimental design that could affect whether the gain is attributable to reference structure versus extra text exposure; it is not a circularity, because the result is not forced by construction. The Limitations section also explicitly concedes that investigating the correlation between TRR and MRR would require multiple training and evaluation iterations of both models, which is an honest statement of an unperformed analysis rather than a hidden circular step. Overall, the paper's derivation chain is self-contained: the framework and losses are defined independently of the headline result, and the reported improvements are empirical findings subject to the usual experimental-control limitations, not predictions that reduce to their own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework contributes no analytical derivation with fitted constants; its empirical claim rests on J-CRe3 annotations, a frozen detector, and a transfer assumption from textual to multimodal reference. The main risk factors are the transfer confound and the detector's upper bound, both reflected above.

free parameters (3)
  • Input subword length p = 256
    Maximum token length for the text encoder, fixed without sensitivity analysis. Appendix C shows that utterance length changes reference cues, so this choice affects the measured gains.
  • Object proposal count q = 128 for J-CRe3, 256 for Flickr30k-Ent-JP
    The MRR model scores only the top q proposals from the frozen Detic detector. The paper states the detector limits the upper bound to 0.799 on Flickr30k, so this choice caps achievable recall.
  • Training epochs and batch size = 16 epochs, batch size 16 or 32 for the proposed models; 2 epochs for MDETR and GLIP
    Hyperparameters were chosen without a reported sensitivity study. Unequal training budgets across compared models could affect the comparison.
assumptions (4)
  • domain assumption J-CRe3 annotations of reference relations are accurate and complete.
    Equations 6 and 7 train against S(l,ground) and U(l,ground) derived from J-CRe3; every reported metric inherits the annotation quality.
  • domain assumption The frozen Detic detector proposes all objects that mentions refer to.
    MRR scoring in Equation 5 operates only over q proposals from the detector. The paper itself notes the upper bound of 0.799 on Flickr30k, so missed objects cannot be recovered by the model.
  • domain assumption Textual reference structure transfers to multimodal grounding through the shared text encoder.
    The framework pretrains the encoder on TRR and then uses it for MRR. If transfer does not occur, the reported gains cannot be attributed to reference structure.
  • ad hoc to paper The first subword of a mention represents the whole mention for dot-product similarity.
    Section 3.2.1 defines this linking for basic phrases; it is an approximation with no dedicated ablation in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures." pith.science (2026). https://pith.science/paper/SGO3BLJ2

@misc{pith2026250511726,
  author       = {Pith},
  title        = {Pith review of: Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SGO3BLJ2}},
  note         = {Machine review of arXiv:2505.11726}
}
read the original abstract

Multimodal reference resolution, including phrase grounding, aims to understand the semantic relations between mentions and real-world objects. Phrase grounding between images and their captions is a well-established task. In contrast, for real-world applications, it is essential to integrate textual and multimodal reference resolution to unravel the reference relations within dialogue, especially in handling ambiguities caused by pronouns and ellipses. This paper presents a framework that unifies textual and multimodal reference resolution by mapping mention embeddings to object embeddings and selecting mentions or objects based on their similarity. Our experiments show that learning textual reference resolution, such as coreference resolution and predicate-argument structure analysis, positively affects performance in multimodal reference resolution. In particular, our model with coreference resolution performs better in pronoun phrase grounding than representative models for this task, MDETR and GLIP. Our qualitative analysis demonstrates that incorporating textual reference relations strengthens the confidence scores between mentions, including pronouns and predicates, and objects, which can reduce the ambiguities that arise in visually grounded dialogues.

Figures

Figures reproduced from arXiv: 2505.11726 by the authors.

Figure 1
Figure 1. Example of textual and multimodal reference [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our framework for J-CRe3: The orange and green indicate the processing flows for TRR and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples of phrase grounding: The green mentions and objects are targets for grounding, and the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Ablation study results of Baseline w/ Ours [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Examples of indirect references in MRR: The orange and blue mentions are targets for the accusative [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablation study results of Baseline w/ Ours in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Ablation study results on utterance length in phrase grounding: We compare GLIP, Baseline, and Baseline [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Ablation study results on utterance length in MRR: We compare Baseline and Baseline w/ Ours by varying [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Violin plots of average confidence scores across the MRR models in phrase grounding and MRR: Each [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 39 canonical work pages

  1. [1]

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. https://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_Bottom-Up_and_Top-Down_CVPR_2018_paper.html Bottom-up and top-down attention for image captioning and visual question answering . In Proceedings of the IEEE Conference on Computer Vision ...

    work page 2018

  2. [2]

    Herbert H. Clark. 1975. https://aclanthology.org/T75-2034/ Bridging . In Theoretical Issues in Natural Language Processing

    1975

  3. [3]

    Moura, Devi Parikh, and Dhruv Batra

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos\'e M.F. Moura, Devi Parikh, and Dhruv Batra. 2017. https://openaccess.thecvf.com/content_cvpr_2017/html/Das_Visual_Dialog_CVPR_2017_paper.html V isual D ialog . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 326--335

    work page 2017

  4. [4]

    Samyak Datta, Karan Sikka, Anirban Roy, Karuna Ahuja, Devi Parikh, and Ajay Divakaran. 2019. https://openaccess.thecvf.com/content_ICCV_2019/html/Datta_Align2Ground_Weakly_Supervised_Phrase_Grounding_Guided_by_Image-Caption_Alignment_ICCV_2019_paper.html Align2Ground : Weakly supervised phrase grounding guided by image-caption alignment . In Proceedings o...

    work page 2019

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , vol...

    doi:10.18653/v1/n19-1423 2019

  6. [6]

    Charles J Fillmore. 1968. The case for case. Universals in Linguistic Theory, pages 21--119

    work page 1968

  7. [7]

    Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.682 Semi-supervised multimodal coreference resolution in image narrations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11067--11081

    work page doi:10.18653/v1/2023.emnlp-main.682 2023

  8. [8]

    Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. 2023 b . https://openaccess.thecvf.com/content/ICCV2023/html/Goel_Who_Are_You_Referring_To_Coreference_Resolution_In_Image_Narrations_ICCV_2023_paper.html Who are you referring to? coreference resolution in image narrations . In Proceedings of the IEEE/CVF International Conference on Computer Vi...

    work page 2023

Show all 57 references
  1. [9]

    Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. 2020. https://doi.org/10.1007/978-3-030-58580-8_44 Contrastive learning for weakly supervised phrase grounding . In Proceedings of the 16th European Conference on Computer Vision (ECCV), pages 752--768

    2020 doi

  2. [10]

    Muzhi Han, Yifeng Zhu, Song-Chun Zhu, Ying Nian Wu, and Yuke Zhu. 2024. https://doi.org/10.15607/RSS.2024.XX.034 InterPreT : Interactive predicate learning from language feedback for generalizable task planning . In Proceedings of Robotics: Science and Systems (RSS)

    2024 doi

  3. [11]

    Masatsugu Hangyo, Daisuke Kawahara, and Sadao Kurohashi. 2012. https://aclanthology.org/Y12-1058/ Building a diverse document leads corpus annotated with semantic relations . In Proceedings of the 26th Pacific Asia Conference on Language, Information, and Computation (PACLIC),...

    2012

  4. [12]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021 a . https://arxiv.org/abs/2111.09543 DeBERTaV3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . arXiv:2111.09543

    2021 arXiv

  5. [13]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021 b . https://openreview.net/forum?id=XPZIaotutsD DeBERTa : Decoding-enhanced BERT with disentangled attention . In Proceedings of the 9th International Conference on Learning Representations (ICLR)

    2021

  6. [14]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. 2019. https://openaccess.thecvf.com/content_CVPR_2019/html/Hudson_GQA_A_New_Dataset_for_Real-World_Visual_Reasoning_and_Compositional_CVPR_2019_paper.html GQA : A new dataset for real-world visual reasoning and compositional question ...

    2019

  7. [15]

    Ryu Iida, Mamoru Komachi, Kentaro Inui, and Yuji Matsumoto. 2007. https://aclanthology.org/W07-1522/ Annotating a J apanese text corpus with predicate-argument and coreference relations . In Proceedings of the Linguistic Annotation Workshop, pages 132--139

    2007

  8. [16]

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. https://openaccess.thecvf.com/content/ICCV2021/html/Kamath_MDETR_-_Modulated_Detection_for_End-to-End_Multi-Modal_Understanding_ICCV_2021_paper.html MDETR - modulated detection...

    2021

  9. [17]

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. https://doi.org/10.3115/v1/D14-1086 R efer I t G ame: Referring to objects in photographs of natural scenes . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNL...

    2014 doi

  10. [18]

    Hideo Kobayashi and Vincent Ng. 2020. https://doi.org/10.18653/v1/2020.coling-main.331 Bridging resolution: A survey of the state of the art . In Proceedings of the 28th International Conference on Computational Linguistics (COLING), pages 3708--3721

    2020 doi

  11. [19]

    Fang Kong and Guodong Zhou. 2010. https://aclanthology.org/D10-1086/ A tree kernel-based unified framework for C hinese zero anaphora resolution . In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 882--891. Association for...

    2010

  12. [20]

    Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.401 SIMMC 2.0: A task-oriented dialog dataset for immersive multimodal conversations . In Proceedings of the 2021 Conference on Empirical Methods in Natural...

    2021 doi

  13. [21]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. https://doi.org/10.1007/s11263-016-0981-7 Visual genome: Connecting language and visio...

    2017 doi

  14. [22]

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. 2022. https://openaccess.thecvf.com/content/CVPR2022/html/Li_Grounded_Language-Image_Pre-Training_CVPR_2...

    2022

  15. [23]

    Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. https://openaccess.thecvf.com/content_cvpr_2017/html/Lin_Feature_Pyramid_Networks_CVPR_2017_paper.html Feature pyramid networks for object detection . In Proceedings of the IEEE...

    2017

  16. [24]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. 2024. https://doi.org/10.1007/978-3-031-72970-6_3 Grounding DINO : Marrying dino with grounded pre-training for open-set object detecti...

    2024 doi

  17. [25]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. https://openaccess.thecvf.com/content/ICCV2021/html/Liu_Swin_Transformer_Hierarchical_Vision_Transformer_Using_Shifted_Windows_ICCV_2021_paper.html Swin Transformer : Hierarchical...

    2021

  18. [26]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In Proceedings of the 7th International Conference on Learning Representations (ICLR)

    2019

  19. [27]

    Panzhong Lu, Xin Zhang, Meishan Zhang, and Min Zhang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.518 Extending phrase grounding with pronouns in visual dialogues . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7...

    2022 doi

  20. [28]

    Toki Migimatsu and Jeannette Bohg. 2022. https://doi.org/10.1109/ICRA46639.2022.9812016 Grounding predicates through actions . In Proceedings of the 2022 International Conference on Robotics and Automation (ICRA), pages 3498--3504

    2022

  21. [29]

    Hideki Nakayama, Akihiro Tamura, and Takashi Ninomiya. 2020. https://aclanthology.org/2020.lrec-1.518/ A visually-grounded parallel corpus with phrase-to-region linking . In Proceedings of the Twelfth Language Resources and Evaluation Conference (LREC), pages 4204--4210

    2020

  22. [30]

    Cennet Oguz, Pascal Denis, Simon Ostermann, Emmanuel Vincent, Natalia Skachkova, and Josef Van Genabith. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.88 MMAR : Multilingual and multimodal anaphora resolution in instructional videos . In Findings of the Association for...

    2024 doi

  23. [31]

    Cennet Oguz, Pascal Denis, Emmanuel Vincent, Simon Ostermann, and Josef van Genabith. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.504 Find-2-Find : Multitask learning for anaphora resolution and object localization . In Proceedings of the 2023 Conference on Empirical Met...

    2023 doi

  24. [32]

    Hikaru Omori and Mamoru Komachi. 2019. https://doi.org/10.18653/v1/N19-1344 Multi-task learning for J apanese predicate argument structure analysis . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human La...

    2019 doi

  25. [33]

    OpenAI. 2024. GPT-4o system card. arXiv:2410.21276

    2024 arXiv

  26. [34]

    Arum Park, Seunghee Lim, and Munpyo Hong. 2015. https://aclanthology.org/Y15-1050/ Zero object resolution in K orean . In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation (PACLIC), pages 439--448, Shanghai, China

    2015

  27. [35]

    Piantadosi, Harry Tily, and Edward Gibson

    Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. https://doi.org/10.1016/j.cognition.2011.10.004 The communicative function of ambiguity in language . Cognition, 122(3):280--291

    2012 doi

  28. [36]

    Plummer, Liwei Wang, Christopher M

    Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2017. https://doi.org/10.1007/s11263-016-0965-7 Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models . Internationa...

    2017 doi

  29. [37]

    Massimo Poesio and Renata Vieira. 1998. https://aclanthology.org/J98-2001/ A corpus-based investigation of definite description use . Computational Linguistics, 24(2):183--216

    1998

  30. [38]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings...

    2021

  31. [39]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf Faster R-CNN: towards real-time object detection with region proposal networks . In Proceedings of the 28th Advanc...

    2015

  32. [40]

    Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. 2016. https://doi.org/10.1007/978-3-319-46448-0_49 Grounding of textual phrases in images by reconstruction . In Proceedings of the 14th European Conference on Computer Vision (ECCV), pages 817--834

    2016 doi

  33. [41]

    Ryohei Sasano, Daisuke Kawahara, and Sadao Kurohashi. 2008. https://aclanthology.org/C08-1097 A fully-lexicalized probabilistic model for J apanese zero anaphora resolution . In Proceedings of the 22nd International Conference on Computational Linguistics (COLING), pages 769--776

    2008

  34. [42]

    Kazuhiro Seki, Atsushi Fujii, and Tetsuya Ishikawa. 2002. A probabilistic method for analyzing J apanese anaphora integrating zero pronoun detection and resolution. In Proceedings of the 19th International Conference on Computational Linguistics (COLING)

    2002

  35. [43]

    Tomohide Shibata and Sadao Kurohashi. 2018. https://doi.org/10.18653/v1/P18-1054 Entity-centric joint modeling of J apanese coreference resolution and predicate argument structure analysis . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis...

    2018 doi

  36. [44]

    Beltran-Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori

    Keisuke Shirai, Cristian C. Beltran-Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori. 2024. https://doi.org/10.1109/ICRA57147.2024.10611112 Vision-language interpreter for robot task planning...

    2024

  37. [45]

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. https://openaccess.thecvf.com/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html Rethinking the inception architecture for computer vision . In Proceedings of th...

    2016

  38. [46]

    Nobuhiro Ueda, Hideko Habe, Akishige Yuguchi, Seiya Kawano, Yasutomo Kawanishi, Sadao Kurohashi, and Koichiro Yoshino. 2024. https://aclanthology.org/2024.lrec-main.829 J - CR e3: A J apanese conversation dataset for real-world reference resolution . In Proceedings of the 2024...

    2024

  39. [47]

    Nobuhiro Ueda, Daisuke Kawahara, and Sadao Kurohashi. 2020. https://doi.org/10.18653/v1/2020.coling-main.114 BERT -based cohesion analysis of J apanese texts . In Proceedings of the 28th International Conference on Computational Linguistics (COLING), pages 1323--1333

    2020 doi

  40. [48]

    Nobuhiro Ueda, Kazumasa Omura, Takashi Kodama, Hirokazu Kiyomaru, Yugo Murawaki, Daisuke Kawahara, and Sadao Kurohashi. 2023. https://doi.org/10.18653/v1/2023.acl-demo.52 KWJA : A unified J apanese analyzer based on foundation models . In Proceedings of the 61st Annual Meeting...

    2023 doi

  41. [49]

    Qinxin Wang, Hao Tan, Sheng Shen, Michael Mahoney, and Zhewei Yao. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.159 MAF : Multimodal alignment framework for weakly-supervised phrase grounding . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language...

    2020 doi

  42. [50]

    Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, and Seungwhan Moon. 2023. https://doi.org/10.18653/v1/2023.acl-long.345 SIMMC - VR : A task-oriented multimodal dialog dataset with situated and immersive VR streams . In Pro...

    2023 doi

  43. [51]

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. https://arxiv.org/abs/2310.11441 Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V . arXiv:2310.11441

    2023 arXiv

  44. [52]

    Juntao Yu and Massimo Poesio. 2020. https://doi.org/10.18653/v1/2020.coling-main.315 Multitask learning-based neural bridging reference resolution . In Proceedings of the 28th International Conference on Computational Linguistics (COLING), pages 3534--3546

    2020 doi

  45. [53]

    Xintong Yu, Hongming Zhang, Ruixin Hong, Yangqiu Song, and Changshui Zhang. 2022. https://doi.org/10.1016/j.patcog.2022.108540 VD-PCR : Improving visual dialog with pronoun coreference resolution . Pattern Recognition, 125:108540

    2022

  46. [54]

    Xintong Yu, Hongming Zhang, Yangqiu Song, Yan Song, and Changshui Zhang. 2019. https://doi.org/10.18653/v1/D19-1516 What you see is what you get: Visual pronoun coreference resolution in dialogues . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language...

    2019 doi

  47. [55]

    a henb \

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr \"a henb \"u hl, and Ishan Misra. 2022. https://doi.org/10.1007/978-3-031-20077-9_21 Detecting twenty-thousand classes using image-level supervision . In Proceedings of the 17th European Conference on Computer Vision (ECCV)...

    2022 doi

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.