Why These Documents? Explainable Generative Retrieval with Hierarchical Category Paths

Dongha Lee; Jinyoung Yeo; Ryang Heo; Sangam Lee; SeongKu Kang; Susik Yoon

arxiv: 2411.05572 · v3 · submitted 2024-11-08 · 💻 cs.IR

Why These Documents? Explainable Generative Retrieval with Hierarchical Category Paths

Sangam Lee , Ryang Heo , SeongKu Kang , Susik Yoon , Jinyoung Yeo , Dongha Lee This is my paper

Pith reviewed 2026-05-23 17:30 UTC · model grok-4.3

classification 💻 cs.IR

keywords generative retrievalexplainable retrievalhierarchical categoriesdocument rankinginformation retrievalpath-aware ranking

0 comments

The pith

Generative retrieval can explain its choices by first decoding hierarchical category paths before document identifiers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes HyPE to overcome the lack of explanations in generative retrieval, where models directly decode document IDs in response to queries. HyPE generates hierarchical category paths step-by-step, moving from broad to specific semantic categories, before producing the final docid. Training uses external semantic hierarchies and LLM-selected paths to build augmented datasets, while inference applies path-aware ranking to aggregate topic information. Experiments show this yields both higher explainability and better retrieval performance.

Core claim

HyPE first generates hierarchical category paths step-by-step then decodes the docid, using paths constructed from external hierarchies and LLM-selected candidates during training, and path-aware ranking at inference to prioritize relevant documents while supplying detailed explanations for each retrieval decision.

What carries the argument

Hierarchical category paths progressing from broader to more specific semantic categories, used as intermediate generation steps that carry explanation and enable path-aware aggregation in ranking.

If this is right

Users receive step-by-step category-based justifications for each retrieved document.
Path-aware ranking aggregates information across multiple candidate paths to improve final docid ordering.
Models can be trained on path-augmented datasets built from external hierarchies without changing the core generative architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same path-generation approach could be tested in other generative tasks such as question answering to add transparency.
Domains lacking pre-existing high-quality hierarchies may see reduced gains unless alternative path sources are developed.
Errors in LLM path selection could be measured by tracking how often generated paths lead to lower-ranked but still relevant documents.

Load-bearing premise

External high-quality semantic hierarchies combined with LLM-selected paths will consistently provide useful, non-noisy signals that improve both explainability and ranking without introducing new biases or errors in path generation.

What would settle it

Compare retrieval performance and explanation quality on the same test set when paths are replaced by random or noisy category sequences versus the LLM-selected paths; gains should disappear if the assumption holds.

Figures

Figures reproduced from arXiv: 2411.05572 by Dongha Lee, Jinyoung Yeo, Ryang Heo, Sangam Lee, SeongKu Kang, Susik Yoon.

**Figure 2.** Figure 2: Overview of HYPE framework. (1) HYPE constructs category paths using an external high-quality semantic hierarchy and employs LLM to select appropriate candidate paths for each document. (2) Then, HYPE links queries to the paths based on semantic relevance to construct path-augmented training set, and uses this to optimize the retrieval system. (3) During inference, HYPE employs path-aware ranking strategy … view at source ↗

**Figure 3.** Figure 3: Human evaluation of pairwise quality com [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: Performance changes of HYPE. The number of decoded category paths to obtain a ranked docid list. lution, which is different from the previously generated path but relevant to the query. This shows that HYPE can provide effective explanations to users by tailoring them to each query. Analysis of Path-Aware Ranking. To validate the effectiveness of path-aware ranking strategy, we analyze the performance cha… view at source ↗

**Figure 5.** Figure 5: Annotator interface of human evaluation on retrieval system output. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗

**Figure 6.** Figure 6: Annotator interface of human reranking on retrieval system output. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

read the original abstract

Generative retrieval directly decode a document identifier (i.e., docid) in response to a query, making it impossible to provide users with explanations as an answer for ``why is this document retrieved?''. To address this limitation, we propose Hierarchical Category Path-Enhanced Generative Retrieval (HyPE), which enhances explainability by first generating hierarchical category paths step-by-step then decoding docid. By leveraging hierarchical category paths which progress from broader to more specific semantic categories, HyPE can provide detailed explanation for its retrieval decision. For training, HyPE constructs category paths with external high-quality semantic hierarchy, leverages LLM to select appropriate candidate paths for each document, and optimizes the generative retrieval model with path-augmented dataset. During inference, HyPE utilizes path-aware ranking strategy to aggregate diverse topic information, allowing the most relevant documents to be prioritized in the final ranked list of docids. Our extensive experiments demonstrate that HyPE not only offers a high level of explainability but also improves the retrieval performance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

HyPE adds category paths to generative retrieval for explanations and claims ranking gains, but the approach stands or falls on external hierarchies plus LLM path selection.

read the letter

The core idea is straightforward: standard generative retrieval just spits out a docid with no built-in reason, so HyPE generates a hierarchical category path first (broad to specific) before the docid, then uses those paths in a ranking step to combine topic signals. Training pulls paths from an external semantic hierarchy, has an LLM pick suitable ones per document, and augments the data; inference does path-aware aggregation before final docid ranking. This framing is new in how it sequences the path generation inside the generative model and ties it to the ranking objective. It directly targets a usability gap that has been noted in the generative IR literature. The pipeline is described clearly enough that someone could try to reproduce the data construction step. The experiments are said to show gains on both explainability and retrieval metrics, which is the main positive signal. The soft spots sit where the method depends on the external hierarchy being high-quality and the LLM selections being reliable and unbiased. If path generation introduces noise or systematic errors, the explanations lose value and the ranking step could amplify them. The abstract gives no numbers on path accuracy, selection error rates, or ablations that isolate the hierarchy contribution from the extra training data or the ranking trick. Without those, it is hard to tell how much of the reported improvement comes from the new components versus better tuning. This is the kind of paper that fits a reading group on generative retrieval or explainable search systems. Readers working on production IR who need transparency features would find the method description useful even if they adapt parts of it. The work shows clear engagement with the problem and prior generative retrieval setups, so it deserves a serious referee to examine the full experimental details, ablations, and path quality metrics rather than a desk reject.

Referee Report

1 major / 0 minor

Summary. The paper proposes Hierarchical Category Path-Enhanced Generative Retrieval (HyPE) to address the lack of explainability in standard generative retrieval, which directly decodes document identifiers. HyPE generates hierarchical category paths (from broad to specific semantic categories) step-by-step before decoding the docid, constructs these paths using external high-quality semantic hierarchies and LLM-based selection for each document, augments the training data accordingly, and applies a path-aware ranking strategy at inference to aggregate topic information. The central claim is that this yields both high explainability and improved retrieval performance, as supported by extensive experiments.

Significance. If the experimental results hold, the work could meaningfully advance explainable generative retrieval in information retrieval by providing a structured, hierarchical mechanism for explaining retrieval decisions. The design choice to combine external hierarchies with LLM path selection and path-augmented training is a concrete contribution that could be tested for generalizability across domains.

major comments (1)

[Abstract] Abstract: The assertion that 'Our extensive experiments demonstrate that HyPE not only offers a high level of explainability but also improves the retrieval performance' provides no information on datasets, baselines, metrics (for either performance or explainability), controls, or statistical significance. This is load-bearing for the central claim, as the paper's primary contribution is framed as an empirical improvement.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback. We address the single major comment below and will revise the manuscript accordingly.

read point-by-point responses

Referee: [Abstract] Abstract: The assertion that 'Our extensive experiments demonstrate that HyPE not only offers a high level of explainability but also improves the retrieval performance' provides no information on datasets, baselines, metrics (for either performance or explainability), controls, or statistical significance. This is load-bearing for the central claim, as the paper's primary contribution is framed as an empirical improvement.

Authors: We agree that the abstract would be strengthened by including concrete details on the experimental setup. In the revised manuscript we will expand the abstract to specify the primary datasets (MS MARCO, Natural Questions), the main generative and non-generative baselines, the core metrics (MRR, Recall@K for performance; path fidelity and human-rated explainability scores), and note that improvements are statistically significant under paired t-tests. This change directly addresses the concern while respecting abstract length limits. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper describes a generative retrieval method (HyPE) that augments training data by constructing hierarchical category paths from external semantic hierarchies and LLM-based selection, then applies a path-aware ranking strategy at inference. No equations, derivations, fitted parameters renamed as predictions, or uniqueness theorems are presented. All load-bearing components (hierarchies, path selection, and performance gains) are positioned as external inputs or empirical outcomes rather than self-referential reductions. The method is therefore self-contained against external benchmarks and does not exhibit any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

Review based on abstract only; no explicit free parameters, axioms, or invented entities detailed beyond the proposed method name.

invented entities (1)

HyPE no independent evidence
purpose: Enhance explainability and performance in generative retrieval via hierarchical paths
The method is introduced in the abstract as the core contribution.

pith-pipeline@v0.9.0 · 5718 in / 1047 out tokens · 28201 ms · 2026-05-23T17:30:48.629228+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/AbsoluteFloorClosure.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

HYPE constructs category paths with external high-quality semantic hierarchy, leverages LLM to select appropriate candidate paths... path-aware ranking strategy
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

hierarchical category paths which progress from broader to more specific semantic categories

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Model Editing for New Document Integration in Generative Information Retrieval
cs.IR 2026-03 unverdicted novelty 7.0

DOME adapts generative IR models to unseen documents via critical-layer identification, hybrid-label edit vector optimization, and parameter updates, achieving strong new-document retrieval with reduced training cost.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages · cited by 1 Pith paper · 2 internal anchors

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

work page
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

work page
[3]

Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. https://aclanthology.org/S12-1051 S em E val-2012 task 6: A pilot on semantic textual similarity . In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics -- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth...

work page 2012
[4]

Avishek Anand, Lijun Lyu, Maximilian Idahl, Yumeng Wang, Jonas Wallat, and Zijian Zhang. 2022. Explainable information retrieval: A survey. arXiv preprint arXiv:2211.02405

work page arXiv 2022
[5]

Michele Bevilacqua, Giuseppe Ottaviano, Patrick S. H. Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/cd88d62a2063fdaf7ce6f9068fb15dcd-Abstract-Conference.html Autoregressive search engines: Generating substrings as document identifiers . In NeurIPS

work page 2022
[6]

Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. https://openreview.net/forum?id=5k8F6UU39V Autoregressive entity retrieval . In ICLR

work page 2021
[7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

work page doi:10.18653/v1/n19-1423 2019
[8]

The Llama 3 Herd of Models

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others. 2024. https://api.semanticscholar.org/CorpusID:271571434 ...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[9]

Ferragina and G

P. Ferragina and G. Manzini. 2000. https://doi.org/10.1109/SFCS.2000.892127 Opportunistic data structures with applications . In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 390--398

work page doi:10.1109/sfcs.2000.892127 2000
[10]

Maarten Grootendorst. 2020. https://doi.org/10.5281/zenodo.4461265 Keybert: Minimal keyword extraction with bert

work page doi:10.5281/zenodo.4461265 2020
[11]

Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.813 Can large language models be good emotional supporter? mitigating preference bias on emotional support conversation . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

work page doi:10.18653/v1/2024.acl-long.813 2024
[12]

SeongKu Kang, Yunyi Zhang, Pengcheng Jiang, Dongha Lee, Jiawei Han, and Hwanjo Yu. 2024 b . https://api.semanticscholar.org/CorpusID:273638526 Taxonomy-guided semantic indexing for academic paper search . ArXiv, abs/2410.19218

work page arXiv 2024
[13]

Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.799 SODA : Million-scale dialogue distillation with social commonsense contextualization . In Proceedings of the 2023 Conference on Empirical Methods in...

work page doi:10.18653/v1/2023.emnlp-main.799 2023
[14]

Lost in the Middle: How Language Models Use Long Contexts

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. https://doi.org/10.1162/tacl\_a\_00276 Natural questions: a ...

work page internal anchor Pith review doi:10.1162/tacl 2019
[15]

Dongha Lee, Jiaming Shen, SeongKu Kang, Susik Yoon, Jiawei Han, and Hwanjo Yu. 2022. https://api.semanticscholar.org/CorpusID:246015411 Taxocom: Topic taxonomy completion with hierarchical discovery of novel topic clusters . Proceedings of the ACM Web Conference 2022

work page 2022
[16]

Sangam Lee, Ryang Heo, SeongKu Kang, and Dongha Lee. 2025. https://api.semanticscholar.org/CorpusID:277452715 Imagine all the relevance: Scenario-profiled indexing with knowledge expansion for dense retrieval . ArXiv, abs/2503.23033

work page arXiv 2025
[17]

Sunkyung Lee, Minjin Choi, and Jongwuk Lee. 2023. https://api.semanticscholar.org/CorpusID:265033969 Glen: Generative retrieval via lexical index learning . In Conference on Empirical Methods in Natural Language Processing

work page 2023
[18]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.acl-main.703 BART : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension . In Proceedings of the 58th Annual Meeting of the Associat...

work page doi:10.18653/v1/2020.acl-main.703 2020
[19]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. https://ceur-ws.org/Vol-1773/CoCoNIPS\_2016\_paper9.pdf MS MARCO: A human generated machine reading comprehension dataset . In NeurIPS

work page 2016
[20]

Rodrigo Nogueira and Jimmy Lin. 2020. From doc2query to doctttttquery. Online preprint

work page 2020
[21]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . J. Mach. Learn. Res., 21:140:1--140:67

work page 2020
[22]

S. E. Robertson and S. Walker. 1997. https://doi.org/10.1145/258525.258529 On relevance weights with little relevance information . In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '97, page 16–24, New York, NY, USA. Association for Computing Machinery

work page doi:10.1145/258525.258529 1997
[23]

Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: Bm25 and beyond . Found. Trends Inf. Retr., 3(4):333–389

work page doi:10.1561/1500000019 2009
[24]

Zhihong Shen, Hao Ma, and Kuansan Wang. 2018. https://doi.org/10.18653/v1/P18-4015 A web-scale system for scientific knowledge exploration . In Proceedings of ACL 2018, System Demonstrations , pages 87--92, Melbourne, Australia. Association for Computational Linguistics

work page doi:10.18653/v1/p18-4015 2018
[25]

Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023. https://doi.org/10.48550/arXiv.2304.04171 Learning to tokenize for generative retrieval . CoRR

work page doi:10.48550/arxiv.2304.04171 2023
[26]

Guo, Jiangui Chen, Zuowei Zhu, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng

Yubao Tang, Ruqing Zhang, J. Guo, Jiangui Chen, Zuowei Zhu, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2023. https://api.semanticscholar.org/CorpusID:258865792 Semantic-enhanced differentiable search index inspired by learning strategies . Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

work page 2023
[27]

Cohen, and Donald Metzler

Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Prakash Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/892840a6123b5ec99ebaab8be1530fba-Abstract-Conference.html Transformer memory as a differentiable search index . In NeurIPS

work page 2022
[28]

Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/a46156bd3579c3b268108ea6aca71d13-Abstract-Conference.html A neural corpus indexer for document retrieval . In NeurIPS

work page 2022
[29]

Zihan Wang, Yujia Zhou, Yiteng Tu, and Zhicheng Dou. 2023. https://doi.org/10.1145/3583780.3614993 Novo: Learnable and interpretable document identifiers for model-based ir . In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM '23, page 2656–2665, New York, NY, USA. Association for Computing Machinery

work page doi:10.1145/3583780.3614993 2023
[30]

Sadler, Michelle T

Chao Zhang, Fangbo Tao, Xiusi Chen, Jiaming Shen, Meng Jiang, Brian M. Sadler, Michelle T. Vanni, and Jiawei Han. 2018. https://api.semanticscholar.org/CorpusID:47017463 Taxogen: Unsupervised topic taxonomy construction by adaptive term embedding and clustering . Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

work page 2018
[31]

Peitian Zhang, Zheng Liu, Yujia Zhou, Zhicheng Dou, and Zhao Cao. 2023. https://api.semanticscholar.org/CorpusID:258841428 Generative retrieval via term set generation . In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

work page 2023
[32]

Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang, and Ji - Rong Wen. 2022. https://doi.org/10.48550/arXiv.2208.09257 Ultron: An ultimate retriever on corpus with a model-based indexer . CoRR

work page doi:10.48550/arxiv.2208.09257 2022
[33]

Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2023. https://doi.org/10.48550/arXiv.2206.10128 Bridging the gap between indexing and retrieval for differentiable search index with query generation . In Gen-IR@SIGIR

work page doi:10.48550/arxiv.2206.10128 2023

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

work page

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

work page

[3] [3]

Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. https://aclanthology.org/S12-1051 S em E val-2012 task 6: A pilot on semantic textual similarity . In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics -- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth...

work page 2012

[4] [4]

Avishek Anand, Lijun Lyu, Maximilian Idahl, Yumeng Wang, Jonas Wallat, and Zijian Zhang. 2022. Explainable information retrieval: A survey. arXiv preprint arXiv:2211.02405

work page arXiv 2022

[5] [5]

Michele Bevilacqua, Giuseppe Ottaviano, Patrick S. H. Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/cd88d62a2063fdaf7ce6f9068fb15dcd-Abstract-Conference.html Autoregressive search engines: Generating substrings as document identifiers . In NeurIPS

work page 2022

[6] [6]

Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. https://openreview.net/forum?id=5k8F6UU39V Autoregressive entity retrieval . In ICLR

work page 2021

[7] [7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

work page doi:10.18653/v1/n19-1423 2019

[8] [8]

The Llama 3 Herd of Models

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others. 2024. https://api.semanticscholar.org/CorpusID:271571434 ...

work page internal anchor Pith review Pith/arXiv arXiv 2024

[9] [9]

Ferragina and G

P. Ferragina and G. Manzini. 2000. https://doi.org/10.1109/SFCS.2000.892127 Opportunistic data structures with applications . In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 390--398

work page doi:10.1109/sfcs.2000.892127 2000

[10] [10]

Maarten Grootendorst. 2020. https://doi.org/10.5281/zenodo.4461265 Keybert: Minimal keyword extraction with bert

work page doi:10.5281/zenodo.4461265 2020

[11] [11]

Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.813 Can large language models be good emotional supporter? mitigating preference bias on emotional support conversation . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

work page doi:10.18653/v1/2024.acl-long.813 2024

[12] [12]

SeongKu Kang, Yunyi Zhang, Pengcheng Jiang, Dongha Lee, Jiawei Han, and Hwanjo Yu. 2024 b . https://api.semanticscholar.org/CorpusID:273638526 Taxonomy-guided semantic indexing for academic paper search . ArXiv, abs/2410.19218

work page arXiv 2024

[13] [13]

Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.799 SODA : Million-scale dialogue distillation with social commonsense contextualization . In Proceedings of the 2023 Conference on Empirical Methods in...

work page doi:10.18653/v1/2023.emnlp-main.799 2023

[14] [14]

Lost in the Middle: How Language Models Use Long Contexts

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. https://doi.org/10.1162/tacl\_a\_00276 Natural questions: a ...

work page internal anchor Pith review doi:10.1162/tacl 2019

[15] [15]

Dongha Lee, Jiaming Shen, SeongKu Kang, Susik Yoon, Jiawei Han, and Hwanjo Yu. 2022. https://api.semanticscholar.org/CorpusID:246015411 Taxocom: Topic taxonomy completion with hierarchical discovery of novel topic clusters . Proceedings of the ACM Web Conference 2022

work page 2022

[16] [16]

Sangam Lee, Ryang Heo, SeongKu Kang, and Dongha Lee. 2025. https://api.semanticscholar.org/CorpusID:277452715 Imagine all the relevance: Scenario-profiled indexing with knowledge expansion for dense retrieval . ArXiv, abs/2503.23033

work page arXiv 2025

[17] [17]

Sunkyung Lee, Minjin Choi, and Jongwuk Lee. 2023. https://api.semanticscholar.org/CorpusID:265033969 Glen: Generative retrieval via lexical index learning . In Conference on Empirical Methods in Natural Language Processing

work page 2023

[18] [18]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.acl-main.703 BART : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension . In Proceedings of the 58th Annual Meeting of the Associat...

work page doi:10.18653/v1/2020.acl-main.703 2020

[19] [19]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. https://ceur-ws.org/Vol-1773/CoCoNIPS\_2016\_paper9.pdf MS MARCO: A human generated machine reading comprehension dataset . In NeurIPS

work page 2016

[20] [20]

Rodrigo Nogueira and Jimmy Lin. 2020. From doc2query to doctttttquery. Online preprint

work page 2020

[21] [21]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . J. Mach. Learn. Res., 21:140:1--140:67

work page 2020

[22] [22]

S. E. Robertson and S. Walker. 1997. https://doi.org/10.1145/258525.258529 On relevance weights with little relevance information . In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '97, page 16–24, New York, NY, USA. Association for Computing Machinery

work page doi:10.1145/258525.258529 1997

[23] [23]

Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: Bm25 and beyond . Found. Trends Inf. Retr., 3(4):333–389

work page doi:10.1561/1500000019 2009

[24] [24]

Zhihong Shen, Hao Ma, and Kuansan Wang. 2018. https://doi.org/10.18653/v1/P18-4015 A web-scale system for scientific knowledge exploration . In Proceedings of ACL 2018, System Demonstrations , pages 87--92, Melbourne, Australia. Association for Computational Linguistics

work page doi:10.18653/v1/p18-4015 2018

[25] [25]

Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023. https://doi.org/10.48550/arXiv.2304.04171 Learning to tokenize for generative retrieval . CoRR

work page doi:10.48550/arxiv.2304.04171 2023

[26] [26]

Guo, Jiangui Chen, Zuowei Zhu, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng

Yubao Tang, Ruqing Zhang, J. Guo, Jiangui Chen, Zuowei Zhu, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2023. https://api.semanticscholar.org/CorpusID:258865792 Semantic-enhanced differentiable search index inspired by learning strategies . Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

work page 2023

[27] [27]

Cohen, and Donald Metzler

Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Prakash Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/892840a6123b5ec99ebaab8be1530fba-Abstract-Conference.html Transformer memory as a differentiable search index . In NeurIPS

work page 2022

[28] [28]

Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/a46156bd3579c3b268108ea6aca71d13-Abstract-Conference.html A neural corpus indexer for document retrieval . In NeurIPS

work page 2022

[29] [29]

Zihan Wang, Yujia Zhou, Yiteng Tu, and Zhicheng Dou. 2023. https://doi.org/10.1145/3583780.3614993 Novo: Learnable and interpretable document identifiers for model-based ir . In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM '23, page 2656–2665, New York, NY, USA. Association for Computing Machinery

work page doi:10.1145/3583780.3614993 2023

[30] [30]

Sadler, Michelle T

Chao Zhang, Fangbo Tao, Xiusi Chen, Jiaming Shen, Meng Jiang, Brian M. Sadler, Michelle T. Vanni, and Jiawei Han. 2018. https://api.semanticscholar.org/CorpusID:47017463 Taxogen: Unsupervised topic taxonomy construction by adaptive term embedding and clustering . Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

work page 2018

[31] [31]

Peitian Zhang, Zheng Liu, Yujia Zhou, Zhicheng Dou, and Zhao Cao. 2023. https://api.semanticscholar.org/CorpusID:258841428 Generative retrieval via term set generation . In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

work page 2023

[32] [32]

Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang, and Ji - Rong Wen. 2022. https://doi.org/10.48550/arXiv.2208.09257 Ultron: An ultimate retriever on corpus with a model-based indexer . CoRR

work page doi:10.48550/arxiv.2208.09257 2022

[33] [33]

Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2023. https://doi.org/10.48550/arXiv.2206.10128 Bridging the gap between indexing and retrieval for differentiable search index with query generation . In Gen-IR@SIGIR

work page doi:10.48550/arxiv.2206.10128 2023