Pith. sign in

REVIEW 4 major objections 6 minor 47 references

Evaluating List Construction and Temporal Understanding capabilities of Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read New TLQA benchmark shows LLMs struggle to answer time-stamped list questions, with retrieval sharply improving results.

desk verdict Useful benchmark idea and honest legwork, but the frozen April 2024 ground truth for "to this day" questions is a real flaw that skews the headline temporal deficits. read the letter →

arxiv 2506.21783 v1 pith:Y27HU742 submitted 2025-06-26 cs.CL cs.AI

Alexandru Dumitru , V Venktesh , Adam Jatowt , Avishek Anand This is my paper
classification cs.CLcs.AI
keywords temporalquestionansweringlistunderstandingretrieval-augmentedgenerationWikipediainfoboxesLLMevaluationreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces TLQA, a benchmark that asks large language models to output a complete list of entities, each paired with the correct time interval, for questions like "List all political positions Joe Biden held to this day." The authors find that current models, tested in closed-book and open-domain settings, frequently produce incomplete lists and imprecise temporal bounds. Retrieval augmentation, especially with BM25 over Wikipedia infoboxes, substantially improves performance, while giving models the exact evidence pushes F1 to 0.934. The paper argues that temporal understanding and list construction must be evaluated together, because real-world queries often require both simultaneously.

What carries the argument

The central object is the TLQA benchmark itself, built automatically from Wikipedia infoboxes for two Wikidata relations: P54 (member of a sports team) and P39 (position held). The pipeline extracts a seed set of entities from TempLAMA, disambiguates them to Wikidata IDs, parses their infoboxes to extract entity–time-interval pairs, and generates template-based questions. Two extensions, TLQA-TS (time-span) and TLQA-TM (temporal-marker), are produced by prompting GPT-4o to rephrase questions with explicit time intervals or implicit temporal references. The evaluation machinery consists of list construction metrics (precision, recall, F1) and two temporal metrics: Temporal Overlap and Temporal Jaccard, both computed on sets of years from matched entities.

What would settle it

Manually audit a random sample of, say, 100 TLQA ground-truth answers against independent sources (official records, biographies) for missing roles/teams or wrong end years; if a substantial fraction (e.g., more than 5%) contain errors, the headline F1 figures would change and the conclusion about model weaknesses would need re-examination.

Watch

Extended reading notes

Core claim

The central claim is that LLMs exhibit significant shortcomings in time-referenced list-based question answering: they rarely generate complete answer lists and they often misalign entities with accurate time intervals. On the TLQA benchmark, closed-book F1 ranges from 0.330 to 0.525 for Mistral v0.2 and GPT-4o-mini, while temporal overlap consistently exceeds temporal Jaccard, indicating that models can identify the relevant time frames but struggle to pinpoint start and end years. Retrieval augmentation with BM25 raises F1 to 0.685, and providing the golden infobox raises it to 0.934, suggesting that the bottleneck is partly retrieval quality and partly the models' parametric temporal knowledge. The authors also show that adding even one distractor document to golden evidence degrades performance noticeably, and that hard negatives retrieved by BM25 degrade performance more than random distractors.

Load-bearing premise

The ground truth answers come from parsing Wikipedia infoboxes from an April 2024 dump, and any role or team whose end year is missing is treated as still held; if infoboxes are incomplete, outdated, or misparsed, the reported scores measure benchmark errors rather than model errors.

Editorial extensions

If this is right

  • If TLQA reflects real-world performance, LLM-based systems should not be trusted to produce complete, time-stamped lists from parametric knowledge alone; retrieval over structured evidence is needed.
  • BM25-style sparse retrieval is currently more effective than dense retrieval for entity-centric temporal list questions, so retrieval pipelines for such queries should favor sparse methods or combine both.
  • The small gap between open-domain performance and golden-evidence performance indicates that better tabular retrieval is a high-leverage improvement for RAG systems.
  • The degradation caused by even a single distractor suggests that retrieval-augmented generation is sensitive to noise, and that systems should be designed to filter or down-weight retrieved evidence.
  • Implicit temporal references, such as "after serving as Minister of National Defence," are the hardest to resolve, pointing to a specific weakness in temporal reasoning that cannot be fixed by retrieval alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The TLQA methodology could be extended beyond P39 and P54 to other Wikidata relations with temporal fields in infoboxes (e.g., awards, education), which would test whether the observed weaknesses generalize.
  • The consistent gap between Temporal Overlap and Temporal Jaccard suggests that the core temporal defect is interval boundary precision; a follow-up could measure error distributions for start versus end years to see which is more error-prone.
  • Because the ground truth uses an April 2024 Wikipedia dump with missing end years interpreted as "currently held," the benchmark will drift over time for questions about living politicians or active athletes; periodic re-snapshots are needed to keep it current.
  • The authors' finding that random distractors hurt less than retrieved hard negatives implies that models may rely on context salience cues, not just relevance, a hypothesis that could be tested systematically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces TLQA, a benchmark for list-form question answering with temporal intervals. Questions are generated from Wikipedia infoboxes for two Wikidata relations (P39, positions held, and P54, sports teams), with two extensions, TLQA-TS and TLQA-TM, produced by GPT-4o. The authors evaluate Mistral v0.2 and GPT-4o-mini under closed-book, open-domain (BM25 and dense retrieval), and golden-evidence settings, reporting precision, recall, F1, and two temporal metrics. They conclude that models produce incomplete lists, imprecise temporal bounds, and benefit from retrieval, while distractors degrade performance.

Significance. If the benchmark is sound, TLQA fills a genuine gap by jointly testing list construction and temporal alignment, and the release of the corpus, code, and manual evaluation of 100 questions are strengths. The headline results (closed-book F1 0.330–0.525 vs. golden-evidence F1 0.934) would be a useful empirical target for the community. However, the frozen April 2024 answer key and the GPT-4o-based generation and filtering of the extended subsets currently make the quantitative claims fragile and in need of revision.

major comments (4)
  1. [Sec. 3.2.5, Sec. 3.3, Figure 1] The gold answers for open-ended 'to this day' questions are fixed by the April 2024 Wikipedia dump, with a missing end year interpreted as 'currently held' up to the dump date. The paper was submitted in June 2025 and the questions in Table 2 explicitly contain the phrase 'to this day'; hence any model with post-April-2024 knowledge is penalized on temporal bounds and sometimes on entity lists. Figure 1 illustrates this: Joe Biden's answer stops at 2024, so a 2025-correct answer including the presidency through 2025 would be marked wrong. This affects all temporal metrics in Table 5 and is a load-bearing threat to the conclusion that models lack temporal understanding. The authors should either explicitly anchor every question to the evidence date (e.g., 'as of April 2024'), update the gold labels to a consistent evaluation date, or report a sensitivity analysis that excludes the open-ended current-tense questions.
  2. [Sec. 3.2.6, Figures 4–6, Table 7] The TLQA-TS and TLQA-TM subsets are generated by GPT-4o and filtered by GPT-4o using the original infobox answer as the correctness baseline, and GPT-4o-mini is then evaluated on these subsets. This creates a model-family alignment: the gold answers are produced and vetted by the same model family that is later scored, so the high open-domain and golden scores for GPT-4o-mini on these subsets in Table 7 may partly reflect prompt-aligned answer formatting rather than temporal reasoning. The manual evaluation in Table 4 covers only 100 questions and does not remove the systematic generation bias. The authors should add an independent generation or filtering step with a different model or human validation of the extended subsets, and should report how often the GPT-4o filtering changed the generated answers.
  3. [Sec. 4.1, Tables 5–7] Every reported number appears to come from a single run at temperature 0.3, with no standard deviations, confidence intervals, or significance tests. This is particularly problematic for the prompting comparisons: in Table 5, the Mistral closed-book F1 difference between FEW-SHOT and KNN FEW-SHOT is 0.036, and between KNN FEW-SHOT and Auto-COT it is 0.001, yet Section 5.1 concludes that 'KNN FEW-SHOT works best.' The authors should run multiple seeds or varying few-shot exemplars, report variance, and apply a paired significance test before drawing comparative conclusions.
  4. [Sec. 4.2, Table 5] Temporal Overlap and Temporal Jaccard are computed only for matched entities ('averaged across all matched entities'). A model that outputs one single correct entity with exact bounds receives temporal scores of 1.0 regardless of how many ground-truth entities it omitted. This decouples the temporal metrics from list completeness, so the observed gap between closed-book temporal scores (TO 0.384–0.610 in Table 5) and golden-evidence temporal scores conflates missing entities with temporal misalignment. The authors should either compute temporal metrics over all ground-truth entities, counting missing entities as zero-overlap, or report them jointly with recall as a composite measure.
minor comments (6)
  1. [Figure 2] The entity 'Christiano Ronaldo' should be spelled 'Cristiano Ronaldo'.
  2. [Table 5] The retrieval model 'mutli-qa-mpnet' is a typo for 'multi-qa-mpnet'.
  3. [Sec. 3.2.5, Table 3] The text reports an average of 8.641 answers per question, but the per-split means in Table 3 imply approximately 8.44; please reconcile the numbers.
  4. [Sec. 5.2] The claim that BM25 documents outperform all-mini-lm retrieved documents by 45.11% in F1 does not match the values in Table 5, where (0.607 − 0.430)/0.430 is approximately 41%; the percentage should be corrected or the comparison clarified.
  5. [Table 4] The manual evaluation table would benefit from stating the number of annotators and the exact Likert-scale instructions; currently only the means and Cohen's kappa are given.
  6. [Sec. 3.2.5] The convention that a missing end year means 'currently holds' should be stated in the evaluation prompts as well; otherwise the temporal drift issue in Major Comment 1 is hidden from the reader.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: TLQA's gold answers are externally grounded in Wikipedia infoboxes; the evaluated model outputs are not used to define the labels.

full rationale

The paper's central product is a benchmark, not a fitted prediction. Gold answers are derived by parsing Wikipedia infoboxes (Section 3.2.5), an external source independent of the models under test; the evaluation then measures model outputs against these fixed labels (Table 5). There is no parameter fitted to a subset and then 'predicted' on a closely related quantity, and no uniqueness theorem or ansatz is imported from the authors' prior work to force a choice. The TLQA-TS/TM subsets are generated by GPT-4o with prompts that deliberately make the new answer a subset of the original answer ('The answer should be a subset of the original answer'), and the same model family is used for filtering; this is a label-quality and potential-bias concern, but not a circularity in the paper's claims, because the filtered labels are still checked against the original infobox-derived answer and the reported deficits are empirical comparisons. Self-citations (e.g., [3,18,19,21,30,41]) are related-work context only and are not load-bearing. The golden-evidence setup feeds the ground-truth infobox to the model as an upper-bound sanity check; by construction it should be high, but the paper's substantive findings come from closed-book and open-domain gaps. A genuine validity limitation, but not circularity, is that 'to this day' labels are frozen at the April 2024 dump with missing end years interpreted as 'currently held' (Section 3.2.5), so models with post-2024 knowledge may be penalized; this affects correctness/fairness, not self-reference.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the infobox-as-ground-truth assumption and on the disambiguation pipeline. The temperature and retrieval depth are hand-chosen experimental parameters that affect the reported numbers. The LLM-generated TS/TM subsets add a model-dependent axiom for those extended test sets. No invented physical or conceptual entities are introduced.

free parameters (2)
  • temperature = 0.3
    Set by tuning on the validation set (Section 4.1). Affects all reported scores and is a hand-chosen experimental setting, not a derived constant.
  • retrieval_top_k = 3
    Open-domain results use top-3 retrieved documents (Section 5.2). Table 6 shows that varying k changes F1 and temporal metrics, so this choice is load-bearing for the open-domain numbers.
assumptions (4)
  • domain assumption Wikipedia infoboxes constitute complete and accurate ground truth for all positions held and teams played, including all time bounds.
    Section 3.2.5 builds answers by parsing infobox temporal markers; no cross-check against Wikidata or any other source is reported.
  • domain assumption A missing end year in an infobox means the person currently holds the position or remains with the team.
    Stated in Section 3.2.5. This convention propagates into all "to this day" ground truth answers and directly affects temporal metrics.
  • domain assumption The Wikidata entity disambiguation procedure correctly resolves subjects to Wikipedia pages.
    Section 3.2.2 uses object-ID overlap and label matching. An incorrect mapping would pair a question with the wrong infobox and produce wrong answers.
  • domain assumption GPT-4o can reliably generate and filter TLQA-TS/TM question-answer pairs.
    Section 3.2.6 uses GPT-4o for generation and filtering. Manual evaluation is performed on only 100 sampled questions, so the correctness of the remaining subsets rests on the untested reliability of this procedure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating List Construction and Temporal Understanding capabilities of Large Language Models." pith.science (2026). https://pith.science/paper/Y27HU742

@misc{pith2026250621783,
  author       = {Pith},
  title        = {Pith review of: Evaluating List Construction and Temporal Understanding capabilities of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y27HU742}},
  note         = {Machine review of arXiv:2506.21783}
}
read the original abstract

Large Language Models (LLMs) have demonstrated immense advances in a wide range of natural language tasks. However, these models are susceptible to hallucinations and errors on particularly temporal understanding tasks involving multiple entities in answers. In such tasks, they fail to associate entities with accurate time intervals, generate a complete list of entities in answers or reason about events associated with specific temporal bounds. Existing works do not extensively evaluate the abilities of the model to perform implicit and explicit temporal understanding in a list answer construction setup. To bridge this gap, we propose the Time referenced List based Question Answering or TLQA benchmark that requires structured answers in list format aligned with corresponding time periods. Our TLQA benchmark, requires both list construction and temporal understanding simultaneously, which to the best of our knowledge has not been explored in prior benchmarks. We investigate the temporal understanding and list construction capabilities of state-of-the-art generative models on TLQA in closed-book and open-domain settings. Our findings reveal significant shortcomings in current models, particularly their inability to provide complete answers and temporally align facts in a closed-book setup and the need to improve retrieval in open-domain setup, providing clear future directions for research on TLQA. The benchmark and code at https://github.com/elixir-research-group/TLQA.

Figures

Figures reproduced from arXiv: 2506.21783 by the authors.

Figure 1
Figure 1. An example for Time referenced List based QA [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. TLQA data collection pipeline 3.2 TLQA generation We propose an automated solution to generate TLQA. Our focus is on generating questions from entities related to Wikidata relations namely P54 (member of a sports team) and P39 (position held). The main intuition behind this decision is that only these topics, the entities are naturally associated with multiple organizations/positions over different time periods, mak… view at source ↗
Figure 3
Figure 3. Infobox Type Distribution for P39 and P54 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Prompt used to generate a new timeline dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Prompt used for ensuring the quality of automatically [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Performance of different retrieval settings over the two [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Comparison of retrieval models (Recall) time frames but have difficulty in precisely delineating the start and end years, resulting in either overshooting (example [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Comparison of F1 Score and Temporal Overlap for answer generation with golden evidence augmented with varying numbers [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages

  1. [1]

    Samuel Joseph Amouyal, Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022. QAMPARI: : An Open-domain Ques- tion Answering Benchmark for Questions with Many Answers from Multiple Paragraphs. CoRR abs/2205.12665 (2022). doi:10.48550/ARXIV .2205.12665 arXiv:2205.12665

    arXiv doi:10.48550/arxiv.2205.12665 2022

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott G...

    work page 2020

  3. [3]

    Jorge, and Adam Jatowt

    Ricardo Campos, Gaël Dias, Alípio M. Jorge, and Adam Jatowt. 2014. Survey of Temporal Information Retrieval and Related Applications. ACM Comput. Surv. 47, 2, Article 15 (aug 2014), 41 pages. doi:10.1145/2619088

    work page doi:10.1145/2619088 2014

  4. [4]

    Kaushik Chakrabarti, Zhimin Chen, Siamak Shakeri, Guihong Cao, and Surajit Chaudhuri. 2020. TableQnA: Answering List Intent Queries With Web Tables. CoRR abs/2001.04828 (2020). arXiv:2001.04828 https://arxiv.org/abs/2001.04828

    work page Pith review arXiv 2020

  5. [5]

    Jiabao Chen and Yongquan Fan. 2023. Improving temporal question answering using temporal knowledge graph embedding. In2023 4th International Conference on Computer Engineering and Application (ICCEA) . 570–575. doi:10.1109/ ICCEA58433.2023.10135342

    work page arXiv 2023

  6. [6]

    Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A Dataset for Answer- ing Time-Sensitive Questions. arXiv:2108.06314 [cs.CL]

    arXiv 2021

  7. [8]

    Ziyang Chen, Xiang Zhao, Jinzhi Liao, Xinyi Li, and Evangelos Kanoulas

    work page

  8. [9]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The Power of Noise: Redefining Retrieval for RAG Systems. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2024, Vol. 17) . ACM, ...

    arXiv 2024

Show all 47 references
  1. [10]

    Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W

    Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-Aware Language Models as Temporal Knowledge Bases. Transactions of the Association for Computational Linguistics 10 (03 2022), 257–

    2022

  2. [11]

    Simon Gottschalk and Elena Demidova. 2018. EventKG: A Multilingual Event- Centric Temporal Knowledge Graph. arXiv:1804.04526 [cs.CL]

    2018 arXiv

  3. [12]

    Green, Alice K

    Bert F. Green, Alice K. Wolf, Carol Chomsky, and Kenneth Laughery. 1961. Base- ball: an automatic question-answerer. In Papers Presented at the May 9-11, 1961, Western Joint IRE-AIEE-ACM Computer Conference(Los Angeles, California) (IRE-AIEE-ACM ’61 (Western)). Association for...

    1961

  4. [13]

    Dhruv Gupta and Klaus Berberich. 2014. Identifying Time Intervals of Interest to Queries. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (Shanghai, China) (CIKM ’14). Association for Computing Machinery, New York, ...

    2014

  5. [14]

    Suchanek, Klaus Berberich, Edwin Lewis-Kelham, Gerard de Melo, and Gerhard Weikum

    Johannes Hoffart, Fabian M. Suchanek, Klaus Berberich, Edwin Lewis-Kelham, Gerard de Melo, and Gerhard Weikum. 2011. YAGO2: exploring and querying world knowledge in time, space, context, and many languages. In Proceedings of the 20th International Conference Companion on Worl...

    2011

  6. [15]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (mar 2023), 1–38. doi:10.1145/3571730

    2023 doi

  7. [16]

    Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018. TempQuestions: A Benchmark for Temporal Question Answering. In Companion Proceedings of the The Web Conference 2018(Lyon, France)(WWW ’18). International World Wide Web Conferences Ste...

    2018

  8. [17]

    Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex Temporal Question Answering on Knowledge Graphs. InProceedings of the 30th ACM International Conference on Information and; Knowledge Man- agement (CIKM ’21). ACM. doi:10.1145/3459637.3482416

    2021

  9. [18]

    Hideo Joho, Adam Jatowt, and Roi Blanco. 2014. NTCIR temporalia: a test collection for temporal information access research. In Proceedings of the 23rd International Conference on World Wide Web (Seoul, Korea) (WWW ’14 Com- panion). Association for Computing Machinery, New Yor...

    2014

  10. [19]

    Hideo Joho, Adam Jatowt, and Blanco Roi. 2013. A survey of temporal web search experience. In Proceedings of the 22nd International Conference on World Wide Web (Rio de Janeiro, Brazil) (WWW ’13 Companion). Association for Computing Machinery, New York, NY , USA, 1101–1108. do...

    2013

  11. [20]

    Subbarao Kambhampati. 2024. Can large language models reason and plan? Annals of the New York Academy of Sciences 1534, 1 (March 2024), 15–18. doi:10.1111/nyas.15125

    2024 doi

  12. [21]

    Nattiya Kanhabua and Avishek Anand. 2016. Temporal Information Retrieval. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) (SIGIR ’16). Association for Computing Machinery, New York, NY , USA, 1235...

    2016 doi

  13. [22]

    Radev, Noah A

    Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir R. Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2022. RealTime QA: What’s the Answer Right Now? ArXiv abs/2207.13332 (2022). https://api.semanticscholar.org/CorpusI...

    2022 arXiv

  14. [23]

    Spencer Kelly. 2024. Dumpster-dip: wikipedia dump parser. https://github.com/ spencermountain/dumpster-dip

    2024

  15. [24]

    Florian Le Bronnec, Alexandre Vérine, Benjamin Negrevergne, Yann Chevaleyre, and Alexandre Allauzen. 2024. Exploring Precision and Recall to assess the quality and diversity of LLMs. In 62nd Annual Meeting of the Association for Computational Linguistics

    2024

  16. [25]

    Seongyun Lee, Hyunjae Kim, and Jaewoo Kang. 2023. LIQUID: A Framework for List Question Answering Dataset Generation. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Appli- cations of Artificial Intelligence, IAAI ...

    2023

  17. [26]

    Alexandria Leto, Cecilia Aguerrebere, Ishwar Bhati, Ted Willke, Mariano Tep- per, and Vy Ai V o. 2024. Toward Optimal Search and Retrieval for RAG. arXiv:2411.07396 [cs.CL] https://arxiv.org/abs/2411.07396

    2024 arXiv

  18. [27]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Arch...

    2022 doi

  19. [29]

    Katerina Margatina, Shuai Wang, Yogarshi Vyas, Neha Anna John, Yassine Be- najiba, and Miguel Ballesteros. 2023. Dynamic benchmarking of masked lan- guage models on temporal concept drift with multiple views. arXiv preprint arXiv:2302.12297 (2023)

    2023 arXiv

  20. [30]

    Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2022. A Survey on Multi-hop Question Answering and Generation. arXiv:2204.09140 [cs.CL] Evaluating List Construction and T emporal Understanding capabilities of Large Language Models ICTIR ’25, July 18, 2025, Padua, Italy

    2022 arXiv

  21. [31]

    Ioannidis, Soji Adeshina, Phillip R

    Costas Mavromatis, Prasanna Lakkur Subramanyam, Vassilis N. Ioannidis, Soji Adeshina, Phillip R. Howard, Tetiana Grinberg, Nagib Hakim, and George Karypis. 2021. TempoQR: Temporal Question Reasoning over Knowledge Graphs. arXiv:2112.05785 [cs.CL]

    2021 arXiv

  22. [32]

    OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]

    2023 arXiv

  23. [33]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100, 000+ Questions for Machine Comprehension of Text. CoRR abs/1606.05250 (2016). arXiv:1606.05250 http://arxiv.org/abs/1606.05250

    2016 arXiv

  24. [34]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Jian Su, Kevin Duh, and Xavier Carreras (Eds.). Assoc...

    2016 doi

  25. [35]

    Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. Question An- swering Over Temporal Knowledge Graphs. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Proces...

    2021 doi

  26. [36]

    Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple Entity-Centric Questions Challenge Dense Retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . 6138–6148

    2021

  27. [37]

    Chao Shang, Guangtao Wang, Peng Qi, and Jing Huang. 2022. Improving Time Sensitivity for Question Answering over Temporal Knowledge Graphs. InProceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Pr...

    2022 doi

  28. [38]

    Sara Mahdavi, Joelle K

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaek- ermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise ...

    2023 doi arXiv

  29. [39]

    Jungbin Son and Alice Oh. 2023. Time-Aware Representation Learning for Time- Sensitive Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Sin...

    2023 doi

  30. [40]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796 [cs.CL] https://ar...

    2024 arXiv

  31. [41]

    Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022. ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collections. arXiv:2109.03438 [cs.CL]

    2022 arXiv

  32. [42]

    Wonjin Yoon, Richard Jackson, Aron Lagerberg, and Jaewoo Kang. 2022. Se- quence tagging for biomedical extractive question answering. Bioinform. 38, 15 (2022), 3794–3801. doi:10.1093/BIOINFORMATICS/BTAC397

    2022 doi

  33. [43]

    Chen Zhang, Jiuheng Lin, Xiao Liu, Yuxuan Lai, Yansong Feng, and Dongyan Zhao. 2023. How Many Answers Should I Give? An Empirical Study of Multi- Answer Reading Comprehension. CoRR abs/2306.00435 (2023). doi:10.48550/ ARXIV .2306.00435 arXiv:2306.00435

    2023 doi arXiv

  34. [44]

    Tingyi Zhang, Jiaan Wang, Zhixu Li, Jianfeng Qu, An Liu, Zhigang Chen, and Hongping Zhi. 2024. MusTQ: A Temporal Knowledge Graph Question Answering Dataset for Multi-Step Temporal Reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, An...

    2024 doi

  35. [45]

    Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, and Tat-Seng Chua. 2024. Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding. InProceedings of the 62nd Annual Meeting of the Association for Computa...

    2024

  36. [46]

    Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettlemoyer. 2022. RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering.CoRR abs/2210.14353 (2022). doi:10.48550/ARXIV .2210.14353 arXiv:2210.14353

    2022 doi arXiv

  37. [273]

    doi:10.1162/tacl_a_00459 arXiv:https://direct.mit.edu/tacl/article- pdf/doi/10.1162/tacl_a_00459/2004543/tacl_a_00459.pdf

    doi

  38. [1606]

    doi:10.18653/v1/2024.acl-long.87

    2024 doi

  39. [2022]

    Knowledge-Based Systems 251 (2022), 109134

    Temporal knowledge graph question answering via subgraph reason- ing. Knowledge-Based Systems 251 (2022), 109134. doi:10.1016/j.knosys.2022. 109134

    2022 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.