REVIEW 8 cited by
Dated Data: Tracing Knowledge Cutoffs in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Released Large Language Models (LLMs) are often paired with a claimed knowledge cutoff date, or the dates at which training data was gathered. Such information is crucial for applications where the LLM must provide up to date information. However, this statement only scratches the surface: do all resources in the training data share the same knowledge cutoff date? Does the model's demonstrated knowledge for these subsets closely align to their cutoff dates? In this work, we define the notion of an effective cutoff. This is distinct from the LLM designer reported cutoff and applies separately to sub-resources and topics. We propose a simple approach to estimate effective cutoffs on the resource-level temporal alignment of an LLM by probing across versions of the data. Using this analysis, we find that effective cutoffs often differ from reported cutoffs. To understand the root cause of this observation, we conduct a direct large-scale analysis on open pre-training datasets. Our analysis reveals two reasons for these inconsistencies: (1) temporal biases of CommonCrawl data due to non-trivial amounts of old data in new dumps and (2) complications in LLM deduplication schemes involving semantic duplicates and lexical near-duplicates. Overall, our results show that knowledge cutoffs are not as simple as they have seemed and that care must be taken both by LLM dataset curators as well as practitioners who seek to use information from these models.
Forward citations
Cited by 8 Pith papers
-
Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
When forecasters are barred from reading post-cutoff text, retrieval still improves Brier score on 8 of 9 LLMs, but only on markets Reddit had discussed beforehand; on speculative topics retrieval makes forecasts worse.
-
ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models
Models leak future knowledge despite explicit temporal cutoffs, as quantified by the ExAnte benchmark across four tasks.
-
AISPA: User-Centric System Prompt Auditing for Large Language Model Applications
A human-in-the-loop audit of system prompts from 88 commercial AI products finds protective instructions nearly universal yet incomplete, with ~40% of products containing at least one user-harmful directive.
-
Benchmarking Open-Weight Foundation Models for Global AI Technical Governance
Open-weight frontier models fabricate ~72% of AI-governance numeric answers, almost never refuse, and show inverted North/South accuracy driven largely by a proportional ±10% scoring rule and sparse high-value indicators.
-
Developer Experience with AI Coding Agents: HTTP Behavioral Signatures in Documentation Portals
AI coding agents produce identifiable HTTP behavioral signatures and compress multi-page navigation into one or two requests, rendering standard engagement metrics unreliable.
-
Sword and Shield: Uses and Strategies of LLMs in Navigating Disinformation
In a 25-participant Werewolf-style game, all roles used an LLM chatbot strategically, as a sword for disinformation and a shield against it.
-
Continually Self-Improving Language Models for Bariatric Surgery Question--Answering
bRAGgen uses a perplexity threshold to trigger web retrieval and LoRA fine-tuning, improving answers on a new bariatric surgery QA dataset, but the evaluation is confounded by test-time adaptation.
-
Hyperbolic Deep Learning for Foundation Models: A Survey
A structured survey of hyperbolic-geometry methods for foundation models, concluding the approach is promising but showing limited independent evidence at scale.
Discussion (0). Continue with ORCID to comment.