Cognitive Limits Shape Language Statistics

Statistical regularities in human language have fascinated researchers for decades, suggesting deep underlying principles governing its evolution and information structuring for efficient communication. While Zipf's Law describes the frequency-rank distribution of words, deviations from this pattern-particularly for less frequent words-challenge the notion of an entirely optimized communication system. Here, we present a theoretical framework that integrates concepts from information theory, network science, and adjacent possible to explain these deviations. We propose that language evolves through optimization processes constrained by the finite cognitive capacities of humans. This results in a dual structure within language: while frequent words form an optimized, highly navigable core, less frequent words reside in a suboptimal regime, requiring more complex combinations to convey meaning. Our findings reveal that Zipf's exponents' deviation to larger values-from 1 to 2-marks a transition from an optimal to a suboptimal state, dictated by cognitive limits. This transition imposes a fundamental limit on communication efficiency, where cognitive constraints lead to a reliance on combinations of words rather than the creation of new vocabulary to express an open-ended conceptual space. A simple model based on the adjacent possible remarkably aligns with the empirical frequency-rank distribution of words in a language. These insights have significant implications for natural language processing and the design of artificial linguistic models, offering new perspectives on optimizing human and machine communication.