{"id":"56585778-0bce-40f1-8dea-472d44e2ceeb","arxiv_id":"2601.11334","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Representation learning limits are restatements of Shannon's source coding, channel coding, and rate-distortion theorems under a dictionary mapping embedding size to code rate.","lead":"This paper maps classical Shannon information-theoretic results onto the last-layer embeddings of neural networks, defining 'representation rate' and 'representation capacity' for regression tasks. A generalist would read it to see whether representation-learning limits are just the old communication-theory bounds wearing new names.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 8's achievability proves a different statement: it uses a designed codebook of 2^{nR} messages, while the theorem requires vanishing error over x∼P_X; in the noiseless limit this contradicts Theorem 6.","rationale":"I read the paper as attempting to translate Shannon source/channel/rate-distortion theorems into statements about last-layer embeddings and to claim these limits are independent of the hypothesis class. The reader's weakest assumption correctly notes that achievability proofs construct arbitrary coding schemes and do not show that a trained network realizes them. My stress test identifies a more fundamental mismatch: even for an arbitrary, unrestricted predictor, Theorem 8 as stated is false under the natural reading that x is drawn from P_X. The proof proves Shannon's channel coding theorem for a designed codebook of M messages, but the theorem's stated error probability is over the source distribution, which has exponentially more typical sequences than the codebook whenever R<H(X). The noiseless binary example makes the inconsistency explicit and yields a direct contradiction with Theorem 6's converse. This is not a mere novelty or framing issue; it invalidates the central capacity claim as written. I would therefore move the verdict from CONDITIONAL to REJECT, unless the theorem is rewritten to state and prove only the channel-coding statement for a message set, which would be correct but would not support the paper's claims about learned representations over a given P_X. Credit is due where the paper follows standard proofs: Theorem 6 and Theorem 12 are essentially classical source-coding and rate-distortion statements, and the dictionary between embedding rate and source entropy is a useful conceptual framing. But the central Theorem 8, and Theorem 13's achievability built on it, do not hold for the operational setting claimed.","tokens_in":81,"tokens_out":15956,"duration_ms":357722,"concrete_test":"Analytical check: take X uniform on {0,1}^n, let V=X and Y=X (noiseless), so I(X;Y)=H(X)=1. Choose R=0.5 and n=100. Any predictor with representation rate R has at most |Z|^q=2^{50} distinct outputs, so under uniform X, Pr[h(Y)=V] ≤ 2^{50}/2^{100}=2^{-50}, hence P_e→1. This directly violates Theorem 8's claim that R<I(X;Y)=1 guarantees P_e→0. Then re-examine Appendix B with the codebook size replaced by the source typical-set size 2^{nH(X)}; the proof's final inequality forces R≥H(X), not R<I(X;Y), confirming that the original proof addresses a different achievability problem.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing defect is that Theorem 8's achievability proof does not prove the stated claim. The proof (Appendix B) follows Shannon's channel coding theorem for a preselected set of M=2^{nR} equiprobable messages: it generates codewords x^n(w) from P_X, transmits them over P_{Y|X}, and shows the decoder can recover the message index w with small error when R<I(X;Y). But the theorem's operational setting requires the predictor to have vanishing error over x∼P_X, whose typical set has size approximately 2^{nH(X)}. With representation rate R=Q_z/n, the predictor can output at most 2^{nR} distinct values. If R<H(X), then even an optimal arbitrary predictor can be correct on at most a fraction 2^{nR}/2^{nH(X)} of the source distribution, so P_e→1, not 0. The contradiction is direct in the noiseless case: set Y=X and g the identity, so I(X;Y)=H(X). Theorem 8's condition R<I(X;Y) becomes R<H(X), asserting P_e→0, while Theorem 6's converse asserts P_e→1 for R<H(X)−ε. The source of the error is that the proof treats the source distribution P_X as a codebook of M messages, whereas in representation learning the source is fixed and the model must cover all of its typical support. Consequently, the central 'representation capacity' theorem is not a theorem about learned predictors for the stated problem; it is the classical channel coding theorem restated for a different, narrower problem. This also undermines Theorem 13's achievability, which inherits the same message-set model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an information-theoretic framework for last-layer embeddings in regression tasks. It defines an embedding representation rate R = Q_z/n, a representation capacity C = max_{P_X} I(X;Y), and a representation rate-distortion function R(D). The central claims are: (i) for a bijective noise-free mapping, a predictor with vanishing error exists iff R ≥ H(X) (Theorem 6); (ii) in a noisy/degraded setting, vanishing error is achievable iff R < I(X;Y) (Theorem 8); (iii) for a compressed output, the minimal embedding rate satisfies Q_z ≥ n I(X;\\hat V) (Theorem 12); and (iv) a unified source-channel separation bound R(D) ≤ I(X;Y) ≤ C (Theorem 13). The paper further claims these limits are independent of the hypothesis class (Remark 3). The proofs in Appendices A–D adapt classical source coding, channel coding, rate-distortion, and source-channel separation arguments.","tokens_in":19378,"tokens_out":6219,"duration_ms":62663,"significance":"If the theorems were correct, they would provide a clean information-theoretic characterization of when a last-layer embedding can reliably represent inputs. The paper has some strengths: it explicitly defines the operational quantities, attempts both achievability and converse arguments, and connects the analysis to known phenomena such as neural collapse and Lipschitz-based generalization. However, the central capacity theorem is not a theorem about the stated representation-learning problem. The achievability proof of Theorem 8 is the classical channel coding theorem for M equiprobable messages, not a statement about a fixed source distribution P_X that the predictor must cover. This is not a local gap: it contradicts the paper's own Theorem 6 in the noiseless case and undermines Theorem 13, which inherits the same message-set model. Because the main claimed contribution rests on this mismatch, the paper cannot be accepted in its present form.","major_comments":[{"comment":"The achievability proof proves a different operational statement. It draws M=2^{nR} messages uniformly, transmits codewords x^n(w), and decodes the message index w. But Theorem 8's predictor must have P_e→0 when y is drawn from P_Y|X(·|x) with x∼P_X. The typical support of P_X has about 2^{nH(X)} sequences, while the predictor has at most 2^{nR} output values. If R<H(X), even an arbitrary predictor can be correct on at most a fraction 2^{nR}/2^{nH(X)} of the source, so P_e→1, not 0. Taking Y=X and g=identity gives I(X;Y)=H(X), so Theorem 8 asserts P_e→0 for R<H(X), while Theorem 6's converse asserts P_e→1 for R<H(X)−ε. Thus Theorem 8 is internally inconsistent with Theorem 6. The correct statement of the proof is Shannon's channel coding theorem for M equiprobable messages, not a fixed-source representation theorem.","section":"§5.2, Theorem 8 and Appendix B"},{"comment":"The claim 'there exists a trained predictor' is not derived anywhere. The achievability constructions are arbitrary assignments on typical sets; no hypothesis class, architecture, optimization procedure, or sample size is specified. This matters because Remark 3 asserts the bounds are independent of the hypothesis class. At most, the proofs show existence in the class of all functions from the typical set to the representation alphabet. Theorem 10 supplies only a Lipschitz-based generalization error bound; it does not show that a Lipschitz network can realize the code constructions. If the hypothesis class is restricted, achievability can fail even though the converses remain valid.","section":"§5.1, Remark 3 and Appendices A–D"},{"comment":"The proof imposes a cost constraint Es(x,y)≤S_max and concludes C(S)=max_{P_X: Es≤S} I(X;Y). However, Theorem 8 is stated without any cost constraint. The proof never shows that the constraint is vacuous or explains how it is removed in the final statement. As written, the theorem and the proof concern different optimization problems. This is especially problematic because the capacity definition in Eq (17) is C=max_{P_X} I(X;Y), with no cost constraint.","section":"Appendix B, Eq (33)"},{"comment":"The sandwich I(X,\\hat V)<\\tilde Q_z/n<I(X;Y) is not established. Equation (23) bounds R(D)≤I(X;\\hat V)≤I(X;Y)≤C, but \\tilde Q_z/n appears nowhere in the achievability or converse proofs of Appendix D. The theorem's condition R(D)<C is about two numerical quantities, not about the embedding's effective support. To justify Eq (24), the proof would need to show that a representation with effective support rate \\tilde Q_z/n can realize, and only can realize, the source-channel code. This is exactly the point where the message-index model versus fixed-source-coverage gap reappears.","section":"§5.4, Theorem 13 and Eq (24)"}],"minor_comments":[{"comment":"The inequality 'd log_2 |V| ≥ 2nH(x)' should presumably be 'nH(X)' rather than '2nH(x)'.","section":"Remark 2 and Appendix A"},{"comment":"The quantifier '∀y∼P_Y|X(·|x)' leaves x free; it should be read as 'for all x∼P_X and y generated according to P_Y|X(·|x)'.","section":"Theorem 8 statement"},{"comment":"In 'generate 2^{nR} code words x^n(w), w=1,2,...,n', the upper limit should be M=2^{nR}, not n. Also, the notation is confusing because w is used both for the message index and (elsewhere) for the source sequence.","section":"Definition 14"},{"comment":"Equation (22) should specify that I(X,\\hat V) is evaluated at the conditional distribution attaining the minimum in Eq (21). The same symbol R is used for the representation rate and the rate-distortion function R(D), which is confusing.","section":"Theorem 12"},{"comment":"The event E3 is defined as 'E3 = (E3 ∪ E3)^c', which is circular and must be corrected. The definitions of E2 and E3 also appear to use complements inconsistently.","section":"Appendix C"},{"comment":"The notation P Y n(y^n_i) is malformed, and H(E) should more precisely be H(E^n) (or the per-symbol entropy rate).","section":"Example 1"}],"recommendation":"reject","confidential_remarks":"For the editor: the paper's central technical contribution is essentially Shannon's source coding, channel coding, and rate-distortion theorems restated in representation-learning terminology. The most serious problem is Theorem 8: its achievability proof establishes a message-transmission result that contradicts the fixed-source Theorem 6 in the noiseless case. This is not a wording issue; the operational setting of the theorem differs from the setting of the proof. Because the claimed representation capacity, the subsequent source-channel theorem, and the 'independence of hypothesis class' claim all depend on this step, the defect is not a local fix. The manuscript might be salvageable only as a much more modest paper that explicitly studies a message-identification model and does not claim to characterize learned predictors for a fixed source distribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper re-frames Shannon's source coding, channel coding, and rate-distortion theorems as limits on last-layer embeddings. The conceptual point that regression embeddings need not compress is worth making, and Theorem 6's entropy bound for bijective mappings is a clean, correct observation. The writing is clear and the AEP arguments are standard.\n\nThe problem is with the central capacity theorem. Theorem 8 claims that if R < I(X;Y), a trained predictor can achieve vanishing error in the noisy setting. The achievability proof (Appendix B) follows Shannon's channel coding theorem: it generates 2^{nR} codewords from P_X and shows a decoder can recover the message index with small error. But that is a different problem. In the operational setting, the source x is drawn from P_X, whose typical set has about 2^{nH(X)} elements. The predictor can output at most 2^{nR} distinct values (since R = log|Z|^q/n). If R < H(X), even an optimal predictor can be correct on at most a fraction 2^{nR}/2^{nH(X)} of the source, so P_e → 1. In the noiseless case Y=X, Theorem 8's condition becomes R < H(X), asserting P_e→0, which directly contradicts Theorem 6's converse (R < H(X) − ε implies P_e→1). The source of the error is that the proof treats the source distribution as a codebook of M=2^{nR} messages, whereas the predictor must cover the entire support of P_X. The paper's Remark 1 asserts \"major differences\" from Shannon's theory, but the math in Theorem 8 is just the channel coding theorem applied to a narrower message-recovery problem.\n\nThere are other soft spots. The achievability proofs assume an arbitrary encoder-decoder pair and then call it a \"trained predictor\"; there is no argument that gradient-based training can realize these codes. Remark 3's claim of hypothesis-class independence is not supported by the proofs. There are also minor typos (e.g., '≥ 2nH(x)' in Appendix A) and an unused cost constraint in Theorem 8's achievability.\n\nI don't think this paper is ready for peer review as it stands. The central theorem is false as stated, and the contradiction with Theorem 6 is glaring. The conceptual discussion and Theorems 6 and 12 could be salvaged into a short note, but the authors need to fix the mismatch between the message-based proof and the source-covering claim. I would not cite it in its current form, and I'd skip it for the reading group.\n\nRecommendation: desk reject, with an invitation to resubmit if the capacity theorem is corrected.","headline":"The paper's central capacity theorem (Theorem 8) is not proved as stated and contradicts its own noiseless result (Theorem 6) in the Y=X limit; the achievability proof solves a message-recovery problem, not the claimed source-covering problem.","tokens_in":19938,"tokens_out":7685,"would_cite":false,"duration_ms":79203,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["94A15","94A17","94A29","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that last-layer embedding reliability is governed by classical information-theoretic quantities: with a bijective input–output map, vanishing error requires embedding rate at least H(X), and no architecture can compensate f","keywords":["representation learning","embedding rate","information theory","rate-distortion","mutual information","neural collapse","regression","Shannon theory"],"falsifier":"On a synthetic ergodic source with known entropy H(X), train a regression model with last-layer embedding rate R just below and just above H(X) under a bijective ground truth; Theorem 6 predicts a sharp transition from test error ~1 to ~0, so any experiment showing reliable recovery for R < H(X) would refute the necessity claim.","tokens_in":18845,"feed_emoji":"🧮","tokens_out":7937,"duration_ms":78277,"temperature":0.7,"pith_summary":"The paper tries to establish that the reliability of a learned last-layer embedding—the representation a network produces just before the final output—is governed by classical information-theoretic limits, not by the architecture or training algorithm. For a bijective input–output mapping, a predictor with arbitrarily small error exists if and only if the embedding rate R = Qz/n (bits per input symbol) is at least the input entropy H(X); below that, error tends to one. In a perturbed or noisy setting, the maximum number of distinguishable signals the embedding can reliably represent is bounded by the mutual information I(X;Y), and in a compressed-output setting the minimal embedding rate is the rate-distortion function R(D). The paper also proves a source–channel separation inequality, R(D) ≤ I(X;Y) ≤ C, and argues that these limits are independent of the hypothesis class. If true, this converts 'how well can this model represent the data?' from a network property into a data property, with direct architectural implications.","feed_headline":"No architecture can fix an undersized last layer","feed_subtitle":"For regression, an embedding must carry at least one bit per bit of input entropy; below that, error tends to one.","key_machinery":"The central object is the embedding representation rate R = Qz/n with Qz = log2|Z|^q, the analogue of code rate in communication theory. The proofs adapt Shannon's achievability and converse arguments: the typical set and joint typicality give achievability, Fano's inequality gives the converse, and the rate-distortion covering argument gives the compressed-output bound. The embedding is treated as a codebook with one distinguishable point per resolvable input, and the input–output pair (or noisy input–clean output pair) as a source–channel pair. This machinery turns the question of how well a neural embedding can perform into a rate question set by the source entropy and mutual information.","core_discovery":"The central claim is that the embedding rate R = q log2|Z|/n — the log-size of the embedding space per input symbol — is the fundamental resource for last-layer representations. Theorem 6: for a bijective target, a predictor with vanishing error exists iff R ≥ H(X); conversely, R < H(X)−ε forces error to one. Theorem 8: with a noisy observation y of x, reliable representation is achievable iff R < I(X;Y), and the effective non-zero embedding support is below 2^{nI(X;Y)}. Theorem 12: for compressed output with distortion ≤ D, minimal embedding rate is R(D) = min I(X;V̂) subject to E[d]≤D. Theorem 13: R(D) ≤ I(X;V̂) ≤ I(X;Y) ≤ C. Proposition 3: class-collapsed representations are insufficient","pith_inferences":["One extension the paper leaves implicit: the theorems are proved for stationary ergodic sources and bijective or surjective maps; extending the rate-distortion limit to non-ergodic or finite-sample settings (for example, natural language sequences) would require a finite-n formulation of R(D), which the paper does not give.","If the independence-of-hypothesis-class claim is taken at face value, it supplies a practical diagnostic: estimate H(X) or I(X;Y) from data, compute the last-layer embedding rate R, and flag models whose R is below the predicted bound—those should fail on any training objective.","A testable corollary of the framework is that an untrained or randomly initialized encoder should not achieve the bound, and that the onset of the transition should be visible as the embedding rate crosses the information-theoretic threshold, independent of architecture."],"forward_implications":["If the embedding rate R falls below H(X), no regression network with a bijective target can drive test error to zero, regardless of width, depth, or training method (Theorem 6).","In noisy settings, increasing the embedding dimension beyond the effective support set by I(X;Y) cannot add reliable discriminability; the embedding's useful capacity is capped by 2^{nI(X;Y)} (Theorem 8).","For compressed outputs, guaranteeing distortion ≤ D requires an embedding rate of at least R(D); any smaller rate forces the minimum achievable distortion to exceed D (Theorem 12).","The separation inequality R(D) ≤ I(X;Y) ≤ C implies that noise and compression combine additively: the information bottleneck is the observation channel, not the network (Theorem 13).","Class-collapsed representations (neural collapse) make regression impossible when the target varies within a class, because all within-class inputs map to the same embedding (Proposition 3)."],"fun_headline_variants":["Last-layer size sets a hard error floor","Regression needs one bit per bit of input entropy","Undersized embeddings guarantee prediction failure","Embedding rate limits last-layer reliability"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The achievability proofs assume that a neural network trained by the learning algorithm can realize the arbitrary typical-set code constructed in the proof; the paper does not establish that gradient-based training finds that code, and with a restricted hypothesis class (e.g., Lipschitz networks) the achievable-rate claims may fail.","fun_headline_variants_meta":{"raw":{"variants":["Last-layer size sets a hard error floor","Regression needs one bit per bit of input entropy","Undersized embeddings guarantee prediction failure","Embedding rate limits last-layer reliability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":991,"prompt_tokens":633,"completion_tokens":358,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":377,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":377,"tokens_out":358,"duration_ms":4659,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:02:47.885216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic ergodic source with known entropy H(X), train a regression model with last-layer embedding rate R just below and just above H(X) under a bijective ground truth; Theorem 6 predicts a sharp transition from test error ~1 to ~0, so any experiment showing reliable recovery for R < H(X) would refute the necessity claim.","supporting_citations":[],"review_version":1}