{"id":"1f1d5858-c6d9-4dc7-8b29-f6d83468306f","arxiv_id":"2607.28263","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"Long-context memory can be organized by caching intermediate residual states and recomputing only query-conditioned upper layers over a bounded retrieved pack.","lead":"CoMem caches mid-layer residual states per context chunk and only recomputes upper transformer layers on a fixed retrieved pack, so model-side read cost stays constant as stored context grows. It offers a practical layer-axis alternative to token-axis KV compression for long-context LLM memory.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"LoCoMo/quality gains vs full context are retrieval-led; depth reuse is an efficiency–fidelity tradeoff that even distilled j=12 does not win on the same pack.","rationale":"The independence half of the strongest claim is architecturally secure and well measured (fixed pack ~6.5k; H20 18.26 GB / 7.83× at 128k; adapter-free arm). Experiments are careful (template-free protocol, frozen backbone, cluster bootstrap, independent judge, same-pack depth sweeps). The soft spot is causal attribution inside the bundled claim: Motivation I3 and Fig. 3 treat selection as the long-range bottleneck, and Table 8 shows that once selection is fixed, deeper caching does not improve quality—distillation only partially restores it. That is adjacent to the reader’s BM25 concern (selector faithfulness) but more central to whether “depth division of labor” earns the accuracy side of the headline versus “RAG + optional mid-layer cache.” Paper Limitations and conclusion already describe a trade-off rather than free accuracy from cutting depth; the stress test asks that the strongest comparative numbers be presented with j=0 retrieval and YaRN-KV controls so readers cannot over-assign LoCoMo/RULER gains to layer-axis memory. Verdict stays CONDITIONAL (accept-shaped systems result, not a clean accuracy win from depth alone). Agreement with the reader is partial: same risk region (selection/quality externalities), different load-bearing hinge (gain attribution vs BM25 surface-form brittleness).","tokens_in":23935,"tokens_out":771,"duration_ms":73425,"concrete_test":"Add one same-protocol panel: identical BM25 top-12 packs for (a) j=0 full recompute, (b) distilled j=12 CoMem, (c) frozen j=12, plus (d) YaRN-extended full-context KV-Direct on LoCoMo (1,986) and RULER 8k–128k 15-cell macro. If (a)≥(b) on LoCoMo Judge and YaRN-KV closes most of the RULER macro gap to CoMem, reframe the quality half of the claim as retrieval-led with depth as efficiency.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim bundles length-independent model-side read (true by construction for fixed k,c) with competitive quality (LoCoMo 38.27 vs KV-Direct 34.59; RULER 97.05). Table 8 holds the retrieved pack fixed: j=0 full recompute on that pack already scores 41.59 LoCoMo—above distilled CoMem j=12 (38.27) and KV-Direct (34.59)—while frozen j=6/9/12 fall to 32.78/29.15/24.52. So the dialogue edge over full context is bounded selection/noise reduction, not mid-depth caching. Depth is a real cost knob (Table 9: ~29% lower Read at j=12) that incurs readout loss; PG19 self-distillation repairs a lot of it but does not surpass same-pack j=0. RULER’s macro gap vs KV-Direct is further inflated at 128k by unextended RoPE collapse (Table 15: KV-Direct 0s), which the paper labels a stress reference but which still feeds the headline 97.05 vs 78.80 contrast. The layer-axis organization is therefore best read as making a retrieval pack cheaper to read, not as the source of the accuracy win the strongest claim pairs with independence.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that transformer depth is used unevenly—semantic content becomes accessible by mid-depth while upper layers specialize for the query—and turns that observation into CoMem: write each context chunk only through split depth j, cache the residual tensor hj, retrieve a fixed top-k pack (flagship iterative BM25, k=12, c=512), and recompute only layers [j:L] with full cross-chunk attention. For fixed retrieval budget, model-side read compute and memory are independent of stored length. On a frozen continued-trained Qwen3-8B under a unified chat-template-free protocol, a rank-32 PG19 self-distillation LoRA at j=12 yields RULER 97.05 and LoCoMo Judge 38.27 (vs KV-Direct 34.59), with adapter-free H20 controls reporting 18.26 GB vs 89.36 GB and 7.83× full-write-inclusive prefill at 128k. Controlled same-pack depth sweeps, selector ablations, block-diagonal vs full attention, and an adapter-free arm separate retrieval, split depth, and distillation.","tokens_in":24306,"tokens_out":1677,"duration_ms":48705,"significance":"If the efficiency measurements and depth-partition design hold, the work supplies a clean complementary axis for long-context memory: reuse along layers rather than only along tokens. Strengths that raise the contribution above a pure systems note include (i) direct hardware measurements of bounded read working set and prefill (Table 3), (ii) same-pack depth and j=0 retrieval controls that make the quality–cost trade-off falsifiable (Tables 8–9), (iii) task-free PG19 bidirectional-KL LoRA that repairs readout without benchmark labels, (iv) conversation-cluster bootstrap and independent-judge checks on LoCoMo, and (v) ports to sparse MoE backbones. The result is practically relevant for multi-query long stores and conceptually useful as evidence that mid-depth residuals are a compressible, resumable memory unit.","major_comments":[{"comment":"Abstract and Table 1 pair CoMem’s LoCoMo 38.27 and RULER 97.05 against KV-Direct 34.59 / 78.80 in a way that invites attributing the accuracy edge to depth partitioning. Table 8 holds the retrieved pack fixed: j=0 full recompute already scores LoCoMo 41.59—above distilled j=12 (38.27) and KV-Direct (34.59)—while frozen j=6/9/12 fall to 32.78/29.15/24.52. Depth is therefore a real query-cost knob (Table 9: ~29% lower Read at j=12) that incurs readout loss partially repaired by distillation, not the source of the dialogue gain over full context. The abstract, contributions, and Table 1 should lead with j=0-on-pack as a primary baseline and state explicitly that bounded selection drives the LoCoMo edge while depth supplies the efficiency–fidelity trade-off.","section":"Abstract; Table 1; §5.5; Table 8"},{"comment":"The RULER headline contrast (97.05 vs KV-Direct 78.80) mixes in-window cells with unextended native-RoPE stress cells where KV-Direct collapses to 0 at 128k (Table 15). The paper labels those cells stress references and disclaims YaRN comparisons, but the five-benchmark Avg and the abstract still absorb that collapse. Report an in-window (e.g., ≤40,960) RULER macro beside the full 8k–128k macro, or exclude post-native full-context zeros from any head-to-head average used in the abstract.","section":"Abstract; Table 1; Table 15; §5 Baselines and position range"},{"comment":"Motivation I3 and Figure 3 conclude that selection—not cached-state fidelity—is the long-range bottleneck because BM25≈oracle on RULER needles. That diagnostic does not automatically extend to multi-fact or low-lexical-overlap dialogue: Table 5 shows wider retrieval can hurt variable tracking via distractors, BABILong qa1/qa2 expose a clear in-window compression tax versus full context (Table 17), and Limitations already flag surface-form retrieval. Either add a dense/learned retriever control on LoCoMo/BABILong or narrow I3 to “on the reported needle sweep, under lexical overlap.”","section":"§3 (I3); Figure 3; Table 5; Table 17; §7"}],"minor_comments":[{"comment":"Figure 1’s accuracy curve uses the distilled flagship while the 7.83×/18.26 GB inset is adapter-free; the caption states this, but a single panel legend marking “distilled quality / adapter-free efficiency” would reduce misreading.","section":"Figure 1"},{"comment":"Eq. (1) writes |hj|/|KV| = d/(2L dkv) = nq/(2L nkv); a one-line derivation that full KV stores 2L nkv-dimensional vectors per token would help readers verify the 1/18 Qwen3-8B figure.","section":"§4 Notation and storage; Eq. (1)"},{"comment":"Table 1 Avg is an unweighted mean of heterogeneous metrics (RULER 15-cell macro, LoCoMo judge, LongBench F1, etc.). Consider dropping Avg or marking it as descriptive only.","section":"Table 1"},{"comment":"MemoryLLM is repeatedly caveated as out-of-backbone and out-of-template; consider moving it to an appendix diagnostic row so main tables stay within-backbone.","section":"Table 1; Table 2; §5"},{"comment":"Minor prose inconsistencies: title uses “Understanding Is Done Early” while the running header compresses spacing (“UNDERSTANDINGISDONEEARLY”); “residual tensor” / “residual states” alternate; arXiv date line says July 31, 2026.","section":"Title page; passim"},{"comment":"Appendix B judge endpoint lacks a dated snapshot (acknowledged). Pinning model version or releasing raw judge judgments would improve auditability.","section":"Appendix B.2"}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is more careful in the body (especially Table 8 and the Conclusion’s trade-off language) than in the abstract/Table 1 packaging. I view this as a citation-hygiene and framing fix, not a soundness failure—hence minor_revision rather than major. The skeptic note about retrieval-led quality is correct and should be absorbed into the camera-ready framing; it does not erase the efficiency contribution or the depth-as-knob evidence. Fit for a solid systems/NLP venue is good if the attribution edits land."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: CoMem is a clean engineering move—write chunks only to depth j, cache one residual per token, retrieve a fixed pack, recompute the upper band—and the efficiency story holds up. At 128k they show 18.26 GB vs 89.36 GB and 7.83× prefill on an H20. That part is not hand-waving.\n\nWhat is actually new is the integrated design, not any single ingredient. HCache already mid-layer caches and recomputes; KV-Direct already treats residual as enough to rebuild KV; RAG-style work already retrieves. CoMem’s contribution is shallow write + tunable split + bounded external selection + full cross-chunk upper attention, plus a frozen-backbone PG19-only LoRA that pushes the usable j deeper. The eval discipline is better than average for this area: chat-template-free protocol, adapter-free arm, same-pack depth sweeps, oracle/recency/BM25 selectors, block-diagonal ablation, conversation-cluster bootstrap, and a second judge. Circularity is low—the adapter never sees benchmark labels.\n\nSoft spot, in proportion: the stress-test note is right on the quality attribution. Table 8 holds the pack fixed; retrieved j=0 already gets 41.59 LoCoMo, above distilled j=12 (38.27) and KV-Direct (34.59). Frozen deeper splits fall off; distillation repairs a lot but does not beat same-pack full recompute. So the dialogue edge over full context is bounded selection (noise reduction), while depth is a real cost–fidelity knob (~29% cheaper Read at j=12) that you pay for and then partly buy back. The RULER 97 vs 79 headline also leans on unextended RoPE collapse for KV-Direct past the native window—the paper labels that a stress reference, and you should keep that label when you cite it. BM25 and ten LoCoMo clusters are real limits; they own them in Limitations.\n\nMath and storage ratio are straightforward (1/18 of full bf16 KV on Qwen3-8B). Citations are fair to the close ancestors. This is for people building long-context serving and memory systems, not for anyone hunting a new theory of depth. I would bring it to reading group, send it to referees, and cite the constant-read construction and the depth-vs-readout measurements. Engage—just don’t sell the accuracy win as coming from the layer cut alone.","headline":"Solid systems paper: constant model-side read via mid-depth residual cache is real; the LoCoMo win over full context is mostly retrieval, not depth magic.","tokens_in":24965,"tokens_out":616,"would_cite":true,"duration_ms":22844,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Long-context memory can be organized along transformer depth, not only along tokens, by caching mid-layer residual states and recomputing only the upper layers on a fixed retrieved pack.","keywords":["long-context memory","transformer depth","residual stream","KV cache","bounded retrieval","self-distillation","CoMem","layer-axis reuse"],"falsifier":"On tasks where the needed evidence shares little surface form with the query, or exceeds the fixed retrieved pack, measure whether CoMem falls far behind full context while an oracle or strong dense selector does not; if so, the constant-read property can hold while the accuracy claims fail.","tokens_in":24771,"feed_emoji":"💾","tokens_out":951,"duration_ms":32535,"temperature":0.7,"pith_summary":"Transformers do not use depth uniformly: lower and middle layers make semantic content accessible, while upper layers increasingly specialize representations for the current query and prediction. This paper turns that division of labor into CoMem, which writes each context chunk only through an intermediate layer, caches the residual tensor, retrieves a fixed number of relevant tensors, and recomputes only the upper layers with the query present. For a fixed retrieval budget, model-side read compute and memory no longer grow with how much context has been stored. On a frozen backbone with a light self-distillation adapter trained on plain text, the method stays competitive on long-context and dialogue-memory benchmarks while cutting peak memory and prefill cost sharply at long lengths. A sympathetic reader cares because unbounded context becomes a depth-and-retrieval design problem rather than an ever-growing full-depth cache.","feed_headline":"Mid-layer caches make long reads cost fixed, not linear","feed_subtitle":"Store residual states early, retrieve a bounded pack, and recompute only the upper layers.","key_machinery":"CoMem (Comprehension Memory): write each chunk through layers [0:j] and cache the residual hj; retrieve a fixed top-k pack (flagship: iterative BM25); recompute layers [j:L] with full cross-chunk attention and the query present. Split depth j is the quality–cost knob; a rank-32 self-distillation LoRA repairs readout fidelity at deeper j without updating the backbone.","core_discovery":"Semantic information becomes usable near mid-depth while upper layers are more query-conditioned, so a model can cache one intermediate residual state per token, retrieve a bounded relevant pack, and resume only the upper transformer. For fixed retrieval breadth, online read compute and memory are then independent of stored-context length, and long-context memory can be organized along the layer axis rather than only the token axis.","pith_inferences":["If content depth stays near mid-network across scale while zero-shot readable depth deepens, very large models may need little or no adapter for deep caching.","Serving systems could treat mid-layer residual stores as first-class reusable objects, amortizing write cost across many queries on the same document.","Pretraining that deliberately shapes compressible mid-depth interfaces could reduce reliance on post-hoc self-distillation.","Replacing lexical chunk selection with dense or residual-key retrieval is the natural control for paraphrased and multi-hop evidence."],"forward_implications":["Split depth becomes a deployable knob: deeper writes prepay work and cut per-query recompute, at the cost of frozen readout fidelity that light adaptation can repair.","One mid-depth residual per token stores far less than full-depth KV (about 1/18 the bytes on the evaluated 8B model).","Bounded relevance plus full upper-layer cross-chunk attention can match or beat full-context on some dialogue-memory settings by excluding distractors.","The same depth partition keeps a roughly constant model-side read as stored context grows, including on sparse MoE backbones.","In-window tasks can still pay a compression tax: bounded read is not uniform accuracy dominance whenever full context fits."],"fun_headline_variants":["Cache mid-depth residuals; long context reads stay fixed-cost","Upper layers specialize prediction; cache earlier for unbounded memory","CoMem: write chunks mid-model, retrieve bounded pack, recompute top","Layer-axis memory: fixed retrieval budget freestores context length","Depth division of labor yields length-independent long-context reads"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Lexical retrieval over chunk text is good enough to surface the evidence the upper layers need, so selection—not the fidelity of the cached mid-depth state—is the main long-range bottleneck.","fun_headline_variants_meta":{"raw":{"variants":["Cache mid-depth residuals; long context reads stay fixed-cost","Upper layers specialize prediction; cache earlier for unbounded memory","CoMem: write chunks mid-model, retrieve bounded pack, recompute top","Layer-axis memory: fixed retrieval budget freestores context length","Depth division of labor yields length-independent long-context reads"]},"model":"grok-4.5","effort":"low","cost_usd":0.002398,"raw_usage":{"total_tokens":1006,"prompt_tokens":855,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":23984000,"prompt_tokens_details":{"text_tokens":855,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":78,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":855,"tokens_out":73,"duration_ms":2989,"temperature":1.0,"reasoning_tokens":78,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T12:53:25.905096+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On tasks where the needed evidence shares little surface form with the query, or exceeds the fixed retrieved pack, measure whether CoMem falls far behind full context while an oracle or strong dense selector does not; if so, the constant-read property can hold while the accuracy claims fail.","supporting_citations":[],"review_version":1}