{"id":"13f5796b-c1db-4875-a5ad-6300a02e2415","arxiv_id":"2605.24786","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"CONF-KV dynamically allocates KV cache budget via model confidence, matching full-KV perplexity within 1.5-2.1 points at sliding-window memory cost and improving retrieval accuracy.","lead":"CONF-KV uses the model's next-token confidence to dynamically set KV cache size during long inference, keeping more tokens when uncertain. This could reduce memory use for extended contexts while staying close to full-cache performance.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Next-token confidence scalar may not reliably signal when additional context retention is needed","rationale":"This matches the reader's weakest_assumption exactly. Because the original verdict was already UNVERDICTED on abstract-only information, surfacing the unvalidated proxy assumption does not alter the verdict category but confirms why more evidence is required.","tokens_in":1826,"tokens_out":298,"duration_ms":33376,"concrete_test":"Ablation: fix the per-step budget to the average budget observed under CONF-KV on each task, rerun NIAH (32K) and the 75 VisualWebArena tasks, and compare accuracy/success rate to the original variable-budget results. If the fixed-budget version matches within statistical noise, the confidence-driven budgeting is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that a scalar derived from the next-token distribution can be used to set a per-step cache budget that retains task-critical tokens better than fixed or attention-only policies. This assumes the model's instantaneous uncertainty is a good proxy for missing long-range context rather than other sources of entropy (ambiguity, model capacity limits). The abstract describes the composite attention-plus-recency ranking and protected window but supplies no ablation, correlation study, or oracle comparison isolating whether the dynamic budget component drives the reported gains on NIAH or VisualWebArena; those gains could arise from the ranking or window alone.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces CONF-KV, a KV-cache eviction policy for long-horizon LLM inference that derives a scalar confidence score from the next-token distribution to dynamically adjust the per-step cache budget, ranks tokens by a composite of accumulated attention and recency within the budget, and incorporates a protected recent window, blockwise attention, and mixed-precision storage. It claims to achieve performance close to full KV cache with memory footprint similar to a 512-token sliding window, with specific gains on Needle-in-a-Haystack (91.4% accuracy) and VisualWebArena (95.3% of full-KV success at 2.8x lower memory).","tokens_in":1951,"tokens_out":561,"duration_ms":25308,"significance":"If the empirical results hold and the confidence signal is shown to drive the gains, the method could provide a practical, low-overhead way to reduce KV-cache memory for long-context inference while preserving most of the full-cache capability across model families. The combination of dynamic budgeting with mixed-precision storage and pyramidal per-layer allocation adds engineering utility for deployment.","major_comments":[{"comment":"Abstract: The reported improvements on Needle-in-a-Haystack (91.4% vs. 53.8% sliding window, 80.6% H2O) and VisualWebArena (95.3% of full-KV success) are attributed to the confidence-aware per-step budget selection, yet no ablation isolating the contribution of the next-token confidence scalar versus the attention-recency ranking or protected window is described. This omission leaves open whether the central innovation is load-bearing for the claimed gains.","section":"Abstract"},{"comment":"Abstract and experimental results: Concrete numbers are given for perplexity (within 1.5--2.1 points of full KV), retrieval accuracy, and memory reduction, but the text supplies no error bars, number of runs, statistical tests, or variance across the four model families and generated lengths up to 4K. This weakens the ability to assess whether the dynamic budget component reliably outperforms fixed or attention-only baselines.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract states results across four model families but does not name them or provide per-family breakdowns; adding this would improve reproducibility.","section":"Abstract"},{"comment":"Notation for the composite ranking score and the exact form of the confidence scalar (e.g., entropy, max-probability, or other) should be defined explicitly in the method section rather than left implicit.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript accordingly to strengthen the presentation of results.","responses":[{"response":"We agree that an explicit ablation isolating the next-token confidence scalar is required to demonstrate its contribution relative to the attention-recency ranking and protected window. In the revised manuscript we will add a dedicated ablation study that disables the confidence-based budget allocation while keeping the remaining components fixed, thereby quantifying the incremental benefit of the central innovation.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The reported improvements on Needle-in-a-Haystack (91.4% vs. 53.8% sliding window, 80.6% H2O) and VisualWebArena (95.3% of full-KV success) are attributed to the confidence-aware per-step budget selection, yet no ablation isolating the contribution of the next-token confidence scalar versus the attention-recency ranking or protected window is described. This omission leaves open whether the central innovation is load-bearing for the claimed gains."},{"response":"We concur that statistical details are needed to support reliability claims. The revised manuscript will report standard deviations as error bars for all primary metrics, state the number of independent runs performed, and include variance summaries across the four model families and sequence lengths up to 4K. Where feasible we will also add statistical significance tests against the fixed and attention-only baselines.","revision_made":"yes","referee_comment":"[Abstract] Abstract and experimental results: Concrete numbers are given for perplexity (within 1.5--2.1 points of full KV), retrieval accuracy, and memory reduction, but the text supplies no error bars, number of runs, statistical tests, or variance across the four model families and generated lengths up to 4K. This weakens the ability to assess whether the dynamic budget component reliably outperforms fixed or attention-only baselines."}],"tokens_in":1500,"tokens_out":423,"duration_ms":22794,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core idea is to turn the model's own next-token distribution into a scalar that decides how large the cache budget should be at each decoding step, keeping more tokens when uncertainty is high. Inside that budget they rank by accumulated attention plus recency and protect a short recent window, then layer on blockwise online-softmax and FP16/INT8 storage with a pyramidal per-layer variant.\n\nThe paper does show usable numbers across four model families. On needle-in-a-haystack up to 32k tokens it reaches 91.4% retrieval accuracy against 53.8% for a fixed 512-token window and 80.6% for H2O. On 75 VisualWebArena tasks it keeps 95.3% of full-KV success rate at 2.8 times lower peak memory while staying within 1.5-2.1 perplexity of the full cache and matching the memory footprint of the sliding-window baseline.\n\nThe soft spot is the missing evidence on whether the confidence signal is actually doing work. The abstract gives no ablation that isolates the dynamic budget from the composite ranking or the protected window, no correlation between the scalar and downstream token importance, and no error bars or statistical tests. Without those, the gains could come from the attention-recency part alone. The stress-test concern lands: the abstract does not address it.\n\nThis is aimed at practitioners who need longer contexts on fixed hardware and are already running attention-based eviction. It is worth sending to a serious referee because the problem is real, the combination is new relative to the cited baselines, and the reported deltas are large enough to justify checking the details, even though the current write-up is thin on verification.","headline":"CONF-KV adds a next-token confidence signal to set dynamic per-step KV budgets on top of attention-recency ranking and mixed-precision storage, with reported gains on NIAH and VisualWebArena, but the abstract supplies no ablations to show the confidence component drives the improvement.","tokens_in":2421,"tokens_out":449,"would_cite":false,"duration_ms":16329,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"CONF-KV uses the model's next-token uncertainty to set a dynamic KV cache budget that keeps perplexity within 2 points of full cache while matching a fixed 512-token window's memory use.","keywords":["KV cache eviction","LLM inference","confidence-aware management","long-context models","mixed-precision storage","attention mechanisms","memory optimization","next-token prediction"],"falsifier":"A controlled test in which high-confidence next-token predictions still require distant tokens that the eviction policy discards, producing a clear drop in accuracy beyond the reported 1.5-2.1 perplexity band or below the 91.4 percent needle retrieval figure.","tokens_in":2722,"feed_emoji":"💾","tokens_out":814,"duration_ms":23270,"temperature":0.7,"pith_summary":"The paper introduces a cache manager that reads the next-token probability distribution as a confidence signal and expands or contracts the allowed number of retained tokens accordingly. When uncertainty is high the system holds more context; when the model is confident it evicts more aggressively. Inside the chosen budget, tokens are kept or dropped by a combination of total attention they have received and their recency, with a protected recent window always preserved. The method is paired with blockwise online-softmax attention and mixed FP16/INT8 storage. Across multiple model families the resulting system matches the memory footprint of a static 512-token window yet stays within 1.5-2.1 perplexity points of an unpruned cache, raises needle-in-a-haystack retrieval accuracy to 91.4 percent, and retains 95.3 percent of full-KV task success on VisualWebArena at 2.8 times lower peak memory.","feed_headline":"Next-token confidence prunes KV cache to sliding-window size","feed_subtitle":"Dynamic budget from model uncertainty keeps perplexity within 2 points of full cache and raises long-sequence retrieval to 91 percent.","key_machinery":"The confidence-derived budget allocator that turns the next-token softmax into a scalar deciding how many tokens to retain at each decoding step.","core_discovery":"CONF-KV converts the next-token distribution into a scalar confidence score that directly determines the per-step cache budget, ranks surviving tokens by accumulated attention mass plus recency, protects a recent window for local coherence, and combines the policy with blockwise online-softmax attention and mixed-precision storage to reduce memory and compute for long-horizon generation.","pith_inferences":["The same uncertainty signal could be used to vary the protected window size itself rather than keeping it fixed.","Layer-wise budget allocation derived from per-layer confidence might further reduce memory on deeper models without hurting coherence.","The approach may allow existing hardware to support longer effective contexts on tasks where the model quickly becomes confident.","If attention scores become unreliable at very large scale, the ranking component could be replaced by a cheaper recency-plus-confidence heuristic."],"forward_implications":["On sequences up to 32K tokens, needle retrieval accuracy reaches 91.4 percent compared with 53.8 percent for sliding windows and 80.6 percent for H2O.","On 75 VisualWebArena tasks the method keeps 95.3 percent of full-KV success rate at 2.8 times lower peak memory.","Perplexity remains within 1.5-2.1 points of the unpruned cache while memory stays near that of a 512-token sliding window.","The same policy works across four model families for generated lengths up to 4K tokens."],"fun_headline_variants":["Confidence score sets per-step KV cache budget","Model uncertainty drives KV cache eviction policy","KV tokens ranked by attention and recency in CONF-KV","Mixed precision KV storage with confidence aware pruning"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The next-token distribution supplies a scalar confidence score that reliably indicates when more context must be retained, and the composite attention-plus-recency ranking preserves task-critical information without the full cache.","fun_headline_variants_meta":{"raw":{"variants":["Confidence score sets per-step KV cache budget","Model uncertainty drives KV cache eviction policy","KV tokens ranked by attention and recency in CONF-KV","Mixed precision KV storage with confidence aware pruning"]},"model":"grok-4.3","cost_usd":0.00388,"raw_usage":{"total_tokens":2012,"prompt_tokens":705,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":38799500,"prompt_tokens_details":{"text_tokens":705,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1251,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":705,"tokens_out":56,"duration_ms":10549,"temperature":1.0,"reasoning_tokens":1251,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T11:50:32.662883+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled test in which high-confidence next-token predictions still require distant tokens that the eviction policy discards, producing a clear drop in accuracy beyond the reported 1.5-2.1 perplexity band or below the 91.4 percent needle retrieval figure.","supporting_citations":[],"review_version":1}