{"id":"e49abe4e-3e2f-43ed-aa6f-0b713c83ce31","arxiv_id":"2501.10054","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"TARDIS folds two feed-forward weight matrices into one by linearly approximating activations in common input ranges, then recomputes outliers with a small predictor, claiming 80 percent FFN parameter reduction and up to 1.6x inference speedup.","lead":"TARDIS shrinks the feed-forward layers of large language models by treating their curved activation functions as straight lines for most inputs. It reports up to 80 percent fewer parameters in those layers and roughly 1.6 times faster text generation, at the cost of about 11 percent accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 80% parameter-reduction claim omits the original FFN weights that the result-fixing kernel must load; if those weights are part of the deployed footprint, the headline compression ratio is not real.","rationale":"The reader's weakest_assumption focuses on the predictor's false-negative rate, which is a genuine correctness risk: a missed out-of-range neuron silently corrupts the layer output, and the paper never reports false-negative rates. However, the more load-bearing problem is prior to the predictor: the method's fallback path requires the original W1/W2 weights, and those weights are excluded from the compression-ratio accounting. Even a perfect predictor does not help if the original weights needed for correction remain part of the deployable model. The paper itself lists 'only the original weights of neurons that require exact computation' as a memory component, yet the headline 80% reduction and the pruning comparison do not include them. Since any neuron can fall outside its linear range, the fix path must be able to access all original FFN weights, so the claimed parameter reduction is not established. The reader's verdict of REJECT remains appropriate; this stress-test does not move it, but it identifies a different primary vulnerability than the reader's stated weakest assumption.","tokens_in":20998,"tokens_out":9132,"duration_ms":100064,"concrete_test":"Compute the total persistent storage actually needed to run the reported vLLM/HuggingFace deployments: serialize the folded matrix C and bias B, the 2-bit GPTQ predictor, and every original W1/W2 entry the fix CUDA kernel can access during the 8+192-token generation of Section 7.4. If any original-weight access occurs, those weights are part of the deployed footprint; sum their bytes and compare with the original Falcon-7B FFN size. If the sum is at least the original FFN size, the '80% parameter reduction' headline fails. If the authors can demonstrate a configuration where only a small fixed subset of original weights is ever accessed and verify this on the evaluation distribution, the claim needs to be restated accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that after folding, each FFN layer occupies roughly d^2 parameters instead of 2dh. But Section 5.4's result-fixing step and Section 6's CUDA kernel 'performs selective loading of FFN weights based on predicted neuron indices', and the fix formula subtracts the approximate contribution of an out-of-range neuron and adds back the exact one. Computing the exact contribution needs that neuron's original W1 column and W2 row. Because the predictor can flag any neuron on any token and no neuron is permanently pruned, a correct deployment must retain (or at least have access to) the full original W1 and W2 for fallback. Section 7.1's compression-ratio accounting counts only the folded matrix and the 2-bit predictor, and Section 5.4 lists the original weights separately, but the headline '80% parameter reduction' does not subtract them. With h=4d, retaining the full original FFN weights (2dh) already equals the original FFN size; adding the folded matrix and predictor cannot yield an 80% reduction. Compared with Wanda and RIA, which actually delete pruned weights, TARDIS's reported ratios are not apples-to-apples, so the 'up to 65% higher accuracy' comparison is largely against methods that genuinely shrink the model. The speedup measurements may stand as a speculative-execution result, but the paper's parameter-reduction claim is not supported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TARDIS proposes to compress the feed-forward network (FFN) blocks of large language models by replacing each neuron's non-linear activation with a per-neuron linear approximation on a \"hot\" input range, pre-folding the two FFN matrices into a single d-by-d matrix C plus a bias vector B, and using a quantized predictor to flag out-of-range neurons whose exact contributions are recomputed online. The paper reports up to 80% parameter reduction in FFNs, accuracy substantially better than Wanda and RIA at high compression ratios, and end-to-end speedups of 1.4x on HuggingFace and 1.6x on vLLM for a 7B model. The constant-folding algebra in Section 5.2 is correct for exactly linear activations, and the evaluation is broad, covering five models, three language-generation benchmarks, three zero-shot tasks, and sensitivity analyses. However, the headline parameter-reduction claim is not supported, because the result-fixing mechanism requires retaining the original FFN weights for every neuron that can be flagged, and the compression-ratio accounting in Section 7.1 omits those weights.","tokens_in":21240,"tokens_out":7546,"duration_ms":77329,"significance":"If the parameter-reduction claim were valid, the paper would introduce a genuinely new compression paradigm: turning a two-matrix FFN with a non-linear activation into a single dense matrix while preserving most of the model's behavior. The constant-folding insight is elegant, the derivation in Section 5.2 is internally consistent for linear activations, and the speedup measurements may reflect a real speculative-execution effect worth studying. The paper also deserves credit for reporting calibration-set sensitivity, predictor-size sensitivity, and floating-point reordering effects, which are useful empirical checks. The central problem is that the comparison to pruning baselines is not commensurable and the reported compression ratio excludes the original weights that the fallback path must load. Because the paper's main advertised contribution is parameter reduction, and that claim is not substantiated by the deployment accounting, the significance of the work as presented is substantially lower than claimed.","major_comments":[{"comment":"The 80% parameter-reduction claim omits the original W1 and W2 weights required by the result-fixing mechanism. Section 5.4 states that fixing subtracts the linear approximation and adds back the exact computation for flagged neurons; this exact computation requires W1:,n and W2n,: (or equivalent data) for every neuron n that the predictor may flag. Because Section 5.3's predictor can flag any neuron on any token and no neuron is permanently excluded, a correct deployment must retain the full original FFN weight matrices. With the typical h = 4d architecture, retaining 2dh = 8d^2 original parameters already equals the original FFN size, and adding the folded matrix (d^2) and the predictor only increases the total footprint. Thus the abstract's '80% parameter reduction' and the Section 7.1 compression-ratio calculation are not supported as stated.","section":"Section 5.4 and Section 7.1"},{"comment":"The accuracy comparisons to Wanda and RIA are not apples-to-apples. Wanda and RIA delete pruned weights, so their reported compression ratios correspond to genuinely smaller models; TARDIS keeps the original weights available for online fallback while counting only the folded matrix and the 2-bit predictor in its compression ratio. The 'up to 65% higher accuracy' claim therefore compares a method that can revert to the exact original computation against methods that have permanently removed weights. The paper should recompute the reported ratios using the true deployed footprint, or explicitly reframe the result as a speculative-I/O reduction rather than a parameter reduction.","section":"Section 7.1"},{"comment":"Correctness depends on the 2-bit GPTQ predictor flagging every out-of-range neuron, but the paper never measures the predictor's false-negative rate. If the predictor misses an out-of-range neuron, the speculative folded-matrix result is kept without correction, so the layer output is silently wrong in a way that is not bounded by the linear-range approximation error. Figure 15 reports only aggregate perplexity versus predictor size, which does not reveal how often such misses occur or whether they are concentrated in particular layers or input regimes. The authors should measure false-negative rates on held-out data and, if nonzero, analyze their effect on the accuracy claims.","section":"Section 5.3 and Section 7.6"},{"comment":"For OPT-6.7B, Tables 3 and 4 report identical results at 50%, 70%, and 80% compression because TARDIS assigns the same linear function in all three cases. This is discussed only as a consequence of the model's activation pattern, but it also means that the reported compression ratio does not correspond to a change in the compressed artifact for this model. The parameter-reduction and speedup claims should be reconciled with the observation that the threshold parameter can leave the folded model unchanged.","section":"Section 7.2"}],"minor_comments":[{"comment":"'therotical analysis' should be 'theoretical analysis'.","section":"Section 2.2"},{"comment":"The third dataset is labeled 'PDB' in Table 3 and Figure 11a; it should be 'PTB' for consistency with the text.","section":"Table 3 and Figure 11"},{"comment":"The phrase 'We first quantifies the performance' should be 'We first quantify the performance'.","section":"Section 7.2"},{"comment":"The explanation that long-prompt generation 'reduces the sparsity patterns that TARDIS leverages' is imprecise: TARDIS does not exploit activation sparsity but rather the concentration of activation inputs in linear ranges. Consider rewording.","section":"Section 7.4"},{"comment":"The runtime breakdown in Figure 14 groups 'mask generation and index conversion' under 'Others', but the text does not define this category precisely; please state what operations are included and how their cost was measured.","section":"Section 7.5"}],"recommendation":"reject","confidential_remarks":"The central parameter-reduction claim is not fixable by a local correction: the fallback design requires the original FFN weights, and once those are counted, the reported 80% reduction disappears. The speedup results could potentially be salvaged as a speculative-execution system with a different title and framing, but that would be a substantially different paper from the one submitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a clever systems idea with a broken headline. The paper proposes per-neuron partial linear approximation of GELU/SiLU, folding the two FFN matrices into one d×d matrix, and using a 2-bit GPTQ predictor to spot neurons whose activation inputs fall outside their linear range, then fixing those results. The constant-folding algebra is correct for the in-range part, and the empirical evaluation on five LLMs is new. The reported 1.6×/1.4× end-to-end speedups in vLLM/HuggingFace are plausible for memory-bandwidth-bound decoding, given that most neurons are processed through the folded matrix.\n\nWhat it does well: the calibration is data-efficient (8 samples), the adaptive per-layer/per-neuron thresholds are a sensible response to the non-uniform importance they document, and the paper is honest about the GLU-variant limitation in the discussion. The speedup breakdown giving predictor, folded matrix, and fixing costs is useful.\n\nThe soft spot is not minor: the 80% parameter-reduction claim is not supported by the paper's own design. Section 5.4 says the memory footprint includes 'the original weights of neurons that require exact computation,' and Section 6 says the CUDA kernel 'performs selective loading of FFN weights based on predicted neuron indices.' Since the predictor can flag any neuron on any token, a correct deployment needs access to the full original W1 and W2. With h=4d, that is 2dh parameters—the entire original FFN—so adding the folded matrix and predictor cannot produce an 80% reduction. Section 7.1's accounting counts only the folded matrix and predictor, which is cherry-picking. This also makes the comparison with Wanda and RIA unfair: those methods actually delete weights, so the 'up to 65% higher accuracy' is against methods that genuinely shrink the model.\n\nA second genuine gap: the predictor's false-negative rate is never measured. The paper reports perplexity versus predictor size (Figure 15), but a single missed out-of-range neuron silently corrupts the layer output with no correction. The accuracy results would be undercut if the predictor misses on some fraction of tokens; that rate needs to be reported.\n\nI'd take the speedup results as a plausible speculative-execution story, but the central compression claim is not established. No code or data are released, and the numbers have no variance.\n\nBottom line: worth a serious referee, but only with major revision. The authors need to state the true storage footprint including fallback weights, either redefine the claim in terms of per-token active parameters or drop it, measure predictor false negatives, and release artifacts. As submitted, I would not cite the headline number.","headline":"The per-neuron linearization idea is genuinely novel and the speedups are plausible, but the 80% parameter-reduction headline doesn't survive contact with the fallback weights, so the paper needs major revision before it stands.","tokens_in":21829,"tokens_out":3407,"would_cite":false,"duration_ms":32034,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TARDIS cuts 80% of feed-forward parameters by folding non-linear activations into a single precomputed matrix.","keywords":["TARDIS","constant folding","feed-forward network compression","activation linear approximation","speculative result fixing","LLM inference acceleration","post-training compression","transformer FFN"],"falsifier":"Run a folded model on held-out tokens and compare the predictor's in-range verdict with the true range membership of every neuron's activation input; any false negative (an out-of-range input the predictor does not flag) violates the correctness premise. A directly checkable version is to count, on a fresh corpus such as the WikiText-2 benchmark used in the paper, how often the predicted fixing set omits a neuron whose actual input lies outside its assigned interval.","tokens_in":20744,"feed_emoji":"⚡","tokens_out":7841,"duration_ms":70110,"temperature":0.7,"pith_summary":"The paper claims that the feed-forward block of a transformer LLM can be compressed by treating its non-linear activation as a linear function on the input range where activations actually land, then folding the two weight matrices into one precomputed matrix. Because roughly 65% of each neuron's activation inputs sit inside about 20% of the total input range, a straight-line fit covers most computations with small error; a small predictor catches the outlier inputs and the system recomputes only those neurons exactly. On models from roughly 1.6B to 11.1B parameters, the paper reports removing 80% of feed-forward parameters, with end-to-end generation speeding up about 1.4x in a standard Python implementation and 1.6x in a production serving engine, at a roughly 10.9% accuracy cost on a 7B model. This matters because parameter-loading I/O, not arithmetic, dominates token generation, so shrinking the feed-forward weight matrices is a direct path to faster serving.","feed_headline":"Constant folding trims 80% of LLM feed-forward weights","feed_subtitle":"Treating nonlinear activations as linear on common inputs speeds a 7B model up to 1.6x.","key_machinery":"The load-bearing object is the partially linear per-neuron activation $\\varphi_n(x)=a_n x+b_n$ for $x\\in[\\ell_1,\\ell_2)$ and $\\varphi_n(x)=\\sigma(x)$ otherwise, together with the folded matrices $C=\\sum_n a_n W_{1:,n}W_{2n,:}$ and $B=\\sum_n b_n W_{2n,:}$ that result from constant folding. A single linear range per neuron avoids the exponential blow-up in folded matrices that a multi-range scheme would create. The online predictor is a compressed version of $W_1$ ($2$-bit quantized in the implementation) that decides, from the layer input, which neurons are outside their linear range; a custom kernel then subtracts the folded contribution and recomputes the exact term for exactly those neurons. A two-level adaptive thresholding scheme assigns wider linear ranges to less important layers and neurons, and a greedy centroid-anchored range search with kernel density estimation finds each neuron's range from calibration data.","core_discovery":"TARDIS's central discovery is that the two-matrix feed-forward computation $FFN(x)=\\sigma(xW_1)W_2$ can be replaced by a single matrix multiplication $xC+B$ whenever $\\sigma$ is replaced by a per-neuron line $ax+b$: associativity lets the constants absorb the weights as $C=\\sum_n a_n W_{1:,n}W_{2n,:}$ and $B=\\sum_n b_n W_{2n,:}$. The paper establishes that this substitution can be made safe in practice by profiling each neuron's activation input distribution on a tiny calibration set, assigning each neuron a 'hot' range and a least-squares linear fit, and using a compressed 2-bit quantized copy of $W_1$ as an online predictor that flags neurons whose current input falls outside their range. For flagged neurons the incorrect linear contribution is subtracted and the true $\\sigma$ computation added back. Across five models, the folded layers keep perplexity close to the dense model at moderate compression, and at 80% FFN compression the method keeps far higher downstream accuracy than two leading pruning baselines, with measured 1.4x-1.6x end-to-end speedups.","pith_inferences":["If the 2-bit predictor has any false negatives on deployment data, silently wrong layer outputs are possible; a cheap safeguard would compute a small always-exact control neuron per layer and trigger full fallback when that neuron's own input is out of range.","The nearly lossless results on a ReLU-based 6.7B model in the evaluation suggest TARDIS is a natural fit for models that already show activation sparsity, and the folded matrix itself could then be quantized or pruned for further gains.","One could replace the greedy centroid-anchored range search with an optimal dynamic-programming interval choice, likely reducing approximation error for the same threshold and shrinking the predictor's workload.","The large accuracy gap over pruning at high compression, if it holds, changes the cost-benefit calculus of extreme compression: the relevant baseline for 80% FFN compression may be a folded linear computation rather than sparse pruning."],"forward_implications":["For a feed-forward block with hidden width $h=4d$, the parameter count drops from $2dh$ to $d^2$ in theory, and the paper reports an 80% measured reduction after accounting for the predictor and folded matrix.","End-to-end token generation on a 7B model speeds up by about 1.4x in a standard transformer implementation and 1.6x in a production serving engine, at a reported accuracy loss around 10.9% on downstream tasks.","At 80% FFN compression, the method claims up to 65% higher downstream accuracy than state-of-the-art pruning baselines, with dramatically lower perplexity (e.g., 13.4 versus 1489-7796 on one benchmark).","Calibration needs only about 8 text samples: the actual in-range fraction lands within 1.8% of target, and swapping calibration corpora moves perplexity by less than 0.4.","The folding trick works for GELU, SiLU, and ReLU feed-forward networks, but the paper states it does not directly extend to gated (GLU-variant) FFNs, where folded matrices grow exponentially."],"supporting_citations":[{"why":"Supplies the primary pruning baseline that TARDIS must beat at each compression ratio.","marker":"[53]"},{"why":"Supplies the second pruning baseline used for comparison on perplexity and zero-shot accuracy.","marker":"[68]"},{"why":"Provides the 2-bit quantization method used to build the online predictor from W1.","marker":"[29]"},{"why":"Provides the Falcon-7B model and FFN parameter share that anchors most experiments and the speedup measurements.","marker":"[19]"},{"why":"Provides the production serving system used to measure end-to-end inference speedup.","marker":"[37]"}],"fun_headline_variants":["LLM speedup via partial linear folding: 80% FFN cut","TARDIS: Linear-fit fold trims 80% of LLM FFN weights","Constant folding goes linear: 80% FFN compression, 1.6x speed","LLM feed-forward: partial linear fold gives 1.6x inference","TARDIS: linearizing activations trims 80% FFN, 1.6x speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the compressed online predictor never misses a neuron whose activation input is outside its linear range, because a single missed outlier would leave the folded-matrix value in the output and silently corrupt the layer.","fun_headline_variants_meta":{"raw":{"variants":["LLM speedup via partial linear folding: 80% FFN cut","TARDIS: Linear-fit fold trims 80% of LLM FFN weights","Constant folding goes linear: 80% FFN compression, 1.6x speed","LLM feed-forward: partial linear fold gives 1.6x inference","TARDIS: linearizing activations trims 80% FFN, 1.6x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2467,"prompt_tokens":1022,"completion_tokens":1445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":1330}},"tokens_in":638,"tokens_out":1445,"duration_ms":10147,"temperature":1.0,"reasoning_tokens":1330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:23:44.447611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a folded model on held-out tokens and compare the predictor's in-range verdict with the true range membership of every neuron's activation input; any false negative (an out-of-range input the predictor does not flag) violates the correctness premise. A directly checkable version is to count, on a fresh corpus such as the WikiText-2 benchmark used in the paper, how often the predicted fixing set omits a neuron whose actual input lies outside its assigned interval.","supporting_citations":[{"cited_title":"A simple and effective pruning approach for large lan- guage models","cited_arxiv_id":null,"evidence_quote":"Supplies the primary pruning baseline that TARDIS must beat at each compression ratio."},{"cited_title":"Plug-and-play: An efficient post-training pruning method for large lan- guage models","cited_arxiv_id":null,"evidence_quote":"Supplies the second pruning baseline used for comparison on perplexity and zero-shot accuracy."},{"cited_title":"Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the 2-bit quantization method used to build the online predictor from W1."},{"cited_title":"The falcon series of open language models, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the Falcon-7B model and FFN parameter share that anchors most experiments and the speedup measurements."},{"cited_title":"Efficient memory man- agement for large language model serving with page- dattention","cited_arxiv_id":null,"evidence_quote":"Provides the production serving system used to measure end-to-end inference speedup."}],"review_version":1}