{"id":"2b73d93a-b90f-4050-9029-005af215d4ec","arxiv_id":"2501.00684","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A gated calculator module inserted into a frozen Llama 3.1 8B model enables near-perfect arithmetic on BigBench, including multiplication, in a single forward pass without external tools.","lead":"This paper inserts a gated calculator module into a frozen language model so the model can perform addition, subtraction, multiplication, and division by emulating a calculator on the GPU. The modified model reaches near-perfect accuracy on the BigBench Arithmetic benchmark, including multiplication, which had been unsolved by prior LLM methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing unknown is the Input Mapping's extraction accuracy: never directly reported, and the paper's own division anomaly shows sensitivity to operand distribution, so the 98-99% reliability claim is supported only for the benchmark's specific distribution.","rationale":"The reader's weakest_assumption identifies the same component: the Input Mapping's generalization from synthetic data to the benchmark distribution. My stress-test concurs and sharpens it in two ways. First, the paper never measures the Input Mapping's extraction accuracy in isolation, so there is no direct evidence about the error rate or its causes. Second, the paper's own anomaly analysis in Section 4 reveals that the Input Mapping is sensitive to the operand-generation distribution, which is a concrete indication that extraction is not uniformly robust. The end-to-end results on the benchmark (98-99%) are strong and internally consistent: the calculator is deterministic, the ablation shows the calculator is necessary, and the same-size finetuning baseline is far worse. Thus the central claim is credible within the tested setting, but the reliability claim is not yet established outside the exact benchmark distribution. No fatal flaw is identified; rather, the conditional verdict is appropriate, with the additional recommendation that future revisions report component-level extraction accuracy and robustness across templates and number distributions. I do not see an internal inconsistency or a reason to reject or unverify the paper.","tokens_in":11875,"tokens_out":14633,"duration_ms":147071,"concrete_test":"On the BIG-Bench test set, compare the Input Mapping's discretized operand and operator outputs (before the calculator) against ground truth, and report per-digit and per-operand accuracy for each arithmetic subtask. Then construct a held-out evaluation with the same calculator but varied prompt templates, operand lengths, and number-generation algorithms (e.g., uniform vs. fixed digit count, leading zeros, different wording) and measure the Input Mapping's extraction accuracy. If operand-level accuracy is below the reported end-to-end accuracy, or if it falls below roughly 99% on the held-out set, the reliability claim is not established beyond the original benchmark distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the learned Input Mapping submodule reliably extracting the correct operands and operator from the frozen LLM's layer-1 activations at the anchor token. This is the only component that must generalize from synthetic training data to the benchmark's natural-language template and operand distribution. The paper does not report the Input Mapping's digit-level or operand-level accuracy, only the end-to-end accuracy. Because the calculator is non-differentiable and the Output Mapping is trained exclusively on the correct calculator output, a wrong extraction at inference produces a wrong calculator result that is injected with the same gating as a correct one; thus end-to-end accuracy is an upper bound on extraction accuracy, but the failure modes and margin are unexamined. Section 4, 'Investigating Anomalies', admits that the benchmark uses a different operand-generation algorithm for division, causing a measurable accuracy drop (98% vs 99%). This is direct evidence that the Input Mapping's error rate is distribution-dependent: if the template or number-generation distribution differed more substantially (e.g., variable operand lengths, leading zeros, alternative word order), extraction accuracy could degrade non-negligibly and the near-perfect benchmark claim would not transfer. The paper's stated robustness to 'alternate templates' is asserted without reporting numbers, so the generality of the extraction mapping is precisely the weakest load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Integrated Gated Calculator (IGC), a module inserted into a frozen pretrained LLM (after layer 1 of Llama 3.1 8B) that extracts operands and operators from the LLM's intermediate activations at an anchor token, computes the exact arithmetic result via a non-differentiable calculator emulated on the GPU, and injects the result back through a gated output mapping. The authors train only the IGC's two submodules on synthetic data and report 98-99% accuracy on the BigBench Arithmetic benchmark, outperforming the previous best results from PaLM 535B and a same-size finetuning baseline, with an ablation suggesting the calculator is responsible for the improvement. The paper also claims the method is efficient, interpretable, avoids side effects on non-arithmetic tasks, and could be extended to other non-differentiable operations.","tokens_in":12039,"tokens_out":6009,"duration_ms":55568,"significance":"If the empirical claims hold up, this is a valuable result: it demonstrates that a small, modular, non-differentiable component can be trained to perform exact arithmetic inside an LLM in a single forward pass, without chain-of-thought or external tools, and it provides a head-to-head finetuning baseline and an ablation that strengthen the attribution of the gain to the calculator. The paper also offers a concrete path toward training such a module during pretraining, which is an interesting idea. However, the significance is currently limited by reproducibility concerns: no code is provided, no seed-level results or error bars are reported, and the accuracy of the learned input-extraction component is never directly measured. These omissions make it impossible to assess the reliability and generality of the central claim.","major_comments":[{"comment":"The manuscript states that code will be provided only upon acceptance and does not report hyperparameters, number of seeds, or per-seed results. Table 1 reports averages without variance, and Figure 4 shows multiple seeds but no numeric values. Because the headline claim is a reliability claim ('reliably achieves 98% to 99% across multiple training runs'), the absence of seed-level data and code prevents independent verification. Please provide the full experimental setup, seed-level accuracies with standard deviations, and release the code (or at least detailed pseudocode for all three submodules).","section":"Appendix B / Section 4"},{"comment":"The end-to-end accuracy is the only metric reported; the digit-level and operand-level accuracy of the Input Mapping submodule is never reported. This matters because the Input Mapping is the only component that must generalize from synthetic training data to the benchmark's template and operand distribution, and the paper itself attributes the slightly lower division accuracy (98% vs 99%) to a distribution mismatch between the training and test operand generators. Please report the Input Mapping's extraction accuracy per subtask, quantify the distribution mismatch (e.g., by evaluating on the training distribution and on the benchmark distribution), and discuss the failure modes when extraction is incorrect.","section":"Section 4, 'Investigating Anomalies'"},{"comment":"The claim that 'we noticed no difference in performance between these templates' is made without reporting any numbers, template descriptions, or results. Since template variation directly stresses the Input Mapping's ability to identify operands and operators, this claim is load-bearing for the generality of the method. Please provide the set of templates used, per-template accuracies, and a comparison to the BigBench template.","section":"Section 4, 'Alternate Templates'"},{"comment":"The claim that the IGC 'causes no destructive interference in tasks where it is not needed' is supported only by an unspecific sentence saying the architecture was tested on non-arithmetic tasks. No task names, metrics, or before/after comparisons are given. Since avoiding side effects is one of the advertised advantages (Table 2), please report the non-arithmetic tasks and the model's performance on them with and without the IGC.","section":"Section 3.1, 'Gated Outputs'"},{"comment":"The text reports that the IGC+shortcut hybrid has 'high variance' and 'final performance is overall lower than the pure IGC,' but no final accuracies are given. Because this ablation is the key evidence that the calculator, not the extra parameters, is responsible for the gain, please provide the final accuracy (mean and standard deviation) for each architecture shown in Figure 4.","section":"Section 4, 'Ablations'"}],"minor_comments":[{"comment":"The model name 'PALM' should be written as 'PaLM' for consistency with the cited reference (Chowdhery et al., 2023).","section":"Table 1 and text"},{"comment":"The symbol '#' is used in the table without a legend; please define it (presumably meaning 'partially' or 'depends on the task').","section":"Table 2"},{"comment":"The caption contains 'afterTt' with a missing space; it should be 'after T_t'.","section":"Figure 2 caption"},{"comment":"The paper inconsistently spells 'finetuning' and 'fine-tuning'; please choose one form and use it consistently.","section":"Throughout"},{"comment":"The statement that for larger numbers 'the LLM can still use the IGC as a very reliable approximator' is vague; please specify what behavior the model exhibits for inputs longer than the fixed maximum length.","section":"Section 8, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is under review at ACL Rolling Review, and the lack of code and seed-level detail is especially problematic for verifying the SOTA claim against PaLM 535B. The core idea is interesting and the ablation is a good start, but the current evidence is not yet at the level of a strong journal paper; the major revisions above should address the reproducibility and component-level evaluation gaps."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is that this paper does something genuinely new: it inserts a non-differentiable GPU calculator into a frozen LLM, trains the input mapping with an auxiliary loss, and gates the result back into the residual stream. The 98–99% numbers on BigBench Arithmetic, including multiplication, are credible enough to take seriously. The finetuning baseline and the ablation without the calculator both support the claim that the calculator is doing the work, not just the finetuning.\n\nWhat I like: the left-aligned digit representation that matches Llama 3.1 tokenization is an important, and seemingly necessary, detail. The paper is honest about the lopsided n-shot comparison and deliberately averages its runs instead of cherry-picking the best. The division anomaly is also openly attributed to a distribution mismatch between training and test operands, which is the right instinct.\n\nWhere I'd push back: the paper as posted has no code, no error bars, and no seed count, even though it claims low variance. The side-effect claim on non-arithmetic tasks is stated in the abstract and Section 3, but no experiment or numbers are shown for it. The stress-test concern about the input mapping is real: end-to-end accuracy is the only reported metric, so we don't know how often the module extracts the wrong operands. The auxiliary loss on the input mapping mitigates this, but it doesn't quantify how well the mapping generalizes across templates or number distributions. The division drop shows it can fail under distribution shift, so the 'near-perfect' claim is tied to the benchmark's particular distribution.\n\nThese are soft spots, not fatal flaws. The architecture is novel and the central mechanism is plausibly sound. But reproducibility is a serious issue for a paper this empirical, and the side-effect claim needs to be evidenced, not just asserted.\n\nWho is this for? Anyone working on LLM arithmetic, tool-augmented models, or non-differentiable modules. It deserves a serious referee, but the referee should condition acceptance on releasing code or at least detailed experimental setup, and on demonstrating the gate behavior on non-arithmetic tasks. I'd bring it to reading group, mainly to discuss whether the input-mapping risk is as load-bearing as it looks.","headline":"The IGC is a genuinely new mechanism for exact arithmetic inside an LLM, and the BigBench results are strong enough to merit refereeing, but the posted paper is not yet reproducible and the side-effect claim is asserted rather than shown.","tokens_in":12643,"tokens_out":1494,"would_cite":true,"duration_ms":15788,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a frozen 8B LLM with a 17M-parameter gated calculator module solves five-digit arithmetic at 98–99% accuracy in a single forward pass, without external tools or intermediate tokens.","keywords":["arithmetic reasoning","large language models","integrated gated calculator","BigBench Arithmetic","tokenization","parameter-efficient finetuning","multiplication","tool use"],"falsifier":"Generate BigBench-style test items with the benchmark's own operand generator, run the trained IGC, and inspect the extractor's digit-wise outputs before the calculator fires; if even a small fraction (say, over 1%) of test inputs are mis-parsed into the wrong operands or operator, the claimed 98–99% accuracy cannot be sustained.","tokens_in":11551,"feed_emoji":"🧮","tokens_out":9371,"duration_ms":79326,"temperature":0.7,"pith_summary":"Arithmetic remains a stubborn weakness of large language models, and no prior method solved the multiplication subtask of the BigBench Arithmetic benchmark without external tools or multi-step generation. This paper proposes the Integrated Gated Calculator (IGC), a small trainable module inserted into a frozen LLM that extracts the operands and operator from the model's hidden states, computes the exact result with a non-differentiable tensor-based calculator on the GPU, and injects the result back through learned gates. On the benchmark, the modified Llama 3.1 8B model reaches 98–99% accuracy across addition, subtraction, multiplication, and division, beating models nearly two orders of magnitude larger and a same-size finetuning baseline. If the result holds, exact arithmetic can be treated as an internal subroutine of an LLM rather than something the model must generate token by token.","feed_headline":"17M calculator module lifts an 8B LLM to 99% arithmetic accuracy","feed_subtitle":"One forward pass, no tools, no chain-of-thought: the module masters 5-digit arithmetic, including multiplication.","key_machinery":"The load-bearing mechanism is the IGC's division of labor: a learned extractor, a deterministic calculator, and a learned injector. The Input Mapping submodule is an attention mechanism that, at the anchor token, gathers the tokens carrying the numbers and operator and produces fixed-length categorical digit distributions in a left-aligned format; left alignment is chosen to match Llama 3.1's chunked left-to-right tokenization of numbers, so each digit position maps to a predictable token. The calculator is a sequence of non-differentiable tensor operations (digit-to-integer conversion, parallel addition, subtraction, multiplication, and division, and integer-to-left-aligned-digit conversion), which blocks gradients and forces the auxiliary loss for the Input Mapping. The Output Mapping submodule uses separate learned gating weights per output token, so it can leave tokens unchanged when no arithmetic is needed. This gating is what lets the module avoid destructive interference on non-arithmetic tasks.","core_discovery":"The central claim is that an LLM's arithmetic failures can be bypassed by routing the calculation through a small, explicitly supervised calculator module rather than by making the network learn arithmetic from examples. The IGC is placed after layer 1 of a frozen Llama 3.1 8B model. At an anchor token marking the end of the user's request, the Input Mapping submodule uses attention over preceding tokens to output categorical distributions over the two operands and the operator; the most probable values are discretized and fed to a GPU-emulated calculator that performs all four operations in parallel and returns a left-aligned one-hot digit result; the Output Mapping submodule then uses learned per-token gates to add the result into subsequent hidden states. Training is split: the Input Mapping is supervised by an auxiliary cross-entropy loss with ground-truth operands, while the Output Mapping learns through the LLM's normal loss. On the BigBench Arithmetic benchmark this configuration reports 99% overall accuracy, 99% on addition, subtraction, and multiplication, and 98% on division, outperforming the best n-shot PaLM 535B result and a parameter-matched finetuning adapter baseline.","pith_inferences":["Editorially, the result suggests that LLM arithmetic errors are dominated by operand extraction and output formatting rather than by the absence of computational machinery; the IGC essentially converts arithmetic into a reading-comprehension task plus a lookup.","A testable extension is to evaluate the Input Mapping's digit-wise accuracy directly on out-of-template phrasings; if it degrades while the calculator remains perfect, extraction, not computation, is the binding constraint.","Another extension is to chain multiple IGC executions for multi-step arithmetic or larger operands; the paper's fixed-length design implies accuracy should stay perfect up to the trained length and then need external tools.","The left-aligned tokenization analysis implies tokenizer design may matter more than model scale for arithmetic; other tokenizers would need their own alignment scheme, which the paper's code is meant to automate."],"forward_implications":["A frozen 8B LLM plus a 17M-parameter module can perform exact 5-digit arithmetic in constant time, with no intermediate output tokens; this removes the main efficiency cost of chain-of-thought for arithmetic.","Because the comparison includes a same-size finetuning adapter baseline at 70% overall and 22% on multiplication, the reported gap attributes the gain to the calculator mechanism rather than to finetuning per se.","The module's gated outputs mean tasks without arithmetic are left nearly unaffected, so adding the IGC should not degrade unrelated capabilities.","The same three-part design (extract, non-differentiable black-box operation, re-inject with gates) can be pointed at other discrete operations, such as lookups, and the paper argues it could be trained during pretraining as a subroutine for word problems."],"supporting_citations":[{"why":"Provides the BigBench Arithmetic benchmark and the n-shot comparison scores the paper's results are measured against.","marker":"bench authors, 2023"},{"why":"Supplies the Llama 3.1 8B backbone and the chunked digit tokenization that motivates the left-aligned internal number format.","marker":"Dubey et al., 2024"},{"why":"Gives the PaLM 535B n-shot baseline that IGC claims to beat on most subtasks, especially multiplication.","marker":"Chowdhery et al., 2023"},{"why":"Describes Toolformer, the external-tool alternative, and the annotation-by-perplexity scheme the paper adapts for IGC training data.","marker":"Schick et al., 2023"},{"why":"Shows finetuning alone can solve addition and subtraction but not multiplication, defining the gap IGC closes.","marker":"Liu and Low, 2023"},{"why":"Defines adapter-based tuning, the family the IGC extends with its non-differentiable calculator and gated outputs.","marker":"Houlsby et al., 2019"},{"why":"Defines chain-of-thought prompting, the main efficiency and capability baseline the paper compares against.","marker":"Wei et al., 2023"}],"fun_headline_variants":["Gated calculator module lifts LLM arithmetic to 99% accuracy","Small module lets 8B LLM beat 535B models at arithmetic","One-pass arithmetic: LLM plus tiny calculator hits 99%","Frozen LLM + gated calculator: 99% on BigBench arithmetic","LLM arithmetic solved with an internal calculator module"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the small learned extractor can reliably read the correct operands and operator out of the LLM's hidden states at the moment the request ends, even when the test inputs come from a different random-number distribution than the synthetic training data; the paper itself attributes its slightly lower division score to exactly that mismatch.","fun_headline_variants_meta":{"raw":{"variants":["Gated calculator module lifts LLM arithmetic to 99% accuracy","Small module lets 8B LLM beat 535B models at arithmetic","One-pass arithmetic: LLM plus tiny calculator hits 99%","Frozen LLM + gated calculator: 99% on BigBench arithmetic","LLM arithmetic solved with an internal calculator module"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1740,"prompt_tokens":959,"completion_tokens":781,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":689}},"tokens_in":575,"tokens_out":781,"duration_ms":6276,"temperature":1.0,"reasoning_tokens":689,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:44:33.035374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate BigBench-style test items with the benchmark's own operand generator, run the trained IGC, and inspect the extractor's digit-wise outputs before the calculator fires; if even a small fraction (say, over 1%) of test inputs are mis-parsed into the wrong operands or operator, the claimed 98–99% accuracy cannot be sustained.","supporting_citations":[],"review_version":1}