{"id":"458c4155-511a-4070-b049-6338276e93db","arxiv_id":"2504.16266","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TeLLMe is the first edge FPGA accelerator that runs a 1.58-bit ternary LLM end-to-end, including prefill and decoding, achieving 9.51 tokens/s and 0.55 to 1.15 second prefill under 7 watts.","lead":"This paper builds TeLLMe, an FPGA accelerator that runs 1.58-bit ternary large language models on a low-power edge board, handling both the prompt-processing prefill stage and token generation. It reports up to about 9.5 tokens per second and roughly one-second first-token latency under 7 watts, which matters for private and energy-constrained AI at the edge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central deployment claim requires a usable 0.7B ternary checkpoint, but Section IV reports no output-quality evaluation; if the checkpoint is degraded, the measured throughput and prefill latency do not constitute usable LLM inference.","rationale":"I read TeLLMe as a hardware-systems claim: a KV260 design that runs ternary LLM prefill and decoding under 7 W. The most load-bearing condition is not the cleverness of the table-lookup matmul or the reverse attention schedule, but whether the executed model is an actual usable ternary LLM. The authors never quantify output quality; they rely on cited BitNet/DeepSeek results, which are about algorithmic accuracy in general, not about their specific 0.7B checkpoint. This is an unverified prerequisite, and the paper itself supplies no evidence against the risk. The hardware description is internally consistent, resource tables add up, and the measurements are plausible, so I do not see a fatal contradiction in the architecture. The gap is lack of validation and lack of artifact release. The reader's weakest_assumption identifies exactly this issue, and the CONDITIONAL verdict already captures it. I would keep the verdict unchanged but make the condition explicit: the authors should provide output-quality evidence and an artifact for reproduction, or soften the claim that this is a usable generative AI deployment.","tokens_in":12350,"tokens_out":8788,"duration_ms":89206,"concrete_test":"Ask the authors to release the exact 0.7B ternary checkpoint used in Section IV and run a standard quality check: WikiText-2 perplexity, 3-5 zero-shot accuracy tasks, and 20 generated samples, compared against the full-precision baseline. The usability claim is supported if the ternary model's perplexity is close to the published BitNet-1.58 range (or within roughly one point of the baseline) and the samples are coherent; if the checkpoint is missing or perplexity collapses, the performance numbers should not be presented as usable LLM inference. An independent alternative is to rerun Section IV with the official BitNet b1.58 0.7B checkpoint and the released bitstream, reporting both quality and throughput.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TeLLMe's central claim is that a real ternary LLM can run end-to-end on a 7 W edge FPGA. This requires the specific 0.7B BitNet-style model to produce acceptable text after 1.58-bit quantization. Section II.B motivates the design by citing BitNet-1.58 and DeepSeek accuracy, but Section IV contains no perplexity, zero-shot benchmark, or generation-quality check; Table V lists only model size, and the text reports only throughput, latency, power, and resources. This is not a minor omission because 1.58-bit weights are normally obtained through training from scratch or carefully designed quantization, not by naively quantizing a pretrained model. If the authors used an ad hoc or unspecified checkpoint, the model could be degraded to the point of unusability, in which case 9.51 tokens/s and 0.55 s prefill are hardware numbers for a system that cannot perform its intended function. No code, bitstream, or checkpoint is released, so the reader cannot currently verify the model's quality from the manuscript alone. The concern is not a disagreement with consensus; it is an unverified prerequisite for the paper's headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TeLLMe, an FPGA accelerator for ternary (1.58-bit) LLM inference on the AMD KV260 edge platform. The design combines a table-lookup-based ternary matrix multiplication engine, a fused attention unit with a reverse-reordering schedule for prefill, and hardware reuse of the decoding attention unit for the LM head. The authors report up to 9.51 tokens/s decoding throughput with 1024-token contexts, prefill latencies of 0.55–1.15 s for 64–128 token prompts, and power below 7 W. The paper claims this is the first edge FPGA accelerator to support both prefill and decoding for ternary LLMs end-to-end.","tokens_in":12515,"tokens_out":2664,"duration_ms":27596,"significance":"If the reported results are validated, TeLLMe would be a meaningful advance: it demonstrates that a complete LLM inference pipeline, including the compute-heavy prefill stage, can run on a low-power edge FPGA with ternary weights and 8-bit activations. The paper's strengths are its concrete hardware implementation (HLS/Vivado bitstream on real hardware), detailed resource breakdowns (Tables I and IV), and an explicit algorithmic contribution in the reverse attention scheduling that avoids causal-mask waste (Section III.B). The measured performance against prior FPGA works (Section IV.C) and mobile SoCs (Section IV.D) gives the paper practical relevance. The central limitation is that no output-quality evaluation of the 0.7B ternary model is provided, and the measurement methodology for latency, throughput, and power is not described; these gaps prevent the headline claims from being fully verified as usable LLM inference.","major_comments":[{"comment":"The paper reports no evaluation of the output quality of the 0.7B BitNet-style ternary model. Table V lists only the model size in MB, and Figure 9 reports throughput and prefill latency, but no perplexity, zero-shot benchmark scores, or generated-sample quality is given anywhere in Section IV. The deployment claim that TeLLMe is a 'practical viability' for real-world applications depends on the 1.58-bit quantized model retaining acceptable language quality. Since ternary weights are typically obtained by training from scratch (as in BitNet-1.58) or by carefully designed quantization, the manuscript must state the checkpoint provenance and quantization procedure and report at least one standard quality metric (e.g., perplexity on a held-out set, or a small set of zero-shot tasks). Without this, the reported 9.51 tokens/s and 0.55 s prefill are hardware figures for a system whose end-to-end functionality is unverified.","section":"Section IV, Table V and Figure 9"},{"comment":"The measurement methodology is not described. There is no statement of how latency, throughput, and power were measured (e.g., board-level power monitor vs. on-chip sensors, number of repeated runs, whether power is total system power or FPGA-only), and no error bars or variance information is reported for any of the headline numbers. Given that the power budget of 7 W is a central part of the contribution (abstract, Section I, Conclusion), the authors should specify the measurement setup and report at least the range or standard deviation across runs for the throughput and prefill latency figures.","section":"Section IV.A, Figure 9, Table III"},{"comment":"The comparison in Table III is not apples-to-apples and the 'up to 16.4x improvement' claim is overstated as stated. TeLLMe runs a 0.7B BitNet W1.58 model, whereas Li et al. run LLaMA2-7B W4, LlamaF runs TinyLLaMA W8, and SECDA runs TinyLLaMA W4; the models differ in parameter count, arithmetic intensity, and memory footprint, so raw tokens/s are not directly comparable. The authors should either normalize throughput by model size or compute/bandwidth requirements, or clearly frame the comparison as 'system-level demonstration' rather than architectural superiority. They should also clarify whether the KV260 bandwidth entry (19.2 GB/s) is the same for all KV260-based designs and how the BW column was obtained for the other works.","section":"Section IV.C, Table III"},{"comment":"The bandwidth and iteration-count formulas in Table II are not derived in the text, and the units are unclear. For example, the reverse scheduling row gives 'N^2/(2p) + N' for both Data Block and Iteration Count, but these quantities have different dimensions (data volume versus number of iterations). The authors should define each column formally and show how the expressions follow from the schedule described in Section III.B.2, particularly the claim that reverse attention reduces both computation and data movement relative to dense scheduling while maintaining constant bandwidth.","section":"Section III.B, Table II"}],"minor_comments":[{"comment":"The abstract reports 'up to 9 tokens/s' while Section IV.C reports '9.51 tokens/s'; these numbers should be made consistent throughout the paper.","section":"Abstract and Section I"},{"comment":"There is a typo in the introduction: 'Tenary' should be 'Ternary'.","section":"Section I"},{"comment":"Reference [8] (DeepSeek-R1) duplicates reference [3], and the description of DeepSeek-R1 as applying 'hybrid quantization strategy' to MoE layers is not accurate for the R1 reasoning model; the authors likely mean DeepSeek-V3 or another work, and should correct the citation and description.","section":"Section II.B, references [3] and [8]"},{"comment":"The superscript notation in Equation (3) is confusing: the text uses both p(1) and p(2) for intermediate max values, but the display mixes m, ℓ, and o with superscripts without a clear convention. Please define each variable and its superscript explicitly.","section":"Section III.B, Equation (3)"},{"comment":"The resource utilization percentages in Table IV are given as (71%), (28%), (66%), (93%), (75%) but the corresponding device capacities are not stated in the table caption; adding the XCK26 resource totals would help readers interpret the utilization.","section":"Section IV.B, Table IV"},{"comment":"The source of the Snapdragon 8 Gen 3 numbers is not cited; please provide a reference or measurement details so readers can assess how those baselines were obtained.","section":"Section IV.D, Table V"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid hardware-design contribution with real measurements on the KV260, but it falls short of the journal bar in its current form because the end-to-end claim (usable ternary LLM inference) is not validated by any model-quality metric, and the measurement methodology is underspecified. The missing quality evaluation is not a minor omission: the entire deployment narrative depends on the 0.7B checkpoint being functional. If the authors can add a model-quality evaluation and clarify the measurement and comparison protocols, the paper would be suitable for publication. I do not see any evidence of circular reasoning or fabricated results; the concern is incomplete validation, not internal inconsistency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTeLLMe is a real build, not a simulation: a ternary (1.58-bit) LLM accelerator on a KV260 FPGA that does both prefill and decode, with measured decode around 9.5 tokens/s and 0.55-1.15s prefill for 64-128 token prompts under 7W. The architecture is genuinely thoughtful. The table-lookup matmul adapts T-MAC's CPU trick to FPGA soft logic with a multi-table dataflow, and the reverse-attention scheduling for prefill is the standout idea: it avoids wasting compute on causal-mask zeros and cuts DRAM traffic compared to naive or dense scheduling. Reusing the decode attention hardware for the LM head is a sensible resource-saving move. The resource tables are detailed and internally consistent.\n\nThe soft spots are real but mostly fixable. The biggest one, as you'd guess, is that Section IV measures throughput, latency, power, and resources, but never evaluates whether the 0.7B ternary model produces acceptable text. No perplexity, no zero-shot benchmarks, no sample outputs. For a paper claiming \"practical viability\" and comparing against mobile SoCs running 1B/3B models, that's a load-bearing omission. If the checkpoint is an open-source BitNet-style model with known quality, say so and cite it. If it was quantized ad hoc, the 9.51 tokens/s might be generating nonsense. This must be addressed before deployment claims carry weight.\n\nSecond, the measurement methodology is thin: no error bars, no power-measurement description, no run counts. Power and latency are headline numbers; the reporting should match that. Third, Table III compares against TinyLlama W8 and LLaMA2-7B W4, so the \"16.4x improvement\" is apples-to-oranges, and the footnote honestly admits the resource numbers aren't directly comparable. None of these are fatal, but they need tightening.\n\nI think the stress-test note lands. The missing quality check is not a side issue; it's a prerequisite for the end-to-end claim. That said, it's easily fixable with a few benchmarks or generated samples. The hardware contribution stands on its own as a first demonstration.\n\nSend it to peer review. Expect a major-revision round focused on model-quality evidence and measurement rigor.","headline":"Real hardware, real measurements, and a clever architecture, but the missing output-quality evaluation is a load-bearing gap that needs fixing before the paper can claim practical deployment.","tokens_in":13122,"tokens_out":2443,"would_cite":true,"duration_ms":24310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Full ternary LLM inference runs on a 7-watt edge FPGA","keywords":["ternary LLM","FPGA accelerator","prefill","decoding","table-lookup matmul","1.58-bit quantization","edge inference","energy efficiency"],"falsifier":"Run the exact 0.7B ternary model deployed on TeLLMe on a standard language-generation or perplexity benchmark against its full-precision counterpart; if quality degrades sharply, the speed and energy numbers do not translate into a usable deployment. Alternatively, measure time-to-first-token and decode throughput on an HBM-enabled FPGA: if decode speed does not scale with memory bandwidth, the bandwidth-bound claim would be contradicted.","tokens_in":12087,"feed_emoji":"⚡","tokens_out":6366,"duration_ms":57044,"temperature":0.7,"pith_summary":"This paper claims that a modest FPGA can run a full ternary (1.58-bit) large language model end to end, including the often-skipped prefill stage, within a 7-watt power budget. It presents TeLLMe, an accelerator for AMD KV260 that reaches 9.51 tokens per second for decoding, supports 1024-token contexts, and prefills 64 to 128-token prompts in 0.55 to 1.15 seconds. The design turns ternary matrix multiplication into table lookups and reorders prefill attention so causal-mask computation and DRAM traffic shrink. If correct, this closes the gap between quantized-model algorithms and practical on-device generative AI, giving battery-conscious edge devices a viable path to running their own LLMs.","feed_headline":"Ternary LLM inference runs on a 7-watt edge FPGA","feed_subtitle":"TeLLMe handles prefill and decoding on an AMD KV260, reaching 9.51 tokens per second under 7 W.","key_machinery":"The load-bearing mechanism is the table-lookup-based ternary matmul: weights in {-1,0,1} are packed into groups of G, each group encoded as a small index; on-chip LUTs are configured as a table of all $3^G$ signed sums for a group of activations, so a multiply-accumulate becomes an indexed add or subtract. Index vectors access multiple tables in parallel, and the table setup is reused across all output dimensions. The second mechanism is reversed attention scheduling for prefill, which iterates query tokens from the end of the sequence and fuses score, softmax, and value aggregation in one pass, eliminating masked computations that naive and dense schedules incur. Together these mechanisms reduce LUT pressure and DRAM traffic enough to fit both prefill and decoding on a 108K-LUT FPGA.","core_discovery":"TeLLMe is, to the authors' knowledge, the first FPGA accelerator for a 1.58-bit ternary LLM that supports both prefill and autoregressive decoding on the same edge device. Using BitNet-1.58-style weights and 8-bit activations, it implements a table-lookup matmul engine that precomputes all add and subtract combinations for groups of ternary weights and reuses grouped activations across projection and feedforward layers. A fused attention unit with reversed reordering processes queries from the end of the sequence, avoiding invalid causal-mask computation and keeping bandwidth near constant. The decoding-phase attention hardware is reused for the language-model head because both are memory-bound matrix-vector operations. On AMD KV260 at 250 MHz, TeLLMe reports up to 9.51 tokens per second decoding, prefill latencies of 0.55 to 1.15 seconds for 64 to 128-token prompts, and under 7 W total power.","pith_inferences":["If 1.58-bit model quality holds up in evaluation, the same table-lookup and reversed-attention machinery could extend to 1-bit or 2-bit quantization with only lookup-table size changes, broadening the design's applicability.","The reversed reordering schedule is independent of ternary quantization and could speed up any causal attention on bandwidth-limited FPGAs or other low-power accelerators.","The paper's comparisons to mobile SoCs use different process nodes and memory systems; a normalized energy-per-token or quality-adjusted throughput metric would be needed to place the 7 W advantage on a fair footing.","The absence of model-quality results suggests an immediate follow-up: benchmark the actual 0.7B ternary model on standard tasks to confirm that the generated tokens are useful, not merely fast."],"forward_implications":["Full on-device LLM inference becomes possible on low-cost FPGAs, removing the need to offload prefill to a host and the associated privacy and latency costs.","The 9.51 tokens per second decode rate and roughly 0.55 second prefill for short prompts make interactive edge applications, such as chatbots and on-device summarization, plausible at 7 W.","Table-lookup matmul provides a hardware template for other ultra-low-bit formats, including pure binary and 2-bit weights, by enlarging the lookup table accordingly.","Because decoding is memory-bandwidth bound, moving the same design to a higher-bandwidth platform should raise tokens per second nearly proportionally, an architectural rather than algorithmic ceiling.","Prior FPGA accelerators that only decode would need to add a prefill path; TeLLMe shows the resource overhead of doing so is affordable at roughly 108K LUTs."],"supporting_citations":[{"why":"Defines the 1.58-bit ternary weight format and training recipe that TeLLMe accelerates.","marker":"[7]"},{"why":"Shows ternary quantization works at scale, supporting the claim that 1.58-bit models retain usefulness.","marker":"[8]"},{"why":"Introduces table-lookup matmul for low-bit LLMs on CPUs, the algorithmic basis TeLLMe ports to FPGA LUTs.","marker":"[15]"},{"why":"Prior KV260 FPGA LLM accelerator limited to decoding; supplies the baseline TeLLMe's prefill support and throughput improve on.","marker":"[9]"},{"why":"Provides the attention data-reuse reordering idea that TeLLMe extends with causal-mask-aware reverse scheduling.","marker":"[17]"},{"why":"Defines the fused kernel computation that TeLLMe adapts to an FPGA-friendly block-size-one form.","marker":"[21]"},{"why":"Characterizes FPGA strength in decoding and weakness in prefill, motivating TeLLMe's co-optimization.","marker":"[10]"},{"why":"Earlier edge FPGA LLM accelerator with low decode throughput; serves as a comparison baseline for TeLLMe's speed.","marker":"[18]"}],"fun_headline_variants":["First ternary LLM accelerator for edge FPGAs: prefill+decoding","9 tokens/s ternary LLM on a 7 W FPGA, prefill included","Edge FPGA runs 1.58-bit LLM at 9.5 tok/s under 7 W","Prefill+decoding ternary LLM on FPGA: 9.5 tok/s at 7 W","TeLLMe: 1.58-bit LLM on FPGA does prefill and decoding at 7 W"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system assumes the 0.7B BitNet-style ternary model used in the experiments retains usable output quality after 1.58-bit quantization; the paper reports no perplexity, benchmark, or sample-quality evaluation, so high throughput and low power could coexist with unusable text.","fun_headline_variants_meta":{"raw":{"variants":["First ternary LLM accelerator for edge FPGAs: prefill+decoding","9 tokens/s ternary LLM on a 7 W FPGA, prefill included","Edge FPGA runs 1.58-bit LLM at 9.5 tok/s under 7 W","Prefill+decoding ternary LLM on FPGA: 9.5 tok/s at 7 W","TeLLMe: 1.58-bit LLM on FPGA does prefill and decoding at 7 W"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000878,"raw_usage":{"total_tokens":3825,"prompt_tokens":1005,"completion_tokens":2820,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":2698}},"tokens_in":621,"tokens_out":2820,"duration_ms":18302,"temperature":1.0,"reasoning_tokens":2698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:08:13.655141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact 0.7B ternary model deployed on TeLLMe on a standard language-generation or perplexity benchmark against its full-precision counterpart; if quality degrades sharply, the speed and energy numbers do not translate into a usable deployment. Alternatively, measure time-to-first-token and decode throughput on an HBM-enabled FPGA: if decode speed does not scale with memory bandwidth, the bandwidth-bound claim would be contradicted.","supporting_citations":[{"cited_title":"Edge-moe: Memory- efficient multi-task vision transformer architecture with task-level spar- sity via mixture-of-experts,","cited_arxiv_id":null,"evidence_quote":"Provides the attention data-reuse reordering idea that TeLLMe extends with causal-mask-aware reverse scheduling."},{"cited_title":"Understanding the potential of fpga-based spatial accel- eration for large language model inference,","cited_arxiv_id":null,"evidence_quote":"Characterizes FPGA strength in decoding and weakness in prefill, motivating TeLLMe's co-optimization."}],"review_version":1}