Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Moving the draft model to the edge and giving the cloud target model early exits lets the client pre-draft tokens during verification, cutting latency by up to 35% with output identical to standard speculative decoding.

desk verdict A genuinely new pre-drafting idea with under-supported empirical claims; the mechanism is real, but the headline speedup is overstated and the latency model needs to be honest about timing. read the letter →

arxiv 2505.21594 v1 pith:QVNYVYOT submitted 2025-05-27 cs.RO cs.AIcs.DC

Yeshwanth Venkatesha , Souvik Kundu , Priyadarshini Panda This is my paper · ORCID
classification cs.ROcs.AIcs.DC
keywords speculativedecodingearlyexitsedge-cloudinferencepre-draftingLLMlatencyreductionvision-languagemodelon-devicedraftAPIcost
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that speculative decoding can be split across a cloud server and a resource-limited edge device without sacrificing output quality or paying for a cloud draft model. The proposed framework runs a small draft model on the client, verifies its tokens on a large server model, and equips the server model with early-exit heads that emit validated token continuations before the final exit completes. Those early continuations let the client start drafting the next round of tokens during what would otherwise be idle waiting, and a cache hit at final verification makes the next draft instantly available. On a Jetson Nano client with a Llama2-7B server, the paper reports up to a 35% latency reduction over cloud autoregressive decoding, a further 4–11% from this preemptive drafting, and a 21% speedup in a vision-language robot navigation demo. The consequence is a combination: lower cloud API cost, lower latency, and text output identical to standard speculative decoding.

What carries the argument

The load-bearing machinery is a set of early-exit adapters: small trainable LM heads inserted after layers of the frozen target model, each producing logits $z^{(l)}$ and confidence $S^{(l)} = \max(\mathrm{softmax}(z^{(l)}))$ used to prioritize candidates. Priority queues on both server and client order early-exit outputs by confidence so scarce client threads work on the most likely continuations, and a pre-draft cache stores the draft-model output for each candidate. The argument's backbone is the cache-miss formula $\frac{\gamma c + 1}{r\gamma c + 1}$ with $c = T_p/T_q$: a cache hit replaces the drafting term $\gamma T_p$ with the synchronization term $T_r$, and under negligible communication this ratio is the whole speedup, so low miss rate $r$ and high latency ratio $c$ make preemptive drafting increasingly valuable.

What would settle it

Instrument the Jetson client to record, for each verification round, the arrival time of the final-exit verification and whether the pre-drafted tokens for the candidate the final exit accepts are already in the pre-draft cache at that moment; if a substantial fraction of cache hits are not ready when the final verification arrives, the Table 4 latency model overstates the FSD speedup. A simpler check is to cap the client to a single pre-drafting thread and rerun the MT-bench and GSM8K evaluations; if the SD-to-FSD speedup does not drop materially, preemptive drafting is not the source of the gain.

Watch

Extended reading notes

Core claim

The central discovery is that early exits turn the slow edge client into a parallel pre-drafter instead of an idle requester. Each round, the target model's early exits verify the client's $\gamma$ draft tokens and return multiple candidate continuations $x^{(i)}_{t:t+\delta^{(i)}+1}$; the client treats each candidate as a new prefix, runs the draft model on it, and stores the resulting next $\gamma$ draft tokens in a pre-draft cache. When the final exit returns the verified continuation, the client looks it up; on a hit it sends the pre-drafted tokens immediately, paying only thread-synchronization latency $T_r$ instead of the full drafting cost $\gamma T_p$. Because the final exit still verifies every token, the output distribution is exactly that of standard speculative decoding, not an approximation. The paper's latency model for the fast scheme is $\frac{n}{\tau}(2T_c + r\gamma T_p + (1-r)T_r + T_q)$ versus $\frac{n}{\tau}(2T_c + \gamma T_p + T_q)$ for vanilla edge-cloud speculative decoding, with $r$ the cache miss rate; this formula is what turns the measured 4–11% speedup into a system claim.

Load-bearing premise

The speedup rests on the assumption that the edge device finishes pre-drafting the winning candidate before the server's final verification arrives, so a cache hit costs only thread synchronization and not a fresh draft pass; if the client is too slow or has too few threads for the number of early exits, the promised 4–11% gain over vanilla speculative decoding will not materialize.

Editorial extensions

If this is right

  • Cloud API cost for a given output can drop by about half, because token drafting happens on the device and only the target model's verification is billed.
  • Latency-sensitive applications on cheap edge hardware, such as interactive assistants and closed-loop robot control, become feasible; the reported robot demo achieves a 21% speedup over cloud autoregressive decoding.
  • Because the final exit still verifies every token, the framework can be added underneath existing speculative-decoding pipelines without changing the generated text.
  • Under the paper's Appendix B projection with near-zero communication latency, large model pairs with latency ratio above 0.5 and cache miss rate below 10% could see multi-fold speedups rather than the current 4–11%.
  • Edge-cloud speculative decoding is not universally faster than cloud autoregressive decoding: on the Jetson with small target models (Vicuna-7B, Vicuna-13B), vanilla edge-cloud SD is slower, and early-exit pre-drafting is what keeps it competitive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the 4–11% gain depends on the test prompts resembling the ShareGPT and SpatialVLM data used to train the early-exit adapters; on out-of-domain inputs the cache miss rate would rise and the pre-drafting benefit could shrink or disappear.
  • Beyond the paper: the method's sweet spot is the asymmetric regime measured here, where the edge draft model is slow but still fast enough to finish inside the verification window; a faster edge device lowers $c$ and makes the AR-to-SD gain larger while making the extra FSD gain smaller.
  • Beyond the paper: the single-client prototype leaves server scheduling and multi-client contention unexplored; with several robots sharing one A100, early-exit queueing and verification batching would interact in ways the Table 4 latency model does not capture.
  • Beyond the paper: pre-drafting every early-exit candidate multiplies on-device memory and energy by the active thread count, which is already capped at seven threads for Vicuna-160M on the Jetson; power-constrained deployments would need a policy that pre-drafts only a few high-confidence candidates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a speculative edge-cloud decoding framework in which a small draft model runs on the client and a large target model with trained early-exit adapters runs on the server. Early-exit outputs from the target model are sent to the client during verification, allowing the client to preemptively draft the next set of tokens and cache them. If the final exit output matches an early exit, the cached draft is used immediately, reducing client idle time and server idle time. The authors train early-exit adapters for Vicuna-7B/13B and Llama2-7B, evaluate on six generation benchmarks using Jetson Nano and RTX 2080 Ti clients with an A100 server, and report speedups of autoregressive decoding to vanilla edge-cloud speculative decoding and further speedups from their fast speculative decoding. A robotics case study on a Unitree Go2 with a Qwen2-VL 7B target is also included.

Significance. The paper addresses a timely and practical problem: reducing the cost and latency of LLM inference for edge devices by splitting drafting and verification between the edge and the cloud. The central idea of using early exits to enable preemptive drafting is novel and potentially useful, and the claimed exactness of the method is an attractive property. The explicit latency model and the system-level measurements on real hardware are valuable, and the robotics deployment adds a credible application demonstration. However, the additional speedup from the fast speculative decoding (FSD) mechanism rests on an unverified timing assumption about pre-drafting completing within the server-verification idle window, and the reported 'average 35% latency reduction' is not supported by the paper's own numbers. If the timing assumption can be validated, the contribution would be a meaningful step toward cost-effective edge-cloud LLM inference.

major comments (3)
  1. [Table 4; Section 4.2; Appendix B] The FSD latency model in Table 4 is (n/τ)(2Tc + rγTp + (1−r)Tr + Tq), which charges only thread-synchronization time Tr on a cache hit and omits the drafting cost γTp entirely. This is valid only if, for every cache-hit round, the client finishes pre-drafting the matching early exit before the final exit output arrives. The paper provides no wall-clock traces or timing analysis demonstrating that this holds. Table 5 caps the client at 6–15 threads while Table 6 reports average early exit indices of 8–14, so for later matching exits the pre-drafting workload may not fit within the idle window. If pre-drafting spills past final-exit arrival, the FSD speedups in Table 6 would be an artifact of the latency formula rather than a real system speedup. Please provide measured end-to-end latency traces showing that pre-drafting completes within the idle window under the reported thread caps, or revise the model to include the time to receive and process early exits before the final output.
  2. [Section 1 contribution bullet; Abstract] The contribution list states 'we show an average 35% latency reduction from autoregressive to vanilla edge cloud speculative decoding,' but Table 6 does not support this. For the cited pair Vicuna-68M/Llama2-7B on Jetson, the average AR→SD speedup is 1.35x, which is a 26% reduction, not 35%; the 35% figure appears only for the HumanEval benchmark (1.53x). On RTX the average is 1.94x (48% reduction). Please correct the stated average or rephrase consistently as 'up to 35%' to match the abstract.
  3. [Section 4.2] The paper does not state whether the speedups in Table 6 are measured end-to-end or computed from the latency model in Table 4. If they are model-computed, the timing assumption in the first major comment applies directly and the FSD gains are not empirically established; if they are measured, the measurement procedure (including how cache-hit and cache-miss rounds are timed and how thread scheduling is handled) should be described explicitly.
minor comments (5)
  1. [Algorithm 1] The pseudocode processes the priority queue in the main thread (lines 9–14), whereas Section 3 Step 4 states that each early exit is processed in a new thread; please clarify the actual threading model.
  2. [Table 6, NQ row] In the Vicuna-160m/Vicuna-13B RTX column, the Speedup SD→FSD entry reads '1.12%' and should be '1.12x'.
  3. [Conclusion] The conclusion says 'up to a 35% speedup' where the abstract and results refer to a 35% latency reduction; please use consistent terminology (speedup versus reduction).
  4. [Section 5] The claim that this is 'the first work to show end-to-end speculative decoding with models split between edge and cloud' is presented without a thorough comparison to recent asynchronous multi-device speculative decoding work; please soften or support this claim with a more complete related-work discussion.
  5. [Section 3, exactness statement] The statement that outputs are 'identical' to standard speculative decoding may be too strong if the draft model uses stochastic sampling; the appropriate guarantee is that the output distribution is identical, and the paper should state this precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FSD speedups are empirical end-to-end measurements, and the analytic latency model (Table 4, Eq. 9) is a stated-assumption projection, not a fitted input renamed as a prediction.

full rationale

The paper's central claims — up to 35% latency reduction from AR to edge-cloud SD and an additional 4–11% SD-to-FSD speedup — are presented as measured system results (Tables 5, 6, and 8), not as quantities derived from fitted parameters. The latency formulas in Table 4 and the speedup projection Eq. 9 in Appendix B depend on measured cache-miss rate r and measured latency ratio c, and Eq. 9 explicitly assumes negligible communication latency and ignores thread synchronization latency; this is an analytic model with stated assumptions, not a circular reduction. The early-exit mechanism cites Panda et al. (2016), a co-author's prior work, but only as background for a well-known technique; the paper trains its own early-exit adapters and the load-bearing novelty is the preemptive drafting mechanism, whose speedup is empirically reported. The reviewer-identified concern that pre-drafting may not finish within the server's idle window is a timing-feasibility and correctness risk, not a circularity: the paper's equations do not define the result in terms of its inputs, and no fitted parameter is renamed as a prediction. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central result is empirical, not derived from new physics or math. It assumes standard speculative decoding correctness, that early-exit prefixes are often identical to final-exit prefixes, that pre-drafting can be hidden in idle client time, and that measured per-round latencies are representative. The invented components are trained early-exit adapters and a pre-draft cache; neither has evidence outside this paper.

free parameters (3)
  • draft length gamma = 4 default; swept to 10 in Fig. 4b
    Controls the number of speculative tokens per round; chosen by hand, not derived; central to the latency model and speedups.
  • number of early-exit threads = Up to 30 on RTX, 15 on Jetson for Vicuna-68M; 6 in robot demo
    Limits how many early-exit outputs can be pre-drafted; chosen by hardware; directly affects cache hit rate and FSD speedup.
  • early-exit adapter count and placement = 31 for Vicuna-7B, 39 for Vicuna-13B, 27 for Qwen2-VL-7B
    Number and placement of exits is a design choice; determines how early pre-drafting can start.
assumptions (4)
  • standard math Verification at the final exit preserves the exact speculative decoding output distribution; early-exit prefixes do not alter accepted tokens.
    Invoked in Section 3 (Steps 5 and 6) and Section 4.2 to claim FSD output is identical to standard SD; this is a known property of speculative decoding, not demonstrated in the paper.
  • domain assumption Early-exit adapters trained on ShareGPT produce prefixes that match final-exit outputs often enough for the pre-draft cache to be useful.
    Section 3 Early Exit Training and Table 6 cache miss rates; adapter quality is only measured internally via cache miss rate and average early exit, not against external benchmarks.
  • domain assumption Pre-drafting on the client can be completed within the idle window while the server is still verifying, so the FSD latency model needs only Tr on cache hit and not gamma*Tp.
    Table 4 FSD latency formula and Appendix B Eq. 9; no wall-clock profiling is provided to show drafting fits in idle time under thread limits.
  • domain assumption Early-exit outputs are transmitted to the client before the final exit output, making pre-drafting possible in the same round.
    Section 3 Step 3; depends on communication being slower than server-side early-exit production and on network conditions.
invented entities (2)
  • Early-exit adapters with LM heads attached to target model layers
    purpose: Produce verified prefix tokens before the final exit to trigger pre-drafting
    Newly trained model components; their predictive quality is only evidenced internally via cache miss rate, with no external benchmark of adapter quality.
  • Pre-draft cache
    purpose: Store draft tokens computed from early-exit prefixes so they can be resent on a cache hit
    Software data structure fundamental to the method; its hit rate is the key empirical quantity and is only measured inside the paper's own system.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits." pith.science (2026). https://pith.science/paper/QVNYVYOT

@misc{pith2026250521594,
  author       = {Pith},
  title        = {Pith review of: Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVNYVYOT}},
  note         = {Machine review of arXiv:2505.21594}
}
read the original abstract

Large Language Models (LLMs) enable various applications on edge devices such as smartphones, wearables, and embodied robots. However, their deployment often depends on expensive cloud-based APIs, creating high operational costs, which limit access for smaller organizations and raise sustainability concerns. Certain LLMs can be deployed on-device, offering a cost-effective solution with reduced latency and improved privacy. Yet, limited computing resources constrain the size and accuracy of models that can be deployed, necessitating a collaborative design between edge and cloud. We propose a fast and cost-effective speculative edge-cloud decoding framework with a large target model on the server and a small draft model on the device. By introducing early exits in the target model, tokens are generated mid-verification, allowing the client to preemptively draft subsequent tokens before final verification, thus utilizing idle time and enhancing parallelism between edge and cloud. Using an NVIDIA Jetson Nano (client) and an A100 GPU (server) with Vicuna-68M (draft) and Llama2-7B (target) models, our method achieves up to a 35% reduction in latency compared to cloud-based autoregressive decoding, with an additional 11% improvement from preemptive drafting. To demonstrate real-world applicability, we deploy our method on the Unitree Go2 quadruped robot using Vision-Language Model (VLM) based control, achieving a 21% speedup over traditional cloud-based autoregressive decoding. These results demonstrate the potential of our framework for real-time LLM and VLM applications on resource-constrained edge devices.

Figures

Figures reproduced from arXiv: 2505.21594 by the authors.

Figure 1
Figure 1. Illustration of traditional cloud-based autoregressive decoding versus cloud-based speculative [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our proposed approach. Given a prefix, the client generates two draft tokens and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of training early exit adapters. Steps 5 & 6 : Once the final output xt:t+δ+1 from the tar￾get model is received, the client checks whether these tokens were already processed in any of the early exits by looking at the pre-draft cache. If there is a hit, the corresponding pre-draft tokens are retrieved from the pre-draft cache and immediately sent to the server for the next round of verifi￾cation, avoi… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation studies: (a) Effect of varying the number of early exit threads, (b) Effect of varying the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Example run of the Unitree Go2 robot performing an object-finding task using vision-language [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Estimated speedup per round of our FSD relative to vanilla SD method assuming no communication [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 5 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    The LISTENER processes the verification requests from the client

    the server consists of two asynchronous threads:LISTENER and SENDER . The LISTENER processes the verification requests from the client. It takes in the prefix x1:t, draft tokens xt:t+γ, and their corresponding probability distributionp1:γ. As shown in Fig. 2, communication typically becomes the bottleneck in the server as early exit outputs are produced f...

    work page 2023

  3. [5]

    Navila: Legged robot vision-language-action model for navigation

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation. arXiv preprint arXiv:2412.04453,

    arXiv

  4. [6]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  5. [7]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  6. [8]

    Not all layers of llms are necessary during inference

    13 Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of llms are necessary during inference. arXiv preprint arXiv:2403.02181,

    arXiv

  7. [9]

    A survey on large language models with multilingualism: Recent advances and new frontiers

    Kaiyu Huang, Fengran Mo, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, et al. A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405.10936,

    arXiv

  8. [10]

    Eagle-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858,

    arXiv

Show all 32 references
  1. [11]

    Pushing large language models to the 6g edge: Vision, challenges, and opportunities

    Zheng Lin, Guanqiao Qu, Qiyuan Chen, Xianhao Chen, Zhe Chen, and Kaibin Huang. Pushing large language models to the 6g edge: Vision, challenges, and opportunities. arXiv preprint arXiv:2309.16739,

    arXiv

  2. [12]

    Small language models: Survey, measurements, and insights

    Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights. arXiv preprint arXiv:2409.15790,

    arXiv

  3. [13]

    Amusd: Asynchronous multi-device speculative decoding for llm acceleration

    Bradley McDanel. Amusd: Asynchronous multi-device speculative decoding for llm acceleration. arXiv preprint arXiv:2410.17375,

    arXiv

  4. [14]

    Abstractive text summarization using sequence-to-sequence rnns and beyond

    Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023,

    arXiv

  5. [16]

    Mobile edge intelligence for large language models: A contemporary survey

    Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang. Mobile edge intelligence for large language models: A contemporary survey. arXiv preprint arXiv:2407.18921,

    arXiv

  6. [18]

    From words to watts: Benchmarking the energy costs of large language model inference

    14 Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Ext...

    2023

  7. [19]

    A survey of neural code intelligence: Paradigms, advances and beyond

    Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403.14734,

    arXiv

  8. [20]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

  9. [21]

    Branchynet: Fast inference via early exiting from deep neural networks

    Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp. 2464–2469. IEEE,

    2016

  10. [22]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  11. [23]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191,

    arXiv

  12. [24]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682,

    arXiv

  13. [26]

    Edge-llm: Enabling efficient large language model adaptation on edge devices via layerwise unified compression and adaptive layer tuning and voting

    Zhongzhi Yu, Zheng Wang, Yuhan Li, Haoran You, Ruijie Gao, Xiaoya Zhou, Sreenidhi Reedy Bommu, Yang Katie Zhao, and Yingyan Celine Lin. Edge-llm: Enabling efficient large language model adaptation on edge devices via layerwise unified compression and adaptive layer tuning and ...

    arXiv

  14. [27]

    Large language models for robotics: A survey

    Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S Yu. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07226,

  15. [28]

    Edgeshard: Efficient llm inference via collaborative edge computing

    15 Mingjin Zhang, Jiannong Cao, Xiaoming Shen, and Zeyang Cui. Edgeshard: Efficient llm inference via collaborative edge computing. arXiv preprint arXiv:2405.14371,

    arXiv

  16. [29]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,

    arXiv

  17. [30]

    A survey on language models for code

    Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. A survey on language models for code. arXiv preprint arXiv:2311.07989,

    arXiv

  18. [32]

    Naturally, having a lowerr will improve speedup, but the usefulness of our FSD method becomes more pronounced as we get to higherc andγ

    For reference, we plot the measuredc and r values based on the Jetson implementation of our current set of models within this landscape. Naturally, having a lowerr will improve speedup, but the usefulness of our FSD method becomes more pronounced as we get to higherc andγ. For...

    2024

  19. [2016]

    Conditional deep learning for energy-efficient and enhanced pattern recognition

    Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. Conditional deep learning for energy-efficient and enhanced pattern recognition. In 2016 design, automation & test in europe conference & exhibition (DATE), pp. 475–480. IEEE,

    2016

  20. [2019]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258,

    arXiv

  21. [2021]

    Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism

    Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. arXiv preprint arXiv:2312.04916,

    arXiv

  22. [2022]

    Empowering 1000 tokens/second on-device llm prefilling with mllm-npu

    Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. Empowering 1000 tokens/second on-device llm prefilling with mllm-npu. arXiv preprint arXiv:2407.05858, 2024a. Minrui Xu, Dusit Niyato, Jiawen Kang, Zehui Xiong, Shiwen Mao, Zhu Han, Dong In...

    arXiv

  23. [2023]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165,

    2005 arXiv

  24. [2024]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.