Pith. sign in

REVIEW 4 major objections 5 minor 61 references

TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applications

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TimelyLLM claims LLM serving can meet many robots' deadlines by pausing plan generation at each executable action and scheduling by time-utility functions, gaining up to 1.97× utility and 84% less waiting time than vLLM.

desk verdict Legitimate segmented-generation system with an honest evaluation, but the headline utility ratio is misleading and the appendix proof has a gap. read the letter →

arxiv 2412.18695 v1 pith:2YUD7FPX submitted 2024-12-24 cs.RO cs.DCcs.LG

Neiwen Ling , Guojun Chen , Lin Zhong This is my paper · ORCID
classification cs.ROcs.DCcs.LG
keywords largelanguagemodelservingtime-sensitiveschedulingsegmentedgenerationtime-utilityfunctionrobotictaskplanningcontinuousbatchingreal-timesystemsedgerobotics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TimelyLLM argues that an LLM service can satisfy many robots with time-sensitive demands at once if generation is no longer treated as one indivisible response. The key observation is an asymmetry: a drone needs seconds to act on a command such as 'move forward', while the LLM produces those tokens in tens of milliseconds, so a server can pause a plan the moment a runnable action appears, lend the GPU to a more urgent request, and still resume before the robot finishes. Scheduling is driven by each request's time-utility function, a user-supplied deadline with utility and tolerance, so urgent tasks such as collision avoidance are favored without starving normal planning. On workloads built from real Tello drone and Neuromeka robot-arm traces, the paper reports up to 1.97× the time utility and 84% less waiting time than the vLLM baseline.

What carries the argument

The load-bearing mechanism is segmented generation: decoding stops at the first executable skill via a regular-expression stop checker, and the suspended context (KV cache plus tokenized output, about 170 MB for Llama3-8B) is stored so resuming costs roughly 9.5 ms of host-to-GPU transfer instead of roughly 133 ms of re-prefilling. On top of it sits the PUD-based priority rule, which scores each initial or suspended generation by its time-utility gain per unit of generation time divided by slack, and for suspended generations sets the expected response time to zero so that resumption urgency is tied to the robot's estimated execution of the preceding segment, profiled per skill from real robot measurements. The third piece is latency-guided batch-size selection, which estimates a segment's worst-case completion from its remaining token budget and recent generation speed and admits new generations only if the most urgent one still meets its deadline. These three pieces convert the time redundancy between plan generation and robot execution into a resource pool the scheduler can spend on urgent requests.

What would settle it

Run the workload on a task set whose robot actions all execute in less time than the per-segment generation time plus context-switch overhead, as the paper's own 'print' skill already does at about 1 ms. If TimelyLLM's aggregate time utility still beats vLLM's FCFS batching on that workload, then execution-time redundancy is not the operative mechanism; if it falls below, the central claim depends entirely on every skill taking longer to execute than to plan, which is exactly the premise the system is built on.

Watch

Extended reading notes

Core claim

The paper's central claim is that time-sensitive LLM serving for robots becomes a tractable scheduling problem once generation is segmented at executable-skill boundaries. TimelyLLM's content-aware stop checker watches the token stream, and the moment a complete robot skill such as 'pick(cyan_box)' appears it suspends decoding, caches the KV cache and tokenized output so that resumption needs no prefill, and hands the segment to the robot for immediate execution. While the robot acts, the scheduler ranks every initial and suspended generation by potential utility density, computed from the request's time-utility function and, for suspended generations, the estimated execution time of the segment the robot is currently performing, while a latency-guided batch-size selector caps the batch so the most urgent generation stays within its remaining time budget. The paper proves that maximizing the sum of segment-level utilities is Pareto optimal with respect to completion time and first-segment utility for monotonically non-increasing TUFs, and reports end-to-end gains of 1.97× in time utility and 84% less waiting time on robot-arm workloads, with per-segment overheads under 6% of generation time.

Load-bearing premise

The benefit rests on the assumption that executing a generated robot action takes longer than generating the next plan segment plus switching context, so that robot execution hides the planning of what comes next; for near-instant actions like printing a message, segmentation only adds waiting time, which the paper itself observes in task type 1.

Editorial extensions

If this is right

  • A single GPU serving many robots can meet mixed deadlines that FCFS batching misses: urgent tasks with a 200 ms expected response time keep near-full utility while normal planning tasks with a 1 s deadline stay at maximum, where the vLLM baseline lets urgent utility collapse to about a fifth of its maximum under load.
  • The longer the generated plan, the larger the gain, because more segments create more preemption opportunities; the robot-arm workloads with ~100-token plans deliver the headline 1.97× utility and 84% waiting-time reduction.
  • Segmented scheduling dominates pure priority policies and plain streaming: on urgent tasks it beats FCFS by 183% and EDF by 142% in utility without sacrificing normal tasks, and beats streaming execution (vLLM-stream) because suspension frees resources for preemption.
  • The overheads stay small enough for edge deployment, with segmentation adding under 6% of generation time per trace and scheduling costing about 2 ms for eight queued tasks, so the utility gains are not consumed by the mechanism itself.
  • The same redundancy idea transfers to chatbot consumption by segmenting at sentence or paragraph boundaries and treating reading time as execution time, cutting delays by up to 96% on the Phi-3 model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The claimed gains scale with the gap between generation speed and execution speed: on slower GPUs, larger models, or fatter prompts, per-segment generation approaches the robot's action duration and the 1.97× advantage would shrink toward ordinary priority queuing, a regime the paper does not measure.
  • The pattern generalizes beyond robots to any LLM consumer with a predictable consumption time, such as code interpreters, UI-automation agents, or speech synthesis, each of which could subscribe a time-utility function; the paper only evaluates robot execution and simulated reading, so this is an extrapolation.
  • The execution-time model is the softest transfer point: it is profiled on exactly two robot platforms and deliberately takes minimum durations for variable skills to stay worst-case safe, so an online learned estimator replacing the offline profile would be a direct test of whether that conservatism is what preserves utility for normal tasks.
  • Frequent suspension raises the chance that a long normal plan is repeatedly pushed back by a stream of urgent arrivals, the same effect the paper observes for the fast 'print' skill in task type 1; a starvation bound or aging term in the priority function would be a natural follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents TimelyLLM, an LLM serving system for time-sensitive robotic requests. TimelyLLM splits LLM-generated robot plans into executable segments, suspends generation after each segment, and schedules suspended generations using a TUF-based priority mechanism that estimates both expected time-utility gain and the robot's execution time for the generated segment. The system is implemented on top of vLLM with a custom stop checker and context caching. The authors build LRTrace, a trace collection system using a Tello drone and a Neuromeka robot arm, and evaluate TimelyLLM against vLLM and a streaming variant on drone, robot-arm, and chatbot workloads. The headline results are an up-to-1.97x improvement in time utility and an 84% reduction in waiting time over vLLM.

Significance. If the central claim holds, TimelyLLM is a useful contribution to LLM serving for robotics: it identifies a real redundancy between plan generation and robot execution, implements it with modest overhead, and evaluates against an external, widely used baseline (vLLM) rather than only against self-constructed variants. The paper also contributes LRTrace, a first attempt at building real-world multi-robot LLM serving traces, and reports overhead numbers for segmentation and scheduling. The idea of semantically grounded segmentation with resumption from cached KV states is plausible and could benefit other streaming-consumer applications. However, the current manuscript does not establish the headline gains for general robotic workloads because the evaluation is single-run and the '1.97x' figure is an additive utility gain computed from a negative baseline, while the core benefit depends on an unquantified domain condition: segment execution time must exceed generation time plus resume overhead.

major comments (4)
  1. [§7.4.1, Figure 9] The headline '1.97x' improvement in time utility is not a ratio. For the Desk Clean task, vLLM achieves utility -0.97 and TimelyLLM achieves 1.00, so the difference is 1.97, but the ratio is negative and would not be reported as a fold improvement. The abstract and conclusion repeat the '1.97x' wording. Please report the additive utility gain or use a non-negative utility transformation, and remove ratio language for this comparison.
  2. [§7.1–§7.4] All evaluation results are point estimates with no error bars, confidence intervals, or number of runs, even though the workloads are stochastic: §6 specifies Poisson event arrivals, and §6 also says execution times are randomly sampled from profiled data. Claims such as '191% improvement' (§7.2), '183% and 142%' (§7.3.2), and '84% waiting-time reduction' (§7.4.1) are therefore not statistically supported. Please report multiple seeds or replicates with variance, or at least a sensitivity analysis showing that the conclusions are stable.
  3. [§4.2, §4.3, §7.2] The core benefit of segmented generation depends on the assumption that the execution time of a generated segment exceeds the time to generate the next segment plus context-switch and resumption overhead. The paper itself acknowledges in §7.2 that task type 1, whose 'print' skill executes in about 1 ms, incurs increased waiting time because generation cannot be hidden. Yet the headline gains come from WID3 robot-arm tasks with long-duration manipulation skills. No analysis quantifies the fraction of segments for which the execution time actually exceeds the generation-plus-resume cost, and no sensitivity experiment varies this ratio. Without that, the claim that TimelyLLM improves time utility for general multi-agent robotic workloads is not established.
  4. [Appendix A, Lemma 2 and Lemma 3] The proof of Theorem 1 has a logical gap. Lemma 2 assumes there exists another solution x' that improves one request's completion time 'without impacting the total time utility of the first segments' and asserts that 'no other requests need to be negatively impacted by this change'. This is essentially the Pareto statement to be proven, not a consequence of optimality of Eq. 3. Similarly, Lemma 3 assumes a first-segment utility increase with no negative impact elsewhere. Furthermore, monotonic non-increasing TUFs do not imply that a strict waiting-time reduction yields a strict utility increase; for TUF1 defined in §4.2 with ERT=0, the function is flat for non-positive waiting times. The proof should be restructured or the theorem stated with the additional non-interference assumption made explicit.
minor comments (5)
  1. [Figure 5] The x-axis labels task types 0–7, but the workloads in Table 3 use trace IDs 1–8; the off-by-one indexing should be clarified in the caption or the axis.
  2. [§1, §2.1] The abstract says existing systems employ 'FCFS batching', but §2.1 correctly notes that vLLM and similar systems use continuous batching; please reconcile this terminology.
  3. [§4.3, Eq. (4)] The slack time L(s_i_k) is used as a scaling factor but its computation is not defined. Please specify how L is calculated for initial and suspended generations.
  4. [§7.5.1] Table 4 reports overhead as a percentage of segmented generation time, but the text does not state the number of runs or the variance across runs; adding this would strengthen the overhead claim.
  5. [§5, §6] No artifact or repository link is provided for TimelyLLM or LRTrace; including one would support reproducibility and allow the community to verify the reported traces.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central comparison is against an external baseline (vLLM) that does not use TimelyLLM's TUF-based scheduler, and the TUF parameters are justified from external robotics and HCI sources.

full rationale

The derivation chain separates the scheduler's internal objective (Eq. 3, Eq. 4) from the externally implemented baseline. TimelyLLM's priority function uses TUF_i_0/TUF_i_1 with user-supplied ERT, alpha, beta, and profiling-based generation/execution estimates; the evaluation then computes the same Eq. 1 TUF from actual response times. This alignment would be circular only if the reported gain were a mathematical identity; it is not, because vLLM does not optimize Eq. 4 and can score arbitrarily badly under the chosen TUF. The TUF shapes are grounded in external references (HCI one-second preference [47]; drone obstacle-avoidance 200 ms reaction [9]) rather than fitted from the outcome. The self-citation to Typefly [10] is used for building the LRTrace dataset and defining the stream-execution baseline, not as a proof of TimelyLLM's claims; the headline comparison is to vLLM, an external open-source server. The paper explicitly admits a boundary of the mechanism in §7.2 for task type 1 (near-instant 'print' skills), which limits generality but is an honest scope limitation, not a circular step. The '1.97x' phrasing is an additive gain from a negative baseline utility, a reporting concern, not circularity. No step in the derivation reduces to its own inputs; score 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The evaluation depends on several hand-set parameters, especially the TUF shapes that define the utility metric. The theoretical justification relies on a monotonicity assumption and an unjustified independence assumption in the appendix proof. No new physical or conceptual entities are introduced; the system itself is a scheduling mechanism.

free parameters (6)
  • TUF parameters for normal tasks (ERT, beta, alpha) = ERT=1s, beta=1, alpha=-2
    Set from HCI literature [47] and a 1.5s cutoff; directly defines the utility metric the paper optimizes and evaluates.
  • TUF parameters for urgent tasks (ERT, beta, alpha) = ERT=200ms, beta=2, alpha=-6.67
    Set from drone obstacle avoidance literature [9] and a 0.5s cutoff; shapes the claimed utility gains.
  • Segment generation time budget = 90 ms
    Used in priority assignment (Eq. 4); profiled from the authors' setup.
  • Maximum token length per segment = 10 tokens
    Used for adaptive batching worst-case estimation.
  • Simulated network latency = 8 ms
    Added to responses in evaluation; based on local network tests.
  • Execution time model parameters = profiled skill durations (e.g., min, average)
    Used to estimate completion time for suspended generations; profiled on Tello drone and Neuromeka arm.
assumptions (4)
  • domain assumption Modern LLMs generate robot plans faster than robots execute them.
    Central insight stated in §1; if false, segmented generation cannot hide resumption latency.
  • domain assumption TUF is monotonically non-increasing in waiting time.
    Required for Theorem 1; the paper notes this holds in many practical scenarios [55].
  • ad hoc to paper Reducing one request's waiting time does not increase another request's waiting time.
    Used in Lemma 2 proof in Appendix A; unjustified on shared GPU resources.
  • domain assumption KV cache and tokenized output can be saved and restored without retokenization mismatches.
    Required by §4.1 context switching; supported by a 9.50ms transfer measurement, but not guaranteed for all models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applications." pith.science (2026). https://pith.science/paper/2YUD7FPX

@misc{pith2026241218695,
  author       = {Pith},
  title        = {Pith review of: TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YUD7FPX}},
  note         = {Machine review of arXiv:2412.18695}
}
read the original abstract

Large Language Models (LLMs) such as GPT-4 and Llama3 can already comprehend complex commands and process diverse tasks. This advancement facilitates their application in controlling drones and robots for various tasks. However, existing LLM serving systems typically employ a first-come, first-served (FCFS) batching mechanism, which fails to address the time-sensitive requirements of robotic applications. To address it, this paper proposes a new system named TimelyLLM serving multiple robotic agents with time-sensitive requests. TimelyLLM introduces novel mechanisms of segmented generation and scheduling that optimally leverage redundancy between robot plan generation and execution phases. We report an implementation of TimelyLLM on a widely-used LLM serving framework and evaluate it on a range of robotic applications. Our evaluation shows that TimelyLLM improves the time utility up to 1.97x, and reduces the overall waiting time by 84%.

Figures

Figures reproduced from arXiv: 2412.18695 by the authors.

Figure 1
Figure 1. presents a high-level system overview of Time￾lyLLM. A robotic agent submits a request, along with its time￾sensitive requirement in terms of the Expected Response Time or deadline (ERT in TUF), the tolerance level for miss￾ing deadlines (𝛼 in TUF), and the time-sensitive degree (𝛽 in TUF). We encapsulate these parameters in the TUF defined in §2.3 to form the basis of the time-sensitive requirement. TimelyLLM uses … view at source ↗
Figure 2
Figure 2. A Case Comparison of Normal Generation vs. Context￾aware Segmented Generation: Under normal generation mode, Ur￾gent Request 1 cannot be executed in time due to the blocking of Normal Request 0 generation. Content-aware segmented generation optimizes this by releasing resources upon completing Segment 0 of Request 0, thereby allowing the system to process Urgent Request 1. Once Request 1 is processed, the system res… view at source ↗
Figure 3
Figure 3. Three user-perceptible latencies for a robotic request: (i) Request response time 𝑊 (𝑠0): the time taken to perform the first action, which is also the waiting time of segment 0. (ii) Robot waiting time Í𝐾 𝑘=0𝑊 (𝑠𝑘 ),𝑊 (𝑠0) +𝑊 (𝑠1) in the figure: the cumula￾tive waiting time introduced by LLM planning, represented as the sum of waiting time for all segments. For normal LLM generation, this equals the request respons… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Robots used in data collection: We utilize a Ryze Tech Tello drone [49] and a Neuromeka Indy7 Pro robotic arm [33]. These platforms are used to profile the actual execution time of various robotic skills. for selecting tasks. Based on the collected time traces from 𝑇 𝑟…
Figure 5
Figure 5. Figure 5: End-to-end Performance under different levels of resource contention. (LW: Low Workloads, HW: High Workloads). Based on (a) and (d), TimelyLLM improves the utility significantly over the vLLM baseline on urgent requests. Additionally, the remaining figures demonstrate …
Figure 6
Figure 6. Figure 6: Time trace for Task 71-72 derived from WID 2. 7.2 Overall Performance of TimelyLLM We start by comparing the end-to-end performance of Time￾lyLLM with the vLLM baseline. For this evaluation, we mea￾sure and record the time utility, actual response time, and the waiting…
Figure 8
Figure 8. Figure 8: Performance of Time￾lyLLM with different scheduling policies, i.e., FCFS, EDF. fine-grained resource allocation and successfully allocates resources to tasks with higher time utility, thus resulting in better time utility for the whole task set ( [PITH_FULL_IMAGE:figu…
Figure 10
Figure 10. Figure 10: Performance for chatbot application with Llama3 and Phi3 models. tasks, thanks to its efficient prioritization approach based on the scaled utility density. 7.4 Scalability of TimelyLLM 7.4.1 Robot arm task [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 39 canonical work pages

  1. [1]

    https://github.c om/vllm-project/vllm, 2024

    vllm: Easy, fast, and cheap llm serving for everyone. https://github.c om/vllm-project/vllm, 2024

    work page 2024

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  3. [3]

    Infercept: Efficient intercept support for augmented large language model inference

    Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiy- ing Zhang. Infercept: Efficient intercept support for augmented large language model inference. In Forty-first International Conference on Machine Learning

    work page

  4. [4]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    Taming throughput-latency tradeoff in llm inference with sarathi-serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve. Proceedings of 18th USENIX Symposium on Operating Systems Design and Implementation, 2024, Santa Clara , 2024

    work page 2024

  6. [6]

    Utility accrual real-time scheduling under variable cost functions

    Umut Balli, Haisang Wu, Binoy Ravindran, Jonathan Stephen Ander- son, and E Douglas Jensen. Utility accrual real-time scheduling under variable cost functions. IEEE Transactions on Computers, 56(3):385–401, 2007

    work page 2007

  7. [7]

    An evaluation model for information distribution in multi-robot systems

    Michal Barciś and Hermann Hellwagner. An evaluation model for information distribution in multi-robot systems. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications Workshops (INFO- COM WKSHPS), pages 824–829. IEEE, 2019

    work page 2019

  8. [8]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

Show all 61 references
  1. [9]

    Drone detection using depth maps

    Adrian Carrio, Sai Vemprala, Andres Ripoll, Srikanth Saripalli, and Pascual Campoy. Drone detection using depth maps. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 1034–1037. IEEE, 2018

    2018

  2. [10]

    Type- fly: Flying drones with large language model

    Guojun Chen, Xiaojing Yu, Neiwen Ling, and Lin Zhong. Type- fly: Flying drones with large language model. arXiv preprint arXiv:2312.14950v2, 2024

    2024 arXiv

  3. [11]

    A scheduling algorithm for tasks described by time value function

    Ken Chen and Paul Muhlethaler. A scheduling algorithm for tasks described by time value function. Real-Time Systems, 10:293–312, 1996

    1996

  4. [12]

    Robots that can chat

    Boston Dynamics. Robots that can chat. https://bostondynamics.com /blog/robots-that-can-chat/, 2024

    2024

  5. [13]

    Number of parameters in gpt-4

    Exploding Topics. Number of parameters in gpt-4. https://explodingt opics.com/blog/gpt-parameters, 2024

    2024

  6. [14]

    Transformers: State-of-the-art machine learning for pytorch, tensorflow, and jax

    Hugging Face. Transformers: State-of-the-art machine learning for pytorch, tensorflow, and jax. https://huggingface.co/docs/transforme rs/index, 2024

    2024

  7. [15]

    Hierarchical neural story generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833, 2018

    2018 arXiv

  8. [16]

    Figure + openai allow speech-to-speech reasoning over learned behaviors

    Figure. Figure + openai allow speech-to-speech reasoning over learned behaviors. https://www.figure.ai/ai, 2024

    2024

  9. [17]

    Efficient llm scheduling by learning to rank

    Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. Efficient llm scheduling by learning to rank. arXiv preprint arXiv:2408.15792, 2024

    2024 arXiv

  10. [18]

    Prompt cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khan- delwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems , 6:325–338, 2024

    2024

  11. [19]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 13 Neiwen Ling, Guojun Chen, and Lin Zhong 2024

    arXiv

  12. [20]

    A time- driven scheduling model for real-time operating systems

    E Douglas Jensen, C Douglass Locke, and Hideyuki Tokuda. A time- driven scheduling model for real-time operating systems. In Rtss, volume 85, pages 112–122, 1985

    1985

  13. [21]

    Coedge: A co- operative edge system for distributed real-time deep learning tasks

    Zhehao Jiang, Neiwen Ling, Xuan Huang, Shuyao Shi, Chenhao Wu, Xiaoguang Zhao, Zhenyu Yan, and Guoliang Xing. Coedge: A co- operative edge system for distributed real-time deep learning tasks. In Proceedings of the 22nd International Conference on Information Processing in Sen...

    2023

  14. [22]

    𝑠3: Increasing gpu utilization during generative inference for higher throughput

    Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. 𝑠3: Increasing gpu utilization during generative inference for higher throughput. Advances in Neural Information Processing Systems , 36:18015–18027, 2023

    2023

  15. [23]

    An llm compiler for parallel function calling

    Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. An llm compiler for parallel function calling. arXiv preprint arXiv:2312.04511, 2023

    2023 arXiv

  16. [24]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  17. [25]

    Mobilegpt: Augment- ing llm with human-like app memory for mobile task automation

    Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y Ko, Sangeun Oh, and Insik Shin. Mobilegpt: Augment- ing llm with human-like app memory for mobile task automation. 2024

    2024

  18. [26]

    A utility accrual scheduling algorithm for real-time activities with mu- tual exclusion resource constraints

    Peng Li, Haisang Wu, Binoy Ravindran, and E Douglas Jensen. A utility accrual scheduling algorithm for real-time activities with mu- tual exclusion resource constraints. IEEE Transactions on Computers, 55(4):454–469, 2006

    2006

  19. [27]

    Reflect: Summarizing robot experiences for failure explanation and correction

    Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724, 2023

    2023 arXiv

  20. [28]

    An example real-time command, control, and battle management application for alpha

    David P Maynard, Samuel E Shipman, Raymond K Clark, J Duane Northcutt, Russell B Kegley, Betsy A Zimmerman, and Peter J Keleher. An example real-time command, control, and battle management application for alpha. Archons Project TR-88121, CMU, 1988

    1988

  21. [29]

    The llama 3 herd of models

    Meta. The llama 3 herd of models. https://ai.meta.com/research/publi cations/the-llama-3-herd-of-models/, 2024

    2024

  22. [30]

    Meta ai assistant built with llama 3

    Meta. Meta ai assistant built with llama 3. https://about.fb.com/news/ 2024/04/meta-ai-assistant-built-with-llama-3/, 2024

    2024

  23. [31]

    Introducing meta llama 3: The most capable openly available llm to date

    Meta AI. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/, 2024

    2024

  24. [32]

    Llama 2 70B: An MLPerf Inference Benchmark for Large Language Models

    MLCommons. Llama 2 70B: An MLPerf Inference Benchmark for Large Language Models. https://mlcommons.org/2024/03/mlperf- llama2-70b/, 2024

    2024

  25. [33]

    Neuromeka indy

    Neuromeka. Neuromeka indy. https://en.neuromeka.com/cobot-1-1, 2024

    2024

  26. [34]

    Tensorrt-llm

    NVIDIA. Tensorrt-llm. https://nvidia.github.io/TensorRT-LLM/over view.html, 2024

    2024

  27. [35]

    Exegpt: Constraint-aware resource scheduling for llm inference

    Hyungjun Oh, Kihong Kim, Jaemin Kim, Sungkyun Kim, Junyeol Lee, Du-seong Chang, and Jiwon Seo. Exegpt: Constraint-aware resource scheduling for llm inference. In Proceedings of the 29th ACM Interna- tional Conference on Architectural Support for Programming Languages and Opera...

    2024

  28. [36]

    One queue is all you need: Resolving head-of-line blocking in large language model serving

    Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Shengkun Cui, Chandra Narayanaswami, Zbigniew Kalbar- czyk, and Ravishankar Iyer. One queue is all you need: Resolving head-of-line blocking in large language model serving. arXiv preprint arXiv:2407.00047, 2024

    2024 arXiv

  29. [37]

    Managing delays in human- robot interaction

    Hannah Pelikan and Emily Hofstetter. Managing delays in human- robot interaction. ACM Transactions on Computer-Human Interaction , 30(4):1–42, 2023

    2023

  30. [38]

    Efficiently scaling transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems , 5:606–624, 2023

    2023

  31. [39]

    Efficient interactive llm serving with proxy model-based sequence length prediction

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T Kalbarczyk, Tamer Başar, and Ravishankar K Iyer. Efficient interactive llm serving with proxy model-based sequence length prediction. arXiv preprint arXiv:2404.08509, 2024

    2024 arXiv

  32. [40]

    Sayplan: Grounding large language models using 3d scene graphs for scalable task planning.arXiv preprint arXiv:2307.06135, 2023

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning.arXiv preprint arXiv:2307.06135, 2023

    2023 arXiv

  33. [41]

    Ravindran, E.D

    B. Ravindran, E.D. Jensen, and P. Li. On recent advances in time/utility function real-time scheduling and resource management. In Eighth IEEE International Symposium on Object-Oriented Real-Time Distributed Computing (ISORC’05), 2005

    2005

  34. [42]

    Robots that ask for help: Uncertainty alignment for large language model planners

    Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928, 2023

    2023 arXiv

  35. [43]

    Average reading speed

    Scholar Within. Average reading speed. https://scholarwithin.com/av erage-reading-speed#, 2024

    2024

  36. [44]

    Revenue-driven scheduling in drone delivery networks with time-sensitive service level agreements

    Shireen Seakhoa-King, Paul Balaji, Nicolas Trama Alvarez, and William J Knottenbelt. Revenue-driven scheduling in drone delivery networks with time-sensitive service level agreements. In Proceed- ings of the 12th EAI international conference on performance evaluation methodolo...

    2019

  37. [45]

    Don’t stop me now: Embedding based scheduling for llms

    Rana Shahout, Eran Malach, Chunwei Liu, Weifan Jiang, Minlan Yu, and Michael Mitzenmacher. Don’t stop me now: Embedding based scheduling for llms. arXiv preprint arXiv:2410.01035, 2024

    2024 arXiv

  38. [46]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024

    2024

  39. [47]

    Response time and display rate in human perfor- mance with computers

    Ben Shneiderman. Response time and display rate in human perfor- mance with computers. ACM Computing Surveys (CSUR), 16(3):265– 285, 1984

    1984

  40. [48]

    Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting

    Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv pr...

    2023 arXiv

  41. [49]

    Tello sdk user guide, 2023

    Tello. Tello sdk user guide, 2023

    2023

  42. [50]

    Tidwell, R

    T. Tidwell, R. Glaubius, W. D. Smart, and C. D. Gill. Optimizing expected time utility in cyber-physical systems schedulers. In 2010 IEEE 31st Real-Time Systems Symposium (RTSS 2010) , 2010

    2010

  43. [51]

    Optimizing expected time utility in cyber-physical systems schedulers

    Terry Tidwell, Robert Glaubius, Christopher D Gill, and William D Smart. Optimizing expected time utility in cyber-physical systems schedulers. In 2010 31st IEEE Real-Time Systems Symposium , pages 193–201. IEEE, 2010

    2010

  44. [52]

    Chatgpt for robotics: Design principles and model abilities

    Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. IEEE Access, 2024

    2024

  45. [53]

    Llmˆ 3: Large language model-based task and motion planning with motion failure reasoning

    Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song- Chun Zhu, and Hangxin Liu. Llmˆ 3: Large language model-based task and motion planning with motion failure reasoning. arXiv preprint arXiv:2403.11552, 2024

    2024 arXiv

  46. [54]

    Autodroid: Llm-powered task automation in android

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and 14 TimelyLLM: S...

    2024

  47. [55]

    Time-utility function — Wikipedia, 2024

    Wikipedia contributors. Time-utility function — Wikipedia, 2024

    2024

  48. [56]

    Fast distributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023

    2023 arXiv

  49. [57]

    Utility accrual scheduling under arbitrary time/utility functions and multi-unit resource constraints

    Haisang Wu, Binoy Ravindran, E Douglas Jensen, and Umut Balli. Utility accrual scheduling under arbitrary time/utility functions and multi-unit resource constraints. Proc. of the 10th Real-Time and Em- bedded Computing Systems and Applications , page 1, 2004

    2004

  50. [58]

    Orca: A distributed serving system for {Transformer-Based} generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , pages 521–538, 2022

    2022

  51. [59]

    {SHEPHERD}: Serving{DNNs} in the wild

    Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. {SHEPHERD}: Serving{DNNs} in the wild. In 20th USENIX Sym- posium on Networked Systems Design and Implementation (NSDI 23) , pages 787–808, 2023

    2023

  52. [60]

    Bootstrap your own skills: Learning to solve new tasks with large language model guidance

    Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. arXiv preprint arXiv:2310.10021, 2023

    arXiv

  53. [61]

    Response length perception and sequence scheduling: An llm-empowered llm inference pipeline.Advances in Neural Information Processing Systems, 36, 2024

    Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. Response length perception and sequence scheduling: An llm-empowered llm inference pipeline.Advances in Neural Information Processing Systems, 36, 2024. APPENDIX A PROOF OF PROBLEM EQUIV ALENCE We refor...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.