{"id":"a00ff521-003d-42ba-b195-ab0a82bd712b","arxiv_id":"2412.18695","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TimelyLLM segments LLM-generated robot plans into executable pieces and schedules those pieces by urgency, reducing response delays for time-critical robot tasks.","lead":"This paper describes TimelyLLM, a system that pauses and resumes LLM text generation mid-plan so that urgent robot commands can jump the queue. It reports up to 1.97x better time utility and 84% less waiting time than a standard LLM server in tests with drones and robot arms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Segmented generation's benefit hinges on segment execution time exceeding generation plus resume overhead; for fast skills like 'print' the paper admits added waiting, so the headline gain is not established for general robotic workloads.","rationale":"After reading the full text, I believe the reader's verdict is on the right track. The single most load-bearing concern is the paper's reliance on the assumption that robot execution of a segment will exceed the LLM generation time for the next segment plus context-switch overhead. This assumption is the basis for segmented generation (§1, §4.3), and the paper's own observation about 'print' actions in §7.2 shows a real regime where it fails. Because the evaluation aggregates across tasks and only demonstrates the benefit on robot-arm workloads with long-duration skills, the headline claim is overgeneralized. The negative-baseline issue with the 1.97x number (from -0.97 to 1.00) is a reporting problem, but even a correct additive statement would not remove the need to scope the claim by skill duration. The Appendix A proof gap is real but less central: the scheduler is a greedy PUD heuristic, not the optimal solution of Eq. 3, so the Pareto result is not what carries the empirical claim. I would keep the reader's CONDITIONAL verdict and add a requested characterization of the break-even condition.","tokens_in":21233,"tokens_out":6218,"duration_ms":54845,"concrete_test":"Construct a workload from LRTrace traces in which the profiled execution time of each skill is less than the profiled segment generation time (≈90 ms) plus resume overhead (≈10–20 ms, Table 4), e.g., replace longer drone/arm moves with 'print' and short rotations. Run WID1 or WID2 under both TimelyLLM and vLLM and compare aggregate TUF and waiting time. If TimelyLLM fails to beat vLLM (or is worse) on this fast-skill workload, the claimed improvement is conditional on skill duration and should be scoped explicitly. As a cheaper analytical check, compute for every segment in the original traces the margin m = execution_time − (generation_time_of_next_segment + context_switch_overhead); if substantial mass has m < 0, the mechanism cannot deliver the advertised gains in those regimes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—up to 1.97x utility and 84% waiting-time reduction over vLLM—is conditional on a hidden workload property: the execution time of a generated segment must be long enough to hide the generation time of the next segment plus context-switch overhead. This is the key insight in §1 and is formalized in §4.3, where resume priority is derived from an estimated execution-time model. The paper's own evaluation exposes the boundary: §7.2 reports that task type 1 uses near-instant 'print' skills (~1 ms execution), and TimelyLLM's waiting time increases there because generation (≈90 ms profiled in §7.1) cannot be hidden. Yet the headline numbers come from WID3 robot-arm tasks (Desk Clean) where pick/place operations take substantial time, not from a representative mixture of robot skills. No analysis quantifies the fraction of segments for which the execution time actually exceeds generation latency plus overhead; without that, the benefit is not established for general multi-agent robotic workloads. The reported 1.97x is also an additive gain from a negative baseline utility (−0.97→1.00), so the fold wording is misleading, but the deeper issue is the unstated domain restriction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TimelyLLM, an LLM serving system for time-sensitive robotic requests. TimelyLLM splits LLM-generated robot plans into executable segments, suspends generation after each segment, and schedules suspended generations using a TUF-based priority mechanism that estimates both expected time-utility gain and the robot's execution time for the generated segment. The system is implemented on top of vLLM with a custom stop checker and context caching. The authors build LRTrace, a trace collection system using a Tello drone and a Neuromeka robot arm, and evaluate TimelyLLM against vLLM and a streaming variant on drone, robot-arm, and chatbot workloads. The headline results are an up-to-1.97x improvement in time utility and an 84% reduction in waiting time over vLLM.","tokens_in":21501,"tokens_out":4016,"duration_ms":36215,"significance":"If the central claim holds, TimelyLLM is a useful contribution to LLM serving for robotics: it identifies a real redundancy between plan generation and robot execution, implements it with modest overhead, and evaluates against an external, widely used baseline (vLLM) rather than only against self-constructed variants. The paper also contributes LRTrace, a first attempt at building real-world multi-robot LLM serving traces, and reports overhead numbers for segmentation and scheduling. The idea of semantically grounded segmentation with resumption from cached KV states is plausible and could benefit other streaming-consumer applications. However, the current manuscript does not establish the headline gains for general robotic workloads because the evaluation is single-run and the '1.97x' figure is an additive utility gain computed from a negative baseline, while the core benefit depends on an unquantified domain condition: segment execution time must exceed generation time plus resume overhead.","major_comments":[{"comment":"The headline '1.97x' improvement in time utility is not a ratio. For the Desk Clean task, vLLM achieves utility -0.97 and TimelyLLM achieves 1.00, so the difference is 1.97, but the ratio is negative and would not be reported as a fold improvement. The abstract and conclusion repeat the '1.97x' wording. Please report the additive utility gain or use a non-negative utility transformation, and remove ratio language for this comparison.","section":"§7.4.1, Figure 9"},{"comment":"All evaluation results are point estimates with no error bars, confidence intervals, or number of runs, even though the workloads are stochastic: §6 specifies Poisson event arrivals, and §6 also says execution times are randomly sampled from profiled data. Claims such as '191% improvement' (§7.2), '183% and 142%' (§7.3.2), and '84% waiting-time reduction' (§7.4.1) are therefore not statistically supported. Please report multiple seeds or replicates with variance, or at least a sensitivity analysis showing that the conclusions are stable.","section":"§7.1–§7.4"},{"comment":"The core benefit of segmented generation depends on the assumption that the execution time of a generated segment exceeds the time to generate the next segment plus context-switch and resumption overhead. The paper itself acknowledges in §7.2 that task type 1, whose 'print' skill executes in about 1 ms, incurs increased waiting time because generation cannot be hidden. Yet the headline gains come from WID3 robot-arm tasks with long-duration manipulation skills. No analysis quantifies the fraction of segments for which the execution time actually exceeds the generation-plus-resume cost, and no sensitivity experiment varies this ratio. Without that, the claim that TimelyLLM improves time utility for general multi-agent robotic workloads is not established.","section":"§4.2, §4.3, §7.2"},{"comment":"The proof of Theorem 1 has a logical gap. Lemma 2 assumes there exists another solution x' that improves one request's completion time 'without impacting the total time utility of the first segments' and asserts that 'no other requests need to be negatively impacted by this change'. This is essentially the Pareto statement to be proven, not a consequence of optimality of Eq. 3. Similarly, Lemma 3 assumes a first-segment utility increase with no negative impact elsewhere. Furthermore, monotonic non-increasing TUFs do not imply that a strict waiting-time reduction yields a strict utility increase; for TUF1 defined in §4.2 with ERT=0, the function is flat for non-positive waiting times. The proof should be restructured or the theorem stated with the additional non-interference assumption made explicit.","section":"Appendix A, Lemma 2 and Lemma 3"}],"minor_comments":[{"comment":"The x-axis labels task types 0–7, but the workloads in Table 3 use trace IDs 1–8; the off-by-one indexing should be clarified in the caption or the axis.","section":"Figure 5"},{"comment":"The abstract says existing systems employ 'FCFS batching', but §2.1 correctly notes that vLLM and similar systems use continuous batching; please reconcile this terminology.","section":"§1, §2.1"},{"comment":"The slack time L(s_i_k) is used as a scaling factor but its computation is not defined. Please specify how L is calculated for initial and suspended generations.","section":"§4.3, Eq. (4)"},{"comment":"Table 4 reports overhead as a percentage of segmented generation time, but the text does not state the number of runs or the variance across runs; adding this would strengthen the overhead claim.","section":"§7.5.1"},{"comment":"No artifact or repository link is provided for TimelyLLM or LRTrace; including one would support reproducibility and allow the community to verify the reported traces.","section":"§5, §6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant problem, and the basic idea is sound enough to warrant a major revision rather than rejection. The main risks are (i) the misleading ratio-based presentation of a negative-baseline utility gain, (ii) lack of statistical validation for stochastic workloads, (iii) an unquantified domain restriction on the benefit of segmentation, and (iv) a non-rigorous optimality proof in the appendix. I would not recommend acceptance without these being addressed. I also note that the paper would benefit from a clear statement of the non-interference assumption in the scheduling formulation, since it is central to both the proof and the practical validity of the Pareto claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good to talk. I've read TimelyLLM. Short version: this is a legitimate systems paper with a real idea, a working implementation, and an honest set of measurements, but the headline utility number is presented in a way that will mislead a casual reader, and the proof in the appendix does not quite close.\n\nWhat is actually new: the combination of content-aware segmented generation with TUF/PUD scheduling for multi-robot serving. The individual pieces exist in streaming execution and utility accrual scheduling, but the integration is new, and the evaluation against vLLM and a streaming variant is reasonable. The overhead measurements are a plus: segmented generation costs under 6% of generation time, and scheduling adds about 2 ms for eight tasks. The authors even point out where the approach fails, the 'print' skill in task type 1, where execution is around 1 ms and segmentation adds waiting time. That honesty suggests they understand the mechanism's boundary.\n\nSoft spots, in order of seriousness.\n\nFirst, the '1.97x' in the abstract is misleading. In the Desk Clean case, vLLM's utility is -0.97 and TimelyLLM's is 1.00. That is an additive gain of 1.97 utility units, not a multiplicative one. Reporting it with a multiplication sign invites a comparison that is not meaningful when the baseline is negative. The body text mostly says 'increases utility by 1.97', which is fine, but the abstract should be fixed.\n\nSecond, no error bars. The workloads sample from Poisson arrivals and profiled execution times, so there is stochastic variance. Point estimates without intervals make it hard to judge whether the differences are robust.\n\nThird, the Appendix A proof has a gap. It assumes, without argument, that an alternative schedule improving one request's completion time can leave all other requests unaffected. In a shared server, that is not generally true. The theorem might still be true, but the proof needs a real argument.\n\nFourth, the stress-test note is right: the benefit rests on segment execution time exceeding generation time plus resume overhead. The paper shows this holds for the drone and robot-arm tasks with slow actions and fails for fast print-like skills. The abstract should qualify the claim to workloads with substantial execution times. That does not undermine the core idea; it just means the scope is narrower than the title suggests.\n\nWho this is for: someone working on LLM serving with timing constraints, or embodied AI systems that put LLM inference behind multiple robots. It deserves a serious referee. I would send it out. For my own work, I probably would not cite it in the next year, but it is worth a reading group discussion.","headline":"Legitimate segmented-generation system with an honest evaluation, but the headline utility ratio is misleading and the appendix proof has a gap.","tokens_in":22010,"tokens_out":5163,"would_cite":false,"duration_ms":44064,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TimelyLLM claims LLM serving can meet many robots' deadlines by pausing plan generation at each executable action and scheduling by time-utility functions, gaining up to 1.97× utility and 84% less waiting time than vLLM.","keywords":["large language model serving","time-sensitive scheduling","segmented generation","time-utility function","robotic task planning","continuous batching","real-time systems","edge robotics"],"falsifier":"Run the workload on a task set whose robot actions all execute in less time than the per-segment generation time plus context-switch overhead, as the paper's own 'print' skill already does at about 1 ms. If TimelyLLM's aggregate time utility still beats vLLM's FCFS batching on that workload, then execution-time redundancy is not the operative mechanism; if it falls below, the central claim depends entirely on every skill taking longer to execute than to plan, which is exactly the premise the system is built on.","tokens_in":21003,"feed_emoji":"🤖","tokens_out":13515,"duration_ms":108888,"temperature":0.7,"pith_summary":"TimelyLLM argues that an LLM service can satisfy many robots with time-sensitive demands at once if generation is no longer treated as one indivisible response. The key observation is an asymmetry: a drone needs seconds to act on a command such as 'move forward', while the LLM produces those tokens in tens of milliseconds, so a server can pause a plan the moment a runnable action appears, lend the GPU to a more urgent request, and still resume before the robot finishes. Scheduling is driven by each request's time-utility function, a user-supplied deadline with utility and tolerance, so urgent tasks such as collision avoidance are favored without starving normal planning. On workloads built from real Tello drone and Neuromeka robot-arm traces, the paper reports up to 1.97× the time utility and 84% less waiting time than the vLLM baseline.","feed_headline":"1.97x time utility: segmented LLM serving for time-critical robots","feed_subtitle":"Pausing generation while robots execute lets urgent requests jump the queue, beating vLLM by up to 1.97x.","key_machinery":"The load-bearing mechanism is segmented generation: decoding stops at the first executable skill via a regular-expression stop checker, and the suspended context (KV cache plus tokenized output, about 170 MB for Llama3-8B) is stored so resuming costs roughly 9.5 ms of host-to-GPU transfer instead of roughly 133 ms of re-prefilling. On top of it sits the PUD-based priority rule, which scores each initial or suspended generation by its time-utility gain per unit of generation time divided by slack, and for suspended generations sets the expected response time to zero so that resumption urgency is tied to the robot's estimated execution of the preceding segment, profiled per skill from real robot measurements. The third piece is latency-guided batch-size selection, which estimates a segment's worst-case completion from its remaining token budget and recent generation speed and admits new generations only if the most urgent one still meets its deadline. These three pieces convert the time redundancy between plan generation and robot execution into a resource pool the scheduler can spend on urgent requests.","core_discovery":"The paper's central claim is that time-sensitive LLM serving for robots becomes a tractable scheduling problem once generation is segmented at executable-skill boundaries. TimelyLLM's content-aware stop checker watches the token stream, and the moment a complete robot skill such as 'pick(cyan_box)' appears it suspends decoding, caches the KV cache and tokenized output so that resumption needs no prefill, and hands the segment to the robot for immediate execution. While the robot acts, the scheduler ranks every initial and suspended generation by potential utility density, computed from the request's time-utility function and, for suspended generations, the estimated execution time of the segment the robot is currently performing, while a latency-guided batch-size selector caps the batch so the most urgent generation stays within its remaining time budget. The paper proves that maximizing the sum of segment-level utilities is Pareto optimal with respect to completion time and first-segment utility for monotonically non-increasing TUFs, and reports end-to-end gains of 1.97× in time utility and 84% less waiting time on robot-arm workloads, with per-segment overheads under 6% of generation time.","pith_inferences":["The claimed gains scale with the gap between generation speed and execution speed: on slower GPUs, larger models, or fatter prompts, per-segment generation approaches the robot's action duration and the 1.97× advantage would shrink toward ordinary priority queuing, a regime the paper does not measure.","The pattern generalizes beyond robots to any LLM consumer with a predictable consumption time, such as code interpreters, UI-automation agents, or speech synthesis, each of which could subscribe a time-utility function; the paper only evaluates robot execution and simulated reading, so this is an extrapolation.","The execution-time model is the softest transfer point: it is profiled on exactly two robot platforms and deliberately takes minimum durations for variable skills to stay worst-case safe, so an online learned estimator replacing the offline profile would be a direct test of whether that conservatism is what preserves utility for normal tasks.","Frequent suspension raises the chance that a long normal plan is repeatedly pushed back by a stream of urgent arrivals, the same effect the paper observes for the fast 'print' skill in task type 1; a starvation bound or aging term in the priority function would be a natural follow-up."],"forward_implications":["A single GPU serving many robots can meet mixed deadlines that FCFS batching misses: urgent tasks with a 200 ms expected response time keep near-full utility while normal planning tasks with a 1 s deadline stay at maximum, where the vLLM baseline lets urgent utility collapse to about a fifth of its maximum under load.","The longer the generated plan, the larger the gain, because more segments create more preemption opportunities; the robot-arm workloads with ~100-token plans deliver the headline 1.97× utility and 84% waiting-time reduction.","Segmented scheduling dominates pure priority policies and plain streaming: on urgent tasks it beats FCFS by 183% and EDF by 142% in utility without sacrificing normal tasks, and beats streaming execution (vLLM-stream) because suspension frees resources for preemption.","The overheads stay small enough for edge deployment, with segmentation adding under 6% of generation time per trace and scheduling costing about 2 ms for eight queued tasks, so the utility gains are not consumed by the mechanism itself.","The same redundancy idea transfers to chatbot consumption by segmenting at sentence or paragraph boundaries and treating reading time as execution time, cutting delays by up to 96% on the Phi-3 model."],"supporting_citations":[{"why":"The vLLM/PagedAttention continuous-batching engine that TimelyLLM is implemented on and the primary baseline it must beat.","marker":"[24]"},{"why":"Typefly supplies the LLM-powered drone application whose streaming execution insight TimelyLLM extends and whose task set feeds the LRTrace dataset.","marker":"[10]"},{"why":"Introduces the time-utility function model that defines TimelyLLM's scheduling objective and the TUF parameters each request carries.","marker":"[20]"},{"why":"Provides the one-second user response-time preference used to set the expected response time for normal robot tasks.","marker":"[47]"},{"why":"Provides the 200 ms drone reaction-time requirement used to set the expected response time for urgent tasks.","marker":"[9]"},{"why":"Supplies the robot skill syntax (e.g., 'pick([obj],{})') the content-aware stop checker matches to decide when a segment is executable.","marker":"[53]"},{"why":"The real-world command-and-control time-utility example from which the paper's generalized TUF equation is extended.","marker":"[28]"},{"why":"Llama 3 is the model used to generate the plans in all robot-control evaluations, so the reported latency and overhead numbers depend on it.","marker":"[29]"},{"why":"The Tello drone used to profile per-skill execution times that feed the execution-time model behind suspended-generation priority.","marker":"[49]"},{"why":"The Neuromeka robot arm used to profile the longer complex-skill execution times for the WID3 evaluations.","marker":"[33]"}],"fun_headline_variants":["Segmented LLM serving boosts robot response by 1.97x","Time-aware scheduling for robots: 84% less waiting, 1.97x utility","Pause generation, let robots act: TimelyLLM's segmented scheduling","TimelyLLM: 1.97x utility via segmented LLM generation","Robot-focused LLM serving: segmented generation cuts waiting 84%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benefit rests on the assumption that executing a generated robot action takes longer than generating the next plan segment plus switching context, so that robot execution hides the planning of what comes next; for near-instant actions like printing a message, segmentation only adds waiting time, which the paper itself observes in task type 1.","fun_headline_variants_meta":{"raw":{"variants":["Segmented LLM serving boosts robot response by 1.97x","Time-aware scheduling for robots: 84% less waiting, 1.97x utility","Pause generation, let robots act: TimelyLLM's segmented scheduling","TimelyLLM: 1.97x utility via segmented LLM generation","Robot-focused LLM serving: segmented generation cuts waiting 84%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000691,"raw_usage":{"total_tokens":3125,"prompt_tokens":940,"completion_tokens":2185,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":2083}},"tokens_in":556,"tokens_out":2185,"duration_ms":14171,"temperature":1.0,"reasoning_tokens":2083,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:32:54.187305+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the workload on a task set whose robot actions all execute in less time than the per-segment generation time plus context-switch overhead, as the paper's own 'print' skill already does at about 1 ms. If TimelyLLM's aggregate time utility still beats vLLM's FCFS batching on that workload, then execution-time redundancy is not the operative mechanism; if it falls below, the central claim depends entirely on every skill taking longer to execute than to plan, which is exactly the premise the system is built on.","supporting_citations":[{"cited_title":"A time- driven scheduling model for real-time operating systems","cited_arxiv_id":null,"evidence_quote":"Introduces the time-utility function model that defines TimelyLLM's scheduling objective and the TUF parameters each request carries."},{"cited_title":"Response time and display rate in human perfor- mance with computers","cited_arxiv_id":null,"evidence_quote":"Provides the one-second user response-time preference used to set the expected response time for normal robot tasks."},{"cited_title":"Drone detection using depth maps","cited_arxiv_id":null,"evidence_quote":"Provides the 200 ms drone reaction-time requirement used to set the expected response time for urgent tasks."},{"cited_title":"An example real-time command, control, and battle management application for alpha","cited_arxiv_id":null,"evidence_quote":"The real-world command-and-control time-utility example from which the paper's generalized TUF equation is extended."},{"cited_title":"The llama 3 herd of models","cited_arxiv_id":null,"evidence_quote":"Llama 3 is the model used to generate the plans in all robot-control evaluations, so the reported latency and overhead numbers depend on it."},{"cited_title":"Tello sdk user guide, 2023","cited_arxiv_id":null,"evidence_quote":"The Tello drone used to profile per-skill execution times that feed the execution-time model behind suspended-generation priority."},{"cited_title":"Neuromeka indy","cited_arxiv_id":null,"evidence_quote":"The Neuromeka robot arm used to profile the longer complex-skill execution times for the WID3 evaluations."}],"review_version":1}