Pith. sign in

REVIEW 6 major objections 4 minor 48 references

Production coding agents are session-structured loops of LLM and tool calls, making the KV cache highly reusable inside a turn but fragile at turn boundaries, model switches, and compaction.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:16 UTC pith:JFJD7BZV

load-bearing objection Large, honestly presented production trace study of coding agents; the numbers are useful, but the cache-lifecycle claims rest on client telemetry the paper itself admits it cannot validate from the server side. the 6 major comments →

arxiv 2608.00101 v1 pith:JFJD7BZV submitted 2026-07-30 cs.AI cs.LG

Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale

Banruo Liu , Haoran Qiu , \'I\~nigo Goiri , Rodrigo Fonseca , Ricardo Bianchini , Esha Choukse This is my paper
classification cs.AI cs.LG
keywords agentic codingproduction workload characterizationKV cacheprefix cachingLLM servingcontext compactionidle-time predictionuser archetypes
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper works from a week of sampled production traces (13.5M sessions, 3.2M users, 761M LLM calls, 95T tokens) to establish that AI coding agents behave structurally differently from chatbots: a user turn unfolds into an autonomous, mostly serial loop of LLM calls and tool calls coupled about 1:1, with 87% of LLM calls agent-initiated. This structure makes prompt-prefix caching highly effective within a turn, with median hit rates near 98% and averages around 90%, but predictable events break it: turn boundaries cut the average to 55%, model switches to 8%, and context compaction, which touches 7.8% of sessions but 44% of tokens, drops cache hit rates by a median 66%. Tool failures in 9% of turns trigger retry loops that amplify compute up to 4x, and user archetypes span a 50x range in per-turn token consumption. The paper argues these quantified patterns are the empirical basis for redesigning serving systems around sessions and turn boundaries rather than independent requests.

Core claim

The central claim is that the unit of work in coding-agent serving is the session-structured turn, not the individual LLM request. Within a turn the prompt grows monotonically as history and tool outputs accumulate, so each successive call reuses the previous prefix; average KV-cache hit rates reach ~90%, with a predictable ramp from 45% on the first call to 92–94% by the third. At turn boundaries, inter-turn idle triggers time-based eviction, dropping the average to 55%; model switching (in ~6.4% of sessions, usually reactive to errors or rate limits) leaves only 8%. Context compaction rewrites the prompt so aggressively (median 72.8% token drop, 66.1% hit-rate drop) that it cold-starts the

What carries the argument

The central object is the user turn: one user prompt plus the agent's full autonomous response chain of LLM calls and tool executions. The paper's key mechanism is the KV-cache lifecycle over these turns. Because each LLM call in a turn extends a shared prompt prefix, prefix caching reuses almost all cached key-value tensors within a turn; because turn boundaries insert minutes of user idle time, time-based eviction, model switches, and context compaction invalidate that prefix. The turn boundary, a clearly identifiable event in the telemetry, is the load-bearing structural marker that the paper uses to separate short, reclaimable idle periods from long, reclaimable ones, and it is also the

Load-bearing premise

The headline cache-lifecycle numbers come from client-reported 'cached prompt tokens', not from measuring the server's actual KV-cache state; if those counts overstate real cache reuse, the 90/55/8% rates and the systems conclusions built on them would shift.

What would settle it

Compare client-reported cached-prompt-token counts with server-side KV-cache hit/miss logs on a sample of the same sessions. If the measured intra-turn hit rate is not near 90%, the turn-boundary drop is not near 55%, or caches are not actually evicted in the 2–10 minute idle window, the central lifecycle claim is unsupported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • KV-cache management becomes a session-level scheduling concern: within a turn the cache should stay pinned to GPU memory, while at turn boundaries it should become a candidate for offloading to DRAM or disk based on predicted idle duration.
  • Capacity planning and admission control must model autonomous agent chains, since 87% of LLM calls are agent-initiated and user-request arrivals alone do not predict load.
  • Model pinning and proactive cache staging on the target model can avoid the near-total cache cold-start (8% hit rate) observed after model switches.
  • Context compaction should be treated as a first-class cache event; incremental, prefix-preserving compaction could cut the observed 66% median cache-hit-rate drop.
  • Uniform eviction and retention policies are suboptimal across a 50x range of per-turn token consumption; tiered, archetype-aware SLOs would reduce tail cost for the heaviest users without wasting memory on light users.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not run this experiment, but its 2–10 minute eviction cliff implies a cheap test: a keep-alive prefill just before the retention deadline could preserve cache for sessions the predictor says will return, converting eviction into a controlled decision.
  • The paper measures that failed builds inject 7–8x more prompt tokens at the median; a natural extension is to test whether streaming or summarizing tool output before it enters the prompt reduces context growth and cache pressure without changing agent behavior.
  • Because the traces come from one product over one week, the generality of the 1:1 coupling and the 90/55/8% lifecycle is best tested by applying the same turn-level analysis to traces from other coding agents or later time windows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. This paper presents a large-scale workload characterization of GitHub Copilot's coding agent using one week of anonymized telemetry from June 2026 (13.5M sessions, 3.2M users, 760.5M LLM calls). The authors document a tightly coupled LLM-tool execution loop, high agent autonomy, heavy-tailed session/turn distributions, session-structured KV-cache reuse, context compaction events, tool failure amplification, user archetypes, and resource idle-time patterns. They also design a lightweight idle-time predictor at turn boundaries and report that it captures 86–90% of total idle time. The stated contributions are empirical: a production-scale description of agentic coding and systems implications for KV-cache management, scheduling, and container reclamation.

Significance. If the quantitative findings hold, this is a valuable and timely dataset for the systems community. The scale is unprecedented for coding-agent traces, and the paper makes concrete, falsifiable claims about turn-boundary cache degradation, model-switch invalidation, compaction cost, and bimodal idle times. The paper also ships a reproducible prediction experiment with a held-out evaluation (subject to the data-window issue noted below). The main risk is that the headline cache-lifecycle numbers are derived from client-reported cached-token counts without server-side validation, which the paper itself acknowledges in Section 11. The systems implications are plausible but conditional on that measurement. With validation or appropriately softened claims, this would be an influential reference point for agent-native serving systems.

major comments (6)
  1. [§3.1, §5.2, Table 4, Fig. 10, §11] The central cache-lifecycle claims (90% intra-turn, 55% turn-boundary, 8% after model switch, 66.1% median drop after compaction) are computed from client-reported cached-token counts in telemetry, but the paper never states the exact definition of 'cache hit rate' or validates the field against server-side KV-cache state. Section 11 explicitly notes 'No server-side view.' If the client counts cacheable-but-missed prefixes, or aggregates multiple candidate prefix matches, all subsequent systems implications shift. Please either provide server-side validation for a subset of sessions, or re-label these as client-reported prefix reuse and remove the eviction-policy and serving-system language from the key findings.
  2. [§5.3, Fig. 15] The sharp drop in cache hit rate between 2 and 10 minutes of idle time is attributed to 'a time-based KV cache eviction policy at the serving system.' This is an inference from client-side telemetry; no server-side eviction events are observed. The same pattern could be produced by client-side timeout accounting, dynamic prompt content, tokenizer changes, or user behavior. The paper should label this as a hypothesis or validate with server logs; as written, the causal claim overstates the evidence.
  3. [§3.1 vs §9.2] Section 3.1 states the dataset covers one week in June 2026, but Section 9.2 says the idle-time predictor is trained on 150K sessions from 'a one-week window' and evaluated on 50K sessions from 'the following week.' These statements are inconsistent unless the dataset actually spans two weeks. Because the 86–90% captured-idle-time claim depends on a genuine temporal holdout, the authors must clarify the data window or rerun the evaluation with a holdout within the available week.
  4. [§9.1, Table 8 vs Fig. 32a] The idle-time statistics are internally inconsistent. Table 8 reports intra-turn P50 container idle = 5.8s and KV-cache idle = 1.2s, while Figure 32a shows overall medians of 6.8s and 5.0s. Since the text says more than 90% of idle intervals are intra-turn, the combined median should be dominated by the intra-turn values. Additionally, cross-turn KV-cache idle median (172s) is an order of magnitude smaller than the reported user idle median (1,512s), despite cross-turn idle being defined to include the user wait between turns. Please reconcile the definitions and ensure the reported numbers describe the same populations.
  5. [§3.1, Table 3] The sampling methodology is under-specified. The paper says 'sampled subset' but gives no sampling rate, method, or inclusion criteria, and notes the traces are US-only (three timezones). With such skewed and heavy-tailed distributions, small selection biases can change percentages such as 87% agent-initiated, 7.8% compaction, or archetype shares. Please provide sampling details, discuss selection bias, and state which numbers are computed on the full population versus the sampled subset, since Section 3.1 says aggregate metrics are computed 'except when computing aggregate metrics.'
  6. [§9.2, Fig. 33b] The metric 'Captured idle time' is central to the predictor claim (86–90%) but is never formally defined. Please specify exactly how it is computed (e.g., dwell time in sessions whose predicted survival crosses a threshold before actual idle ends, divided by total idle time), the operating-point selection rule, and how the result relates to the reported ROC-AUC and F1. Without this definition, the gap between decaying accuracy and stable captured idle time cannot be interpreted.
minor comments (4)
  1. [Abstract vs Table 3] The abstract says '95T tokens', but Table 3 reports 44.9T prompt tokens and 39.3B completion tokens, which totals roughly 45T. Please correct the headline token count.
  2. [§4.2] The text says 'the median turn triggers 3 LLM calls and 3 tool invocations, consuming 160.2K prompt tokens and 265 completion tokens,' which conflicts with Table 4 and Figure 4e (median per-turn LLM calls 4.5, prompt tokens 227.6K, completion tokens 1.9K). Please reconcile.
  3. [§1] Typo: 'reclaimation' should be 'reclamation' in the Introduction.
  4. [§9.2] The conditional survival formula S(t|idle>t0)=S(t)/S(t0) requires S(t0)>0; the paper should state how boundary cases are handled in practice.

Circularity Check

0 steps flagged

No significant circularity: the paper's findings are direct telemetry measurements, and the learned idle-time predictor is evaluated on held-out data.

full rationale

This is a workload characterization paper, not a derivation. The central quantitative claims (KV cache hit rates by call position, turn-boundary degradation, model-switch invalidation, compaction impact) are computed from telemetry fields such as 'cached prompt tokens' and timestamps, as described in Sections 3.1, 5.2, and 6. These are measurements, not quantities fitted to a conclusion. The one predictive component, the idle-time predictor, is trained on 150K sessions and evaluated on 50K sessions from a separate week (Section 9.2), with ROC-AUC and captured-idle metrics reported against baselines; 'captured idle time' is a held-out evaluation metric, not a restatement of a training objective or of a fitted constant. The paper's own limitation statement, 'No server-side view' (Section 11), identifies a validity concern about whether client-reported cached tokens reflect server-side KV-cache state, but this is a measurement-threat critique, not circularity: it does not show that any conclusion is equivalent by construction to its input. Self-citations appear (e.g., Refs. [8], [32], [31], [30]) but none is load-bearing for the headline findings; the workload statistics stand on the trace data, and related-work comparisons cite external benchmarks and prior characterizations. No step reduces an output to an input by definition, no fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported from prior work. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The characterization is a direct measurement from telemetry, so no fitted constants are needed for the headline statistics. However, the paper assumes telemetry accuracy, equivalence between client-side cached-token counts and server-side cache hits, representativeness of the US-only sample, and that the telemetry turn segmentation matches user intent. The idle-time predictor adds hand-chosen hyperparameters and operating thresholds, but these only affect the auxiliary prediction claim.

free parameters (2)
  • LightGBM quantile-regressor hyperparameters = 12 quantile targets; 400 trees per model
    Chosen for the idle-time predictor (Section 9.2); not part of the workload characterization, but affect the reported 86–90% captured idle time.
  • Idle-time prediction operating points = 30s to 30m evaluation windows; 60s binary threshold
    Hand-picked thresholds for evaluating accuracy/F1 and captured-idle-time (Section 9.2).
axioms (4)
  • domain assumption Telemetry metadata (timestamps, token counts, success/failure flags) is accurate and complete for sampled sessions.
    Section 3.1: all statistics are derived from this structural metadata; any misreporting or truncation propagates into every headline number.
  • domain assumption Client-reported 'cached prompt tokens' correspond to actual server-side KV-cache hits.
    Section 5.2 defines cache hit rate from cached-token counts; Section 11 acknowledges there is no server-side view to validate this.
  • domain assumption Sampled US-only traces from one week generalize to the broader coding-agent population.
    Section 3.1 notes all traces come from US regions across at most three timezones and are sampled; Section 4 phrases findings as general 'coding-agent' workload properties.
  • domain assumption The telemetry session/turn/step segmentation matches user-intent boundaries.
    Section 3.2 defines the hierarchy; turn-boundary cache degradation and the idle-time predictor depend on this segmentation.

pith-pipeline@v1.3.0-alltime-deepseek · 25153 in / 16055 out tokens · 150480 ms · 2026-08-04T01:16:54.125646+00:00 · methodology

0 comments
read the original abstract

AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution, creating a workload different from chatbots. We present the first production-scale characterization of this workload using sampled GitHub Copilot traces from June 2026, comprising 3.2M users, 13M sessions, 761M LLM calls, and 95T tokens. Our analysis reveals distinctive workload properties with important systems implications. For example, agentic coding sessions consist of sparse user-initiated turns, each unfolding into an autonomous agent loop of LLM calls almost always coupled with tool execution. This structure yields KV cache hit rates averaging 90% within a turn, but falling to 55\% across turn boundaries and drastically invalidated after events like model switches or context compaction. Diverse workflows and user behaviors are observed with variable and long-tailed token consumption, time span, and tool calls. We highlight the difference between quick agentic turnaround times and the minutes-long user idle periods at turn boundaries, and design a lightweight idle-time predictor that captures 86-90\% of total idle time, enabling proactive decisions for efficient resource orchestration. These findings challenge assumptions underlying current LLM-serving systems and provide an empirical foundation for agent-native infrastructure.

Figures

Figures reproduced from arXiv: 2608.00101 by Banruo Liu, Esha Choukse, Haoran Qiu, \'I\~nigo Goiri, Ricardo Bianchini, Rodrigo Fonseca.

Figure 1
Figure 1. Figure 1: Traffic trend from uniformly sampled subset of traces over one week in June, 2026. All metrics are sampled from the full population and normalized to day 1 hour 1. Jun 01 Jun 02 Jun 03 Jun 04 Jun 05 Jun 06 Jun 07 0 1 2 3 4 Normalized Calls Extremely Long (>120K) Long (80K–120K) Medium (40K–80K) Short (≤40K) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-session average metrics from sampled traces during the data collection period [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of the number of LLM invocations and tool calls per turn during weekdays vs. weekends. 1 10 100 Calls per Turn (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 CDF of Sessions 4.5 4 LLM calls/turn Tool calls/turn [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: CDFs of per-turn LLM calls and tool invocations track each other closely, confirming the 1:1 coupling across the entire range in every turn and session. 4.3 Execution Structure The 1:1 LLM↔Tool Ratio. Across the full population, the ratio of LLM calls to tool invocations is remarkably close to 1:1 (mean 40.6 vs. 43.6 per session, Figure 4d). This observa￾tion also holds at the per-turn level, as shown in … view at source ↗
Figure 4
Figure 4. Figure 4: Distributions of session and turn metrics. All dis￾tributions show a right-skew tail behavior to various extents. sessions complete within 10 minutes, yet a persistent heavy tail extends for multiple hours. At the turn level within each session, the median turn trig￾gers 3 LLM calls and 3 tool invocations, consuming 160.2K prompt tokens and 265 completion tokens. However, the up￾per tail is substantial: P9… view at source ↗
Figure 7
Figure 7. Figure 7: Measured LLM call parallelism. Takeaway 2: 87% of LLM calls are agent-initiated. User request arrivals alone do not predict LLM load; capacity planning requires session- or turn-level modeling of au￾tonomous agent execution chains. LLM Call Parallelism. Although agentic workflows can spawn parallel subtasks, LLM execution is overwhelmingly serial [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of inference usage of top 15 models. The remainder of the paper quantifies the resource im￾plications of this workload for the LLM itself (Section 5), context compaction (Section 6), tool execution (Section 7), and resource idleness (Section 9). 5 LLM Footprints Having established the session-structured execution pattern, we now zoom in on the LLM calls that dominate agentic sessions. We first… view at source ↗
Figure 8
Figure 8. Figure 8: Timeline of a “Deep-loop w/failures” turn: repeated tool failures trigger autonomous retry loops, generating 36 LLM calls and 35 tool calls in total. repeatedly alternates between reasoning, tool execution, di￾agnosis, and corrective actions. This behavior highlights a key distinction from traditional chatbot workloads: task completion frequently involves trial-and-error and iterative validation rather tha… view at source ↗
Figure 10
Figure 10. Figure 10: CDFs of per-call token counts: Prompt tokens (median 68K); Cached prompt tokens (median 63K); output tokens (median 247). The >275:1 input-to-output ratio makes KV-cache reuse the critical serving lever. 0 20 40 60 80 100 Token Share (%) Avg per Call System History FuncCalls Context RepoInstr [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: CDF of session wall-clock time spent in LLM execution, tool execution, and user idle periods for single￾turn sessions (6% of all sessions) and multi-turn sessions (94%, with user-time in between two turns). interactions, where models frequently process large code￾bases, execution logs, and accumulated context. Takeaway 5: Coding-agent workloads are highly token￾intensive: both prompt and completion length… view at source ↗
Figure 13
Figure 13. Figure 13: Cache hit rate distribution (left) and its average￾rate progression during a turn (right). Rates rise from an average cache hit rate of 45% initially to ∼90% at plateau as new content offsets prefix reuse gains. Takeaway 7: Prefix caching rate is high overall (median 98%), and follows a predictable trajectory within a turn— 45% on the cold-start call, jumping to 86% by the second call, and plateauing at 9… view at source ↗
Figure 14
Figure 14. Figure 14: Cache hit rate in LLM calls before and after turn boundaries. Same-model boundaries cause a ↓26% average drop; model-switch boundaries cause near-complete invali￾dation (↓67%). <1s 1-5s 5-30s 30s-2m 2-10m 10m-1h >1h Idle Time Between Turns 0 20 40 60 80 100 Cache Hit Rate (%) [PITH_FULL_IMAGE:figures/full_fig_p010_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Cache hit rate drops with inter-turn idle time. 5.4 Model Switches Since a session’s model is fixed for the duration of a turn, model switches only occur at turn boundaries, and when they do, they destroy the cache entirely rather than merely degrading it: a KV cache built for one model’s weights and attention layout cannot be reused by another model. We observe that model switches affect ∼6.4% of session… view at source ↗
Figure 17
Figure 17. Figure 17: Distribution of compaction trigger points. one compaction event account for only 7.8% of sessions but 44.2% of total tokens, 37.1% of LLM calls, and 38.9% of tool calls ( [PITH_FULL_IMAGE:figures/full_fig_p011_17.png] view at source ↗
Figure 19
Figure 19. Figure 19: Distribution of prompt token drop percentage after context compaction. 0 20 40 60 80 100 Cache Hit Rate Drop (%) 0.0 0.2 0.4 0.6 0.8 1.0 CDF of Events Median: 66.1% [PITH_FULL_IMAGE:figures/full_fig_p011_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Distribution of cache hit rate drop percentage after context compaction. Context Compaction Impact on KV cache. Beyond the added latency, compaction also perturbs the KV cache: the summarized history invalidates the previously cached prefix, forcing a costly cache-cold prefill on the next turn and elimi￾nating the reuse benefits that prefix caching would otherwise provide. When compaction does occur, it i… view at source ↗
Figure 22
Figure 22. Figure 22: Distribution of tool call vs. LLM call duration. 1ms 10ms 1s 10s 1min 1h 1d Duration (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 CDF of Tool Calls get_file run_command_in_terminal code_search replace_string_in_file file_search run_build apply_patch create_file update_plan_progress get_symbols_by_name [PITH_FULL_IMAGE:figures/full_fig_p012_22.png] view at source ↗
Figure 21
Figure 21. Figure 21: Most invoked tools and their frequency. Compaction is therefore a hidden cost of long agentic sessions: the agent fills the context window during deep ex￾ploration, triggers compaction, loses nearly all cached state, and then must rebuild the cache from scratch on subsequent calls, incurring both higher latency and higher cost exactly when the session is most complex. Takeaway 10: Context compaction affec… view at source ↗
Figure 25
Figure 25. Figure 25: Tokens added from tool call outputs grouped by success/failure status of tool execution. and failed calls contribute a broadly similar number of to￾kens, with failures occasionally trending lower since error responses are often terse compared to full file or search con￾tents. One clearest exception to this pattern is run_build: successful builds return minimal output (a median of roughly 60 tokens), where… view at source ↗
Figure 26
Figure 26. Figure 26: Tool call concurrency during a turn. 93% of batches contain a single tool (issued in batches), but a power￾law tail extends to 80+ parallel tools. 0 10 20 30 40 50 Parallelism Rate (%) multi_replace_string_in_file apply_patch run_build update_plan_progress replace_string_in_file run_command_in_terminal edit_file get_errors create_file code_search file_search get_projects_in_solution get_file get_symbols_b… view at source ↗
Figure 27
Figure 27. Figure 27: Per-tool parallelism for the 15 most-invoked tools. Bars show each tool’s parallelism rate (i.e., the fraction of its invocations issued in a parallelized batch). The annotation at the end of each bar (e.g., 1.9×) is the tool’s average batch size. Read/lookup tools are frequently batched; write/run￾command tools are almost always serial. Parallelism is concentrated in read-only operations. Fig￾ure 27 show… view at source ↗
Figure 29
Figure 29. Figure 29: Users of different types varying with time. 1 10 100 1K Per User (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 CDF of Users 2 11 Sessions/user Turns/user [PITH_FULL_IMAGE:figures/full_fig_p015_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Distribution of the number of sessions and turns per user during the data collection period. and rely on the agent’s autonomy to iterate without inter￾vention. As shown in [PITH_FULL_IMAGE:figures/full_fig_p015_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Distribution of total prompt token and comple￾tion token consumption per user [PITH_FULL_IMAGE:figures/full_fig_p016_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Idle time distribution. sleeping (when tool execution is done in sandbox containers) and KV cache eviction/offloading. 9.1 Idle Time Distributions Coding agent workflows naturally alternate among LLM inference, tool execution, and user input, leaving different resources temporarily unused. To quantify these opportu￾nities for resource reclamation, we define and estimate idle time from the perspective of e… view at source ↗
Figure 33
Figure 33. Figure 33: Idle time prediction evaluation. session will remain idle, it reliably identifies that the session will remain idle long enough to be worth reclaiming. 9.3 Multiplexing Opportunity based on Predictions For agent serving platforms where LLM inference runs on a shared GPU cluster and tool execution runs in container￾s/sandboxes with either dedicated CPU cluster (e.g., Kuber￾netes [2, 25]) or FaaS services [… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 15 linked inside Pith

  1. [1]

    Amazon. 2025. AWS Lambda.https://aws.amazon.com/lambda/

    2025

  2. [2]

    Amazon Web Services. 2025. Elastic Kubernetes Service (EKS).https: //aws.amazon.com/eks/

    2025

  3. [3]

    Anthropic. 2025. Claude Code.https://docs.anthropic.com/en/docs/ claude-code

    2025

  4. [4]

    Anthropic. 2026. Anthropic Economic Index report: Cadences.https: //www.anthropic.com/research/economic-index-june-2026-report

    2026

  5. [5]

    Anthropic. 2026. Prompt Caching.https://platform.claude.com/docs/ en/build-with-claude/prompt-caching. Claude Platform Documenta- tion. Accessed: 2026-07-09

    2026

  6. [6]

    Arena Team. 2026. Agent Arena: Causal Evaluation of Agents in the Real World.https://arena.ai/blog/agent-arena-methodology. Arena Blog

    2026

  7. [7]

    Samriddhi Biswas, Sagar Goel, Ranjita Mohan, Samarth Khare, Ran- jita Ramjee, and Mohit Bansal. 2026. Sutradhara: Orchestrator- Engine Co-Design for Tool-Based Agentic Inference.arXiv preprint arXiv:2601.12967(2026)

    Pith/arXiv arXiv 2026

  8. [8]

    Gohar Irfan Chaudhry, Esha Choukse, Haoran Qiu, Íñigo Goiri, Ro- drigo Fonseca, Adam Belay, and Ricardo Bianchini. 2026. Murakkab: Resource-efficient agentic workflow orchestration in cloud platforms. InProceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI)

    2026

  9. [9]

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Bao- quan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. ReTool: Reinforcement learning for strategic tool use in LLMs.arXiv preprint arXiv:2504.11536(2025)

    Pith/arXiv arXiv 2025

  10. [10]

    GitHub. 2025. GitHub Copilot.https://github.com/features/copilot

    2025

  11. [11]

    GitHub Copilot. 2026. GitHub Copilot CLI Context Management and Compaction.https://docs.github.com/en/copilot/concepts/agents/ copilot-cli/context-management. GitHub Docs. Accessed: 2026-07-09

    2026

  12. [12]

    Dongxin Guo, Jikun Wu, and Siu Ming Yiu. 2026. SAGA: Workflow- Atomic Scheduling for AI Agent Inference on GPU Clusters.arXiv preprint arXiv:2605.00528(2026)

    Pith/arXiv arXiv 2026

  13. [13]

    Qinghao Hu, Peng Sun, Shengen Yan, Yonggang Wen, and Tianwei Zhang. 2021. Characterization and Prediction of Deep Learning Work- loads in Large-Scale GPU Datacenters. InProceedings of the Interna- tional Conference for High Performance Computing, Networking, Storage and Analysis (SC)

    2021

  14. [14]

    Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Representations, Vol. 2025. 58791–58831

    2025

  15. [15]

    Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. InProceed- ings of the 2019 USENIX Annual Technical Conference (ATC)

    2019

  16. [16]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?arXiv preprint arXiv:2310.06770(2024)

    Pith/arXiv arXiv 2024

  17. [17]

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Wei- dong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly effi- cient gradient boosting decision tree.Advances in neural information processing systems30 (2017)

    2017

  18. [18]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica

  19. [19]

    Hanchen Li, Azalia Mirhoseini, and Ion Stoica. 2025. Efficient Agentic LLM Inference with KV Cache Retention.arXiv preprint arXiv:2511.02230(2025)

    Pith/arXiv arXiv 2025

  20. [20]

    Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Liu, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLM- based Applications with Semantic Variable.Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI)(2024)

    2024

  21. [21]

    Banruo Liu, Wei-Yu Lin, Minghao Fang, Yihan Jiang, and Fan Lai. 2026. Compass: SLO-aware Query Planner for Compound AI Serving at Scale. arXiv:2504.16397 [cs.DB]https://arxiv.org/abs/2504.16397

    Pith/arXiv arXiv 2026

  22. [22]

    Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaot- ing Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, et al

  23. [23]

    Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E Gonzalez, et al. 2025. Autellix: An efficient serving engine for LLM agents as general programs.arXiv preprint arXiv:2502.13965(2025)

    Pith/arXiv arXiv 2025

  24. [24]

    Prateek Majgaonkar et al. 2025. Understanding Code Agent Behaviour: An Empirical Study.arXiv preprint arXiv:2511.00197(2025)

    arXiv 2025

  25. [25]

    Microsoft. 2025. Azure Kubernetes Service (AKS).https://azure. microsoft.com/en-us/services/kubernetes-service/

    2025

  26. [26]

    Microsoft. 2025. Serverless on Azure.https://azure.microsoft.com/en- us/solutions/serverless. 19 Banruo Liu, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, and Esha Choukse

    2025

  27. [27]

    Microsoft. 2026. Microsoft 365 Copilot AI for Enterprise Productivity. https://www.microsoft.com/en-us/microsoft-365-copilot/enterprise

    2026

  28. [28]

    OpenAI. 2025. Codex.https://openai.com/index/codex/

    2025

  29. [29]

    OpenAI. 2025. Introducing Deep Research.https://openai.com/index/ introducing-deep-research/

    2025

  30. [30]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. 2024. Charac- terizing Power Management Opportunities for LLMs in the Cloud. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (La Jolla, CA, USA)(ASP...

    arXiv 2024

  31. [31]

    Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ramachandran Ramjee, and Rodrigo Fonseca. 2025. ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Mul- timodal Model Serving. InProceedings of the 2025 ACM Symposium on Cloud Computing (SoCC 2025)(V...

    2025

  32. [32]

    Yeonju Ro, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bian- chini, Aditya Akella, Zhangyang Wang, Mattan Erez, and Esha Choukse. 2025. Sherlock: Reliable and Efficient Agentic Workflow Execution.arXiv preprint arXiv:2511.00330(2025)

    arXiv 2025

  33. [33]

    Mohammad Shahrad, Rodrigo Fonseca, Íñigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. 2020. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider. InProceedings of the 2020 USENIX Annual Technical Conference (ATC)

    2020

  34. [34]

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2025. DynamoLLM: Designing LLM inference clusters for performance and energy efficiency. In2025 IEEE International Sym- posium on High Performance Computer Architecture (HPCA). IEEE, 1348–1362

    2025

  35. [35]

    Xingyao Wang et al. 2024. OpenHands: An Open Platform for AI Soft- ware Developers as Generalist Agents.arXiv preprint arXiv:2407.16741 (2024)

    Pith/arXiv arXiv 2024

  36. [36]

    Yawen Wang, Kapil Arya, Marios Kogias, Manohar Vanga, Aditya Bhandari, Neeraja J Yadwadkar, Siddhartha Sen, Sameh Elnikety, Chris- tos Kozyrakis, and Ricardo Bianchini. 2021. SmartHarvest: Harvesting idle CPUs safely and efficiently in the cloud. InProceedings of the Sixteenth European Conference on Computer Systems. 1–16

    2021

  37. [37]

    Yechen Xu, Xinhao Kong, Tingjun Chen, and Danyang Zhuo. 2024. Conveyor: Efficient tool-aware LLM serving with tool partial execu- tion.arXiv preprint arXiv:2406.00059(2024)

    Pith/arXiv arXiv 2024

  38. [38]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Liber, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer In- terfaces Enable Automated Software Engineering.arXiv preprint arXiv:2405.15793(2024)

    Pith/arXiv arXiv 2024

  39. [39]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models.arXiv preprint arXiv:2210.03629(2023)

    Pith/arXiv arXiv 2023

  40. [40]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. InProceedings of the 16th USENIX Symposium on Operating Systems Design and Implementa- tion (OSDI)

    2022

  41. [41]

    Shan Yu, Junyi Shu, Yuanjiang Ni, Kun Qian, Xue Li, Yang Wang, Jinyuan Zhang, Ziyi Xu, Shuo Yang, Lingjun Zhu, et al. 2026. Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving.arXiv preprint arXiv:2604.25899(2026)

    Pith/arXiv arXiv 2026

  42. [42]

    Yichao Yuan, Ankita Nayak, Souvik Kundu, and Nishil Talati. 2026. Agentic AI Workload Characteristics.arXiv preprint arXiv:2605.26297 (2026)

    Pith/arXiv arXiv 2026

  43. [43]

    Wei Zhang, Zhiyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. 2025. JITServe: SLO-aware LLM Serving with Imprecise Request Information. arXiv:2504.20068 [cs.DC]https: //arxiv.org/abs/2504.20068

    arXiv 2025

  44. [44]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shi Cao, Christos Kober, Liang Shi, Ziniu Wu, et al. 2024. SGLang: Efficient Execution of Structured Language Model Programs.arXiv preprint arXiv:2312.07104(2024)

    Pith/arXiv arXiv 2024

  45. [45]

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. DeepResearcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 414–431

    2025

  46. [46]

    Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2026. TraceLab: Characterizing Coding Agent Workloads for LLM Serving. arXiv:2606.30560 [cs.LG] https://arxiv.org/abs/2606.30560 20

    Pith/arXiv arXiv 2026

  47. [2023]

    InProceedings of the 29th Symposium on Operating Systems Principles (SOSP)

    vLLM: Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP)

  48. [2025]

    LMCache: An efficient KV cache layer for enterprise-scale LLM inference.arXiv preprint arXiv:2510.09665(2025)

    arXiv 2025