{"id":"aa6daf7e-700e-42db-802c-69fba87248f8","arxiv_id":"2606.20374","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"ARGUS delivers always-on, sub-2% overhead, multi-layer tracing plus 3700× kernel compression and progressive diagnosis that localizes fail-slow faults in 10k-GPU LLM training clusters.","lead":"ARGUS is a production tracing system that watches 10,000+ GPU LLM training jobs continuously at under 2% overhead, compresses kernel traces thousands of times, and automatically narrows fail-slow problems to a few ranks and kernels. It matters because silent slowdowns already waste thousands of GPU-hours per job and existing tools are either too coarse or too expensive to leave on.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The load-bearing fidelity claim for L3 is only weakly stress-tested: production cases that needed kernel-level localization either bypassed L3 or never quantified reconstruction error under non-stationary kernels.","rationale":"The Reader correctly isolates the weakest assumption: that online KDE clustering plus three-statistic log-normal CDF reconstruction preserves enough distributional signal for reliable W1-based L3 detection. That assumption is load-bearing for the strongest claim (always-on kernel-level online diagnosis at 10k scale). The paper supplies strong engineering evidence for overhead (<2%), compression ratio (~3700×), and progressive narrowing via L1/L2, plus six months of deployment and five qualitative cases. It does not, however, supply quantitative validation that the lossy summaries remain faithful under the irregular or non-stationary kernels that appear in real VLM/MoE jobs. Because the overall system still delivers value through L1/L2 + on-demand L4 even if L3 is imperfect, the appropriate verdict remains CONDITIONAL rather than REJECT; the missing fidelity numbers simply keep it from unconditional ACCEPT. My concern is essentially the same as the Reader’s, so agreement is full and the recommended verdict is UNCHANGED (still CONDITIONAL).","tokens_in":18538,"tokens_out":687,"duration_ms":7398,"concrete_test":"On a held-out production window (or controlled injection of 1.5–3× slowdowns on selected GEMM/NCCL kernels), recompute pairwise W1 both from full raw duration samples and from the (count, p50, p99) log-normal reconstructions; report rank correlation of the two distance matrices and L3 IQR detection F1. If correlation drops below ~0.9 or F1 falls materially under non-stationary regimes, the L3 fidelity claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that ARGUS can perform always-on kernel-level online cross-rank diagnosis at 10k-GPU scale rests on the fidelity of the online KDE + (count, p50, p99) → log-normal mixture CDF pipeline (§5.2, §6.2). The paper asserts that multimodal but regular kernel durations make this lossy summary sufficient for Wasserstein-1 detection. Yet the five production case studies do not stress this assumption: Case 2 is the only pure L3 success (stable communication kernels); Cases 1 and 5 are caught by L2 phase CV; Case 3 never fires L1–L3 (masked by PP alignment and VLM shape variation); Case 4 is intermittent and diluted inside L3 windows, requiring manual L4. No ablation reports reconstruction error (true W1 on raw samples vs. reconstructed CDFs), false-positive/false-negative rates under controlled kernel slowdowns, or behavior when modes drift inside a window (JIT, dynamic MoE routing, shape-varying attention). If the reconstruction systematically under- or over-estimates tails for non-stationary kernels, L3 alerts become unreliable precisely when fine-grained localization is most needed, leaving the system dependent on coarser L1/L2 or manual L4.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"ARGUS is a production tracing and diagnosis system for LLM training on 10,000+ GPU clusters. It decomposes observation into CPU call stacks (py-spy), framework phase semantics (CUDA Events), and GPU kernel activity (CUPTI), claiming combined always-on overhead under 2%. A host-side Processor converts traces to Perfetto and compresses kernel events online via KDE valley clustering into (count, p50, p99) summaries (~3,700×, 10 MB → 2.7 KB per rank per step). Progressive diagnosis runs L1 (iteration-time anomalies), L2 (parallelism-group-aware phase CV/z-score), and L3 (log-normal CDF reconstruction + Wasserstein-1 + IQR) in parallel, with L4/L5 for manual deep-dive. The system has been deployed for over six months; five production case studies cover compute stragglers, link degradation, pipeline-bubble masking, FlashAttention JIT stalls, and compute faults misread as network issues.","tokens_in":18845,"tokens_out":1517,"duration_ms":24424,"significance":"If the claims hold, ARGUS fills a genuine gap between always-on coarse monitors (Greyhound, C4, Minder, etc.) and high-overhead or triggered fine profilers (nsys, PyTorch Profiler, EROICA). Always-on kernel-level cross-rank comparison at 10k-GPU scale with sub-2% overhead and multi-month production use would be a substantial systems contribution for LLM training reliability and efficiency. Strengths include concrete overhead and memory measurements against strong baselines, quantified compression, a clear progressive diagnosis design, and five production cases with remediation outcomes (node replacement, PCIe repair, JIT disk cache). The work is empirical and falsifiable rather than circular: numbers are measured against external tools and production ground truth.","major_comments":[{"comment":"§5.2–§6.2 (KDE clustering, log-normal mixture CDF from (count,p50,p99), W1 + IQR): The paper’s central differentiator is always-on kernel-level online cross-rank diagnosis at 10k-GPU scale. That claim rests on lossy summaries preserving enough fidelity for W1 detection. There is no reconstruction-error measurement (true W1 on raw samples vs. reconstructed CDFs), no controlled kernel-slowdown injection with precision/recall, and no evaluation under non-stationary modes (JIT, dynamic MoE routing, shape-varying attention). Case 2 is the only clean L3 success on stable communication kernels; Cases 1 and 5 are caught at L2; Case 3 never fires L1–L3; Case 4 is diluted in L3 windows and needs L4. Without fidelity/ablation evidence, the L3 path—and thus the claim of kernel-level online comparison at scale—is under-supported relative to how strongly it is positioned in the abstract, Table 1, and","section":null},{"comment":"§8.2 and Figures 8–9: Overhead and RSS are measured only on 8- and 32-GPU runs. The production claim is 10,000+ GPUs for six months, yet there is no quantitative always-on overhead, memory, or Processor/pipeline cost at that scale (or even at a few hundred GPUs). Small-scale <2% is encouraging and the streaming design is sound, but the load-bearing “always-on under 2% at production scale” claim needs at least production-side overhead evidence or a larger controlled run; otherwise the evaluation does not match the scale of the claim.","section":null},{"comment":"§6 and §9 / Appendix D: Diagnosis effectiveness is shown only via five narrative case studies. There is no aggregate report of alert volume, false-positive rate, time-to-localization, or coverage over the six-month deployment (e.g., how often L3 alone localized vs. L2 vs. manual L4). Table 6 maps fault classes to levels but is not backed by measured detection rates. For a production diagnosis system, some quantitative operational metrics—or a controlled fault-injection suite covering the Table 6 categories—are needed to substantiate continuous fail-slow detection beyond selected successes.","section":null},{"comment":"§6.2 Eq. (4) and §5.2 (Scott bandwidth, valley filters): Free parameters (IQR coefficient α, KDE bandwidth constant, cluster-level and spacing filters) are not sensitivity-analyzed. Because L3 anomaly flags depend on these choices, the paper should state default values used in production and show that detection outcomes are stable under reasonable variation, or justify them from data rather than leaving them as unexamined knobs.","section":null}],"minor_comments":[{"comment":"Table 1: FLARE is marked Always-on Cont. with Fine granularity ✓, yet the text says coverage is limited to a predefined operator set; a footnote clarifying “fine” vs. “arbitrary kernel” would avoid overstating the comparison.","section":null},{"comment":"Figure 1 caption and body: GPU-hours wasted (23,758) and 7% figure would be clearer if the job duration and expected iteration time baseline method were stated in one place.","section":null},{"comment":"§4.2: Stream selection for NCCL (device vs. peer group rank) is important; a short pseudocode or table of rules would help reproducibility.","section":null},{"comment":"§5.2 / Figure 6: Units and axes (log2 duration) are fine, but stating the window length used for clustering in production would aid interpretation of compression numbers in Table 4.","section":null},{"comment":"§8.1 vs. Appendix C: Evaluation uses HunYuan-V3 Preview with EP=8; case studies use different hybrid parallelisms. A sentence noting that overhead results are for one MoE config and may vary with PP-heavy or VLM shape diversity would set expectations.","section":null},{"comment":"Typos/clarity: “LLMS” → “LLMs” (Intro); “aoutlier-based nomaly” → “outlier-based anomaly” (§6.2); “Cont.=continuous” formatting in Table 1 is slightly cramped.","section":null},{"comment":"Related work: Aegis [12] is mentioned briefly; if it is concurrent production work, a clearer capability contrast (beyond communication-layer bound) would strengthen §2.2.","section":null}],"recommendation":"major_revision","confidential_remarks":"Strong production systems paper with real deployment evidence; the main risk for the venue is that the novel L3 compression+W1 pipeline is marketed as the scale enabler but is the least rigorously validated component. If the authors add reconstruction-error numbers, a small controlled injection study, and production overhead/alert stats, this could become a clear accept. Scope fit for a systems/DC journal is good; not a theory paper."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is the first system I've seen that honestly tries to keep always-on kernel-level traces under 2% overhead and still do online cross-rank comparison at 10k GPUs. The hierarchy split (py-spy stacks + CUDA-event phases + CUPTI kernels), the Vector/Processor pipeline, and the KDE valley clustering that turns 10 MB of kernels into 2.7 KB of (count, p50, p99) summaries are concrete engineering, not just packaging. Six months of production use and five real cases (compute stragglers, PCIe link faults, PP bubble masking, FlashAttention JIT, and compute faults that look like network) give it more weight than most academic systems papers.\n\nWhat works: overhead numbers on 8/32-GPU runs are clean and constant-memory; the progressive L1–L3 funnel is sensible; Case 2 is a genuine L3 win that coarser monitors would miss; Case 5 shows the value of not trusting out-of-band network alerts. Table 1 is fair. Citations cover the right prior art (Greyhound, Holmes, C4, MegaScale, FLARE, etc.).\n\nSoft spots, in proportion: the L3 fidelity claim (log-normal mixture CDFs + W1 + IQR) is only weakly stress-tested. Most cases are caught by L1/L2 or need manual L4; there is no reconstruction-error ablation or controlled slowdown study. That is a real gap for the kernel-level claim, but not fatal—the system still delivers value through the coarser levels and the always-on traces. Overhead micro-benchmarks are far smaller than the claimed scale, and there is no public code or detector precision/recall. Free parameters (α, Scott bandwidth, valley filters) are standard and not hidden.\n\nThis is for people who run or build large training clusters and for systems researchers who care about production observability. It deserves a serious referee. I would engage with it, cite the production evidence and the compression design, and push for the missing L3 ablations in revision.","headline":"Solid production systems paper that actually closes the always-on + kernel-level + 10k-scale gap; L3 fidelity is under-tested but not load-bearing for the main claim.","tokens_in":19509,"tokens_out":538,"would_cite":true,"duration_ms":6108,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"ARGUS keeps always-on, kernel-level tracing under 2% overhead on 10,000-GPU LLM clusters and compresses events ~3,700× so real-time straggler diagnosis becomes practical.","keywords":["LLM training","fail-slow detection","GPU tracing","kernel compression","distributed systems","performance diagnosis","10k-GPU clusters","always-on observability"],"falsifier":"Run a production-scale job whose kernel duration distributions are highly irregular or non-stationary; if the compressed (count, p50, p99) summaries plus Wasserstein-1 tests then fail to flag known injected stragglers while raw-trace analysis succeeds, the central compression claim collapses.","tokens_in":19418,"feed_emoji":"🔍","tokens_out":915,"duration_ms":8611,"temperature":0.7,"pith_summary":"Large-scale LLM training wastes huge amounts of GPU time on silent fail-slow faults: a few slow ranks, links, or kernels drag every synchronous step without raising an error. Coarse monitors can see that something is wrong but not which kernel or why; full profilers cost 5–30% overhead and produce terabytes of data, so they cannot stay on. ARGUS claims a middle path that works in production: it splits observation into three light complementary streams (CPU stacks, framework phase durations, and GPU kernel activity), keeps total overhead under 2%, and compresses the high-volume kernel stream online by roughly 3,700\times. A progressive automated pipeline then narrows the search from tens of thousands of ranks down to a handful of suspect windows, ranks and kernels. Deployed for more than six months on a 10,000+ GPU cluster, the system has already diagnosed representative failures that other approaches miss.","feed_headline":"Always-on kernel tracing under 2% cost for 10k GPUs","feed_subtitle":"Compresses events 3,700× so real-time straggler diagnosis works in production LLM training","key_machinery":"Online KDE valley clustering of log-duration samples per (kernel, stream, rank) window, followed by log-normal CDF reconstruction from only (count, p50, p99) per cluster and Wasserstein-1 distance matrices for cross-rank anomaly detection. This compresses ~10 MB of raw kernel events to ~2.7 KB per rank per step while retaining enough distributional fidelity for automated L3 localization.","core_discovery":"A hierarchical, three-signal tracing design plus online KDE-based statistical compression of kernel durations makes always-on, fine-grained, real-time cross-rank diagnosis feasible at 10,000-GPU scale under a combined overhead of less than 2%, and the resulting progressive L1–L3 pipeline isolates anomalous windows, straggler ranks and degraded kernels in production.","pith_inferences":["The same hierarchical-signal plus lossy-but-fidelity-preserving compression pattern could be ported to large-scale inference serving or reinforcement-learning training with only modest re-instrumentation.","If the log-normal mixture reconstruction proves robust, other distributed systems that generate repetitive multimodal event streams (databases, ML serving) could adopt analogous online statistical summaries instead of raw traces.","Masking effects under pipeline parallelism shown in the case studies suggest that purely iteration-time or phase-level detectors will systematically under-detect stragglers whenever a global sync is present; future detectors may need explicit dependency models."],"forward_implications":["Always-on kernel-level observability becomes practical rather than a short-window sampling luxury at 10k-GPU scale.","Fail-slow diagnosis can move from machine- or link-level alerts to named kernels and host-side stack frames without waiting for offline post-mortems.","The same progressive pipeline can surface performance-optimization opportunities (imbalance, poor overlap, JIT stalls) in addition to hard faults.","Trace data volumes drop three orders of magnitude, bringing online cross-rank comparison inside ordinary time-series storage budgets.","Production clusters can keep continuous fail-slow detection running for months instead of only during suspected incidents."],"fun_headline_variants":["Always-on tracing under 2% for 10k-GPU LLM clusters","Hierarchical signals plus 3700× kernel compression at scale","Real-time straggler isolation via L1–L3 on 10000+ GPUs","Fine-grained always-on diagnosis under 2% overhead","Three-signal ARGUS pipeline finds degraded kernels live"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Kernel timings on ranks that share the same parallel role stay regular enough that a few per-mode percentiles still capture the differences needed to spot a true straggler.","fun_headline_variants_meta":{"raw":{"variants":["Always-on tracing under 2% for 10k-GPU LLM clusters","Hierarchical signals plus 3700× kernel compression at scale","Real-time straggler isolation via L1–L3 on 10000+ GPUs","Fine-grained always-on diagnosis under 2% overhead","Three-signal ARGUS pipeline finds degraded kernels live"]},"model":"grok-4.5","effort":"low","cost_usd":0.004884,"raw_usage":{"total_tokens":1390,"prompt_tokens":816,"num_sources_used":0,"completion_tokens":78,"cost_in_usd_ticks":48840000,"prompt_tokens_details":{"text_tokens":816,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":496,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":816,"tokens_out":78,"duration_ms":4891,"temperature":1.0,"reasoning_tokens":496,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T13:13:54.189480+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run a production-scale job whose kernel duration distributions are highly irregular or non-stationary; if the compressed (count, p50, p99) summaries plus Wasserstein-1 tests then fail to flag known injected stragglers while raw-trace analysis succeeds, the central compression claim collapses.","supporting_citations":[],"review_version":2}