{"id":"8fb5ad9c-9ff3-45b1-874b-8c8329adfe02","arxiv_id":"2601.20408","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OptiKIT combines automated quantization and SLO-driven serving-parameter tuning to more than double per-GPU throughput on some production workloads while preserving near full-precision accuracy.","lead":"OptiKIT is an automated pipeline that quantizes large language models and tunes their serving settings without manual expert effort. At eBay it reports more than 2x per-GPU throughput gains and cuts optimization time from roughly 80-100 hours to 15-25 hours.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 1's overload criterion looks algebraically inverted: with arrivals regressed on completions, backlog gives slope <1, not >1, so the Benchmarker may certify early-overload trials as steady-state and inflate the reported SLO-compliant throughput.","rationale":"The reader correctly identifies the Benchmarker's steady-state detector as the weakest link in the paper's central throughput claim. However, the reader's stated concern is that the hand-set tolerance τβ may be too optimistic and that the Benchmarker is not independently validated. My review finds a more fundamental and more concrete defect: as printed, the regression in Equation 1 has the overload direction reversed. If arrival timestamps are regressed on completion timestamps, an overloaded queue produces a slope below 1, not above 1. The paper's bullet interpretation is only correct for the reverse regression. This means the acceptance rule |β − 1| ≤ τβ, even with a well-chosen τβ, can certify trials that are already accumulating queue backlog, and all SLO-compliant throughput numbers in Tables 5 and 7 are downstream of this detector. I do not claim the system is fraudulent or that the reported gains are impossible; quantization and runtime tuning of the kind described can plausibly yield large gains. The issue is that the paper's own formal description of the measurement instrument is internally inconsistent, so the empirical backbone of the abstract claim is not yet trustworthy as written. A single controlled simulation or independent re-derivation would settle whether the sign is simply a typographical error in the paper or a genuine flaw in the detector. Because the concern is concrete and addressable, the appropriate verdict remains conditional: the paper should not be accepted without correcting or clarifying Equation 1 and re-validating the reported throughput numbers against the corrected criterion. This is a partial agreement with the reader because we converge on the same subsystem but identify different root causes.","tokens_in":16275,"tokens_out":7484,"duration_ms":67078,"concrete_test":"Simulate a fixed-rate open-loop trial with arrival rate 20% above service rate (e.g., λ_a = 1.2 requests/s, λ_s = 1.0 requests/s), collect per-request arrival and completion timestamps for roughly 1000 requests, and fit Equation 1 exactly as printed (r_i = α + β c_i). If the fitted β is < 1 in the overloaded run, the paper's β > 1 overload criterion is inverted; then re-evaluate the Benchmarker's acceptance of a borderline trial with β = 0.96 under |β − 1| ≤ 0.05 and recompute the Table 7 rates under a one-sided stability condition (or with the corrected c-on-r regression) to see whether the 2.87x Mistral gain survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing link in the central throughput claim is the Benchmarker's steady-state detector (Section 4.3, Equation 1). The paper fits r_i = α + β c_i, regressing arrival timestamps on completion timestamps, and states that β > 1 means 'arrivals faster than completions.' Under standard queueing algebra, if arrival rate λ_a exceeds service rate λ_s, the i-th completion time grows as c_i ≈ i/λ_s while the arrival time grows as r_i ≈ i/λ_a, so the fitted slope is β ≈ λ_s/λ_a < 1; backlog accumulates exactly when β < 1, not β > 1. The stated interpretation corresponds to the reverse regression, c_i = α + β r_i. Because the acceptance rule is only |β − 1| ≤ τβ, with τβ hand-set to 0.02–0.05, a trial in early overload with β ≈ 0.96 would be classified as steady-state. Every SLO-compliant rate in Tables 5 and 7 is selected by this detector, so the headline 'more than 2x GPU throughput' claim rests on a criterion whose printed direction is internally inconsistent with the claimed semantics. This is not merely a question of tuning τβ; it is a question of whether the detector measures what the paper says it measures. The throughput numbers cannot be taken at face value until this is resolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OptiKIT, an end-to-end distributed framework for automated LLM optimization developed at eBay. OptiKIT automates model compression through quantization recipes, statistical quality evaluation, regression-based inference benchmarking under SLOs, and Bayesian runtime tuning, and it is integrated with enterprise infrastructure via Ray-based actor pools. The authors report that OptiKIT achieves more than 2x per-GPU throughput improvement over a default FP16 vLLM baseline while preserving near-full-precision quality on GSM8K, IFEval, and Do-Not-Answer, and that it cuts engineering time from 80-100 to 15-25 hours per model. The paper also states that the system is open-sourced.","tokens_in":16738,"tokens_out":5956,"duration_ms":51556,"significance":"If the claims hold, OptiKIT would be a valuable contribution to production LLM serving. It addresses a real operational bottleneck, combines quantization and serving-parameter tuning in a single automated workflow, evaluates on externally recognized quality benchmarks, and reports trial-level variance for statistical quality. The architecture is described in enough detail to be a useful reference for similar systems, and the authors explicitly acknowledge open questions such as long-context fidelity and domain-tuned models. However, the headline throughput claim is supported by only two of four reported workloads, and the benchmark acceptance criterion contains a direction inconsistency that calls into question every SLO-compliant throughput number. The strengths here are real, but the central claim cannot be taken at face value without resolving that issue.","major_comments":[{"comment":"The overload criterion is algebraically inverted as written. With r_i and c_i the arrival and completion timestamps, fitting r_i = α + β c_i gives β ≈ λ_s / λ_a under rate mismatch (assuming r_i ≈ i/λ_a and c_i ≈ i/λ_s), so queue growth corresponds to β < 1, whereas the paper states that β > 1 denotes overload. Under the acceptance rule |β−1| ≤ τβ with τβ ∈ [0.02, 0.05], a mildly overloaded trial with β ≈ 0.96 would be certified as steady state. Since every SLO-compliant rate in Tables 5 and 7 is selected by this detector, the reported throughput gains cannot be validated until the regression direction is corrected (or the definitions of r_i and c_i are swapped) and the experiments are rerun; the paper should also report how the headline gains vary with τβ.","section":"Section 4.3, Eq. (1), Figure 4"},{"comment":"The unqualified claim that OptiKIT 'in production delivers more than 2x GPU throughput improvement' is not supported by the paper's own data. Table 5 reports end-to-end gains of 2.13x for Qwen, 2.87x for Mistral with the p95 latency SLO, 1.55x for Mistral with the TTFT/TPOT SLO, and 1.25x for Llama 3 70B. Only two of the four workloads exceed 2x, so the abstract and conclusion should be rephrased to state the full range and to explicitly acknowledge the workloads that do not meet the 2x threshold.","section":"Abstract, Section 7.1, Table 5"},{"comment":"The throughput results are reported as point estimates without replication counts, error bars, or confidence intervals. Given that the benchmark acceptance depends on a hand-set stability tolerance, the paper should report the number of repeated measurements, the variance across runs, and the sensitivity of the per-GPU gains to the choice of τβ. Without this information, the reader cannot assess whether the reported 1.25x-2.87x range is statistically distinguishable from the baseline or whether the gains are robust to small changes in the benchmark criterion.","section":"Section 5.4, Appendix B, Tables 6-7"},{"comment":"The human-hour comparison ('80-100 hours manual vs 15-25 hours OptiKIT') is presented as a headline result in Figure 1 and in the title, but it is only described as 'estimated on internal data.' The estimation procedure, sample size, task scope, and uncertainty are not specified. The paper should either provide the underlying data and a counting methodology or clearly downgrade the claim to an illustrative estimate so that the reader can judge its reliability.","section":"Section 6, Figure 1"},{"comment":"The claim of 'near full-precision performance' with 'average recovery rates exceeding 99%' obscures notable per-task degradations: Llama 3.3 70B drops to 95.4% recovery on Do-Not-Answer and Mistral 24B INT8 drops to 93.5% on IFEval. If 95% recovery is considered production-ready, the paper should state the acceptance threshold explicitly, since the text elsewhere describes these results as robust and production-ready without defining what level of quality degradation is acceptable.","section":"Section 5.3, Table 3, Discussion"}],"minor_comments":[{"comment":"There is a duplicated sentence fragment: 'Mistral exhibited the greatest degradation on the IFEval task ... compared to the full-precision counterpart. as well as the full-precision counterpart.' This should be cleaned up.","section":"Section 5.3"},{"comment":"The sentence 'The StatEval package (Figures 2, 3) package is a core component' repeats the word 'package'; this should be corrected to 'The StatEval package (Figures 2, 3) is a core component.'","section":"Section 4.2"},{"comment":"The terms 'asynchronous closed-loop trial' and 'asynchronous open-loop trial' are used without definitions, and the update rule 'LB←E[latency−1]' is unclear. These should be explained in the text so the reader can understand the search logic.","section":"Algorithm 1"},{"comment":"Table 6 is not referenced in the main text, while Table 7 is referenced in Section 5.4. The authors should either cite Table 6 where it is relevant or remove it to avoid confusion.","section":"Appendix B, Table 6"},{"comment":"The calibration dataset is cited as 'Magic, 2024' with a URL, but the in-text citation should use the full author/name ('Neural Magic, 2024') consistently, as is done elsewhere.","section":"References"},{"comment":"The paper claims that the system is open-sourced, but no repository URL, artifact identifier, or release information appears anywhere in the manuscript. For a systems paper whose reproducibility argument rests on the open-source claim, this omission should be fixed.","section":"Abstract and Section 7.1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark direction issue is serious and load-bearing; I would ask the authors to correct it and rerun the throughput evaluation before the paper can be accepted. The missing open-source artifact should also be addressed in revision, as the paper claims open-sourcing as a contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me get straight to it: this is a plausible production-systems paper, but the headline throughput claim is not yet trustworthy because the steady-state detector in Section 4.3 has the regression direction backwards. Equation 1 fits r_i = α + β c_i, arrivals as a function of completions, and says β>1 means arrivals are outrunning completions. In overload, completion timestamps grow at the service rate while arrivals grow at the faster arrival rate, so the fitted slope is β ≈ λ_s/λ_a < 1. The intended overload signal would come from the reverse regression, c_i = α + β r_i. With the printed equation, a mild overload at λ_a/λ_s = 1.05 gives β ≈ 0.952, which passes a τβ tolerance of 0.02–0.05 and gets certified as steady. Because every SLO-compliant throughput in Tables 5 and 7 is selected by this Benchmarker, the “more than 2× GPU throughput” claim in the abstract and conclusion is not supported until the detector is corrected and the measurements re-run.\n\nWhat is actually new and good: the integrated pipeline — recipe-based compression, StatEval quality gates, TPE-based tuning over vLLM runtime parameters, staged Ray actor pools, and enterprise integration — is a real contribution. The statistical-evaluation section is the strongest part: five-trial RSDs, a determinism study, external benchmarks (GSM8K, IFEval, Do-Not-Answer), and an honest note that INT4 is inconsistent. That is more care than most systems papers take.\n\nSmaller soft spots: the “more than 2×” wording is unqualified even though Table 5 shows 1.25× and 1.55× on two of the four workloads; the engineering-hour savings are an unverified estimate; the abstract promises an open-source release but no artifact link is provided; and “adaptive calibration” is listed as an algorithmic novelty although Section 4.1 describes it as future work.\n\nBottom line: the paper deserves a serious referee. The system design and production context are valuable, the quality data are carefully reported, and the detector flaw is fixable — but I would not cite the throughput numbers until the equation is corrected and the key results are re-benchmarked.","headline":"A well-built production pipeline, but the Benchmarker's steady-state equation is backwards, so the headline throughput gains cannot be trusted until that detector is fixed and the experiments re-run.","tokens_in":17173,"tokens_out":4200,"would_cite":false,"duration_ms":34057,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OptiKIT is a distributed pipeline that automatically quantizes, benchmarks, and tunes LLMs for deployment, reporting more than 2x per-GPU throughput in production with near full-precision quality.","keywords":["LLM optimization","quantization","SLO-driven benchmarking","Bayesian hyperparameter tuning","GPU throughput","automated pipeline","enterprise deployment","calibration data"],"falsifier":"Run an independent closed-loop load test at the rates OptiKIT reports as SLO-compliant—for instance Mistral Small 3 24B with FP8 at tensor-parallelism 4, sustained for several hours with realistic arrival variability—and check whether p95 latency stays under 1500 ms and queue length stays flat; or re-run the full sweep with the stability tolerance $\\tau_\\beta$ set to 0.01 and 0.1 and compare the reported per-GPU gains. If the gains collapse or the latency targets break, the steady-state detector is the load-bearing assumption.","tokens_in":16085,"feed_emoji":"⚙️","tokens_out":11201,"duration_ms":87735,"temperature":0.7,"pith_summary":"OptiKIT is a distributed pipeline that takes a raw large-language model and, with no optimization expert in the loop, returns a quantized model plus a runtime configuration that meets the application's latency targets. The paper's central claim is that automating the whole optimization lifecycle—calibration, quantization, statistical quality checks, SLO-driven load testing, and runtime tuning—is enough to make enterprise-grade LLM deployment routine: in production it reports more than 2x per-GPU throughput improvement over a default FP16 serving setup, with FP8 and INT8 models recovering over 99% of full-precision benchmark accuracy on average. This matters because the real bottleneck for enterprise AI is not GPU hardware alone but the scarce craft knowledge needed to compress and tune models; a system that removes that dependency changes how many teams can ship LLM features within a fixed compute budget.","feed_headline":"Automated LLM pipeline reports 2x+ GPU throughput in production","feed_subtitle":"Without expert help, OptiKIT delivers a quantized, SLO-compliant model at roughly 2x per-GPU throughput.","key_machinery":"The load-bearing machinery is a staged actor-pool pipeline on a distributed cluster, whose three core subsystems form the actual optimization loop. The Optimizer is a backend-agnostic, recipe-based compression engine: a recipe is a declarative specification (int w8a8, int w4a16, fp8 dynamic) bundling a quantization scheme, calibration-data requirements, and layer-selection policy, and each trial draws its own calibration subset to capture sampling variance. The Benchmarker sweeps candidate request rates with exponential search and accepts a rate only if the fitted linear regression of completion timestamps on arrival timestamps has slope $\\beta$ near 1, $|\\beta - 1| \\le \\tau_\\beta$, meaning the queue is neither emptying nor backing up; this is the mechanism that decides what \"SLO-compliant throughput\" means. The Tuner wraps this measurement in a Bayesian search (tree-structured Parzen estimators) over runtime parameters such as tensor-parallel size, max concurrency, and max batched tokens, optimizing per-GPU throughput with a large negative penalty for SLO violations. The loop returns a model artifact plus a tuned configuration, and the actor-pool design destroys and re-creates pools between stages to reclaim GPUs deterministically.","core_discovery":"On its own terms, the paper's discovery is that model compression and serving optimization compose cleanly into a single automated workflow, and that the combined workflow is what unlocks the gains. Individually, quantization buys some throughput and tuning buys some throughput, but the end-to-end configuration—measured by OptiKIT's own SLO-compliant per-GPU throughput—reaches gains such as 2.87x for Mistral Small 3 24B on a latency-p95 workload and 2.13x for Qwen 2.5 7B, while also making previously infeasible tensor-parallel configurations meet their latency targets. The statistical claim is that a generic calibration dataset is sufficient: across 7B, 24B, and 70B models, FP8 Dynamic and INT W8A8 quantization stay within about 1% of full-precision scores on GSM8K, IFEval, and Do-Not-Answer, whereas INT W4A16 is workable but more variable. The paper frames the result as democratization: a non-expert team submits a model and gets a deployable artifact, not a report of recommendations.","pith_inferences":["Beyond the paper, the headline gain should be tested for sensitivity to the stability tolerance $\\tau_\\beta$: re-running the sweep with $\\tau_\\beta$ set to 0.01 and 0.1 would show whether the reported 2x is a fragile artifact of the detector or a stable property of the configurations.","The paper reports that tuning gains are largest under strict SLOs; this suggests a natural routing rule where latency-critical interactive workloads go through the full pipeline, while bulk throughput workloads may need quantization only, which would save pipeline time.","The sequential synchronization barrier between stages, which the paper flags as a limitation, is the obvious next scalability lever; removing it becomes even more important if the same framework extends to pruning and distillation, since the joint search space grows combinatorially.","Because every tuning trial is measured against synthetic load, the paper's configurations are certified only against that pattern; a shadow-deployment validation step under real arrival variability would be a direct, testable extension of the claim."],"forward_implications":["A non-expert team can go from a raw model to a production deployment configuration in roughly 15–25 hours of automated pipeline time, versus the 80–100 person-hours the paper estimates for manual optimization.","The same GPU fleet can serve more than twice the traffic per GPU, or serve a larger model within an existing resource envelope, while keeping near-full-precision quality.","A single generic calibration dataset is enough for FP8/INT8 quantization across at least the 7B, 24B, and 70B families tested, so calibration-data curation ceases to be a per-model expert task.","Latency-critical workloads benefit most from the tuning stage: configurations that fail strict SLOs at low tensor-parallelism become feasible after tuning and/or quantization, giving operators cheaper paths to meet latency targets."],"supporting_citations":[{"why":"Provides GPTQ, the post-training quantization method behind the int w8a8 and int w4a16 recipes.","marker":"Frantar et al., 2023"},{"why":"Supplies SmoothQuant's activation balancing, used in the integer-quantization recipes.","marker":"Xiao et al., 2023"},{"why":"Defines FP8 formats, the basis of the fp8 dynamic recipe that needs no calibration data.","marker":"Micikevicius et al., 2022"},{"why":"Provides vLLM/PagedAttention, the serving stack used for benchmarking and the source of the determinism discussion.","marker":"Kwon et al., 2023"},{"why":"Gives Ray, the distributed actor framework that executes OptiKIT's staged pipelines.","marker":"Moritz et al., 2018"},{"why":"Supplies Optuna with TPE, the Bayesian search algorithm used by the runtime tuner.","marker":"Akiba et al., 2019"},{"why":"Provides the generic calibration dataset used for all integer-quantization trials.","marker":"Magic, 2024"},{"why":"Supplies GSM8K, one of the three open benchmarks used in statistical evaluation.","marker":"Cobbe et al., 2021"},{"why":"Supplies IFEval, the instruction-following benchmark used in statistical evaluation.","marker":"Zhou et al., 2023"},{"why":"Supplies Do-Not-Answer, the safety benchmark used in statistical evaluation.","marker":"Wang et al., 2024b"}],"fun_headline_variants":["OptiKIT automates LLM tuning, 2x GPU throughput for non-experts","No expert needed: OptiKIT yields 2x+ GPU throughput automatically","Automated LLM optimization delivers 2x GPU gains, no expertise needed","Non-experts get 2x GPU throughput via automated OptiKIT pipeline","2x GPU throughput from automated LLM tuning, no expert required"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported throughput gains rest on the Benchmarker's steady-state detector, which accepts a trial as stable when the fitted slope $\\beta$ lies within a hand-chosen tolerance (typically 0.02–0.05), and the paper does not validate that detector against an independent load generator or report how the 2x gains depend on that tolerance.","fun_headline_variants_meta":{"raw":{"variants":["OptiKIT automates LLM tuning, 2x GPU throughput for non-experts","No expert needed: OptiKIT yields 2x+ GPU throughput automatically","Automated LLM optimization delivers 2x GPU gains, no expertise needed","Non-experts get 2x GPU throughput via automated OptiKIT pipeline","2x GPU throughput from automated LLM tuning, no expert required"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2862,"prompt_tokens":963,"completion_tokens":1899,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1794}},"tokens_in":579,"tokens_out":1899,"duration_ms":10618,"temperature":1.0,"reasoning_tokens":1794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:38:53.688139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an independent closed-loop load test at the rates OptiKIT reports as SLO-compliant—for instance Mistral Small 3 24B with FP8 at tensor-parallelism 4, sustained for several hours with realistic arrival variability—and check whether p95 latency stays under 1500 ms and queue length stays flat; or re-run the full sweep with the stability tolerance $\\tau_\\beta$ set to 0.01 and 0.1 and compare the reported per-GPU gains. If the gains collapse or the latency targets break, the steady-state detector is the load-bearing assumption.","supporting_citations":[],"review_version":2}