{"id":"168aa660-88a6-4e5f-ae7a-d53fea977766","arxiv_id":"2505.19739","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Justin extends Flink's DS2 auto-scaler to decide between scaling out (more tasks) and scaling up (more memory per task), using RocksDB cache hit rates and state access latency.","lead":"Justin is a new auto-scaling system for distributed stream processing that adjusts both CPU (number of tasks) and memory per task, instead of allocating the same fixed memory to all operators. In tests on Apache Flink with the Nexmark benchmark, it reached target processing rates with up to 48% less CPU and 28% less memory than the existing DS2 auto-scaler.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline savings on Q8/Q11 may be an artifact of threshold tuning: Δθ=80% and Δτ=1ms are set on the same Nexmark workloads used for evaluation, with no sensitivity analysis.","rationale":"The paper makes a strong quantitative claim: for Q8/Q11, Justin reaches the target rate with 48% less CPU and 27-28% less memory than DS2 in no more reconfiguration steps. Inspection of Algorithm 1 shows that this outcome is produced by replacing DS2's horizontal scaling decision with a vertical one whenever an operator's cache hit rate is below 80% or state access latency exceeds 1 ms. These two constants are the entire trigger for the scale-up branch. The text states that they were identified via experimental analysis in Section 5, which is the same experimental setup used for the headline results. The paper gives no threshold sensitivity analysis and does not report the θ/τ values observed for Q8/Q11, so the reader cannot tell whether the chosen thresholds are comfortably separated from the operating point or whether the savings would vanish under a slightly different threshold. Because the reported 48% CPU reduction is the central evidence for the contribution, this is the most load-bearing assumption. The concern does not impugn the system's engineering; it is a question of whether the reported advantage is a property of the hybrid-scaling idea or of the specific tuned constants. A threshold sweep plus, ideally, a held-out read-heavy workload would settle it. The reader's weakest_assumption points to the same thresholds; I agree, and I do not see grounds to change the CONDITIONAL verdict.","tokens_in":12577,"tokens_out":8668,"duration_ms":94965,"concrete_test":"Run a sensitivity sweep on Q8 and Q11: vary Δθ over {60%, 70%, 90%} and Δτ over {0.5ms, 2ms}, holding other parameters and the random seed fixed, and record final CPU/memory usage and reconfiguration step counts. If the 48%/27-28% advantage over DS2 shrinks materially, e.g., below 20%, or step counts increase for any tested threshold, the central claim is threshold-dependent; if results stay within a small band, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Justin supports target rates with 48% less CPU and 27-28% less memory than DS2 on Q8/Q11 depends on the policy's choice to replace DS2's scale-out with scale-up. That choice is gated by two fixed thresholds in Algorithm 1 (Δθ=80%, Δτ=1ms) that the paper says were selected 'through experimental analysis' in Section 5, i.e., on the same Nexmark setup used for the headline evaluation. No sensitivity analysis is reported, and the paper does not report the observed cache-hit or latency values for Q8/Q11. If those values lie near the thresholds, a different threshold could flip the first reconfiguration from scale-up to scale-out and eliminate most of the reported savings. The 2-minute metric window adds a further aggregation-time assumption, but the threshold dependence is the primary load-bearing point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Justin, an auto-scaler for Apache Flink that extends the DS2 CPU-only elastic scaling policy with memory-aware decisions. Justin monitors per-operator busyness, RocksDB cache hit rate, and state-access latency, and uses these signals to choose between horizontal scaling (changing parallelism) and vertical scaling (changing per-task managed memory) during a reconfiguration. The implementation modifies the Flink Kubernetes Operator and uses a bin-packing scheduler to place tasks with heterogeneous memory demands. The evaluation on the Nexmark benchmark reports that Justin reaches the target rate in as many or fewer reconfiguration steps than DS2, with up to 48% lower CPU usage and 27-28% lower memory usage on queries Q8 and Q11, and with memory savings on stateless queries. The paper's central claim is that decoupling CPU and memory scaling improves resource efficiency for stateful streaming workloads.","tokens_in":12771,"tokens_out":6996,"duration_ms":73866,"significance":"If the reported results are robust, this is a valuable contribution to the DSP elasticity literature. The paper identifies a real limitation of CPU-only auto-scalers (the coupling of memory with parallelism) and proposes a practical, implementable policy that uses storage-layer metrics to arbitrate between horizontal and vertical scaling. The microbenchmarks in Section 3 provide an instructive characterization of how memory affects RocksDB-backed state access under read-, write-, and update-dominated workloads. The authors have made the artifact publicly available (DOI in the footnote), which strengthens reproducibility. However, the evidence for the headline savings is currently weakened by the absence of a sensitivity analysis for the two decision thresholds and by an evaluation that appears to be based on a single run per configuration, with no error bars or statistical comparison.","major_comments":[{"comment":"The policy's decision to prefer scale-up over scale-out is gated by two fixed thresholds, Δθ = 80% and Δτ = 1 ms, which the authors state were selected 'through experimental analysis' on the same Nexmark setup used for the evaluation. The paper does not report the observed cache-hit rate and state-access latency trajectories for Q8 and Q11, and it provides no sensitivity analysis showing how the qualitative results (48% CPU savings, 27-28% memory savings, and the number of reconfiguration steps) vary with these thresholds. Since the first reconfiguration's choice between vertical and horizontal scaling is determined by these thresholds, the headline savings could be an artifact of threshold tuning. I request that the authors report the time series of θ and τ for the stateful operators in Q8 and Q11 and include a sensitivity study (e.g., varying Δθ from 60% to 95% and Δτ from 0.5 ms to 2 ms) to demonstrate that the resource-savings and step-count claims are not confined to the chosen values.","section":"Section 5, Experimental setup; Algorithm 1"},{"comment":"The experimental comparison against DS2 appears to be based on a single run per query and per auto-scaler. The paper states that metrics are collected and aggregated at 5-second granularity and that scaling decisions occur at the end of 2-minute windows, but no repeated trials, error bars, confidence intervals, or statistical tests are reported. The claims that Justin uses '48% less CPU' and '27-28% less memory' are point estimates from single trajectories, and the number of reconfiguration steps is similarly a single observation. I ask the authors to run each configuration multiple times (at least three to five repetitions) and report the variance (or a paired statistical comparison) to establish that the differences are not due to run-to-run variability in the scaling dynamics.","section":"Section 5.1, Figure 5"},{"comment":"The discussion of Q8 states that Justin needs three scale-out reconfigurations while DS2 needs four, but also that Justin's intermediate configurations take longer to stabilize, so that DS2 actually converges sooner. This is an important qualification that is not fully reconciled in the paper's abstract, which claims only that Justin reaches the target rate 'in as many or fewer reconfiguration steps.' The convergence time is a practically meaningful dimension of auto-scaler quality; I ask the authors to report the wall-clock convergence time for all queries (not only Q8 and Q11) and to clarify whether the trade-off between fewer steps and longer stabilization is consistent across queries.","section":"Section 5.1, Q8 paragraph"}],"minor_comments":[{"comment":"The comment on line 14 ('Roll-back scale in') is confusing; the surrounding text says 'cancel the previous scale-up.' Please align the comment with the intended action, which is to revert to the memory level before the last vertical scaling if the vertical scaling did not improve metrics.","section":"Section 4.2, Algorithm 1"},{"comment":"The box plots in Figure 4 are described as showing the distribution of the measured rate, but the number of samples per configuration and the precise aggregation method are not stated. Adding this information (e.g., number of 5-second measurements per 10-minute run, whether all samples are used) would help readers interpret the spread.","section":"Section 3, Microbenchmarks"},{"comment":"The paper says that the elastic scaling trigger parameters are set to keep the average busyness between 20% and 80%, but it does not specify the target busyness used by DS2 after scaling, nor the exact values of the DS2 trigger thresholds. Please provide these parameters or a reference to the DS2 configuration used.","section":"Section 5, Experimental setup"},{"comment":"The phrase 'finegrain memory allocation' should be 'fine-grained memory allocation' for consistency with the rest of the text.","section":"Abstract"},{"comment":"The paper does not discuss the cost of state migration during reconfigurations, even though vertical scaling and horizontal scaling may impose different migration overheads. Since the policy explicitly alternates between scale-up and scale-out, a sentence on how migration costs were handled (or why they were negligible) would strengthen the evaluation.","section":"Section 5.1, Q11 and Q8"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a distributed-systems venue and the artifact is a plus. The core idea is sound and the engineering effort is substantial. The main risk is that the headline results depend on thresholds that are tuned on the evaluation workload without a sensitivity analysis. I would ask the authors to add the sensitivity study and repeated trials in the revision; if these cannot be provided, the strength of the claims should be scaled back accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nJustin is a solid systems paper: it decouples CPU and memory scaling for stream processing operators, a genuinely useful idea that, to my knowledge, hasn't been done before. The policy is simple: use cache hit rate and state access latency to decide between scale-up (more memory per task) and scale-out (more tasks). The microbenchmarks in Section 3 are well-designed and give three concrete takeaways. The implementation is real — about 1,500 LoC on top of Flink and DS2, with artifacts on Zenodo. Evaluating on Nexmark against DS2 is the right choice.\n\nThe headline claims — 48% less CPU and 27-28% less memory on Q8 and Q11 — are plausible but not yet airtight. Two issues. First, the decision thresholds Δθ=80% and Δτ=1ms are selected via 'experimental analysis' on the same Nexmark workloads, and no sensitivity study is reported. If the observed cache-hit rates or latencies for Q8/Q11 sit near those thresholds, slightly different values could flip the first reconfiguration from scale-up to scale-out and erase much of the savings. The paper doesn't report the raw metric values, so a reviewer can't check. Second, the main evaluation appears to be single runs per query: no error bars, no repeated trials. The difference between Justin and DS2 is large, so I wouldn't expect noise to reverse it, but the point estimates deserve a confidence interval.\n\nThere are minor soft spots: only six Nexmark queries, no skew or co-location scenarios, and the policy cancels DS2's scale-out before trying scale-up, which could delay convergence if the scale-up fails. The paper addresses the last one by showing equal or fewer reconfigurations, so that's fine.\n\nOverall, the central contribution is clean, the related work is fair, and the limitations are honestly stated. This paper deserves a serious peer review. I'd recommend accepting it conditionally, with the main revisions being a sensitivity analysis for the thresholds, repeated runs with variance, and ideally reporting the observed metric values that drive the decisions. If those come out, the paper will be a solid citation for hybrid scaling in DSP.\n\nBring it to the next reading group.","headline":"A genuinely useful hybrid CPU/memory auto-scaler for Flink with a clean policy and real implementation, but the headline savings rest on hand-tuned thresholds and single-run evaluations that need tighter evidence.","tokens_in":13289,"tokens_out":2990,"would_cite":true,"duration_ms":30099,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid CPU/memory auto-scaler that decides between scaling up and scaling out can support target stream rates with up to 48% less CPU and 27–28% less memory than a CPU-only auto-scaler.","keywords":["distributed stream processing","elastic scaling","hybrid scaling","resource management","Apache Flink","RocksDB","Nexmark","DS2"],"falsifier":"Run a read-heavy stateful Nexmark query on the same seven-node testbed with a target rate, then vary the per-task managed memory so that the cache hit rate stays just above 80% while the state working set still forces disk accesses; if Justin fails to scale up and instead scales out, adding CPU without reducing state-access latency, the threshold choice is falsified. More directly, reproducing the Q8/Q11 experiments and finding that Justin uses no less CPU or memory than DS2 at the same target rate would contradict the paper's central claim.","tokens_in":12405,"feed_emoji":"⚡","tokens_out":5371,"duration_ms":51247,"temperature":0.7,"pith_summary":"Justin is an auto-scaler for distributed stream processing that treats CPU and memory as separable resources instead of scaling them together. It extends the existing DS2 autoscaler, which adjusts only the number of parallel tasks, by watching RocksDB state-access metrics: cache hit rate and average state-access latency. When an operator is CPU-saturated, Justin decides whether the cheaper fix is more memory per task (scale up) or more tasks (scale out), and it strips managed memory from stateless operators. In Nexmark benchmarks on Apache Flink, Justin supports the same target rates with up to 48% less CPU and 27–28% less memory on the stateful queries Q8 and Q11, in the same or fewer reconfiguration steps.","feed_headline":"Hybrid autoscaler saves 48% CPU on stateful stream queries","feed_subtitle":"Justin scales stream operators up or out separately, matching target rates with far fewer resources than DS2.","key_machinery":"The load-bearing object is the Justin decision rule (Algorithm 1): a set of conditions layered on DS2's busyness-based trigger. For each operator that DS2 wants to scale out, Justin reads two RocksDB-derived metrics, the cache hit rate $\\theta$ and the average state-access latency $\\tau$, and applies two thresholds, $\\Delta_\\theta = 80\\%$ and $\\Delta_\\tau = 1\\,\\mathrm{ms}$, together with a history flag marking whether the last action was a vertical scale-up and a memory level counter that doubles per-task managed memory up to a maximum. The rule cancels the horizontal scale-out and issues a scale-up when memory pressure is indicated, and rolls back a previous scale-up when the metrics did not improve. Stateless operators are given no managed memory. These mechanisms do the work of separating memory-bound operators from CPU-bound ones and of enacting heterogeneous memory allocations through task-slot placement with bin packing.","core_discovery":"The paper's central claim is that a stream-processing auto-scaler can reduce resource consumption substantially by making vertical memory scaling an explicit alternative to horizontal parallelism scaling. The mechanism is a policy layered on DS2: when DS2 proposes to increase an operator's parallelism, Justin checks whether the operator's cache hit rate is below 80% or its average state-access latency exceeds 1 ms; if so, it cancels the scale-out and instead increases the per-task managed memory, provided memory is not already at the maximum level. Subsequent decisions use the history of past scale-ups to keep or roll back that choice. The evaluation reports that on the Nexmark Q8 and Q11 queries this policy reaches target rates with 48% less CPU and 27–28% less memory than DS2 while taking the same or fewer reconfiguration steps, and that on queries with small or absent state it saves memory without harming throughput.","pith_inferences":["Beyond the paper: the fixed 80% cache-hit and 1 ms latency thresholds were selected from experiments on one testbed, so a natural extension is an adaptive policy that adjusts them from observed workload characteristics without retuning.","Beyond the paper: coupling Justin's memory decisions with key-skew rebalancing could handle cases where a few hot keys carry most of the state; the paper notes skew as a limitation but does not solve it.","Beyond the paper: the per-task memory levels could be predicted ahead of reconfiguration (for example, from the operator's working-set size) rather than discovered reactively, reducing the number of scale-up attempts.","Beyond the paper: co-scheduling memory-intensive tasks of one query with stateless tasks of another on the same Task Manager could consolidate memory further, a direction the paper lists as future work."],"forward_implications":["Deployments that currently use DS2 can adopt Justin without lengthening convergence: reported reconfiguration counts are the same or fewer on the tested queries.","Read-heavy stateful operators should receive additional per-task memory before extra parallel tasks, because the cache hit rate and access latency identify cases where scale-out wastes CPU.","Write-dominated or stateless operators should be handled with parallelism only, and stateless operators should run with zero managed memory.","The memory savings are not limited to RAM: less managed memory per task also reduces the memory footprint of each Task Manager, allowing denser packing of tasks on the same nodes.","Queries with large state and frequent reads, such as sliding/session windows and joins, are the main beneficiaries; queries with small state are not penalized."],"supporting_citations":[{"why":"DS2, the CPU-only auto-scaler whose busyness-based scaling decisions Justin extends and compares against.","marker":"[17]"},{"why":"The Flink Kubernetes Operator that hosts the DS2 implementation and the Justin policy and mechanisms.","marker":"[3]"},{"why":"RocksDB, the LSM-based state backend whose cache and MemTable behavior Justin monitors.","marker":"[4]"},{"why":"The RocksDB evolution paper supporting the claim that read and write workloads respond differently to memory.","marker":"[12]"},{"why":"The Nexmark benchmark supplying the Q1–Q11 workloads used to evaluate Justin versus DS2.","marker":"[35]"},{"why":"Prometheus, used to collect the cache hit rate and state-access latency metrics that drive the policy.","marker":"[11]"},{"why":"The LSM storage survey explaining read amplification and cache size effects that motivate memory-aware scaling.","marker":"[21]"}],"fun_headline_variants":["Autoscaler splits CPU and memory scaling for stream queries","Stream autoscaler scales memory instead of CPUs to save 48%","Hybrid autoscaler: scale memory not just parallelism","Justin autoscaler cuts CPU 48% by scaling memory separately","Memory-aware autoscaler for streams saves 48% CPU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The policy's two thresholds — a cache hit rate below 80% or an average state-access latency above 1 ms — were chosen by the authors from their own experimental analysis; if those values do not transfer to other workloads or hardware, Justin will sometimes scale up when scaling out would be better, or miss a profitable scale-up, and the resource savings may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Autoscaler splits CPU and memory scaling for stream queries","Stream autoscaler scales memory instead of CPUs to save 48%","Hybrid autoscaler: scale memory not just parallelism","Justin autoscaler cuts CPU 48% by scaling memory separately","Memory-aware autoscaler for streams saves 48% CPU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2645,"prompt_tokens":901,"completion_tokens":1744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1652}},"tokens_in":517,"tokens_out":1744,"duration_ms":12230,"temperature":1.0,"reasoning_tokens":1652,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:06:27.906754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a read-heavy stateful Nexmark query on the same seven-node testbed with a target rate, then vary the per-task managed memory so that the cache hit rate stays just above 80% while the state working set still forces disk accesses; if Justin fails to scale up and instead scales out, adding CPU without reducing state-access latency, the threshold choice is falsified. More directly, reproducing the Q8/Q11 experiments and finding that Justin uses no less CPU or memory than DS2 at the same target rate would contradict the paper's central claim.","supporting_citations":[{"cited_title":"In: OSDI 2018","cited_arxiv_id":null,"evidence_quote":"DS2, the CPU-only auto-scaler whose busyness-based scaling decisions Justin extends and compares against."},{"cited_title":"https://github.com/apache/ flink-kubernetes-operator (Feb 2025)","cited_arxiv_id":null,"evidence_quote":"The Flink Kubernetes Operator that hosts the DS2 implementation and the Justin policy and mechanisms."},{"cited_title":"https://rocksdb.org/ (Feb 2025)","cited_arxiv_id":null,"evidence_quote":"RocksDB, the LSM-based state backend whose cache and MemTable behavior Justin monitors."},{"cited_title":"ACM Trans","cited_arxiv_id":null,"evidence_quote":"The RocksDB evolution paper supporting the claim that read and write workloads respond differently to memory."},{"cited_title":"Technical report (2008)","cited_arxiv_id":null,"evidence_quote":"The Nexmark benchmark supplying the Q1–Q11 workloads used to evaluate Justin versus DS2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prometheus, used to collect the cache hit rate and state-access latency metrics that drive the policy."},{"cited_title":"The VLDB Journal 29(1), 393–418 (2020) Hybrid CPU/Memory Elastic Scaling for Distributed Stream Processing 17","cited_arxiv_id":null,"evidence_quote":"The LSM storage survey explaining read amplification and cache size effects that motivate memory-aware scaling."}],"review_version":1}