{"id":"9205cd61-bef7-4c67-a6a1-19af73e7a113","arxiv_id":"2411.18657","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"ScaleViz uses reinforcement learning to pick the cheapest, most influential data statistics for a visualization recommendation model, achieving up to 10.3x speedup with small divergence from the full-feature model.","lead":"ScaleViz is a system that chooses a small, dataset-specific set of statistical features for visualization recommendation models, cutting computation time by up to about 10x while keeping recommendations close to the original model's output. A reader might care because automated chart recommendation is currently too slow on large tables, and this is a practical attempt to make it faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 Sample-baseline speedups are irreconcilable with the stated 1–5% sampling and with Table 7's own timings, making the headline ~10x speedup comparison unsupported.","rationale":"The reader's weakest assumption about sample-to-full generalization is reasonable, but it is at least directly exercised by the full-data evaluations in Table 3 and Table 7 (up to 1M rows), so it is not the most immediate threat. The sharper, more load-bearing concern is the internal arithmetic of the timing comparison itself. If the Sample baseline row in Table 3 is wrong, the comparative claim 'about 10X faster compared to baseline approaches' is unsupported. If it is right, the only consistent explanations create a different problem for ScaleViz's absolute speedup or for the cost model that the whole method depends on. The paper gives no timing breakdown, so the reader cannot resolve this. This is an internal consistency issue, not a disagreement with consensus or an attack on the authors. A corrected timing re-run could confirm the claims or invalidate them, so a conditional verdict requiring that re-run is the appropriate outcome.","tokens_in":8278,"tokens_out":12192,"duration_ms":113293,"concrete_test":"Re-run the Flights rows of Table 3 for both VizML and ML VR using exactly the stated 1%, 2%, 3%, and 5% sample fractions for the Sample baseline, recording separate wall-clock times for feature computation, model inference, and any imputation step, and compare against Table 7's B_MAX values. Also record the actual error achieved by each method at each budget. If a 5% sample yields a speedup near 20x, the Table 3 Sample entries are wrong; if it yields ~1.3x, identify the dominant non-scalable cost and show that ScaleViz still achieves ~10x after paying that same cost. This check is decisive because it determines whether the reported speedup comparison is arithmetically possible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is purely empirical, and its evidence is Table 3. That table cannot be reconciled with the paper's own protocol and timings. Section 5.1 says the Sample baseline computes all features on 1%, 2%, 3%, and 5% uniform samples. Under the paper's own cost model (Section 4.1 assumes polynomial cost growth with rows; Table 7 gives B_MAX on Flights/VizML as 13,259 ms), a 5% sample should take roughly 5% of full feature-computation time, giving a speedup of about 20x, and a 1% sample about 100x. Yet Table 3 reports Sample speedups of only 1.30x for Flights/VizML and 1.40x for Flights/ML VR. The only ways to explain this are (a) a large fixed overhead that does not scale with rows, but then ScaleViz's own 10.3x speedup is impossible because it would pay the same overhead on top of its B'=1,285 ms from Table 7; or (b) feature costs do not scale as assumed, which undermines the cost-profiler extrapolation that defines the budgets. The paper provides no per-stage timing breakdown to distinguish these. Additionally, it is unclear how a method capped at 5% sampling can satisfy the very tight error thresholds (0.0002 for VizML, 3.43e-05 for ML VR) imposed on all methods in Table 3. Because Table 3 is the direct support for the headline speedup claim, the empirical comparison is currently untrustworthy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ScaleViz, a reinforcement-learning framework that selects a small, dataset-specific subset of statistical features for a given visualization-recommendation (Vis-Rec) model under a user-specified time budget. It profiles feature-computation costs with polynomial regression, trains a DQN-based agent on progressively larger data samples, and at inference computes only the selected features on the full dataset. Experiments with VizML and ML VR on four datasets report speedups up to 10.3x while keeping prediction divergence below thresholds of 0.0002 and 3.43e-05, and low IoU values between selected feature sets across datasets are used to argue that dataset-specific selection is necessary.","tokens_in":8620,"tokens_out":4922,"duration_ms":46293,"significance":"If the empirical claims held, this would be a useful practical contribution: it directly addresses the scalability bottleneck of feature-heavy Vis-Rec models and makes a concrete, falsifiable speedup claim. The explicit problem formulation, the released anonymous code link, and the experimental design of comparing against Random, Greedy, and Sample baselines are strengths. However, the central speedup claim rests on Table 3, whose Sample-baseline numbers are internally inconsistent with the paper's own sampling protocol and cost tables, and on error thresholds that are asserted without sensitivity analysis or repeated runs. These issues must be resolved before the contribution can be considered established.","major_comments":[{"comment":"Table 3 reports Sample speedups of 1.30x for Flights/VizML and 1.40x for Flights/ML VR, but Section 5.1 defines Sample as computing all features on 1%, 2%, 3%, and 5% uniform samples. Under the paper's own polynomial cost model, a 5% sample of Flights should cost roughly 5% of the full feature-computation time, which Table 7 gives as B_MAX = 13,259 ms for VizML, implying a speedup around 20x rather than 1.30x. If a large fixed overhead explains the low Sample speedups, the same overhead would also penalize ScaleViz, whose Table 7 B' = 1,285 ms is a full-dataset selected-feature cost, making the reported 10.3x hard to reconcile without a per-stage timing breakdown. Please provide separate timings for feature computation, imputation, model scoring, and overhead for every method, and state explicitly what the speedup denominator includes.","section":"Table 3; Section 5.1; Table 7"},{"comment":"The headline comparison imposes error thresholds of 0.0002 for VizML and 3.43e-05 for ML VR, but these thresholds are asserted rather than derived, and no sensitivity analysis is given. Furthermore, the paper reports no error bars or repeated runs anywhere; because the RL agent uses epsilon-greedy exploration and experience replay, the speedups in Table 3 and the curves in Figs. 3-4 should be reported as means with variance over multiple random seeds, and the threshold choice should be justified with a sensitivity study over a range of epsilon values.","section":"Section 5.2; Figs. 3-4"},{"comment":"Equation (1) defines the objective as minimizing divergence from P(f), the full-feature output of the same model, so the 'error' throughout the paper is fidelity to the base model's predictions, not a direct measure of visualization quality. The paper should state this explicitly and avoid wording that implies an external ground truth for correct recommendations. Relatedly, Eq. (3) rewards each feature by its immediate absolute change in P's score divided by its cost, but no argument or experiment shows that greedily accumulating these local rewards minimizes the global loss in Eq. (1) under a budget; an ablation comparing the learned policy against a policy that directly optimizes final loss under budget, or against a lookahead variant, would substantiate the design.","section":"Eq. (1); Eq. (3)"},{"comment":"The agent is trained only on progressively larger samples and then applied to the full dataset, relying on the assumption that the important features learned on samples remain important at full scale. The paper provides no direct validation of this assumption. Please add an experiment that compares features selected on samples against features selected with access to the full dataset or a substantially larger held-out sample, and report how often the sample-selected feature set achieves error <= epsilon on the full dataset under the same budget.","section":"Section 4.2; Algorithm 1"}],"minor_comments":[{"comment":"The caption states 'milliseconds for VizML, seconds for ML VR'; please verify and state units consistently per entry, since the 1k-row ML VR entry of 592 seconds appears difficult to reconcile with the reported timing elsewhere and with the sample-based training protocol.","section":"Table 7"},{"comment":"The y-axis labeled 'normalized error' in Figs. 3-4 is not defined; please specify the normalization procedure and the exact error metric (cross-entropy loss for VizML versus MSE on top-k scores for ML VR) used in each plot.","section":"Figs. 3-4"},{"comment":"In Fig. 1, the axes are not labeled, so the CDF of feature computation time cannot be read; please label both axes and state how features were bucketed or counted when constructing the CDF.","section":"Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the internal inconsistency of Table 3 with the stated Sample-baseline protocol and the cost model. I expect the authors can fix this by reporting per-stage timings and rerunning with repeated seeds, but until then the headline 10x speedup claim should not be taken at face value. The paper's contribution is potentially useful, so I would prefer revision over rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a reasonable engineering idea with a broken empirical centerpiece. The approach—budget-aware RL feature selection with cost profiling for Vis-Rec models—is coherent, and the dataset-specific angle is worth taking seriously. But the paper's main evidence for the 10x speedup does not survive contact with its own numbers.\n\nWhat's genuinely new here is the packaging: using cost profiling to estimate feature costs, then an RL agent to pick a small, dataset-specific feature subset under a time budget, so a base Vis-Rec model can run faster without retraining. The components are all known (double DQN, greedy budgeted selection, polynomial cost models), but the combination for runtime Vis-Rec feature selection is new. The paper also does something useful in showing that selected feature sets have low overlap across datasets (IoU mostly 10-20%), which argues for dataset-specific selection over a one-size-fits-all pruning. The experiments cover two SOTA models and four datasets, and the code is linked.\n\nThe soft spots are real, and the biggest one is fatal for the current version. The stress-test note about Table 3 is correct. The Sample baseline computes all features on 1%, 2%, 3%, and 5% uniform samples. Under the paper's own cost model, where full feature computation on Flights/VizML takes 13,259 ms, a 5% sample should be roughly 20x faster, not 1.4x as reported. Either the cost model is wrong, or there is a large fixed overhead that would also eat ScaleViz's claimed 10.3x speedup. The paper gives no per-stage timing to disambiguate. That makes the headline comparison unsupported.\n\nAlso missing: no error bars or repeated runs, no sensitivity analysis for the error thresholds (0.0002 and 3.43e-05), and all experiments top out at 1M rows despite talk of much larger datasets. The RL training time is never reported, so the speedup is apparently inference-only; if the agent must be trained per dataset, the amortized cost is a substantial missing piece. The circularity—minimizing divergence from the same model P that defines the reward—is a legitimate design choice, but it means the paper demonstrates \"same output faster,\" not better visualization quality.\n\nWho this is for: people building fast Vis-Rec pipelines for BI tools. They'll get value from the framing and the RL formulation, but they should not trust Table 3 until it's fixed. The paper deserves a serious referee because the problem is practical and the approach is worth engaging with, but it needs major revision: reconcile the Sample baseline with the cost model, add repeated trials and threshold sensitivity, report training time, and test at larger scale.\n\nRecommendation: send to peer review, but with the expectation of substantial revision before publication.","headline":"A sensible engineering idea for budgeted feature selection in Vis-Rec, but the headline 10x speedup rests on a Table 3 that contradicts the paper's own cost model and needs major empirical rework before it can be trusted.","tokens_in":9206,"tokens_out":2970,"would_cite":false,"duration_ms":28778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learning agent can pick the few statistics a visualization recommender truly needs, cutting compute up to 10x.","keywords":["visualization recommendation","feature selection","reinforcement learning","budget-aware optimization","scalability","statistical features","time budget","large datasets"],"falsifier":"Take a large dataset and, using only a 1% sample, let ScaleViz choose its feature subset; then compute those features on the full data and compare the recommender's top-k output with the output from all features. The claim fails if the error exceeds the paper's thresholds for datasets where rare categories or outliers disproportionately appear in the full data, or if a rank correlation of per-feature reward between sample and full dataset is low.","tokens_in":8044,"feed_emoji":"📊","tokens_out":6053,"duration_ms":49892,"temperature":0.7,"pith_summary":"Automated visualization recommenders score many candidate charts by first computing hundreds or thousands of statistical features per column; on large tables this feature computation dominates runtime and can make the models impractical. ScaleViz claims that for a given dataset and a user-supplied time budget, a reinforcement-learning agent can learn which subset of those statistical features matters most to the recommender, so only that subset has to be computed on the full data. On two established recommenders and four real-world datasets, the paper reports up to 10.3x faster recommendation generation with error below 0.0002 for VizML and below 3.43e-5 for ML VR. If correct, it turns a costly one-size-fits-all preprocessing step into a cheap, dataset-specific one without changing the underlying model.","feed_headline":"Agent trims chart-recommender statistics for 10x speedup","feed_subtitle":"ScaleViz selects a dataset-specific subset of statistical features, preserving recommendation accuracy while cutting computation.","key_machinery":"The load-bearing mechanism is a budget-aware reinforcement-learning feature-acquisition loop. A cost profiler first estimates each statistic's compute time on the full dataset by fitting polynomial regressions to measurements on small samples; zero-cost features (e.g., column type, category count, min/max) are always kept. Then a double deep-Q-network with experience replay, explored by epsilon-greedy action selection, sequentially chooses features; state is the set of features acquired so far, action is the next unmasked feature, and reward is the absolute change in the recommender's predicted score from adding that feature divided by its estimated cost. Training episodes run on successively larger samples, and the trained agent is finally run at inference time to pick features for the full dataset under the user budget.","core_discovery":"The paper's central claim is that dataset-specific feature selection, framed as budgeted sequential acquisition, preserves a visualization recommender's output while drastically reducing computation. The agent treats each candidate statistical feature as an item to purchase with a known compute cost, and gives reward proportional to the change the feature causes in the recommender's predicted scores divided by that cost. After training on deliberately growing samples of the target dataset, the agent selects the feature subset for the full dataset. Evaluations with two models and four datasets show speedups between 8.1x and 10.3x versus full-feature computation at fixed error thresholds (0.0002 for VizML, 3.43e-5 for ML VR), and intersection-over-union values of 3-22% across datasets, which the paper reads as evidence that the important features really are dataset-specific.","pith_inferences":["If the small-sample-to-full-scale transfer assumption holds broadly, the same budgeted feature-acquisition loop could be applied to other pipelines whose front-end computes many expensive statistics, such as automated data cleaning or data profiling tools.","The reward definition, change in model score per unit cost, is a general principle; one could test whether using importance measured on a held-out validation set instead of the training score change improves selection stability on very noisy datasets.","The reported IoU values suggest an even stronger claim than the paper makes explicitly: not only is feature importance dataset-dependent, it may also be unstable under small perturbations of the data, which would make any static pruning baseline systematically worse.","A natural stress test would run ScaleViz on datasets where a rare category or high-leverage outlier only appears at scale; if the sample-based agent misses that statistic, errors should spike above the reported thresholds."],"forward_implications":["Users of existing Vis-Rec models can specify a wall-clock budget and get recommendations from the same model at roughly 8-10x lower compute, with prediction error held below the paper's thresholds.","The approach is model-agnostic: it wraps a Vis-Rec model's feature front-end rather than retraining the recommender itself.","Because selected feature sets differ sharply across datasets (IoU 3-22%), a fixed or hand-picked feature subset will not reproduce the speedup; selection must be done per dataset.","Costs grow nonlinearly with dataset size, so the relative saving increases with scale: on the Flights dataset the budget needed for converged recommendations drops from about half of full cost at 1k rows to about one-tenth at 1M rows.","Zero-cost metadata features are always included, so the agent never wastes budget on statistics like column type that carry predictive signal for free."],"supporting_citations":[{"why":"Target recommender; establishes the 81 column-level features and encoding/visualization prediction tasks that ScaleViz prunes.","marker":"[6]"},{"why":"Target recommender; establishes the 1006 per-column statistical features whose computational cost motivates the budgeted feature selection.","marker":"[13]"},{"why":"Supplies the double deep Q-learning with experience replay used to train the feature-acquisition agent.","marker":"[12]"},{"why":"Justifies polynomial regression for extrapolating feature computation cost from samples to full dataset size.","marker":"[16]"},{"why":"Baseline greedy budgeted feature selection is inspired by this method and compared against ScaleViz.","marker":"[17]"}],"fun_headline_variants":["Reinforcement learning trims chart-recommender stats for 10x speedup","RL selects only the statistics that matter for chart recommendations","Agent cuts chart-reco compute 10x by picking dataset-specific stats","ScaleViz: RL selects key stats for 10x faster chart recommendations","Learn which stats matter: RL agent speeds chart recommendations 10x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The feature importance learned on small samples of a dataset remains the feature importance on the full dataset, so features selected under the budget still carry the signal the recommender depends on.","fun_headline_variants_meta":{"raw":{"variants":["Reinforcement learning trims chart-recommender stats for 10x speedup","RL selects only the statistics that matter for chart recommendations","Agent cuts chart-reco compute 10x by picking dataset-specific stats","ScaleViz: RL selects key stats for 10x faster chart recommendations","Learn which stats matter: RL agent speeds chart recommendations 10x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3300,"prompt_tokens":912,"completion_tokens":2388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":2293}},"tokens_in":528,"tokens_out":2388,"duration_ms":16249,"temperature":1.0,"reasoning_tokens":2293,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:28:46.443648+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a large dataset and, using only a 1% sample, let ScaleViz choose its feature subset; then compute those features on the full data and compare the recommender's top-k output with the output from all features. The claim fails if the error exceeds the paper's thresholds for datasets where rare categories or outliers disproportionately appear in the full data, or if a rank correlation of per-feature reward between sample and full dataset is low.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Target recommender; establishes the 81 column-level features and encoding/visualization prediction tasks that ScaleViz prunes."},{"cited_title":"KDD ’21, ACM (2021)","cited_arxiv_id":null,"evidence_quote":"Target recommender; establishes the 1006 per-column statistical features whose computational cost motivates the budgeted feature selection."},{"cited_title":"nature (2015)","cited_arxiv_id":null,"evidence_quote":"Supplies the double deep Q-learning with experience replay used to train the feature-acquisition agent."},{"cited_title":"Statistics and its interface 9(4), 399 (2016)","cited_arxiv_id":null,"evidence_quote":"Justifies polynomial regression for extrapolating feature computation cost from samples to full dataset size."},{"cited_title":"The Greedy Miser: Learning under Test-time Budgets","cited_arxiv_id":"1206.6451","evidence_quote":"Baseline greedy budgeted feature selection is inspired by this method and compared against ScaleViz."}],"review_version":1}