ScaleViz: Scaling Visualization Recommendation Models on Large Data

Ghazi Shazan Ahmad; Manav Doshi; Ryan Rossi; Shubham Agarwal; Subrata Mitra; Syam Manoj Kumar Paila; Vibhor Porwal

REVIEW 4 major objections 3 minor 17 references

ScaleViz: Scaling Visualization Recommendation Models on Large Data

T0 review · 4 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A learning agent can pick the few statistics a visualization recommender truly needs, cutting compute up to 10x.

desk verdict A sensible engineering idea for budgeted feature selection in Vis-Rec, but the headline 10x speedup rests on a Table 3 that contradicts the paper's own cost model and needs major empirical rework before it can be trusted. read the letter →

arxiv 2411.18657 v1 pith:C6OYPZPS submitted 2024-11-27 cs.AI cs.HCstat.ML

Ghazi Shazan Ahmad , Shubham Agarwal , Subrata Mitra , Ryan Rossi , Manav Doshi , Vibhor Porwal , Syam Manoj Kumar Paila This is my paper · ORCID

classification cs.AIcs.HCstat.ML

keywords visualizationrecommendationfeatureselectionreinforcementlearningbudget-awareoptimizationscalabilitystatisticalfeaturestimebudgetlargedatasets

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automated visualization recommenders score many candidate charts by first computing hundreds or thousands of statistical features per column; on large tables this feature computation dominates runtime and can make the models impractical. ScaleViz claims that for a given dataset and a user-supplied time budget, a reinforcement-learning agent can learn which subset of those statistical features matters most to the recommender, so only that subset has to be computed on the full data. On two established recommenders and four real-world datasets, the paper reports up to 10.3x faster recommendation generation with error below 0.0002 for VizML and below 3.43e-5 for ML VR. If correct, it turns a costly one-size-fits-all preprocessing step into a cheap, dataset-specific one without changing the underlying model.

What carries the argument

The load-bearing mechanism is a budget-aware reinforcement-learning feature-acquisition loop. A cost profiler first estimates each statistic's compute time on the full dataset by fitting polynomial regressions to measurements on small samples; zero-cost features (e.g., column type, category count, min/max) are always kept. Then a double deep-Q-network with experience replay, explored by epsilon-greedy action selection, sequentially chooses features; state is the set of features acquired so far, action is the next unmasked feature, and reward is the absolute change in the recommender's predicted score from adding that feature divided by its estimated cost. Training episodes run on successively larger samples, and the trained agent is finally run at inference time to pick features for the full dataset under the user budget.

What would settle it

Take a large dataset and, using only a 1% sample, let ScaleViz choose its feature subset; then compute those features on the full data and compare the recommender's top-k output with the output from all features. The claim fails if the error exceeds the paper's thresholds for datasets where rare categories or outliers disproportionately appear in the full data, or if a rank correlation of per-feature reward between sample and full dataset is low.

Watch

Extended reading notes

Core claim

The paper's central claim is that dataset-specific feature selection, framed as budgeted sequential acquisition, preserves a visualization recommender's output while drastically reducing computation. The agent treats each candidate statistical feature as an item to purchase with a known compute cost, and gives reward proportional to the change the feature causes in the recommender's predicted scores divided by that cost. After training on deliberately growing samples of the target dataset, the agent selects the feature subset for the full dataset. Evaluations with two models and four datasets show speedups between 8.1x and 10.3x versus full-feature computation at fixed error thresholds (0.0002 for VizML, 3.43e-5 for ML VR), and intersection-over-union values of 3-22% across datasets, which the paper reads as evidence that the important features really are dataset-specific.

Load-bearing premise

The feature importance learned on small samples of a dataset remains the feature importance on the full dataset, so features selected under the budget still carry the signal the recommender depends on.

Editorial extensions

If this is right

Users of existing Vis-Rec models can specify a wall-clock budget and get recommendations from the same model at roughly 8-10x lower compute, with prediction error held below the paper's thresholds.
The approach is model-agnostic: it wraps a Vis-Rec model's feature front-end rather than retraining the recommender itself.
Because selected feature sets differ sharply across datasets (IoU 3-22%), a fixed or hand-picked feature subset will not reproduce the speedup; selection must be done per dataset.
Costs grow nonlinearly with dataset size, so the relative saving increases with scale: on the Flights dataset the budget needed for converged recommendations drops from about half of full cost at 1k rows to about one-tenth at 1M rows.
Zero-cost metadata features are always included, so the agent never wastes budget on statistics like column type that carry predictive signal for free.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the small-sample-to-full-scale transfer assumption holds broadly, the same budgeted feature-acquisition loop could be applied to other pipelines whose front-end computes many expensive statistics, such as automated data cleaning or data profiling tools.
The reward definition, change in model score per unit cost, is a general principle; one could test whether using importance measured on a held-out validation set instead of the training score change improves selection stability on very noisy datasets.
The reported IoU values suggest an even stronger claim than the paper makes explicitly: not only is feature importance dataset-dependent, it may also be unstable under small perturbations of the data, which would make any static pruning baseline systematically worse.
A natural stress test would run ScaleViz on datasets where a rare category or high-leverage outlier only appears at scale; if the sample-based agent misses that statistic, errors should spike above the reported thresholds.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a reasonable engineering idea with a broken empirical centerpiece. The approach—budget-aware RL feature selection with cost profiling for Vis-Rec models—is coherent, and the dataset-specific angle is worth taking seriously. But the paper's main evidence for the 10x speedup does not survive contact with its own numbers.

What's genuinely new here is the packaging: using cost profiling to estimate feature costs, then an RL agent to pick a small, dataset-specific feature subset under a time budget, so a base Vis-Rec model can run faster without retraining. The components are all known (double DQN, greedy budgeted selection, polynomial cost models), but the combination for runtime Vis-Rec feature selection is new. The paper also does something useful in showing that selected feature sets have low overlap across datasets (IoU mostly 10-20%), which argues for dataset-specific selection over a one-size-fits-all pruning. The experiments cover two SOTA models and four datasets, and the code is linked.

The soft spots are real, and the biggest one is fatal for the current version. The stress-test note about Table 3 is correct. The Sample baseline computes all features on 1%, 2%, 3%, and 5% uniform samples. Under the paper's own cost model, where full feature computation on Flights/VizML takes 13,259 ms, a 5% sample should be roughly 20x faster, not 1.4x as reported. Either the cost model is wrong, or there is a large fixed overhead that would also eat ScaleViz's claimed 10.3x speedup. The paper gives no per-stage timing to disambiguate. That makes the headline comparison unsupported.

Also missing: no error bars or repeated runs, no sensitivity analysis for the error thresholds (0.0002 and 3.43e-05), and all experiments top out at 1M rows despite talk of much larger datasets. The RL training time is never reported, so the speedup is apparently inference-only; if the agent must be trained per dataset, the amortized cost is a substantial missing piece. The circularity—minimizing divergence from the same model P that defines the reward—is a legitimate design choice, but it means the paper demonstrates "same output faster," not better visualization quality.

Who this is for: people building fast Vis-Rec pipelines for BI tools. They'll get value from the framing and the RL formulation, but they should not trust Table 3 until it's fixed. The paper deserves a serious referee because the problem is practical and the approach is worth engaging with, but it needs major revision: reconcile the Sample baseline with the cost model, add repeated trials and threshold sensitivity, report training time, and test at larger scale.

Recommendation: send to peer review, but with the expectation of substantial revision before publication.

Referee Report

4 major / 3 minor

Summary. The paper proposes ScaleViz, a reinforcement-learning framework that selects a small, dataset-specific subset of statistical features for a given visualization-recommendation (Vis-Rec) model under a user-specified time budget. It profiles feature-computation costs with polynomial regression, trains a DQN-based agent on progressively larger data samples, and at inference computes only the selected features on the full dataset. Experiments with VizML and ML VR on four datasets report speedups up to 10.3x while keeping prediction divergence below thresholds of 0.0002 and 3.43e-05, and low IoU values between selected feature sets across datasets are used to argue that dataset-specific selection is necessary.

Significance. If the empirical claims held, this would be a useful practical contribution: it directly addresses the scalability bottleneck of feature-heavy Vis-Rec models and makes a concrete, falsifiable speedup claim. The explicit problem formulation, the released anonymous code link, and the experimental design of comparing against Random, Greedy, and Sample baselines are strengths. However, the central speedup claim rests on Table 3, whose Sample-baseline numbers are internally inconsistent with the paper's own sampling protocol and cost tables, and on error thresholds that are asserted without sensitivity analysis or repeated runs. These issues must be resolved before the contribution can be considered established.

major comments (4)

[Table 3; Section 5.1; Table 7] Table 3 reports Sample speedups of 1.30x for Flights/VizML and 1.40x for Flights/ML VR, but Section 5.1 defines Sample as computing all features on 1%, 2%, 3%, and 5% uniform samples. Under the paper's own polynomial cost model, a 5% sample of Flights should cost roughly 5% of the full feature-computation time, which Table 7 gives as B_MAX = 13,259 ms for VizML, implying a speedup around 20x rather than 1.30x. If a large fixed overhead explains the low Sample speedups, the same overhead would also penalize ScaleViz, whose Table 7 B' = 1,285 ms is a full-dataset selected-feature cost, making the reported 10.3x hard to reconcile without a per-stage timing breakdown. Please provide separate timings for feature computation, imputation, model scoring, and overhead for every method, and state explicitly what the speedup denominator includes.
[Section 5.2; Figs. 3-4] The headline comparison imposes error thresholds of 0.0002 for VizML and 3.43e-05 for ML VR, but these thresholds are asserted rather than derived, and no sensitivity analysis is given. Furthermore, the paper reports no error bars or repeated runs anywhere; because the RL agent uses epsilon-greedy exploration and experience replay, the speedups in Table 3 and the curves in Figs. 3-4 should be reported as means with variance over multiple random seeds, and the threshold choice should be justified with a sensitivity study over a range of epsilon values.
[Eq. (1); Eq. (3)] Equation (1) defines the objective as minimizing divergence from P(f), the full-feature output of the same model, so the 'error' throughout the paper is fidelity to the base model's predictions, not a direct measure of visualization quality. The paper should state this explicitly and avoid wording that implies an external ground truth for correct recommendations. Relatedly, Eq. (3) rewards each feature by its immediate absolute change in P's score divided by its cost, but no argument or experiment shows that greedily accumulating these local rewards minimizes the global loss in Eq. (1) under a budget; an ablation comparing the learned policy against a policy that directly optimizes final loss under budget, or against a lookahead variant, would substantiate the design.
[Section 4.2; Algorithm 1] The agent is trained only on progressively larger samples and then applied to the full dataset, relying on the assumption that the important features learned on samples remain important at full scale. The paper provides no direct validation of this assumption. Please add an experiment that compares features selected on samples against features selected with access to the full dataset or a substantially larger held-out sample, and report how often the sample-selected feature set achieves error <= epsilon on the full dataset under the same budget.

minor comments (3)

[Table 7] The caption states 'milliseconds for VizML, seconds for ML VR'; please verify and state units consistently per entry, since the 1k-row ML VR entry of 592 seconds appears difficult to reconcile with the reported timing elsewhere and with the sample-based training protocol.
[Figs. 3-4] The y-axis labeled 'normalized error' in Figs. 3-4 is not defined; please specify the normalization procedure and the exact error metric (cross-entropy loss for VizML versus MSE on top-k scores for ML VR) used in each plot.
[Fig. 1] In Fig. 1, the axes are not labeled, so the CDF of feature computation time cannot be read; please label both axes and state how features were bucketed or counted when constructing the CDF.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ScaleViz's objective is explicitly to approximate the target Vis-Rec model P's full-feature output, so minimizing divergence from P is the stated engineering goal, not a hidden fit or self-derivation.

full rationale

The paper's central claim is that, given a fixed Vis-Rec model P, ScaleViz selects a small subset of statistical features so that P's recommendations computed from those features closely match P's recommendations computed from all features, within a user-specified time budget. Equations (1) and (3) define the objective and reward in terms of divergence from P's own full-feature output. This is a distillation-style approximation objective, not circularity: the paper never claims to validate the quality of the recommendations against an independent ground-truth visualization quality measure. The RL agent is trained on samples and then applied to the full dataset, which is a genuine generalization step rather than a by-construction equivalence. The cost profiler extrapolates feature costs using polynomial regression, but that is an auxiliary prediction about compute time, not the target result. The paper's citations to prior work, including ML VR by co-author Ryan Rossi, are used as black-box target models and as related work; no load-bearing argument reduces to an unverified self-citation or to a uniqueness theorem imported from the authors' own prior work. The observed inconsistencies in Table 3 versus the stated sampling protocol are empirical correctness concerns, not circularity. Overall, the derivation chain is self-contained with respect to its stated goal of approximating a given model's output faster.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four domain assumptions about cost growth, transfer of feature importance across data sizes, the validity of the base model's output as ground truth, and imputation. There are also several user-chosen thresholds and hyperparameters that affect the reported speedup. No invented entities are introduced.

free parameters (7)

epsilon (termination threshold)
Algorithm 1 terminates when episode loss is below epsilon, but no value is reported; the choice affects selected features and reported speedup.
VizML acceptable error threshold = 0.0002
Used to define 'insignificant error' and compute speedups in Table 3; no justification or sensitivity analysis is given.
ML VR acceptable error threshold = 3.43e-05
Used for ML VR in the same way; arbitrary without ablation.
Training sample schedule S
The RL agent trains on samples of increasing size, but exact sizes and spacing are not specified; this affects feature selection.
Imputation sample fraction = 0.01%
Non-selected features are imputed from a 0.01% sample; sensitivity to this fraction is not analyzed.
Polynomial degree for cost regression
Section 4.1 says polynomial regression is used, but the degree is not stated; it affects budget allocation.
Q-network architecture = [512, 128, 64]
Fixed without ablation; network capacity may affect learned feature selection.

assumptions (4)

domain assumption Feature computation costs grow polynomially with dataset size.
Section 4.1 invokes this 'as proved in [16]', but that general result is not shown to hold per feature and per dataset.
domain assumption Feature importance learned on small samples transfers to the full dataset.
Section 4.2 says the agent must predict important features on the full dataset it has not been trained on; no direct stability check is provided.
ad hoc to paper The full-feature base model output is the correct ground truth for visualization quality.
Eq. (1) defines the objective as agreement with P(f), so the paper optimizes fidelity to the base model, not human-rated visualization quality.
domain assumption Imputing non-selected features from a 0.01% sample preserves the base model's predictions.
Section 5.1 states this imputation is used, but its effect on recommendation error is not measured.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ScaleViz: Scaling Visualization Recommendation Models on Large Data." pith.science (2026). https://pith.science/paper/C6OYPZPS

@misc{pith2026241118657,
  author       = {Pith},
  title        = {Pith review of: ScaleViz: Scaling Visualization Recommendation Models on Large Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C6OYPZPS}},
  note         = {Machine review of arXiv:2411.18657}
}

read the original abstract

Automated visualization recommendations (vis-rec) help users to derive crucial insights from new datasets. Typically, such automated vis-rec models first calculate a large number of statistics from the datasets and then use machine-learning models to score or classify multiple visualizations choices to recommend the most effective ones, as per the statistics. However, state-of-the art models rely on very large number of expensive statistics and therefore using such models on large datasets become infeasible due to prohibitively large computational time, limiting the effectiveness of such techniques to most real world complex and large datasets. In this paper, we propose a novel reinforcement-learning (RL) based framework that takes a given vis-rec model and a time-budget from the user and identifies the best set of input statistics that would be most effective while generating the visual insights within a given time budget, using the given model. Using two state-of-the-art vis-rec models applied on three large real-world datasets, we show the effectiveness of our technique in significantly reducing time-to visualize with very small amount of introduced error. Our approach is about 10X times faster compared to the baseline approaches that introduce similar amounts of error.

Figures

Figures reproduced from arXiv: 2411.18657 by the authors.

**Figure 1.** CDF of computation time (in seconds) of various features on datasets of different sizes. With increase in dataset size, the computation time of complex features drastically increases. Datasets VizML MLVR Flights (1M, 12) 972 12072 Income (200k, 41) 3321 41246 Housing (20k, 10) 810 10060 Cars (10k, 9) 729 9054 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Pipeline overview: The cost profiler estimates the computational cost for [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Evaluation of VizML on different datasets [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Evaluation of MLVR on different datasets [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 16 canonical work pages

[1]

In: The 2012 Inter- national Joint Conference on Neural Networks (IJCNN)

Deng, H., Runger, G.: Feature selection via regularized trees. In: The 2012 Inter- national Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2012)

work page 2012
[2]

In: ICMD (2019)

Ding, R., Han, S., Xu, Y., Zhang, H., Zhang, D.: Quickinsights: Quick and auto- matic discovery of insights from multi-dimensional data. In: ICMD (2019)

work page 2019
[3]

In: ICDM

Farahat, A.K., Ghodsi, A., Kamel, M.S.: An efficient greedy method for unsuper- vised feature selection. In: ICDM. pp. 161–170. IEEE (2011)

work page 2011
[4]

IEEE TKDE (2016)

Godfrey, P., Gryz, J., Lasek, P.: Interactive visualization of large data sets. IEEE TKDE (2016). https://doi.org/10.1109/TKDE.2016.2557324

work page arXiv 2016
[5]

Insight-centric Visualization Recommendation

Harris, C., Rossi, R.A., Malik, S., Hoffswell, J., Du, F., Lee, T.Y., Koh, E., Zhao, H.: Insight-centric visualization recommendation. arXiv:2103.11297 (2021)

work page Pith review arXiv 2021
[6]

Hu, K., Bakker, M.A., Li, S., Kraska, T., Hidalgo, C.: Vizml: A machine learning approach to visualization recommendation. In: CHI. pp. 1–12 (2019)

work page 2019
[7]

Hulsebos, M., Demiralp, c., Groth, P.: Gittables: A large-scale corpus of relational tables. Proc. ACM Manag. Data (2023)

work page 2023
[8]

In: SIGMOD (2015)

Idreos, S., Papaemmanouil, O., Chaudhuri, S.: Overview of data exploration tech- niques. In: SIGMOD (2015)

work page 2015

Show all 17 references

[9]

arXiv preprint arXiv:1901.00243 (2019)

Kachuee, M., et al., G.: Opportunistic learning: Budgeted cost-sensitive learning from data streams. arXiv preprint arXiv:1901.00243 (2019)

2019 arXiv
[10]

ACM computing surveys (CSUR) (2017)

Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R.P., Tang, J., Liu, H.: Feature selection: A data perspective. ACM computing surveys (CSUR) (2017)

2017
[11]

In: ICDE

Luo, Y., Qin, X., Tang, N., Li, G.: Deepeye: Towards automatic data visualization. In: ICDE. pp. 101–112. IEEE (2018)

2018
[12]

nature (2015)

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, e.a.: Human-level control through deep reinforcement learning. nature (2015)

2015
[13]

KDD ’21, ACM (2021)

Qian, X., Rossi, R.A., Du, F., Kim, S., Koh, E., Malik, S., Lee, T.Y., Chan, J.: Learning to recommend visualizations from data. KDD ’21, ACM (2021)

2021
[14]

CoRR abs/2101.09460 (2021), https://arxiv.org/abs/2101.09460

Sali, R., Adewole, S., Akakpo, A.: Feature selection using reinforcement learning. CoRR abs/2101.09460 (2021), https://arxiv.org/abs/2101.09460

2021 arXiv
[15]

Acm Sigmod Record (2017)

Vartak, M., Huang, S., Siddiqui, T., Madden, S., Parameswaran, A.: Towards vi- sualization recommendation systems. Acm Sigmod Record (2017)

2017
[16]

Statistics and its interface 9(4), 399 (2016)

Wang, C., Chen, M.H., Schifano, E., Wu, J., Yan, J.: Statistical methods and computing for big data. Statistics and its interface 9(4), 399 (2016)

2016
[17]

arXiv preprint arXiv:1206.6451 (2012) 12

Xu, Z., Weinberger, K., Chapelle, O.: The greedy miser: Learning under test-time budgets. arXiv preprint arXiv:1206.6451 (2012) 12

2012 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

In: The 2012 Inter- national Joint Conference on Neural Networks (IJCNN)

Deng, H., Runger, G.: Feature selection via regularized trees. In: The 2012 Inter- national Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2012)

work page 2012

[2] [2]

In: ICMD (2019)

Ding, R., Han, S., Xu, Y., Zhang, H., Zhang, D.: Quickinsights: Quick and auto- matic discovery of insights from multi-dimensional data. In: ICMD (2019)

work page 2019

[3] [3]

In: ICDM

Farahat, A.K., Ghodsi, A., Kamel, M.S.: An efficient greedy method for unsuper- vised feature selection. In: ICDM. pp. 161–170. IEEE (2011)

work page 2011

[4] [4]

IEEE TKDE (2016)

Godfrey, P., Gryz, J., Lasek, P.: Interactive visualization of large data sets. IEEE TKDE (2016). https://doi.org/10.1109/TKDE.2016.2557324

work page arXiv 2016

[5] [5]

Insight-centric Visualization Recommendation

Harris, C., Rossi, R.A., Malik, S., Hoffswell, J., Du, F., Lee, T.Y., Koh, E., Zhao, H.: Insight-centric visualization recommendation. arXiv:2103.11297 (2021)

work page Pith review arXiv 2021

[6] [6]

Hu, K., Bakker, M.A., Li, S., Kraska, T., Hidalgo, C.: Vizml: A machine learning approach to visualization recommendation. In: CHI. pp. 1–12 (2019)

work page 2019

[7] [7]

Hulsebos, M., Demiralp, c., Groth, P.: Gittables: A large-scale corpus of relational tables. Proc. ACM Manag. Data (2023)

work page 2023

[8] [8]

In: SIGMOD (2015)

Idreos, S., Papaemmanouil, O., Chaudhuri, S.: Overview of data exploration tech- niques. In: SIGMOD (2015)

work page 2015