REVIEW 3 major objections 3 minor 1 cited by
Dataset-Agnostic Recommender Systems
T0 review · 3 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read DAReS proposes that a single zero-configuration codebase, guided only by a small dataset-description file, can replace per-dataset configuration in recommender systems.
desk verdict Position paper with a concrete but under-specified metadata schema; the zero-configuration claim is asserted, not demonstrated, but the proposal is coherent and worth a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Dataset Description Language (DsDL) is the mechanism that carries the argument. It is a small grammar, formally defined in EBNF, describing a dataset as a list of features with types plus optional user_id, item_id, timestamp, and label fields. DsDL is meant to give the system enough context to treat all datasets uniformly: the type tags drive encoding and preprocessing choices, the presence of user/item columns selects between task families, and the label field determines the prediction target and evaluation metric.
What would settle it
Run the proposed zero-configuration pipeline on a fixed task across datasets that differ in feature types, sparsity, and size, using only a DsDL file as input, and compare against per-dataset tuned baselines. If any dataset produces unusable predictions or requires hidden dataset-specific fixes, the central zero-configuration claim fails.
Extended reading notes
Core claim
The central claim is that dataset-specific human labor in recommender systems can be replaced by a standardized dataset description. DAReS reads a DsDL file, a YAML/JSON document listing each feature's name and type (categorical, ordinal, numeric, binary, textual, URL), plus optional user ID, item ID, timestamp, and labels, and uses that metadata alone to autonomously perform feature selection, missing-value imputation, feature transformation, noise removal, model selection, hyperparameter tuning, and evaluation. The paper positions this as level-1 automation: dataset-agnostic but still task-specific, with the task (CTR prediction, rating prediction, Top-N) supplied by the user. It argues this eliminates dataset-specific configurations and improves reusability and reproducibility compared to traditional pipelines, while acknowledging a trade-off: less deep customization for specialized datasets and higher computational cost.
Load-bearing premise
The entire proposal rests on the assumption that a short list of column names and types is enough information for an automated system to make all preprocessing, feature-engineering, model-selection, and hyperparameter decisions correctly, without unacceptable performance loss.
Editorial extensions
If this is right
- The same DAReS codebase could be applied to a new dataset by writing only a short DsDL file, removing per-dataset code changes.
- Non-expert or entry-level users could build recommendation models without domain-specific feature engineering or tuning expertise.
- Standardized DsDL descriptions would make experiments easier to reproduce, since dataset treatment is no longer buried in undocumented code.
- For highly specialized datasets, DAReS would likely underperform hand-tuned pipelines, and the automated search adds computational cost.
- The level-2 extension would let the system infer the task itself from the dataset, removing even the task-specific assumption.
Reading between the lines
- The untested core is whether column-name and type metadata alone is enough to make good model-selection decisions; the most direct way to test that is a fixed task run across many datasets with only DsDL as input.
- One could learn the metadata-to-configuration mapping across a corpus of datasets, turning the position paper's proposal into a meta-learning system; the paper does not claim this is done.
- The same DsDL schema could be extended with dataset statistics (sparsity, cardinality, skew) that would make the zero-configuration promise more realistic; that extension is not in the paper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new paradigm, Dataset-Agnostic Recommender Systems (DAReS), and a metadata format, Dataset Description Language (DsDL), intended to let a single codebase adapt to arbitrary recommender-system datasets without dataset-specific configuration. DsDL describes column names, coarse feature types, optional user/item/timestamp fields, and labels. The paper claims that this metadata enables autonomous feature engineering, missing-value imputation, feature transformation, noise removal, model selection, and hyperparameter tuning. It also introduces level-1 (dataset-agnostic, task-specific) and level-2 (dataset- and task-agnostic) automation levels. The manuscript is a position paper: it contains no implementation, experiments, benchmarks, or formal correctness arguments.
Significance. If the proposed system worked as claimed, it would be a useful contribution to recommender-system reproducibility, reusability, and accessibility for non-experts. The DsDL grammar is a concrete, citable artifact, and the level-1/level-2 distinction is a reasonable conceptual framing. However, the paper does not establish that DsDL is sufficient to drive the preprocessing, model-selection, and hyperparameter decisions it describes; the core capability claim remains a vision rather than a demonstrated result. There are no machine-checked proofs, reproducible code, or benchmark results to verify the central claim.
major comments (3)
- [Abstract, Section 1, and Listing 1] The central claim that DsDL metadata is sufficient for zero-configuration autonomy is unsupported. Listing 1 records only column names and coarse types (categorical, ordinal, numeric, binary, textual, url), plus optional user/item/timestamp/label fields. It does not record missingness rates, cardinalities, skew, distributions, feature interactions, or dataset size, all of which materially affect preprocessing and model choice. Consequently, datasets with identical DsDL descriptions can require different pipelines, and the paper gives no rule or learned mapping from DsDL to the decisions described in Sections 2.2 and 2.3. Section 2.1 itself says the schema may need extension. Without a mechanism or evidence, the abstract's and Section 1's assertions that DAReS eliminates the need for dataset-specific configurations are not established.
- [Sections 2.2 and 2.3, Table 1] The manuscript contradicts its own central claim. Sections 2.2 and 2.3 state that autonomous feature engineering, model selection, and hyperparameter tuning are optional, saying it is still possible to create a functioning DAReS without these capabilities, albeit with potentially reduced performance. The abstract and Section 1 promise automatic, zero-configuration operation. Table 1 further concedes suboptimal performance for DAReS due to the generalization-specialization trade-off. The paper needs to either scope the claim to the fully automated configuration and justify that configuration, or weaken the abstract and Section 1 accordingly.
- [Section 2.4 and Section 4] No empirical validation, implementation, or falsifiable prediction is provided. The paper describes what DAReS may do and lists evaluation metrics, but it does not run any experiment on diverse datasets, does not provide code, and does not quantify the acknowledged performance and overhead trade-offs. Because the central contribution is an empirical capability claim, a proof-of-concept on even a small set of standard datasets, or a concrete design with testable predictions, is necessary to support it. The future-work statement in Section 4 cannot substitute for this.
minor comments (3)
- [Abstract] The abstract contains stray spacing in 'personali zed' and 'significan t'; please proofread the abstract.
- [Section 2.1] The sentence 'can be derived from other available features and.' appears to be missing a continuation after 'and'; in addition, the EBNF terminal 'String ::= < any string >' is informal and should be defined more precisely if the grammar is meant to be formally checkable.
- [Section 3.1] The statement that AutoML approaches 'still require dataset-specific configurations and adaptations' is too broad and is not supported by the cited survey [8]; consider distinguishing DAReS from general AutoML more carefully.
Circularity Check
No circularity: DAReS is a position-paper proposal with no derivation chain; its capability claims are unvalidated rather than circular.
full rationale
The paper contains no equations, fitted parameters, or empirical predictions that reduce to its own inputs. DsDL is introduced as a new metadata schema via an EBNF grammar and a YAML example; it is not derived from any claimed output, nor is any result claimed to follow mathematically from it. The central statements about autonomous feature engineering, model selection, and hyperparameter tuning are phrased as possibilities ('may autonomously handle') and the paper explicitly concedes that a functioning DAReS is possible without these capabilities, albeit with reduced performance. Self-citations appear only as background references (e.g., RBoard, federated learning for CTR prediction, multi-modal collaborative filtering) and are not load-bearing for the paper's proposal. The absence of experiments or external benchmarks is an evidentiary gap, not circular reasoning. Under the provided scoring rubric, this is a non-finding: score 0.
Assumptions & free parameters
assumptions (4)
- domain assumption DsDL metadata is sufficient to drive all preprocessing, feature engineering, model selection, and hyperparameter decisions.
- domain assumption Zero-configuration adaptation does not incur unacceptable performance loss across datasets.
- domain assumption Existing AutoML and feature engineering techniques can be integrated behind DsDL without dataset-specific reconfiguration.
- ad hoc to paper The DsDL schema is complete enough for all recommender tasks.
invented entities (2)
-
DAReS (Dataset-Agnostic Recommender System)
-
DsDL (Dataset Description Language)
Cite this review
Pith. "Pith review of Dataset-Agnostic Recommender Systems." pith.science (2026). https://pith.science/paper/AYNGESKX
@misc{pith2026250107294,
author = {Pith},
title = {Pith review of: Dataset-Agnostic Recommender Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/AYNGESKX}},
note = {Machine review of arXiv:2501.07294}
}
read the original abstract
Recommender systems have become a cornerstone of personalized user experiences, yet their development typically involves significant manual intervention, including dataset-specific feature engineering, hyperparameter tuning, and configuration. To this end, we introduce a novel paradigm: Dataset-Agnostic Recommender Systems (DAReS) that aims to enable a single codebase to autonomously adapt to various datasets without the need for fine-tuning, for a given recommender system task. Central to this approach is the Dataset Description Language (DsDL), a structured format that provides metadata about the dataset's features and labels, and allow the system to understand dataset's characteristics, allowing it to autonomously manage processes like feature selection, missing values imputation, noise removal, and hyperparameter optimization. By reducing the need for domain-specific expertise and manual adjustments, DAReS offers a more efficient and scalable solution for building recommender systems across diverse application domains. It addresses critical challenges in the field, such as reusability, reproducibility, and accessibility for non-expert users or entry-level researchers.
Forward citations
Cited by 1 Pith paper
-
Universal Reusability in Recommender Systems: The Case for Dataset- and Task-Independent Frameworks
The paper proposes a dataset- and task-independent recommender framework (DTIRS) with a schema language (DsDL) and an automation-level roadmap, but provides no implementation or experiments.
Reference graph
Works this paper leans on
-
[1]
Language models are few-shot learners
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020
arXiv 2005
-
[2]
Secon d workshop on information hetero- geneity and fusion in recommender systems (hetrec2011)
Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik. Secon d workshop on information hetero- geneity and fusion in recommender systems (hetrec2011). In Proceedings of the fifth ACM conference on Recommender systems , pages 387–388, 2011
work page 2011
-
[3]
Item-based top-n r ecommendation algorithms
Mukund Deshpande and George Karypis. Item-based top-n r ecommendation algorithms. ACM Transactions on Information Systems (TOIS) , 22(1):143–177, 2004
work page 2004
-
[4]
Neu ral architecture search: A survey
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neu ral architecture search: A survey. Journal of Machine Learning Research , 20(55):1–21, 2019
work page 2019
-
[5]
Recommenders in the wild - pr actical evaluation methods
Kim Falk and Morten Arngren. Recommenders in the wild - pr actical evaluation methods. In Proceedings of the 17th ACM Conference on Recommender Syste ms, RecSys ’23, page 1, New Y ork, NY , USA, 2023. Association for Computing Machinery
work page 2023
-
[6]
A trou- bling analysis of reproducibility and progress in recommen der systems research
Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremone si, and Dietmar Jannach. A trou- bling analysis of reproducibility and progress in recommen der systems research. ACM Trans. Inf. Syst., 39(2), January 2021. 5
work page 2021
-
[7]
Are we really making much progress? a worrying analysis of recent neural recomme ndation approaches
Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much progress? a worrying analysis of recent neural recomme ndation approaches. In Proceed- ings of the 13th ACM Conference on Recommender Systems , RecSys ’19, page 101–109, New Y ork, NY , USA, 2019. Association for Computing Machinery
work page 2019
-
[8]
Automl: A survey of the state-of-the-art
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. Knowledge-based systems, 212:106622, 2021
2021
Show all 29 references
-
[9]
The effect of third pa rty implementations on repro- ducibility
Balázs Hidasi and Ádám Tibor Czapp. The effect of third pa rty implementations on repro- ducibility. In Proceedings of the 17th ACM Conference on Recommender Syste ms, pages 272– 282, 2023
2023
-
[10]
Erase: Benchmarking fe ature selection methods for deep recommender systems
Pengyue Jia, Y ejing Wang, Zhaocheng Du, Xiangyu Zhao, Y ichao Wang, Bo Chen, Wanyu Wang, Huifeng Guo, and Ruiming Tang. Erase: Benchmarking fe ature selection methods for deep recommender systems. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Dat...
-
[11]
Which is more effective in label noise cleaning, correction or filtering? In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 12866–12873, 2024
Gaoxia Jiang, Jia Zhang, Xuefei Bai, Wenjian Wang, and D eyu Meng. Which is more effective in label noise cleaning, correction or filtering? In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 12866–12873, 2024
2024
-
[12]
P osition: A call to action for a human-centered AutoML paradigm
Marius Lindauer, Florian Karl, Anne Klier, Julia Moosb auer, Alexander Tornede, Andreas C Mueller, Frank Hutter, Matthias Feurer, and Bernd Bischl. P osition: A call to action for a human-centered AutoML paradigm. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian ...
2024
-
[13]
Feature transformation an d subset selection
Huan Liu and Hiroshi Motoda. Feature transformation an d subset selection. IEEE Intell Syst Their Appl, 13(2):26–28, 1998
1998
-
[14]
Online Meta-Learning for Model Update Aggregation in Federated Learning for Clic k-Through Rate Prediction
Xianghang Liu, Bartłomiej Twardowski, and Tri Kurniaw an Wijaya. Online Meta-Learning for Model Update Aggregation in Federated Learning for Clic k-Through Rate Prediction. 28th ACM SIGKDD 2022 W orkshop on AdKDD, 2022
2022
-
[15]
Exploring the limits of tr ansfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Le e, Sharan Narang, Michael Matena, Y anqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of tr ansfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020
2020
-
[16]
Intr oduction to recommender systems handbook
Francesco Ricci, Lior Rokach, and Bracha Shapira. Intr oduction to recommender systems handbook. In Recommender systems handbook , pages 1–35. Springer, 2010
2010
-
[17]
Handling m issing values when applying classifi- cation models
Maytal Saar-Tsechansky and Foster Provost. Handling m issing values when applying classifi- cation models. 2007
2007
-
[18]
Selecting a classification method by c ross-validation
Cullen Schaffer. Selecting a classification method by c ross-validation. Machine learning , 13:135–143, 1993
1993
-
[19]
Generic base standards
Roger S Scowen. Generic base standards. In Proceedings 1993 Software Engineering Stan- dards Symposium, pages 25–34. IEEE, 1993
1993
-
[20]
RBoard: A Unified Platform for Reproducible and Reusable Recommender System Benchmarks
Xinyang Shao, Edoardo D’Amico, Gábor Fodor, and Tri Kur niawan Wijaya. RBoard: A Unified Platform for Reproducible and Reusable Recommender System Benchmarks. CoRR, abs/2409.05526, 2024
2024 arXiv
-
[21]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, F aisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023
2023 arXiv
-
[22]
Advancing automation of design decision s in recommender system pipelines
Tobias V ente. Advancing automation of design decision s in recommender system pipelines. In Proceedings of the 17th ACM Conference on Recommender Syste ms, RecSys ’23, page 1355–1360, New Y ork, NY , USA, 2023. Association for Computing Machinery
2023
-
[23]
Toward e fficient automated feature engineering
Kafeng Wang, Pengyang Wang, and Chengzhong Xu. Toward e fficient automated feature engineering. In 2023 IEEE 39th International Conference on Data Engineerin g (ICDE), pages 1625–1637. IEEE, 2023
2023
-
[24]
Au- torec: An automated recommender system
Ting-Hsiang Wang, Xia Hu, Haifeng Jin, Qingquan Song, X iaotian Han, and Zirui Liu. Au- torec: An automated recommender system. In Proceedings of the 14th ACM Conference on 6 Recommender Systems , RecSys ’20, page 582–584, New Y ork, NY , USA, 2020. Associat ion for Computing ...
2020
-
[25]
MM- GEF: Multi-modal representation meet collaborative filter ing
Hao Wu, Alejandro Ariza-Casabona, Bartlomiej Twardow ski, and Tri Kurniawan Wijaya. MM- GEF: Multi-modal representation meet collaborative filter ing. CoRR, abs/2308.07222, 2023
2023 arXiv
-
[26]
On hyperparameter optimization of machine learning algorithms: Theory and practice
Li Y ang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415:295–316, 2020
2020
-
[27]
Click-through rate predicti on in online advertising: A literature review
Y anwu Y ang and Panyu Zhai. Click-through rate predicti on in online advertising: A literature review. Information Processing & Management , 59(2):102853, 2022
2022
-
[28]
Openfe: automated feature generation with expert-level performance
Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Lu o, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: automated feature generation with expert-level performance. In International Conference on Machine Learning , pages 41880–41901. PMLR, 2023. 7
2023
-
[2024]
Association for Computing Machinery
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.