{"id":"dcb686f1-fa9e-4d5b-b976-ef81351a933d","arxiv_id":"2501.07294","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"DAReS is a proposed zero-configuration recommender system framework that uses a Dataset Description Language (DsDL) metadata schema to automate feature engineering, model selection, and hyperparameter tuning.","lead":"This paper proposes DAReS, a framework meant to run the same recommender system code on any dataset by reading a metadata description called DsDL. It is a position paper with no implementation, experiments, or benchmarks, so its central capability claim is unverified.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central zero-configuration claim is an unsupported empirical capability claim: DsDL metadata in Listing 1 is too coarse to determine preprocessing and model decisions, and the paper provides no experiment or mechanism showing that it can.","rationale":"I agree with the reader's weakest_assumption: the load-bearing premise is that DsDL metadata is sufficient to drive correct autonomous decisions, and this premise is unsupported. My stress-test adds concreteness by pointing to the gap between the schema's coarse type labels and the decisions required by Sections 2.2 and 2.3, and by noting the paper's own admission of potential performance loss. The paper is a coherent position paper, and its proposed DsDL syntax is clear, but the central capability claim is empirical and no evidence is supplied. Under the claim-support rubric, REJECT is the appropriate verdict because the abstract's promise of a single codebase that autonomously adapts without fine-tuning is not established. I see no internal contradiction that would justify a different technical objection; the issue is missing evidence for a specific, testable capability, which is exactly what a benchmark would supply.","tokens_in":6483,"tokens_out":3010,"duration_ms":31665,"concrete_test":"Construct a minimal DAReS implementation faithful to Sections 2.1-2.4 for one task, such as CTR prediction, with DsDL files as the only dataset-specific input. Run the identical unmodified code on two public datasets, for example Criteo and Avazu, and compare against dataset-specific tuned baselines using the same metrics. If either dataset requires manual changes to the code or configuration, or if the performance gap to tuned baselines exceeds a pre-registered tolerance, the claim that DsDL enables zero-configuration competitive adaptation fails. A negative or inconclusive result would support the reader's REJECT verdict.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DAReS's central assertion — in the abstract and Section 1 — is that one codebase, given only DsDL, can autonomously perform feature engineering, model selection, and hyperparameter tuning without fine-tuning. For this to hold, the metadata in Listing 1 must be sufficient to select among several plausible decisions: how to impute missing values, which features to keep, whether to normalize or encode, which model family to use, and which hyperparameters to try. The schema only records column names and coarse types (categorical, ordinal, numeric, binary, textual, url), plus optional user/item/timestamp and label columns. It does not describe missingness rates, cardinality, feature interactions, skew, or the distributional structure that actually drives preprocessing and model choice. Two datasets with identical DsDL can therefore require different pipelines; the paper gives no rule, learned mapping, or evidence that the metadata-to-decision function is learnable. Section 2.2 even concedes that a functioning DAReS is possible without the autonomous capabilities, 'albeit with potentially reduced performance,' and Table 1 acknowledges the trade-off of suboptimal performance. Since this is a position paper with no implementation or benchmark, the load-bearing zero-configuration capability claim is neither derived nor demonstrated. This is a gap in evidence, not a disagreement with current consensus; AutoML systems exist, but the specific claim that DsDL alone suffices is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new paradigm, Dataset-Agnostic Recommender Systems (DAReS), and a metadata format, Dataset Description Language (DsDL), intended to let a single codebase adapt to arbitrary recommender-system datasets without dataset-specific configuration. DsDL describes column names, coarse feature types, optional user/item/timestamp fields, and labels. The paper claims that this metadata enables autonomous feature engineering, missing-value imputation, feature transformation, noise removal, model selection, and hyperparameter tuning. It also introduces level-1 (dataset-agnostic, task-specific) and level-2 (dataset- and task-agnostic) automation levels. The manuscript is a position paper: it contains no implementation, experiments, benchmarks, or formal correctness arguments.","tokens_in":6776,"tokens_out":4789,"duration_ms":49477,"significance":"If the proposed system worked as claimed, it would be a useful contribution to recommender-system reproducibility, reusability, and accessibility for non-experts. The DsDL grammar is a concrete, citable artifact, and the level-1/level-2 distinction is a reasonable conceptual framing. However, the paper does not establish that DsDL is sufficient to drive the preprocessing, model-selection, and hyperparameter decisions it describes; the core capability claim remains a vision rather than a demonstrated result. There are no machine-checked proofs, reproducible code, or benchmark results to verify the central claim.","major_comments":[{"comment":"The central claim that DsDL metadata is sufficient for zero-configuration autonomy is unsupported. Listing 1 records only column names and coarse types (categorical, ordinal, numeric, binary, textual, url), plus optional user/item/timestamp/label fields. It does not record missingness rates, cardinalities, skew, distributions, feature interactions, or dataset size, all of which materially affect preprocessing and model choice. Consequently, datasets with identical DsDL descriptions can require different pipelines, and the paper gives no rule or learned mapping from DsDL to the decisions described in Sections 2.2 and 2.3. Section 2.1 itself says the schema may need extension. Without a mechanism or evidence, the abstract's and Section 1's assertions that DAReS eliminates the need for dataset-specific configurations are not established.","section":"Abstract, Section 1, and Listing 1"},{"comment":"The manuscript contradicts its own central claim. Sections 2.2 and 2.3 state that autonomous feature engineering, model selection, and hyperparameter tuning are optional, saying it is still possible to create a functioning DAReS without these capabilities, albeit with potentially reduced performance. The abstract and Section 1 promise automatic, zero-configuration operation. Table 1 further concedes suboptimal performance for DAReS due to the generalization-specialization trade-off. The paper needs to either scope the claim to the fully automated configuration and justify that configuration, or weaken the abstract and Section 1 accordingly.","section":"Sections 2.2 and 2.3, Table 1"},{"comment":"No empirical validation, implementation, or falsifiable prediction is provided. The paper describes what DAReS may do and lists evaluation metrics, but it does not run any experiment on diverse datasets, does not provide code, and does not quantify the acknowledged performance and overhead trade-offs. Because the central contribution is an empirical capability claim, a proof-of-concept on even a small set of standard datasets, or a concrete design with testable predictions, is necessary to support it. The future-work statement in Section 4 cannot substitute for this.","section":"Section 2.4 and Section 4"}],"minor_comments":[{"comment":"The abstract contains stray spacing in 'personali zed' and 'significan t'; please proofread the abstract.","section":"Abstract"},{"comment":"The sentence 'can be derived from other available features and.' appears to be missing a continuation after 'and'; in addition, the EBNF terminal 'String ::= < any string >' is informal and should be defined more precisely if the grammar is meant to be formally checkable.","section":"Section 2.1"},{"comment":"The statement that AutoML approaches 'still require dataset-specific configurations and adaptations' is too broad and is not supported by the cited survey [8]; consider distinguishing DAReS from general AutoML more carefully.","section":"Section 3.1"}],"recommendation":"reject","confidential_remarks":"To the editor: This is a short position statement that does not currently meet the evidentiary standard for a journal article. The central claim is not merely under-polished; it is an empirical assertion that DsDL metadata can drive all downstream decisions, and the paper itself hedges that assertion. I would be open to a resubmission that adds a proof-of-concept implementation and benchmark evaluation, but as submitted I recommend rejection. The paper may be better suited to a workshop venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this is a position paper, not a systems paper. The reader's take is basically right: the zero-configuration capability claim is asserted, not demonstrated. But I'd push back on the strength of the reject. The paper is honest about being a proposal, it acknowledges the trade-offs in Section 2.5, and the DsDL schema is a concrete first step. If you read it as a vision statement with a proposed metadata standard, it's coherent and readable.\n\nWhat's actually new: the DsDL grammar (Listing 1) is a small but concrete artifact — six feature types, optional user/item/timestamp, labels. That's not deep, but it's formal enough to implement against. The level-1/level-2 automation distinction is a useful framing for talking about what's achievable now versus later. The writing is clear, and the references to AutoML and reproducibility work are appropriate. I don't see any citation inflation or self-citation problems.\n\nThe soft spot is exactly where the stress-test puts it. DsDL records column names and coarse types, but not missingness rates, cardinality, skew, or interaction structure. Two datasets with identical DsDL can easily require different imputation, normalization, and model choices. The paper gives no mechanism, learned mapping, or experiment showing that this metadata is sufficient to drive those decisions. In fact, Sections 2.2 and 2.3 say DAReS 'may' include these components and that a functioning system is possible without them — which reads as the authors themselves not committing to the strong version of the claim. Table 1 admits suboptimal performance. So the abstract's 'autonomously manage' overstates what is actually argued.\n\nThat said, this is a position paper, and the absence of experiments is not a fatal flaw if the venue accepts that genre. The paper is a legitimate proposal that could generate useful discussion about metadata standards for recommender systems. I'd send it to review, but with a clear request that the authors either soften the capability claim to match the evidence or provide a proof-of-concept implementation and benchmark. The main value is for researchers working on AutoML for recsys and on reproducibility infrastructure. A serious referee would find the proposal coherent but would need to push on the sufficiency of the metadata.\n\nMy recommendation: engage with it as a position paper, not as a claim of results. Peer review can help sharpen the distinction.","headline":"Position paper with a concrete but under-specified metadata schema; the zero-configuration claim is asserted, not demonstrated, but the proposal is coherent and worth a serious referee.","tokens_in":7291,"tokens_out":2040,"would_cite":false,"duration_ms":19219,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DAReS proposes that a single zero-configuration codebase, guided only by a small dataset-description file, can replace per-dataset configuration in recommender systems.","keywords":["dataset-agnostic recommender systems","Dataset Description Language","zero-configuration","automated feature engineering","model selection","hyperparameter tuning","position paper","reproducibility"],"falsifier":"Run the proposed zero-configuration pipeline on a fixed task across datasets that differ in feature types, sparsity, and size, using only a DsDL file as input, and compare against per-dataset tuned baselines. If any dataset produces unusable predictions or requires hidden dataset-specific fixes, the central zero-configuration claim fails.","tokens_in":6290,"feed_emoji":"⚙️","tokens_out":5688,"duration_ms":49842,"temperature":0.7,"pith_summary":"This paper proposes a new paradigm called DAReS (Dataset-Agnostic Recommender Systems): one codebase that, for a fixed recommendation task, adapts to any new dataset without dataset-specific configuration, feature engineering, or hyperparameter tuning. The load-bearing idea is DsDL, a small structured description of a dataset's columns, types, user/item identifiers, timestamp, and labels, which the system uses to decide preprocessing, model selection, and tuning. The paper is a position paper: it defines the architecture, the metadata grammar, and a roadmap from level-1 (dataset-agnostic, task-specific) to level-2 (task-agnostic) automation. If the claim holds, recommender system development would become reusable and accessible to non-experts, at the cost of reduced per-dataset optimization and higher compute overhead that the paper explicitly accepts.","feed_headline":"A metadata file could replace hand-tuning in recommender systems","feed_subtitle":"DAReS uses the DsDL metadata format to automate feature engineering, model selection, and tuning with zero manual configuration.","key_machinery":"The Dataset Description Language (DsDL) is the mechanism that carries the argument. It is a small grammar, formally defined in EBNF, describing a dataset as a list of features with types plus optional user_id, item_id, timestamp, and label fields. DsDL is meant to give the system enough context to treat all datasets uniformly: the type tags drive encoding and preprocessing choices, the presence of user/item columns selects between task families, and the label field determines the prediction target and evaluation metric.","core_discovery":"The central claim is that dataset-specific human labor in recommender systems can be replaced by a standardized dataset description. DAReS reads a DsDL file, a YAML/JSON document listing each feature's name and type (categorical, ordinal, numeric, binary, textual, URL), plus optional user ID, item ID, timestamp, and labels, and uses that metadata alone to autonomously perform feature selection, missing-value imputation, feature transformation, noise removal, model selection, hyperparameter tuning, and evaluation. The paper positions this as level-1 automation: dataset-agnostic but still task-specific, with the task (CTR prediction, rating prediction, Top-N) supplied by the user. It argues this eliminates dataset-specific configurations and improves reusability and reproducibility compared to traditional pipelines, while acknowledging a trade-off: less deep customization for specialized datasets and higher computational cost.","pith_inferences":["The untested core is whether column-name and type metadata alone is enough to make good model-selection decisions; the most direct way to test that is a fixed task run across many datasets with only DsDL as input.","One could learn the metadata-to-configuration mapping across a corpus of datasets, turning the position paper's proposal into a meta-learning system; the paper does not claim this is done.","The same DsDL schema could be extended with dataset statistics (sparsity, cardinality, skew) that would make the zero-configuration promise more realistic; that extension is not in the paper."],"forward_implications":["The same DAReS codebase could be applied to a new dataset by writing only a short DsDL file, removing per-dataset code changes.","Non-expert or entry-level users could build recommendation models without domain-specific feature engineering or tuning expertise.","Standardized DsDL descriptions would make experiments easier to reproduce, since dataset treatment is no longer buried in undocumented code.","For highly specialized datasets, DAReS would likely underperform hand-tuned pipelines, and the automated search adds computational cost.","The level-2 extension would let the system infer the task itself from the dataset, removing even the task-specific assumption."],"supporting_citations":[{"why":"It documents reproducibility problems in recommender research, motivating the standardized dataset description DsDL.","marker":"[6]"},{"why":"It surveys AutoML and is the baseline that DAReS contrasts, since AutoML still needs dataset-specific configuration.","marker":"[8]"},{"why":"It supplies the EBNF standard used to define the DsDL grammar in Listing 1.","marker":"[19]"},{"why":"It presents an existing automated recommender system whose dataset-specific limitations DAReS aims to overcome.","marker":"[24]"},{"why":"It defines the Top-N recommendation task that requires user and item identifiers in the DsDL schema.","marker":"[3]"},{"why":"It defines the click-through rate prediction task where user and item columns may be unnecessary, justifying their optionality in DsDL.","marker":"[27]"},{"why":"It supplies automated feature engineering methods that DAReS can incorporate in preprocessing.","marker":"[23]"}],"fun_headline_variants":["DAReS: one script, any dataset, zero tuning","Dataset descriptions replace hand-tuning in recommender systems","Automate recommender pipelines with a single metadata file","No more dataset-specific configs: DAReS reads a DsDL file","From metadata to model: DAReS handles all tuning automatically"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire proposal rests on the assumption that a short list of column names and types is enough information for an automated system to make all preprocessing, feature-engineering, model-selection, and hyperparameter decisions correctly, without unacceptable performance loss.","fun_headline_variants_meta":{"raw":{"variants":["DAReS: one script, any dataset, zero tuning","Dataset descriptions replace hand-tuning in recommender systems","Automate recommender pipelines with a single metadata file","No more dataset-specific configs: DAReS reads a DsDL file","From metadata to model: DAReS handles all tuning automatically"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001113,"raw_usage":{"total_tokens":4611,"prompt_tokens":893,"completion_tokens":3718,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":3633}},"tokens_in":509,"tokens_out":3718,"duration_ms":25478,"temperature":1.0,"reasoning_tokens":3633,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:45:07.589474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed zero-configuration pipeline on a fixed task across datasets that differ in feature types, sparsity, and size, using only a DsDL file as input, and compare against per-dataset tuned baselines. If any dataset produces unusable predictions or requires hidden dataset-specific fixes, the central zero-configuration claim fails.","supporting_citations":[{"cited_title":"A trou- bling analysis of reproducibility and progress in recommen der systems research","cited_arxiv_id":null,"evidence_quote":"It documents reproducibility problems in recommender research, motivating the standardized dataset description DsDL."},{"cited_title":"Generic base standards","cited_arxiv_id":null,"evidence_quote":"It supplies the EBNF standard used to define the DsDL grammar in Listing 1."},{"cited_title":"Au- torec: An automated recommender system","cited_arxiv_id":null,"evidence_quote":"It presents an existing automated recommender system whose dataset-specific limitations DAReS aims to overcome."},{"cited_title":"Item-based top-n r ecommendation algorithms","cited_arxiv_id":null,"evidence_quote":"It defines the Top-N recommendation task that requires user and item identifiers in the DsDL schema."},{"cited_title":"Click-through rate predicti on in online advertising: A literature review","cited_arxiv_id":null,"evidence_quote":"It defines the click-through rate prediction task where user and item columns may be unnecessary, justifying their optionality in DsDL."},{"cited_title":"Toward e fficient automated feature engineering","cited_arxiv_id":null,"evidence_quote":"It supplies automated feature engineering methods that DAReS can incorporate in preprocessing."}],"review_version":1}