{"id":"9ac25c66-a6b5-4f07-a7f5-aa509d8767d4","arxiv_id":"2506.03391","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper proposes a dataset- and task-independent recommender framework (DTIRS) with a schema language (DsDL) and an automation-level roadmap, but provides no implementation or experiments.","lead":"This paper proposes DTIRS, a framework that lets the same recommendation code run across different datasets and tasks without manual rewiring. It introduces DsDL, a schema for describing dataset columns and prediction targets, and a roadmap for automating feature engineering, model selection, and tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reusability claim hinges on DsDL carrying enough information for fully automatic pipeline configuration; DsDL omits user/item role semantics, and no implementation or benchmark demonstrates schema sufficiency.","rationale":"The paper is a position and roadmap paper, and its strongest claim is about what DsDL plus DTIRS could do rather than about a delivered system. As such, the absence of experiments is not by itself a logical flaw; however, it means the central claim is an unsupported existential assertion: that a competitive pipeline can be configured from the schema alone. The least secure point is the information content of DsDL. The schema records column names, coarse types, target type, key_col, and for list tasks list_size and relevance_col. It does not record which columns are users, items, or context, nor the row semantics such as whether each row is an event or a user-item pair. These distinctions determine whether F_T should contain matrix-factorization models, sequence models, or tree-based classifiers. If F_T is instead made so broad that every model class is tried for every dataset, then 'task-independent model selection' collapses into generic AutoML, and the only new information DsDL contributes is the target type. That may still be useful standardization, but it is a much weaker claim than eliminating reconfiguration. The reader's weakest assumption identifies the same hinge, and my analysis agrees with it. The concern can be settled only by a working prototype and benchmark, which is exactly what the reader's conditional acceptance requires. I therefore leave the verdict unchanged.","tokens_in":16307,"tokens_out":7367,"duration_ms":85441,"concrete_test":"Implement the minimal DTIRS described in §5.3 on 6–10 public datasets spanning the four target types (e.g., MovieLens, Criteo, Instacart, ML-1M ranking). For each dataset, run the pipeline using only the fields permitted by Listing 1, and compare model choice and performance against (a) per-dataset tuned baselines and (b) a generic AutoML system (e.g., AutoGluon) on the same flattened table. In addition, run an ablation in which user/item columns are explicitly tagged: if model architecture or final performance changes materially, the DsDL schema alone does not supply the information needed for autonomous model selection, and the central reusability claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a DsDL schema alone lets DTIRS 'autonomously' perform feature engineering, model selection, and optimization (abstract; §4.5). The load-bearing condition is that §5.1's schema contains enough information to choose a competitive pipeline. That condition is not established, and there is a concrete gap: DsDL identifies column types and a target, but not the semantic roles required for recommender model selection. In Listing 2, user_id and ad_id are just categorical columns and key_col is index_id, not a user or item key; in Listing 3, key_col is also index_id, so the schema does not tell §4.4's candidate set F_T whether the task is collaborative filtering (user×item matrix) or supervised regression over features. A system could still fall back to generic AutoML on the flattened table, but then DsDL adds no selection information beyond what AutoML already gets from column types, and the claimed 'task-aware model selection' must decide among very different architectures from T alone. Since no prototype, benchmark, or comparison to reference [69] is supplied, the central reusability claim is not yet falsifiable; the paper's own §8.2 and §8.5 acknowledge missing relational and domain knowledge, but not this role-identification gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dataset- and Task-Independent Recommender Systems (DTIRS), a conceptual framework intended to let a single recommendation pipeline adapt to new datasets and task types without manual reconfiguration. The enabling component is a proposed Dataset Description Language (DsDL), a JSON-like schema that records column types, a target type, a label column, a key column, and optional list-size and relevance columns. The central claims are that, given a DsDL schema, DTIRS autonomously performs feature engineering, model selection, and optimization (Sections 4.3–4.5), and that these capabilities define a new automation level, Level-2, in which the same core code handles the four target types of Table 1: binary, numeric, ordered-list, and unordered-list prediction. The paper is explicitly a position/roadmap paper: the appendix FAQ A2 states that experimental results are not the focus and refers to a work-in-progress website for a prototype. The formal content is Equations (1)–(5), which are definitions and an uninstantiated optimization statement, plus the DsDL grammar and four illustrative listings.","tokens_in":16573,"tokens_out":4450,"duration_ms":56488,"significance":"If the central claim were established, DTIRS would be a genuinely useful community resource: it could lower the barrier to entry for recommender systems, improve reproducibility by providing a universal baseline, and reuse pipeline code across datasets and tasks. The four-way task taxonomy in Table 1 is a clean, practical organizing device, and the DsDL grammar is a concrete, testable proposal for a machine-readable dataset description format. The paper is also honest in its limitations: Section 8.2 acknowledges the flat-table restriction, Section 8.5 acknowledges missing domain-knowledge integration, and FAQ A2 openly states that no experiments are included. However, the manuscript currently ships no implementation, benchmark, or formal argument, so the paper's principal claim remains an unverified architectural vision rather than a demonstrated framework.","major_comments":[{"comment":"The central claim that DTIRS autonomously performs feature engineering, model selection, and optimization is not supported by an implementation, a benchmark, or a formal argument. Equation (5) is an uninstantiated arg-min over an unspecified candidate family F_T; the paper does not specify F_T, the search algorithm, or any empirical protocol. FAQ A2 acknowledges the absence of experimental results and asserts that a traditional experiment can be replicated in DTIRS with the same results, but this is a definitional statement, not evidence that DTIRS can select competitive models in practice. The authors should either provide a prototype with experiments on multiple datasets spanning at least two of the four task types in Table 1, or rewrite the abstract and Section 4.5 claims as explicit research goals rather than achieved capabilities.","section":"Section 4.5, Eq. (5), and Appendix A2"},{"comment":"DsDL as specified does not carry enough semantic information for the task-aware model selection claimed in Section 5.3. In Listing 2, user_id and ad_id are merely categorical columns and key_col is index_id, not a user or item key; similarly, in Listing 3, key_col is index_id rather than user_id or movie_id. A DsDL schema therefore cannot tell the model-selection function of Section 4.4 whether the problem is collaborative filtering over a user-item interaction matrix or supervised regression over a flattened feature table. Since Equation (4) selects among F_T using only the task descriptor T, the manuscript needs either to add explicit semantic roles such as user_key and item_key to DsDL, or to give a concrete mapping from TargetType to candidate model families and demonstrate that this mapping produces competitive models. Without this, the central reusability claim fails: the schema adds no selection information beyond what generic AutoML already obtains from column types.","section":"Section 5.2.1–5.2.2, Listings 2 and 3"},{"comment":"The Level-2 definition of task-independence is circular as stated: a system is task-independent if it can solve 'any recommendation task defined in DsDL,' but DsDL is introduced by the same paper and is currently restricted to flat tables (Section 8.2). Table 1 covers four target structures but omits tasks such as session-based or sequential recommendation, multi-task objectives, and tasks requiring relational structure. The claim of universal reusability is therefore bounded by an arbitrary taxonomy that the authors themselves control. The paper should either narrow the contribution to the four stated task types on flat tables and say so explicitly in the title and abstract, or extend DsDL with the mechanisms (e.g., sequence columns, relational joins) needed to represent the omitted task classes.","section":"Section 6.3 and Section 8.2"},{"comment":"The automated feature engineering claim is underspecified. Equation (3) defines X' = Phi(S), but the surrounding text only lists generic AutoML techniques such as missing-value handling, categorical encoding, and feature selection. The manuscript never specifies which transformations are triggered by which schema entries, nor does it offer any evidence that column types alone determine the correct preprocessing. This matters because Section 4.3 is one of the three pillars of Level-2 autonomy (feature engineering, model selection, optimization). A concrete decision rule, or at least a worked example showing how a schema entry such as list_of_categorical leads to a specific transformation, is needed to make the claim falsifiable.","section":"Section 4.3, Eq. (3)"}],"minor_comments":[{"comment":"The sentence 'Instead of treating datasets as rigid structures, we represents a dataset D' contains a subject-verb agreement error and should read 'we represent.'","section":"Section 4.2"},{"comment":"The EBNF grammar contains typographical artifacts that obscure the definition, including broken tokens such as 'la be l_ co l' and the missing spacing around 'list_size'; these should be cleaned and the grammar should be machine-checked or compiled to a parser.","section":"Section 5.1, Listing 1"},{"comment":"The sentence 'In the following, we discusses the practical limitations' should read 'we discuss.'","section":"Section 2"},{"comment":"In Listing 2, the sentence 'The model then need to perform' should be 'The model then needs to perform.'","section":"Section 5.2.1"},{"comment":"Reference [69] (the authors' own 'Dataset-agnostic recommender systems') is cited in the reproducibility discussion but is not positioned in Section 3 relative to DTIRS; the authors should state how DTIRS differs from that prior proposal and what new contribution DsDL makes beyond it.","section":"Section 3 and reference [69]"},{"comment":"The claim that a prototype is available at https://dtirs.gitlab.io would be more verifiable if the manuscript included a versioned repository identifier, a minimal working example, or a link to the code at the time of submission; the current description as 'work in progress' makes the reproducibility argument difficult to assess.","section":"Appendix A2"}],"recommendation":"major_revision","confidential_remarks":"This is a vision/position paper with no experimental results, so its fit depends on whether the journal accepts papers of that genre. The main risk is that the abstract and Section 4.5 state autonomous capabilities as established facts, while the evidence is limited to Equations (1)–(5) and illustrative listings. The authors' own reference [69] and the companion RBoard platform suggest that the group has relevant infrastructure, and the revision should leverage that to provide at least a small-scale prototype evaluation. The self-citation to [69] should also be made explicit in the related-work narrative so that the novelty of DsDL relative to prior work is clear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mateo, here's my take on 2506.03391. The genuinely new things are small but real: the DsDL grammar (an EBNF schema for flat tables with column types and target) and the Level-0/1/2 automation taxonomy are useful packaging for a problem that hurts practitioners. The paper is also refreshingly honest: Section 8 lists flat-table limits and missing domain knowledge, and FAQ A2 admits there are no experiments. That candor counts.\n\nWhat the paper does not do is what its abstract promises. The claim that DsDL enables 'autonomous feature engineering, model selection, and optimization' is unsupported. There is no prototype, no benchmark, and no comparison to existing AutoML/reproducibility frameworks, or to the authors' own earlier dataset-agnostic preprint [69]. Equations (1)-(5) are definitions, not derivations, so there is no formal argument either.\n\nThe stress-test note lands. The schema records column types and target type but not semantic roles. In Listings 2 and 3, key_col is interaction index rather than user/item; user_id and ad_id are just categorical columns. T therefore doesn't tell the candidate set F_T whether the problem is collaborative filtering over a user-item matrix or supervised regression on flattened features. A system can fall back to generic AutoML, but then DsDL adds almost no selection information over column types alone, and the claimed task-aware model selection remains underdetermined.\n\nThat said, I don't think the idea is dead. The task taxonomy by output structure is sensible, and 'raise the floor' is a defensible goal. The route to publication is a concrete Level-1 prototype on several datasets, with DsDL stated explicitly and a comparison against conventional AutoML baselines and [69]. That would test whether DsDL carries enough information; right now, schema sufficiency is an assumption, not a result.\n\nWho should read this: practitioners and researchers thinking about standardized dataset descriptors for recsys, and organizers of reproducibility efforts. It is not a methods paper. A serious editor could send it to peer review as a position/vision paper, but the referees should insist on the missing evidence before treating any of the automation claims as established.","headline":"A clear, honest position paper that names a real deployment pain and proposes a plausible schema, but the central reusability claim is a promissory note until a prototype and benchmark arrive.","tokens_in":17085,"tokens_out":1712,"would_cite":false,"duration_ms":18783,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A schema-based framework aims to end manual recommender reconfiguration.","keywords":["recommender systems","dataset- and task-independent","DsDL","automated machine learning","feature engineering","model selection","task taxonomy","reusability"],"falsifier":"Take a set of flat-table datasets spanning all four target types, write DsDL schemas for them, and run a Level-2 DTIRS implementation against manually tuned baselines for each dataset; if the automated pipeline frequently needs human overrides or falls far below generic defaults on a meaningful share of datasets, the claim that DsDL alone suffices for configuration is refuted.","tokens_in":16098,"feed_emoji":"⚙️","tokens_out":8994,"duration_ms":92098,"temperature":0.7,"pith_summary":"The paper argues that a recommender system can be made dataset- and task-independent: given only a short machine-readable description of a dataset's columns and prediction target, the framework should automatically engineer features, choose a model, and tune hyperparameters. That description language, DsDL, classifies every recommendation task into one of four output structures — binary, numeric, ordered list, or unordered list — and uses that classification to pick preprocessing, loss, and evaluation. The payoff would be code that transfers across datasets without rewriting, lowering the barrier for non-experts and giving researchers a universal baseline to build on. The paper positions this as a road map from Level-1 automation (dataset-agnostic but task-specific) to Level-2 automation (fully dataset- and task-independent), with the caveat that the goal is to raise the floor of baseline performance rather than the ceiling of specialized accuracy. The paper presents this as a conceptual proposal rather than a benchmarked system.","feed_headline":"A short schema could end manual recommender reconfiguration","feed_subtitle":"The proposed DTIRS framework reads a DsDL description of any flat-table dataset and configures its own pipeline.","key_machinery":"The central object is the Dataset Description Language (DsDL), an EBNF grammar for $S' = (C, T)$: it lists columns with types (numeric, binary, categorical, ordinal, textual, URL, and list variants) plus a target block declaring the target type (binary, numeric, ordered_list, unordered_list), label_col, key_col, and optional list_size and relevance_col. DsDL carries the argument by doing four jobs: parsing the schema, mapping label_col and key_col to the learning objective, selecting a model architecture from the target type, and configuring feature transformations and hyperparameter search. The task taxonomy in Table 1 links each target structure to a typical loss and evaluation metric, so the schema alone determines the pipeline's objective and yardstick.","core_discovery":"The central claim is that reusability does not require a single universal model; it requires a universal interface. Representing a dataset as $S' = (C, T)$ — columns plus a task descriptor — is enough for the system to run $\\Phi(S)$ for automated feature engineering and to solve $f_S^*, \\theta^* = \\arg\\min_{f,\\theta} \\mathbb{E}[L(f(X';\\theta), Y)]$ for model selection and optimization. The four-way task taxonomy (binary, numeric, ordered list, unordered list) is what lets one pipeline read a new dataset's DsDL and configure itself, even though models still need retraining on new data. The paper calls this Level-2 automation and presents DsDL as the foundational tool, explicitly accepting that specialized systems may still outperform it on individual tasks.","pith_inferences":["Beyond the paper's claims, DsDL schemas could double as a benchmark harness: by scoring how much automated configuration improves over a fixed default pipeline on each schema, the community could quantify the 'raise the floor' claim directly.","Beyond the paper's claims, if DsDL were adopted as a shared standard, the same loss functions and metrics (NDCG for ordered lists, Jaccard for unordered lists, AUC for binary) could be wired automatically across implementations, so reproducibility studies would focus on model quality instead of glue code.","Beyond the paper's claims, the four-type taxonomy suggests an architecture the paper does not specify: a shared feature-encoding backbone with task-specific heads and losses, which would be a natural testbed for whether one pipeline can really serve all four target types.","Beyond the paper's claims, a stress test the paper leaves open is datasets where the target type is ambiguous (e.g., a rating column that could be treated as numeric or ordered), which would reveal whether DsDL is unambiguous enough to automate model selection without human judgement."],"forward_implications":["A single codebase using DsDL could accept a new flat-table dataset and output a configured predictor for any of the four target types without manual feature engineering or model selection.","Non-experts could deploy recommender systems by writing only a short DsDL description, removing the domain expertise currently required for per-dataset tuning.","Research reproducibility would improve because published results could be re-run on new datasets by swapping the DsDL schema rather than reimplementing the pipeline.","Level-2 automation trades away the top of the performance distribution: DTIRS aims to improve the universal baseline, not to beat specialized per-dataset systems.","Moving from Level-1 to Level-2 requires solving task recognition, task-specific loss and metric configuration, and the computational overhead of automated search."],"supporting_citations":[{"why":"Supplies the large-language-model precedent that a single model can perform many tasks with minimal reconfiguration, which motivates DTIRS.","marker":"[9]"},{"why":"Further evidence of cross-task generalization in pretrained language models that DTIRS seeks to emulate for recommender systems.","marker":"[16]"},{"why":"Survey of automated machine learning for recommendations, supporting the feasibility of automating feature engineering and model selection.","marker":"[13]"},{"why":"Example of automated model construction for sequential recommendation, a building block for Level-1 dataset-independent pipelines.","marker":"[38]"},{"why":"Survey of AutoML for deep recommender systems that grounds the proposed levels of automation in prior work.","marker":"[80]"},{"why":"Existing automated recommender library performing model search and hyperparameter tuning but remaining task-specific, defining the gap DTIRS targets.","marker":"[3]"},{"why":"Shows automatic algorithm selection and optimization for recommenders, evidence that automation is possible within a single task.","marker":"[60]"},{"why":"Automated recommender system for rating and CTR tasks that DTIRS extends by removing task specificity.","marker":"[64]"},{"why":"A unified framework for many recommendation algorithms that still ties models to specific datasets or tasks, serving as the comparison point for DTIRS.","marker":"[79]"}],"fun_headline_variants":["Universal interface, not model, unlocks recommender reusability","DsDL: one schema to configure any recommendation pipeline","Stop rebuilding recommender pipelines: describe the dataset instead","Universal recommender interface: describe data, not code","The secret to reusable recommenders? A dataset description language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a DsDL schema — column names, column types, a target type, a key column, and optional list size and relevance column — contains enough information for an automated system to choose features, models, and hyperparameters with acceptable performance; if it does not, DTIRS still requires human expertise and the reusability claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Universal interface, not model, unlocks recommender reusability","DsDL: one schema to configure any recommendation pipeline","Stop rebuilding recommender pipelines: describe the dataset instead","Universal recommender interface: describe data, not code","The secret to reusable recommenders? A dataset description language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2416,"prompt_tokens":996,"completion_tokens":1420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":1341}},"tokens_in":612,"tokens_out":1420,"duration_ms":12476,"temperature":1.0,"reasoning_tokens":1341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:04:08.434585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of flat-table datasets spanning all four target types, write DsDL schemas for them, and run a Level-2 DTIRS implementation against manually tuned baselines for each dataset; if the automated pipeline frequently needs human overrides or falls far below generic defaults on a meaningful share of datasets, the claim that DsDL alone suffices for configuration is refuted.","supporting_citations":[{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"Further evidence of cross-task generalization in pretrained language models that DTIRS seeks to emulate for recommender systems."},{"cited_title":"A comprehensive survey on automated machine learning for recommendations.ACM Transactions on Recommender Systems, 2(2):1–38, 2024","cited_arxiv_id":null,"evidence_quote":"Survey of automated machine learning for recommendations, supporting the feasibility of automating feature engineering and model selection."},{"cited_title":"Automlp: Automated mlp for sequential recommendations","cited_arxiv_id":null,"evidence_quote":"Example of automated model construction for sequential recommendation, a building block for Level-1 dataset-independent pipelines."},{"cited_title":"Automl for deep recommender systems: A survey","cited_arxiv_id":null,"evidence_quote":"Survey of AutoML for deep recommender systems that grounds the proposed levels of automation in prior work."},{"cited_title":"Introducing lenskit-auto, an experimental au- tomated recommender system (autorecsys) toolkit","cited_arxiv_id":null,"evidence_quote":"Shows automatic algorithm selection and optimization for recommenders, evidence that automation is possible within a single task."},{"cited_title":"Au- torec: An automated recommender system","cited_arxiv_id":null,"evidence_quote":"Automated recommender system for rating and CTR tasks that DTIRS extends by removing task specificity."},{"cited_title":"Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms","cited_arxiv_id":null,"evidence_quote":"A unified framework for many recommendation algorithms that still ties models to specific datasets or tasks, serving as the comparison point for DTIRS."}],"review_version":1}