{"id":"af9f6458-e61a-430d-96c0-96df160739f1","arxiv_id":"2501.08591","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OpenMLDB unifies offline and online feature computation with one SQL compiler plus specialized window and join optimizations, reporting large latency and memory gains over Spark, Flink, DuckDB, and Redis/Trino pipelines.","lead":"OpenMLDB is an open-source system that computes machine-learning features from relational tables in both offline batches and online real-time requests, using a single SQL plan generator to keep the two stages consistent. The paper reports millisecond latencies and large speedups over Spark, Flink, DuckDB, and Redis/Trino pipelines in microbenchmarks and production workloads.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pre-aggregation under §5.1 assumes binlog_offset increases monotonically, which orders arrival, not event time; late tuples can silently vanish from already-merged aggregate levels, breaking the offline/online consistency claim.","rationale":"The reader's weakest_assumption matches the weakest point I can find, so I agree. I read the architecture in good faith: the unified plan generator, shared C++ execution functions, compact encoding, and deployment evidence are plausible and give the paper real independent support; the open-source artifact could allow verification. Nevertheless, the central claim is not just speed but consistency ('unified query plan generator for consistent computation results across the offline and online stages'). The pre-aggregation design in §5.1 is the one place where the online engine deliberately stops scanning raw tuples and instead merges stale, precomputed summaries. Any inconsistency between those summaries and raw data is therefore not a benchmark artifact or a tuning issue; it is a semantic error in the core mechanism. A late-arriving tuple is the minimal case that exposes it, and the paper itself flags out-of-order data as relevant in §5.2, making the omission concrete rather than hypothetical. Multiple secondary concerns exist (missing error bars, baseline configuration details, the abstract's ratios versus plotted data), but they affect the strength of the performance claims, not the logical soundness of the design. The pre-aggregation correctness gap can be settled by a single differential test, so I would not reject the paper; I would keep it conditional on demonstrating or documenting correct handling of late tuples in pre-aggregated windows.","tokens_in":21353,"tokens_out":4749,"duration_ms":46139,"concrete_test":"Using the public OpenMLDB benchmark, deploy a long-window query with DEPLOY test1 OPTIONS(long_windows='w1:1d') over a 100-day window. Insert several days of tuples and wait until the aggregator advances past day 1; then inject a tuple whose event timestamp is in day 1 but arrives late. Issue an online feature request and compare its output with a fresh offline batch computation over the full data set including the late tuple. If the online value does not reflect the late tuple's contribution, the §5.1 pre-aggregation is incorrect under out-of-order streams; if it does match, the paper should document the update/reconciliation mechanism that makes it match.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §5.1, aggregator updates are 'designed by assuming the binlog_offset increases monotonically,' and are protected by a replicator lock. That assumption guarantees a total order on updates by arrival/ingestion order, not by event timestamp. If a tuple with an old event timestamp arrives after its time bucket has already been merged into a coarser pre-aggregation level, the asynchronous update path cannot correctly revise the materialized higher-level aggregate. The paper motivates out-of-order stream data in §5.2 but gives no mechanism for pre-aggregation to handle such tuples. Since long-window features are served by merging pre-aggregates, a late tuple can be silently omitted from every subsequent online query, whereas an offline batch recomputation over the same final dataset would include it. This directly breaks the paper's central claim of offline/online consistency in precisely the long-window regime that pre-aggregation is designed to accelerate. The replicator lock does not rescue the argument: it serializes binlog entries, but the old bucket's aggregate was already consumed to build a higher level before the late entry is applied, and no compensation, invalidation, or lazy re-materialization is described. If OpenMLDB intentionally excludes out-of-order event-time arrivals for pre-aggregated windows, that semantic restriction is not stated and still needs a consistency argument relative to offline execution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OpenMLDB is an industrial feature computation system that unifies offline and online feature extraction through a single SQL plan generator and shared C++ execution functions, and adds an online real-time executor with long-window pre-aggregation and self-adjusting multi-table window unions, plus an offline batch executor with multi-window parallelization and skew-aware repartitioning, and compact in-memory data encoding and indexing. The evaluation reports microbenchmark and production workload results against Spark, Flink, DuckDB, GreenPlum, MySQL, and Redis, claiming 10-20x online speedups, 6x offline speedups, and 40-75% memory savings, supported by deployments in 100+ scenarios. The paper's core technical pitch is that consistency is obtained architecturally, by using the same query plan and C++ libraries offline and online, rather than by reconciliation, and that performance is obtained by compilation plus specialized data structures.","tokens_in":21596,"tokens_out":4852,"duration_ms":45332,"significance":"If the results hold, the paper addresses a real and expensive operational gap: offline/online feature skew in ML pipelines. The strengths are concrete: the system is open source, has production deployments, and the consistency story, a shared SQL compiler and shared C++ implementations, is a credible architectural mechanism rather than an empirical coincidence. The pre-aggregation and self-adjusting window-union designs are also relevant to streaming systems beyond ML. However, the quantitative claims as currently presented are weaker than the abstract suggests: most comparisons are single-point latency and throughput numbers without repetitions or error bars, and several abstract claims do not correspond to directly reported experiments. The correctness of the fast long-window path under out-of-order event-time data is not established. With those gaps closed, this would be a solid systems contribution.","major_comments":[{"comment":"The pre-aggregation update protocol only guarantees a total order on binlog entries, not on event-time order. As the text states, 'aggregator updates are designed by assuming the binlog_offset increases monotonically' and are protected by the replicator lock; this is an arrival/ingestion order. Section 5.2 explicitly motivates out-of-order stream data, but Section 5.1 gives no mechanism for revising an aggregate level after its bucket has already been merged into a coarser level. A tuple with an old event timestamp that arrives late will therefore be silently absent from the materialized long-window aggregate, while an offline batch computation over the same final relation would include it. That directly undermines the central offline/online consistency claim in the long-window regime that pre-aggregation is designed to accelerate. The paper should either describe the late-data handling for pre-aggregated windows, for example compensation, invalidation, or lazy re-materialization, or explicitly restrict pre-aggregation to ordered event-time streams and explain why the offline/online consistency guarantee still covers that restriction.","section":"§5.1 (Aggregator Update)"},{"comment":"The abstract's headline numbers are not all directly supported by the reported experiments. The claimed '10x-20x higher online performance than Flink and DuckDB' is not shown as a Flink-versus-DuckDB comparison in the online microbenchmark; Figure 6 compares OpenMLDB with MySQL(in-mem), DuckDB, and Trino+Redis, while Flink appears only in the RTP workload (Figure 7) and in Section 9.3.2 as a throughput discussion without reported measurements. Likewise, '6x faster offline performance than Spark and MPP databases like GreenPlum' is supported only by the 6.3x Spark multi-window number in Figure 8; no GreenPlum offline experiment appears. In addition, all latency figures (Figures 6-17, Table 3) are single-point measurements with no repetitions, error bars, or significance testing, making it impossible to assess variance or whether the improvements are robust. The evaluation should be aligned with the claims and should include repeated runs with variance or confidence information.","section":"Section 9 and Abstract"},{"comment":"The Subtract-and-Evict mechanism in Section 5.2 inherits the same ordering assumption as Section 5.1 but is presented as supporting out-of-order data. The description says outdated tuples are subtracted as they leave the window and new tuples are added incrementally; this is only correct if tuples are evicted and inserted in timestamp order. The paper cites [66] for load balancing and [58] for Subtract-and-Evict, but it does not state how a late tuple whose timestamp falls inside an already-evicted or already-merged interval is handled. Without that, the self-adjusting window union's high throughput in Section 9.3.2 may come at the cost of omitted tuples, and the claimed 'out-of-order stream data' support in Section 3.1 is not demonstrated. Please specify the event-time semantics precisely, including any watermarking or buffering, for both the pre-aggregation path and the window-union path.","section":"§5.2 (Incremental Computation)"}],"minor_comments":[{"comment":"References [65] and [66] are the same paper with identical title and venue; one duplicate should be removed.","section":"References"},{"comment":"The contribution bullet 'reduces feature deployment time from months to days or even less [?]' contains an unresolved placeholder; please either remove it or replace it with measured numbers.","section":"Contributions list, Section 1"},{"comment":"Several evaluation figures, especially Figure 10 and Figures 14-17, lack readable axis labels and units in the provided text; adding them is necessary for the results to be interpretable.","section":"Figures 10, 14-17"},{"comment":"The memory estimation model defines C by table type and says K varies between 1 and n_index, but it gives no calibration procedure or validation against measured memory usage, so the example calculation (1.568 GB) cannot be reproduced or checked; please provide calibration guidance or measured validation.","section":"Section 8.1, Eq. (1)"},{"comment":"The long-window pre-aggregation experiment reports a 45x latency reduction but only says the optimization introduces a 'slightly higher data loading overhead' without quantifying it; please report the loading overhead so readers can assess the trade-off.","section":"Section 9.3.1"}],"recommendation":"major_revision","confidential_remarks":"This is primarily an industrial-systems paper with strong vendor involvement. The production deployments and open-source repository are relevant evidence, but they should not substitute for independent reproducibility of the benchmarks; the paper references a benchmark harness but gives no artifact evaluation details. I see no derivational circularity in the consistency claim, as it is architectural rather than fitted to performance. The duplicated reference should be caught in copyediting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nOpenMLDB is worth your time if you care about offline/online feature consistency in ML pipelines. The authors have built and deployed a real system, the architecture is coherent, and the open-source codebase gives the qualitative claims more weight than the average system paper. The central idea—compile one SQL plan into C++ code shared by batch and streaming engines, then add pre-aggregation, self-adjusting window unions, parallel offline windows, and a compact in-memory layout—is a reasonable integration of known techniques, and the production results at Akulaku suggest it works. I believe the paper deserves peer review, but not without fixing the following.\n\nThe biggest issue is correctness under out-of-order event-time data. Section 5.1 says aggregator updates assume binlog_offset increases monotonically, protected by a replicator lock. That serializes ingestion order, not event time. If a tuple with an old timestamp arrives after its time bucket has been merged into a coarser aggregate level, the higher-level aggregate has already consumed that bucket. The paper describes no compensation, invalidation, or lazy re-materialization for this case. The result is that long-window features served from pre-aggregation can silently omit late tuples, while an offline batch rerun over the final dataset would include them—exactly the offline/online inconsistency the system claims to eliminate. Section 5.2 discusses out-of-order data for window unions, but the pre-aggregation path does not address it. If OpenMLDB assumes strictly in-order arrival for pre-aggregated windows, that restriction needs to be stated and compared to the offline semantics. As written, this is a load-bearing gap.\n\nThe experimental section is the second problem. There are no error bars, repetitions, or significance tests. Baseline tuning is underspecified. The abstract's '10x–20x' online speedup is not uniformly supported by the plots; offline '6x' is only the multiple-window case, with 2.6x on the single-window microbenchmark. Memory savings are measured against Trino+Redis, but the abstract says 'in-memory databases like Redis.' Those are fixable presentation issues, but they matter for a systems paper making deployment claims.\n\nNovelty is modest—most ingredients have prior art, including the authors' own ICDE 2023 interval-join paper—but the integration into a production-grade, open-source system with 100+ scenarios is a legitimate contribution.\n\nRecommendation: send to peer review with a request for major revision. Specifically: state the event-time semantics for pre-aggregation, add a differential consistency test between online and offline execution under out-of-order arrivals, and report repetitions with variance. With those, this could be a solid systems paper.","headline":"Credible industrial feature computation system with real deployments, but the pre-aggregation path silently drops late tuples and the reported speedups are not uniformly reproduced by the figures.","tokens_in":22169,"tokens_out":2987,"would_cite":true,"duration_ms":28822,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenMLDB compiles one extended-SQL feature script into shared offline/online C++ execution, and reports 10x-20x online speedups over Flink and DuckDB via long-window pre-aggregation and self-adjusting window unions.","keywords":["feature computation","online machine learning","offline-online consistency","pre-aggregation","window aggregation","data skew","in-memory data structure","SQL compilation"],"falsifier":"Insert a tuple whose timestamp lies in the past after the daily or monthly pre-aggregation level covering that time has already been computed, then run a long-window feature query over that interval and compare its result with a recomputation over raw data; a mismatch would show that low latency came at the cost of feature correctness.","tokens_in":21141,"feed_emoji":"⚡","tokens_out":9134,"duration_ms":84850,"temperature":0.7,"pith_summary":"OpenMLDB is a feature-computation system built to close the gap between the two stages where ML features are made: offline batch computation for training and online real-time computation for serving. The paper argues that current practice—writing features in a batch engine, then manually re-implementing them in a streaming or database engine—creates semantic drift and deployment delays of months. OpenMLDB's answer is a unified plan generator that compiles one extended-SQL feature script into C++ machine code shared by its offline and online executors, making consistency structural rather than a verification burden. The system then targets the performance bottlenecks unique to ML features, namely long windows, multi-table window unions, and skewed partitions, with pre-aggregation, a self-adjusting union scheduler, and time-aware repartitioning. If the reported results hold, the payoff is automatic offline/online feature consistency at millisecond latency, with 10x-20x online speedups over Flink and DuckDB and roughly half the memory of in-memory key-value stores.","feed_headline":"OpenMLDB claims 10x-20x faster online feature computation","feed_subtitle":"One extended-SQL script compiles for both training and serving, with pre-aggregated long windows and self-adjusting unions.","key_machinery":"The load-bearing mechanism is the unified plan generator plus the online long-window pre-aggregator. The plan generator compiles one extended-SQL feature script into C++ machine code shared by the offline and online executors, so offline/online consistency is produced by construction rather than checked after deployment. The pre-aggregator maintains a hierarchy of aggregators at chosen time granularities, updated asynchronously in the order enforced by a monotonic update log; a long-window request merges a few pre-aggregated levels with the unaggregated head of the stream instead of scanning raw tuples. Supporting the online engine are a refined skiplist that pre-ranks tuples by key and timestamp, a compact row encoding that stores variable-length strings by offset, and a self-adjusting window-union scheduler that rebalances key-to-thread mappings and applies subtract-and-evict increments.","core_discovery":"The central claim is that one relational system can serve both the offline and online stages of ML feature computation without sacrificing either consistency or latency, by treating ML features as first-class SQL constructs. The unified query plan generator compiles one extended-SQL feature script—including window functions, WINDOW UNION, and LAST JOIN—into optimized machine code shared by both engines, so the feature definition used at training time is the one executed at serving time. For online requests, the decisive mechanism is multi-level pre-aggregation: long-window aggregations are rewritten into merges of precomputed aggregates, and multi-table window unions are executed with dynamic key-to-thread rebalancing plus subtract-and-evict incremental updates. For offline jobs, multi-window parallelism and data-aware repartitioning remove serial window evaluation and skew bottlenecks. The paper reports 10x-20x higher online performance than Flink and DuckDB, 6x faster offline performance than Spark and GreenPlum, and roughly half the memory usage of in-memory key-value stores.","pith_inferences":["Beyond the paper: the correctness of pre-aggregation under out-of-order arrival is not settled; a minimal test is to insert an old-timestamp tuple after a coarse aggregate level has advanced and check whether a long-window query still includes it.","Beyond the paper: the unified-plan approach may generalize beyond SQL; the same one-definition, two-executors compilation strategy could apply to Python or DataFrame feature DSLs, with the same consistency guarantee.","Beyond the paper: combining long-window pre-aggregation with self-adjusting window unions should be tested jointly, since both optimize overlapping time windows and both must agree on how late tuples are reflected in precomputed aggregates.","Beyond the paper: if the compact encoding and skiplist indexing carry the memory savings, they could be adopted as a storage substrate for existing feature stores, decoupling retrieval from computation."],"forward_implications":["Feature deployment time drops from months to days or less, because the offline feature script is the online feature script; no separate streaming re-implementation is needed.","Long-window features (years of history or hotspot keys) go from hundreds of milliseconds to single-digit milliseconds by merging aggregator levels instead of rescanning raw tables.","Multi-table window unions keep stable throughput near 1 million tuples per second as window sizes grow, where static key-based streaming degrades toward about 1,000 tuples per second.","Offline feature jobs with multiple windows or skewed partitions run several times faster, reducing the number of machines needed for daily batch computation.","Memory savings of roughly half make it feasible to hold larger working sets in RAM, lowering hardware cost and reducing spilling."],"supporting_citations":[{"why":"Apache Flink, the primary streaming baseline whose static key distribution and O(log n) eviction motivate the self-adjusting window union.","marker":"[37]"},{"why":"Apache Spark, the main offline baseline that lacks multi-window parallelism and salting-safe skew handling.","marker":"[63]"},{"why":"DuckDB, the embedded analytical engine used as an online MicroBench baseline.","marker":"[52]"},{"why":"Redis, the in-memory store baseline for latency and memory comparisons and the hash-table structure replaced by the refined skiplist.","marker":"[8]"},{"why":"GreenPlum, the MPP database baseline in the real-time RTP comparison.","marker":"[9]"},{"why":"Prior interval-join work on OpenMLDB that motivates the self-adjusting load-balancing strategy for skewed keys.","marker":"[66]"},{"why":"Subtract-and-Evict sliding-window aggregation, the incremental computation used by the self-adjusting window union.","marker":"[58]"},{"why":"TalkingData dataset, the public real-world workload used for memory and performance evaluation.","marker":"[23]"}],"fun_headline_variants":["OpenMLDB: 10x-20x faster online feature computation","One SQL for training and serving, 10x-20x faster","OpenMLDB: pre-aggregation cuts long-window latency","From training to serving: one feature SQL, 10x-20x faster","OpenMLDB: unified SQL for ML features, 10x-20x speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The online speedups rest on the assumption that updates reach the long-window pre-aggregator in timestamp order; the paper enforces this with a lock-protected monotonic update log, but does not specify how a late tuple with an old timestamp is applied after a coarser aggregate level has already advanced, so such a tuple could be silently missed.","fun_headline_variants_meta":{"raw":{"variants":["OpenMLDB: 10x-20x faster online feature computation","One SQL for training and serving, 10x-20x faster","OpenMLDB: pre-aggregation cuts long-window latency","From training to serving: one feature SQL, 10x-20x faster","OpenMLDB: unified SQL for ML features, 10x-20x speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2744,"prompt_tokens":1054,"completion_tokens":1690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":1590}},"tokens_in":670,"tokens_out":1690,"duration_ms":12062,"temperature":1.0,"reasoning_tokens":1590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:22:32.997227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Insert a tuple whose timestamp lies in the past after the daily or monthly pre-aggregation level covering that time has already been computed, then run a long-window feature query over that interval and compare its result with a recomputation over raw data; a mismatch would show that low latency came at the cost of feature correctness.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Apache Flink, the primary streaming baseline whose static key distribution and O(log n) eviction motivate the self-adjusting window union."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Apache Spark, the main offline baseline that lacks multi-window parallelism and salting-safe skew handling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DuckDB, the embedded analytical engine used as an online MicroBench baseline."},{"cited_title":"https://github.com/redis/redis","cited_arxiv_id":null,"evidence_quote":"Redis, the in-memory store baseline for latency and memory comparisons and the hash-table structure replaced by the refined skiplist."},{"cited_title":"https://greenplum.org","cited_arxiv_id":null,"evidence_quote":"GreenPlum, the MPP database baseline in the real-time RTP comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior interval-join work on OpenMLDB that motivates the self-adjusting load-balancing strategy for skewed keys."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Subtract-and-Evict sliding-window aggregation, the incremental computation used by the self-adjusting window union."},{"cited_title":"https://www.kaggle.com/competitions/talkingdata-adtracking-fraud- detection","cited_arxiv_id":null,"evidence_quote":"TalkingData dataset, the public real-world workload used for memory and performance evaluation."}],"review_version":1}