{"id":"8dca73dd-7de3-4c6d-87ec-ac513a59b242","arxiv_id":"2411.09242","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"FluidML combines graph splitting, dynamic programming, and greedy memory allocation to optimize ML inference memory layout, but its reported improvements are inconsistent across models and its headline numbers contradict its own tables.","lead":"FluidML is a compiler and runtime framework that picks tensor memory layouts and allocates memory to make neural network inference faster and smaller on CPUs. The paper reports speedups on some transformer models, but its headline claims of consistent gains do not match its own tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 contradicts the abstract's 'consistently reduce' latency claim: FluidML is slower than ONNX-MLIR on GPT-NEOX and VGG, and the advertised 'up to 25.38%' has no identifiable source in the reported data.","rationale":"The reader's strongest_claim correctly identifies the contradiction between the abstract and the evaluation. My stress-test agrees that this is the most load-bearing issue: the paper's own tables disprove 'consistently reduce ... compared to state-of-the-art approaches' because FluidML loses to ONNX-MLIR on GPT-NEOX and VGG, and loses to TVM on I-BERT memory. This is a direct falsification of the central claim, not a subtle assumption. The reader's weakest_assumption about DP cost additivity is plausible but secondary; even if the DP is perfectly calibrated, the advertised results are not achieved on the tested models, so the abstract's claims would still be false. My proposed test is a simple arithmetic check on the published tables that would settle whether the concern lands; it does not require rerunning experiments. Since the issue is a mismatch between claims and evidence, the paper cannot be accepted as is; the reader's REJECT verdict is appropriate. I mark agreement_with_reader as 'partial' because I do not identify the DP additivity assumption as the load-bearing failure, although I agree with the overall verdict. The verdict_should_be is UNCHANGED because my analysis does not alter the reader's decision.","tokens_in":13883,"tokens_out":6135,"duration_ms":56498,"concrete_test":"Perform a reanalysis of Tables 2 and 4: compute (ONNX-MLIR - FluidML)/ONNX-MLIR for every model/platform row in Table 2, and (TVM - FluidML)/TVM for every model in Table 4. Check two conditions: (1) the minimum latency speedup across all rows is positive; (2) the maximum latency speedup equals 25.38% and the maximum memory reduction equals 41.47%. If either condition fails, the abstract's quantitative claims are not supported by the paper's own data. Also attempt to locate any reported experiment yielding exactly 25.38% latency and 41.47% memory reduction; if none exists, the headline numbers are unexplained.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that FluidML consistently reduces end-to-end inference latency compared to state-of-the-art approaches. Its own Table 2 shows the opposite on two of five models: on AMD, GPT-NEOX runs in 9s with FluidML vs 6s with ONNX-MLIR (50% slower), and VGG runs in 19.0s vs 10.1s (88% slower); the Intel rows show the same pattern. The abstract's 'up to 25.38%' number is also inconsistent with the tables, which show much larger speedups on BERT (e.g., 80.9% on Intel) and no 25.38% value anywhere. The memory claim has a similar issue: Table 4 shows TVM uses 142.1 MB on I-BERT vs FluidML's 348.0 MB, so FluidML is worse than a state-of-the-art compiler on that model. These are not matters of interpretation; the paper's own evidence refutes the 'consistently' and 'compared to state-of-the-art' qualifiers. The scheduling DP's additivity assumption (Eq. 1) is a secondary concern about transferability, but the reported results already fail the advertised claim on the tested models, so the paper is not acceptable without either corrected claims or additional evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FluidML, an MLIR/LLVM-based framework that takes ONNX models and applies a two-part optimization: a dynamic-programming schedule that chooses per-tensor memory layouts using kernel timings collected from a custom virtual machine, and a greedy-by-size static memory allocator that reuses memory blocks across non-overlapping tensor lifetimes. The evaluation reports end-to-end latency and peak memory usage for five models (BERT, ConvBERT, GPT-NEOX, I-BERT, VGG) on Intel, AMD, and Apple platforms, comparing against ONNX-MLIR for latency and against ONNXRuntime, PyTorch, and TVM for memory.","tokens_in":14121,"tokens_out":4937,"duration_ms":47012,"significance":"If the reported gains were consistent and reproducible, FluidML would be a genuinely useful addition to the edge-inference toolchain, particularly for transformer models where layout and memory-access optimization can matter as much as kernel arithmetic. The idea of collecting real-world kernel timings through a VM and feeding them into a scheduling DP is pragmatic and reasonably standard as autotuning practice, and the static-allocation result is a sensible engineering contribution. The toy-model brute-force comparison in Table 3 is also a welcome attempt to bound the optimality loss from conflict resolution. However, the paper's central quantitative claims are not supported by its own data, and the transferability of the DP cost model to large real graphs is not adequately validated, so the current contribution is significantly overstated.","major_comments":[{"comment":"The abstract claims that FluidML can 'consistently reduce the end-to-end inference latency by up to 25.38%' for popular language models compared to state-of-the-art approaches. Table 2 directly contradicts this: on GPT-NEOX, FluidML takes 9 ms versus ONNX-MLIR's 6 ms on AMD and 8 ms versus 4 ms on Intel, and on VGG it takes 19.0 s versus 10.1 s on AMD and 10.0 s versus 5.5 s on Intel. The paper itself acknowledges in §5.1 that FluidML 'slightly lags behind on GPT-NEOX and VGG.' The claim of consistency is false as stated, and the 25.38% figure does not match the much larger or differently-directioned numbers in the table. Please correct the abstract to describe model-dependent results or provide additional evidence that reconciles these discrepancies.","section":"Abstract vs. §5.1, Table 2"},{"comment":"The abstract claims a peak memory reduction 'by up to 41.47%' compared to state-of-the-art approaches, but Table 4 shows that TVM uses 142.1 MB on I-BERT while FluidML uses 348.0 MB, making FluidML more than twice as memory-hungry on that model. Furthermore, the 41.47% figure does not correspond to any comparison in Table 4; for example, versus PyTorch on BERT the reduction is about 86.7%, and versus ONNXRuntime on GPT-NEOX it is about 94.8%. The memory claim should be recomputed, qualified by which baseline and which model, and explicitly acknowledge the I-BERT/TVM case.","section":"Abstract vs. §5.4, Table 4"},{"comment":"The scheduling DP relies on eval(node, layouts) returning real-world kernel execution times and assumes that the cost T(u,l'),(v,l) is stable and additive, i.e., independent of neighboring kernels, cache state, and concurrent memory traffic. The paper validates this only through two toy graphs in Table 3, reporting optimality gaps of about 17% and 9%. Since the paper claims a 'holistic' and 'graph-agnostic' optimization, a stronger validation is needed on the actual benchmark models, for example by comparing the DP-chosen layout schedule against a per-operator greedy layout choice or against a second heuristic, to establish that the cost model transfers beyond the toy settings. Without that, the end-to-end speedups on large graphs could be influenced by factors the DP does not capture.","section":"§3, Eq. (1), Alg. 3 and §5.3"}],"minor_comments":[{"comment":"The paper uses 'FLUID ML' in the body but the title and abstract use 'FluidML'; please standardize the spelling.","section":"Throughout"},{"comment":"There are several typos: 'sequnece' in Algorithm 2, 'untractable' in Section 2, 'FLU-IDML' in Section 5.4, and 'GPT-NEXO' in the Table 4 caption. These should be fixed.","section":"Alg. 2, §2, §5.4"},{"comment":"The table header is difficult to parse: the units are mixed (BERT, I-BERT, ConvBERT, and VGG in seconds, GPT-NEOX in milliseconds) and the column labels wrap confusingly. Please make the model names and units explicit in each column.","section":"Table 2"},{"comment":"The sentence 'At the same time, we noticed that for the Gemm operator. However, the loop reorder has also brought significant improvements;' is grammatically incomplete and should be rewritten.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The abstract's quantitative claims are not merely imprecise; they are contradicted by the paper's own tables. This is a serious reporting issue that the authors must fix before the paper can be considered acceptable. The technical core—layout DP with VM-based timings and static memory planning—could be a valid contribution if presented honestly with model-by-model results, but the current framing overstates consistency and the DP cost model is under-validated. I would not reject outright because the underlying idea is defensible, but the revision will need to either substantially correct the claims or add the missing evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper describes a real 30K-line CPU inference optimizer (ONNX front-end, MLIR/LLVM) that jointly schedules tensor memory layouts via longest-path graph decomposition, per-sequence DP using VM-measured kernel times, majority-voting conflict resolution, and greedy memory allocation. The combination of those known pieces into one generic framework is genuinely new in this form, and the authors did the field a service by building it and by admitting in Section 5.1 that they lag on GPT-NEOX and VGG. The abstract, however, says 'consistently reduce' and quotes 'up to 25.38%'—numbers their own Table 2 does not support and, in places, contradicts.\n\nWhat's good: memory planning beats PyTorch and ONNXRuntime by a wide margin on all tested models, and beats TVM on three of five. The two-graph brute-force comparison (17%/9% loss from conflict resolution) is a smart check, even if tiny. The ablation separating layout DP from loop reordering is useful. Citations of prior work (Liu et al. 2019, Pisarchyk & Lee) are fair.\n\nWhere it's soft: the headline claims. Table 2 shows FluidML 50–88% slower than ONNX-MLIR on GPT-NEOX and VGG; the advertised 25.38% appears nowhere. Section 5.1 admits this, but the abstract doesn't. Second, the DP cost model assumes a kernel's measured time is additive and independent of neighboring kernels/cache state; no validation is provided beyond two toy graphs. Third, no code or error bars. The 'linear time' claim in the conclusion has no complexity argument.\n\nWho's it for: systems/ML practitioners interested in CPU inference; a competent reader can extract useful design ideas, but the current paper would mislead anyone quoting the headline numbers. It deserves a serious referee, but only with the expectation of a major revision: correct the claims, release the artifacts, and add error bars or at least multiple runs. My own verdict is that the contribution is plausible but not acceptable in this form.","headline":"A serious CPU-inference optimization system whose own tables contradict the abstract's 'consistently reduce' latency and memory claims.","tokens_in":14668,"tokens_out":2225,"would_cite":false,"duration_ms":23262,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FluidML argues that a model's execution blueprint—memory layout for every tensor plus loop order for every kernel—can be globally optimized to cut inference latency by up to 25.38% and peak memory by up to 41.47% on CPU platforms.","keywords":["memory layout scheduling","inference runtime optimization","transformer inference","edge deployment","MLIR compiler","dynamic programming","static memory allocation","loop reordering"],"falsifier":"Compile BERT with the released FluidML code on the paper's Intel i7-13700 setup and compare end-to-end latency and peak resident-set size against ONNX-MLIR and the unoptimized version; if the latency reduction does not approach 25.38% or the memory reduction does not approach 41.47%, the central performance claim fails.","tokens_in":13592,"feed_emoji":"⚡","tokens_out":12037,"duration_ms":112601,"temperature":0.7,"pith_summary":"FluidML is a compiler-and-virtual-machine framework that argues a consequential point: for modern neural networks, the memory layout of every tensor and the loop order of every kernel form a global optimization problem, not a per-operator detail. The paper's method splits the computation graph into long operator sequences, uses dynamic programming over each sequence with real measured kernel timings to choose input and output memory layouts, and resolves conflicting layout choices across sequences by majority vote; a greedy-by-size static allocator then packs tensors with non-overlapping lifetimes into shared buffers. On three CPU platforms, FluidML reports end-to-end latency reductions up to 25.38% for MatMul-heavy transformers and peak-memory reductions up to 41.47% relative to ONNX-MLIR, ONNXRuntime, PyTorch, and TVM, with the largest gains on BERT-family models and small or negative gains on MatMul-light GPT-NeoX and VGG. If the reported numbers hold, edge deployments gain a model-agnostic way to speed up inference and shrink footprint without touching the model weights.","feed_headline":"FluidML cuts model latency up to 25% and memory use up to 41%","feed_subtitle":"The framework reshapes how tensors are laid out in memory, cutting latency for transformer models on edge CPUs.","key_machinery":"The load-bearing object is the per-edge memory layout schedule produced by the recursive longest-sequence decomposition and the dynamic-programming recurrence $$t_{v,l} = \\min_{u \\in N(v),\\, l' \\in L(u)} \\left( t_{u,l'} + T_{(u,l'),\\ldots,(v,l)} \\right),$$ where $T$ is the measured time of one kernel given its input and output layouts. The DP assumes those measured kernel times are stable and additive, so the cost of the whole schedule is the sum of per-kernel costs. After each sequence gets its optimal layout plan, a majority-vote pass resolves conflicts on shared edges, and a Greedy-by-Size allocator packs tensors into shared buffers by descending size subject to lifetime overlap. The virtual machine's role is to supply the real-world timings that make the cost model concrete.","core_discovery":"The central claim is that a neural network's execution blueprint can be transformed globally and cheaply. Instead of tuning each operator's kernel in isolation, FluidML decomposes the graph into the longest linear sequences of operators, applies a dynamic program whose cost model is populated by the virtual machine's real-world kernel timings (the recurrence in Eq. 1), and then reconciles the sequences' conflicting layout choices by majority vote. The resulting schedule fixes one memory layout per tensor edge and, for MatMul and GEMM, a cache-friendly loop order; the schedule is lowered through MLIR to LLVM IR and run on a JIT engine. The authors report that this blueprint transformation reduces end-to-end BERT-family inference latency by up to 25.38% and peak memory by up to 41.47% against the compared frameworks, while acknowledging that MatMul-light graphs such as GPT-NeoX and VGG see little or no latency benefit. The memory reduction is achieved by a static Greedy-by-Size allocator that reuses memory across non-overlapping tensor lifetimes and performs only a handful of allocation calls at runtime.","pith_inferences":["The same blueprint transformation could be retargeted to GPUs or NPUs by swapping the virtual machine's eval() oracle for device-specific timing, but the additivity assumption would need revalidation because cache hierarchies and concurrent streams make per-kernel timings less stable.","Because the cost model is populated by measured timings, FluidML could be extended into a per-device autotuner that recompiles the layout schedule for each target hardware and batch shape, using the JIT engine's measurements as feedback.","The majority-vote conflict-resolution policy is a heuristic; weighting votes by the time lost in each sequence or by sequence length might close part of the 9–17% gap to the brute-force optimum that the paper reports on toy graphs.","Extending loop reordering to bias-carrying GEMM and rectangular MatMul shapes is the most direct route to improving the weak GPT-NeoX and VGG results."],"forward_implications":["On MatMul-heavy transformer models like BERT, ConvBERT, and I-BERT, the joint layout and loop-order schedule should cut end-to-end latency by up to 25.38% relative to ONNX-MLIR without changing model weights.","Static Greedy-by-Size allocation should reduce peak memory by up to 41.47% compared with allocator-based frameworks, because non-overlapping tensor lifetimes share buffers and runtime allocation calls drop to single digits.","The longest-sequence decomposition plus per-sequence DP keeps the scheduling overhead linear, so the same compile-time machinery should scale to large graphs.","Ablations imply that loop reordering for MatMul/GEMM is the dominant source of speedup in transformer graphs, contributing roughly 67–77% of the MatMul gains, while layout DP alone contributes smaller gains.","For MatMul-light graphs such as GPT-NeoX and VGG, the framework's latency advantage should be small or negative, which the paper attributes to few, rectangular, or bias-carrying matrix multiplications."],"supporting_citations":[{"why":"TVM is a baseline compiler for latency and memory comparisons and the main deep-learning compiler prior art.","marker":"Chen et al., 2018"},{"why":"ONNX-MLIR is the primary compiler baseline whose latency FluidML is compared against on x86 platforms.","marker":"Le et al., 2020"},{"why":"PyTorch serves as a baseline for peak memory footprint measurements.","marker":"Paszke et al., 2019"},{"why":"ONNXRuntime is a baseline inference engine for both latency and memory comparisons.","marker":"developers, 2021"},{"why":"BERT is the main transformer benchmark model for the reported latency and memory results.","marker":"Devlin et al., 2018"},{"why":"GPT-NeoX is the benchmark model where FluidML reports small or negative gains, bounding the generality claim.","marker":"Andonian et al., 2023"},{"why":"Establishes that compile-time memory optimization is NP-complete, motivating the greedy static allocation approach.","marker":"Gergov, 1999"},{"why":"Supplies the greedy memory-allocation method for deep neural net inference that FluidML adapts.","marker":"Pisarchyk & Lee, 2020"},{"why":"ConvBERT is one of the benchmark models used to evaluate end-to-end latency and memory.","marker":"Jiang et al., 2020"},{"why":"I-BERT is a benchmark model for the reported latency reductions.","marker":"Kim et al., 2021"}],"fun_headline_variants":["FluidML reshapes memory layouts to slash edge inference latency and peak RAM","FluidML cuts LLM inference latency up to 25% and peak memory up to 41%","FluidML reorders tensor layouts to speed edge inference and cut memory","FluidML's global tensor scheduling cuts latency and peak memory","FluidML: open-source runtime that optimizes inference by reshaping tensors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dynamic program assumes that a kernel's measured execution time with a chosen memory layout is stable and additive—that it does not change depending on neighboring kernels, other memory traffic, cache state, or operating-system scheduling.","fun_headline_variants_meta":{"raw":{"variants":["FluidML reshapes memory layouts to slash edge inference latency and peak RAM","FluidML cuts LLM inference latency up to 25% and peak memory up to 41%","FluidML reorders tensor layouts to speed edge inference and cut memory","FluidML's global tensor scheduling cuts latency and peak memory","FluidML: open-source runtime that optimizes inference by reshaping tensors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001015,"raw_usage":{"total_tokens":4278,"prompt_tokens":930,"completion_tokens":3348,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":3245}},"tokens_in":546,"tokens_out":3348,"duration_ms":25198,"temperature":1.0,"reasoning_tokens":3245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:51:19.220615+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compile BERT with the released FluidML code on the paper's Intel i7-13700 setup and compare end-to-end latency and peak resident-set size against ONNX-MLIR and the unoptimized version; if the latency reduction does not approach 25.38% or the memory reduction does not approach 41.47%, the central performance claim fails.","supporting_citations":[{"cited_title":"GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch , 9 2023","cited_arxiv_id":null,"evidence_quote":"GPT-NeoX is the benchmark model where FluidML reports small or negative gains, bounding the generality claim."},{"cited_title":"Algorithms for compile-time memory optimization","cited_arxiv_id":null,"evidence_quote":"Establishes that compile-time memory optimization is NP-complete, motivating the greedy static allocation approach."},{"cited_title":"W., and Keutzer, K","cited_arxiv_id":null,"evidence_quote":"I-BERT is a benchmark model for the reported latency reductions."}],"review_version":1}