{"id":"b960d876-911f-4eda-957c-ddf301982656","arxiv_id":"2501.05460","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EPD Disaggregation separates encoding from prefill and decode in LMM serving, enabling parallel encoding, cached token transfer, and dynamic resource shifts that improve TTFT, memory, batch size, and SLO attainment.","lead":"This paper builds a serving system that separates the picture-encoding, text-prefill, and word-decoding stages of a multimodal AI model onto different GPUs, so image-heavy requests no longer block text processing and each stage can use its own memory budget. The result is lower first-token latency and higher fractions of requests meeting service-level targets, which matters for anyone operating image, video, or audio understanding models at scale.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IRP patch-sharding may change encoder semantics; without output-equivalence checks, reported gains may be for a modified model.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap. The system's novelty is architectural; its value depends on serving the same model. The paper provides no output-quality or token-equivalence validation, and the claim that patches are independent is questionable for the listed encoders. This is a concrete, testable concern, not a stylistic preference. I do not recommend changing the reader's conditional verdict: the concern is serious enough to block unconditional acceptance, but a targeted equivalence/accuracy experiment could settle it. I also note the reader's other points (e.g., SLO threshold selection in Appendix E.3) are secondary: even with fair SLOs, if IRP changes model semantics the gains are for a different model; conversely, if equivalence is established, the SLO calibration can be fixed in reruns. The paper does include real system implementation and released code, which makes the proposed test feasible.","tokens_in":95,"tokens_out":4284,"duration_ms":56456,"concrete_test":"Using the released EPD code, run the same Video-MME or NextQA inputs through (a) monolithic encoder+prefill and (b) EPD with IRP enabled, with all other settings identical. Compare the merged multimodal token embeddings (mean/max cosine distance) and final generated answers (e.g., exact match or LLM-judged correctness) for MiniCPM-V 2.6 and InternVL2-8B/26B at 2 and 4 high-res images. If token cosine distance exceeds a small tolerance (e.g., 1e-3) or answer accuracy drops by more than 1-2%, IRP is not lossless and headline gains must be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance gains (TTFT reductions, SLO improvements) rely on intra-request parallelization (IRP) in §3.2.2, which shards a request's image patches across encoding workers and merges the resulting tokens in prefill. This is only lossless if the LMM's vision encoder already treats patches independently. MiniCPM-V 2.6's SigLIP and InternVL2's InternViT are standard ViT-style encoders in which self-attention mixes information across all patches of an image (or of each tile); splitting patches across GPUs removes that cross-patch context. The paper asserts 'patches are encoded independently' without proof, and no experiment compares EPD's merged tokens or final outputs against monolithic encoding. If the merged tokens differ, EPD serves a different vision encoder, so the measured memory/SLO/TTFT gains do not hold for the original LMM; they hold only for a modified model whose output quality is unverified. This is a correctness risk, not a consensus disagreement: the architecture itself can be checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EPD Disaggregation, a serving framework that separates the multimodal encoding, prefill, and decode stages of an LMM onto dedicated GPU resources, and couples this with asynchronous token transfer, intra-request parallelization (IRP) that shards image patches across encoding workers, a black-box optimizer for resource allocation, and dynamic role switching. The claims are evaluated on MiniCPM-V 2.6, InternVL2-8B, and InternVL2-26B with synthetic workloads, NextQA, and Video-MME, reporting up to 15x lower peak memory, 22x larger batch sizes, 10x more images per request, 2.2x larger KV caches, and substantial TTFT and SLO-attainment improvements over vLLM and a multimodal extension of DistServe.","tokens_in":18911,"tokens_out":11350,"duration_ms":91715,"significance":"If the results hold, EPD is a valuable contribution to LMM serving: it treats the encoding stage as a first-class pipeline component, ships a released implementation, and reports broad improvements across three models and multiple workloads. The inclusion of a limitations section, an NPU deployment, and ablations of the optimizer and role-switching strengthens the empirical package. However, the significance is conditional on two evaluation properties the paper does not currently establish: that IRP's sharded encoding is semantically equivalent to monolithic encoding for the evaluated ViT-based encoders, and that the SLO thresholds used to compare goodput are not chosen post hoc in a way that favors EPD. The evidence is suggestive but not yet conclusive.","major_comments":[{"comment":"The IRP mechanism shards a request's image patches across encoding workers and merges patch-level tokens only at prefill, justified by the assertion that 'patches are encoded independently.' This property is not established for MiniCPM-V 2.6 (SigLIP) or InternVL2-8B/26B (InternViT), which are ViT-style encoders with self-attention across patches of an image or tile. If cross-patch attention is removed, the merged tokens differ from the monolithic encoder's output, and the measured TTFT and SLO gains apply to a modified vision encoder whose output quality is unverified. No experiment compares EPD's token representations or downstream accuracy (e.g., on NextQA or Video-MME) against monolithic encoding. Please add an equivalence check (task accuracy, logit-level comparison, or an explicit proof of patch independence for each encoder) and report results; if equivalence fails, the IRP gains should be reframed as serving a modified model.","section":"§3.2.2, §4.1"},{"comment":"Table 9 states that the SLO thresholds are 'empirically derived' in part from 'what is realistically achievable by both our method and the baselines on a fixed number (8 GPUs) used in experiments.' This makes the SLO-attainment comparison circular: the headline '90–100% improvement' is partly determined by selecting thresholds that EPD meets while baselines miss. Derive the thresholds from an independent, pre-specified rule (e.g., a target percentile of the monolithic single-request TTFT/TPOT distribution, or a fixed multiple of the no-load latency) and re-report SLO attainment and goodput under that rule. The NPU experiment (Appendix F.3) inherits this issue and should be treated the same way.","section":"Appendix E.3, §4.1"},{"comment":"The text states: 'at 4032×3024 resolution, 7× more for InternVL2-26B and 10× more for InternVL2-8B.' Table 2 shows the opposite: InternVL2-26B goes from 1 to 10 images (10×), while InternVL2-8B stays at 19 for both DistServe and EPD (no gain). The abstract's '10x more images per request' claim is therefore not supported by the table for InternVL2-8B. Please correct this mismatch and qualify the headline claim per model (e.g., '10× more for InternVL2-26B at 4K resolution').","section":"§4.3, Table 2"}],"minor_comments":[{"comment":"The TPOT entry for InternVL-26B at 6 images per request (0.95 s) is an order of magnitude larger than neighboring entries (0.08 and 0.15 s) and appears to be a typo (likely 0.095 s); please check.","section":"Appendix E.3, Table 9"},{"comment":"The first data row is labeled 'EDP' rather than 'EPD', and the table shows TPOT improving (0.031 to 0.025 s) when the optimizer is disabled; the text explains the goodput and TTFT degradation but does not comment on this TPOT difference, so please add an explanation or correct the table.","section":"§4.4, Table 5"},{"comment":"The statement that 'the vLLM baseline is equivalent to DistServe and is thus omitted' when decoding is excluded is not self-evident, since vLLM and DistServe may use different numbers of GPUs for the encode+prefill phases; please state the GPU allocation or include vLLM's TTFT result.","section":"§4.2"},{"comment":"Each condition appears to be a single 100-request run with no repeated trials or confidence intervals; for a goodput comparison, reporting run-to-run variance (e.g., across seeds or bootstrap resamples) would materially strengthen the SLO-attainment claims.","section":"§4.1, §4.4, Appendix E.1"},{"comment":"For InternVL2-8B, the maximum images per request is reported as 19 in both DistServe and EPD because of a context-length limit; the sentence 'Without this constraint, a larger number of images per request could be supported' is speculative and should be labeled as such.","section":"§4.3, Table 2"},{"comment":"The phrase 'up to 90-100% improvement' is ambiguous; using 'up to 100 percentage points improvement in SLO attainment' would be clearer.","section":"Abstract"},{"comment":"The limitations section acknowledges pipeline bubbles and throughput trade-offs but does not discuss the risk that IRP's sharded encoding may change the encoder's output semantics; please add a sentence addressing the equivalence concern raised in the major comments.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The main risk to the paper's central claim is the IRP equivalence issue; if the authors can supply accuracy-equivalence measurements or convincingly restrict IRP to a setting where patch independence provably holds, the paper would be substantially stronger. The SLO-threshold selection also needs a transparent, pre-specified rule. I do not see a novelty or authorship problem; the issues are technical and, in my view, correctable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it is a real system contribution. Disaggregating the multimodal encoding stage from prefill is a natural but underexplored extension of prefill-decode disaggregation, and the authors actually build it: asynchronous MM cache transfer, intra-request parallel encoding, a black-box config optimizer, and role switching, all evaluated on three LMMs, multiple datasets, and both GPUs and NPUs, with code released. Second, the headline numbers need a skeptical read, because the main latency and SLO gains may be for a modified model. Section 3.2.2 asserts that image patches are 'encoded independently' and can be sharded across workers. For the encoders used here—SigLIP and InternViT—that is only true if the sharding happens at the level of whole images or tiles the model already encodes independently. The paper does not say which granularity it uses, and it reports no output-equivalence check: no accuracy comparison, no token-similarity check between merged IRP tokens and monolithic encoding. If patch-level sharding is actually happening, the model being served is not the original LMM. This is the weakest point in the paper, and it is load-bearing for the TTFT and SLO claims. The SLO attainment numbers are also weaker than they look. Appendix E.3 says the TTFT/TPOT thresholds were chosen based on 'what is realistically achievable by both our method and the baselines.' That is circular calibration. Re-running with externally fixed SLOs would make the 90-100% improvement claim meaningful. The direct memory, batch-size, and TTFT measurements are more solid, though they lack error bars. There is also a smaller inconsistency: Section 4.3 says 10x more images for InternVL2-8B, but Table 2 shows 19 images for both DistServe and EPD at every resolution for that model. The 10x appears only for InternVL2-26B at the lowest and highest resolutions. Minor, but sloppy. Who is this for? People working on serving infrastructure for LLMs and LMMs will get value from the disaggregation design, cache management, and NPU results. The IRP equivalence question should be settled before the quantitative claims are taken at face value. I would send it to a serious referee—the engineering is real and the question matters—and I would also bring it to reading group, because the gap between assertion and evidence about patch independence is worth dissecting.","headline":"Useful disaggregated serving system for LMMs, but the patch-parallelism claim needs an output-equivalence check before the headline gains can be trusted.","tokens_in":19489,"tokens_out":7431,"would_cite":true,"duration_ms":58079,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-way split of encode, prefill, and decode onto dedicated GPUs lets multimodal servers handle 10x more images per request and cut first-token latency by up to 71%.","keywords":["large multimodal models","disaggregated serving","encode-prefill-decode","intra-request parallelization","time to first token","memory efficiency","dynamic role switching","multimodal token cache"],"falsifier":"Run identical prompts through the monolithic encoder and the IRP-sharded encoder and compare generated outputs on a benchmark such as Video-MME; if the sharded pipeline changes answers or accuracy, or if end-to-end TTFT at high concurrency no longer beats the monolithic baseline once token-transfer costs are included, the paper's central claim is refuted.","tokens_in":18521,"feed_emoji":"⚙️","tokens_out":7918,"duration_ms":60208,"temperature":0.7,"pith_summary":"Serving a large multimodal model means encoding images or video into tokens before the language model can generate text, and that extra stage collides with prefill on the same GPUs, inflating memory and delaying the first token. This paper claims that separating encode, prefill, and decode onto dedicated resources removes the collision, and that the split is practical because multimodal tokens can be cached and moved cheaply, encoding work within one request can be spread across GPUs, and instance roles can be reassigned as demand shifts. If the claim holds, multimodal serving can support more images per request, far larger batches, and tighter latency targets on the same hardware. The paper reports up to 15x lower peak memory, 22x larger batch sizes, 10x more images per request, and up to 71% lower time to first token against baselines that bundle or partially aggregate these stages.","feed_headline":"Three-stage split cuts LMM first-token time by up to 71%","feed_subtitle":"Encoding, prefill, and decode on separate GPUs also lifts batches 22x and SLO attainment to 90-100 percent.","key_machinery":"The load-bearing object is the EPD pipeline itself: encode, EP-migration, prefill, PD-migration, and decode, each on dedicated GPU instances. Three mechanisms carry the argument: the MM cache and asynchronous token transfer that decouple encode from prefill; intra-request parallelization (IRP), which shards one request's image patches across encoders and merges their tokens at prefill; and the dynamic role switcher plus black-box optimizer that keep the stage-wise resource split matched to the workload. IRP is the mechanism that directly converts the disaggregation into lower TTFT, while the MM cache is what makes the extra transfer step affordable.","core_discovery":"The central claim is that the encoding stage of an LMM can be disaggregated from prefill and decode, and that this three-way split is the right systems abstraction for multimodal serving. The paper argues that bundling encoding with prefill creates interference and forces the LLM's weights and KV cache to share memory with the vision encoder, which caps batch sizes, image counts, and cache sizes. EPD assigns each stage its own GPU instances, moves multimodal tokens through an MM cache with asynchronous transfer, shards a single request's image patches across encoders via intra-request parallelization, picks per-stage configurations with a black-box optimizer, and switches instance roles dynamically when the workload shifts. Measured against monolithic and prefill-decode-disaggregated baselines on MiniCPM-V 2.6 and InternVL2-8B and InternVL2-26B, the paper reports up to 15x lower peak memory, 22x larger batches, 10x more images per request, 2.2x larger KV caches, 90-100% SLO attainment gains, and up to 71% lower TTFT.","pith_inferences":["Beyond the paper: the decisive test the paper leaves undone is an output-quality comparison between monolithic and IRP-sharded encoding; until that exists, the speedups are for a pipeline whose equivalence to the original model is unverified.","Beyond the paper: the memory savings suggest a heterogeneous deployment where low-memory high-compute devices handle encoding and high-memory devices handle prefill, turning the 15x memory reduction into a cost saving rather than only a capacity gain.","Beyond the paper: in edge-cloud settings the same split could keep raw images on edge devices and transmit only multimodal tokens, extending the paper's privacy direction into a concrete architecture.","Beyond the paper: a natural stress test is whether IRP's TTFT gains hold when the bottleneck moves from encode compute to token-transfer bandwidth, such as many small images spread across distant nodes."],"forward_implications":["Encoding becomes independently scalable: cluster operators can add encoder GPUs without touching LLM weights or KV-cache memory.","Intra-request parallelization converts encoding latency into parallel work, shrinking TTFT by up to 71% on the models tested.","Freeing encoding workers from LLM weights and KV caches raises the ceiling on batch size (up to 22x), images per request (up to 10x), and KV-cache allocation (up to 2.2x).","Dynamic role switching lets a deployed cluster reallocate instances among encode, prefill, and decode in under 0.7 seconds, absorbing workload shifts without restarting in-flight requests.","The framework extends to audio inputs and to NPU hardware, with larger gains when encoding is a greater share of end-to-end latency."],"supporting_citations":[{"why":"Supplies the prefill-decode disaggregation baseline and the simulator that EPD extends to multimodal workloads.","marker":"Zhong et al., 2024"},{"why":"Supplies the monolithic serving baseline and the execution engine that EPD repurposes for its distributed pipeline.","marker":"Kwon et al., 2023"},{"why":"Provides the phase-splitting motivation and baseline for separating compute stages in LLM serving.","marker":"Patel et al., 2024"},{"why":"Defines the MiniCPM-V 2.6 architecture evaluated in the SLO, TTFT, and memory experiments.","marker":"Yao et al., 2024"},{"why":"Defines the InternVL2-8B and InternVL2-26B models used across evaluation.","marker":"Chen et al., 2024"},{"why":"Supplies the Video-MME video-understanding benchmark used to measure SLO attainment and TTFT.","marker":"Fu et al., 2024"},{"why":"Supplies the NextQA video question-answering benchmark for real-world workload evaluation.","marker":"Xiao et al., 2021"}],"fun_headline_variants":["EPD splits LMM into encode, prefill, decode for 71% faster TTFT","Three-stage disaggregation cuts LMM memory 15x and TTFT 71%","Disaggregating LMM encoding boosts batch 22x and SLOs","Role-switching EPD framework improves LMM serving efficiency","Separate encode, prefill, decode to slash LMM first-token time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a request's image patches can be encoded independently across GPUs and then merged without changing the token representations or the model's output, but the paper reports no accuracy or output-equivalence check.","fun_headline_variants_meta":{"raw":{"variants":["EPD splits LMM into encode, prefill, decode for 71% faster TTFT","Three-stage disaggregation cuts LMM memory 15x and TTFT 71%","Disaggregating LMM encoding boosts batch 22x and SLOs","Role-switching EPD framework improves LMM serving efficiency","Separate encode, prefill, decode to slash LMM first-token time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00085,"raw_usage":{"total_tokens":3753,"prompt_tokens":1059,"completion_tokens":2694,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":2589}},"tokens_in":675,"tokens_out":2694,"duration_ms":17374,"temperature":1.0,"reasoning_tokens":2589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:24:09.103371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run identical prompts through the monolithic encoder and the IRP-sharded encoder and compare generated outputs on a benchmark such as Video-MME; if the sharded pipeline changes answers or accuracy, or if end-to-end TTFT at high concurrency no longer beats the monolithic baseline once token-transfer costs are included, the paper's central claim is refuted.","supporting_citations":[{"cited_title":"Splitwise: Efficient generative llm inference using phase splitting","cited_arxiv_id":null,"evidence_quote":"Provides the phase-splitting motivation and baseline for separating compute stages in LLM serving."},{"cited_title":"Next-qa: Next phase of question-answering to explaining temporal actions","cited_arxiv_id":null,"evidence_quote":"Supplies the NextQA video question-answering benchmark for real-world workload evaluation."}],"review_version":1}