{"id":"862a8d3d-302a-4437-94a9-ea19acfa4454","arxiv_id":"2505.00335","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"NVTM uses flow-guided coordinate alignment and shared modulation codes from 2D grids to speed up and slim down implicit neural video representation.","lead":"This paper proposes NVTM, a video representation model that aligns frames with optical-flow-like motion before fetching pixel codes from 2D grids, so matching pixels across time share one modulation code. It reports roughly 3x faster encoding than NeRV-style models and better quality per parameter than the grid-based NVP baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Encoding-speed comparison omits RAFT flow-precomputation time; Table 2 counts only training time, so the '3x faster' claim may not hold end-to-end.","rationale":"The paper's central contribution is fast and parameter-efficient video INR, so the encoding-speed claim is the place where correctness matters most. Reading the full text, the reported timing is explicitly 'training time', while the method's encoding pipeline additionally needs RAFT flow computation. That omission is not an internal inconsistency, but it is a missing measurement exactly where the headline claim lives, and it can be settled by a wall-clock reproduction. I agree with the reader that flow-alignment accuracy is a real assumption, but it is partially supported by the empirical gains and the disclosed low-motion weakness (Appendix E.3, where NVP wins on Beauty), so I do not consider it the most decisive issue. The undefined log(0) in Eq. (3) is a genuine formal gap that should be fixed or clarified, but it is likely a specification typo rather than a reason to reject the experimental findings. Since both issues are addressable with additional measurements or a clarification, the reader's CONDITIONAL verdict stands unchanged.","tokens_in":18379,"tokens_out":8653,"duration_ms":90259,"concrete_test":"On a single A100, measure and report the full wall-clock for NVTM encoding of one UVG Dynamic sequence (e.g., Bosphorus, 600 HD frames): RAFT optical-flow generation plus the training steps/epochs listed in Table 2 for the ~1min, ~5min, and ~10min checkpoints. Compare that end-to-end time against NeRV/HNeRV/NVP training-only times under identical conditions. If NVTM's end-to-end time to reach roughly 30 dB is not at least 3x shorter than NeRV's, the headline speed claim must be restated as training-time-only. Also record whether the implementation special-cases t=tk in Equation (3) and what flow value is enforced for the first non-keyframe frame.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central speed claim (abstract; Section 4.2) is backed by Table 2, whose caption defines the comparison as training time. NVTM, however, requires RAFT optical flow as auxiliary supervision to train the alignment flow network (Section 4.1), and Section 3.2 explicitly says the optical flow is needed only in the encoding phase ('In the decoding phase, the optical flow is not necessary'). The paper never reports the wall-clock cost of generating RAFT flows on 600-frame HD UVG sequences or 100-frame MCL-JCV sequences. If that precomputation is comparable to the training times shown in Table 2, the 'fastest encoding speed' and 'over 3 times' statements describe only a sub-part of the encoding pipeline, not the end-to-end cost a practical service would pay. A secondary formal gap compounds this: Equation (3) writes Flow_{t->tk}(x,y) = log(t-tk) F_H(t)(x,y), which is undefined at the keyframe t=tk and, for integer frame indices, forces zero flow at t=tk+1; the paper does not state how the keyframe case is handled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NVTM, an implicit neural representation for video that groups temporally corresponding pixels via a learned alignment flow guided by RAFT optical flow, so that a shared modulation latent from per-GOP 2D grids can modulate a base SIREN network. The method is evaluated on dynamic subsets of UVG and MCL-JCV, where it reports faster encoding times and higher reconstruction quality than NVP and NeRV-style baselines, plus results on video compression, super resolution, frame interpolation, and inpainting. The central claims are empirical: a speed advantage of over 3x versus NeRV-style methods and parameter-efficiency gains of about 1.5-1.8 dB over NVP on dynamic sequences.","tokens_in":18614,"tokens_out":7075,"duration_ms":69294,"significance":"If the empirical claims hold, NVTM is a meaningful advance in video INR: it is the first grid-type parametric encoding that explicitly removes temporal redundancy by coordinate alignment, achieving both faster training and better parameter efficiency on dynamic HD content than prior grid-based methods. The paper is unusually thorough: it includes ablations over GOP size, neighbor set, adaptive normalization, and static features; reports random-seed stability; provides sequence-level tables; and extends to multiple downstream tasks. There is no equation-level circularity, since the alignment network is trained against an external pretrained RAFT system and reconstruction is measured on held-out coordinates. The main reservations are that the headline speed comparison omits the cost of RAFT flow precomputation, and that the formal definition of the flow scaling in Eq. (3) is incomplete.","major_comments":[{"comment":"The speed comparison is defined as training time only (Table 2 caption: 'i.e., the training time'), but NVTM requires RAFT optical flow as auxiliary supervision for the alignment flow network (Section 3.2, Section 4.1). The paper never reports the wall-clock cost of computing RAFT flows on 600-frame HD UVG sequences or 100-frame MCL-JCV sequences, nor does it state whether such cost is included in the 'encoding time' values. If, as is typical, flow estimation takes a non-negligible fraction of the 1-minute or 5-minute training intervals, the abstract's 'fastest encoding speed' and 'speed increase of over 3 times' would not describe the end-to-end encoding pipeline. The paper should either include the RAFT precomputation time in the speed comparison and re-evaluate the 3x claim, or explicitly qualify all speed statements as training-time-only and justify that the excluded preprocessing is acceptable in the intended deployment scenario.","section":"Section 4.2 and Table 2"},{"comment":"Eq. (3), Flow_{t->tk}(x,y) = log(t - tk) F_H(t)(x,y), is undefined at the keyframe t = tk because log(0) is undefined. Since the keyframe is defined as the first frame of each GOP and every pixel in that frame has t = tk, the paper must specify how the alignment flow and the aligned coordinate (x_k, y_k) are computed for the keyframe (e.g., by setting flow to zero). Additionally, the domain of t is not stated: if t is a normalized time in [0,1], log(t-tk) is negative for frames immediately after the keyframe, reversing the sign of the learned flow; if t is an integer frame index, log(t-tk)=0 at t=tk+1, which would force zero flow for the adjacent frame and contradict the intended temporal alignment. This is a load-bearing formal gap in the method definition and needs clarification.","section":"Section 3.2, Eq. (3)"}],"minor_comments":[{"comment":"The conclusion's statement that 'NVTM outperforms existing methods of implicit neural video representation on various video related tasks' is too broad: Table 16 shows NVP outperforms NVTM on the Beauty sequence (36.24 vs 35.63 PSNR). The claim should be scoped to dynamic video sequences, in line with the datasets actually used.","section":"Section 5 and Section E.3"},{"comment":"In the 'Output type' column, 'Pxiel' should be 'Pixel'.","section":"Table 17"},{"comment":"The sentence 'we analysis on zxyz corresponding to pixels' should be 'we analyze the modulation latent z_xyz corresponding to pixels.' Also, the notation z_xyz appears in the analysis but z_xyt is used in the methodology; this inconsistency should be fixed.","section":"Section 4.6"},{"comment":"The caption says 'random-valued flow in a notated scale of source video resolution'; it is unclear what 'notated scale' means and how the random flow was generated. Please clarify.","section":"Figure 4(b)"},{"comment":"The Clip operator is not defined; it should state whether clipping is to the closed interval [0,1] and whether the value is clamped after normalization.","section":"Eq. (5)"},{"comment":"There are several typos and grammatical issues, e.g., 'demostrate' in Section 2.2, 'a 10-' and 'a diminishing duration of the entire iterations' in Section B.3, and Figure 1's '1 minutes' should be '1 minute'.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The speed-claim omission is the main obstacle: if the authors report RAFT precomputation times and the 3x speedup holds end-to-end, the paper would be suitable for acceptance. The Eq. (3) issue is fixable with a few clarifying sentences. I would encourage the editor to ask for a re-submission with these points addressed rather than a rejection, as the core idea and empirical evaluation are otherwise solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: NVTM is a legitimate architectural contribution, not a repackaging. The new piece is combining flow-guided coordinate alignment with per-GOP 2D latent grids, and the empirical gains on dynamic HD video are real. But the headline speed claim describes training time only, and the paper should be clearer that RAFT precomputation is extra. There's also a small formal bug in Eq. (3) that needs a stated convention.\n\nThe paper does well: the ablation isolates each component (adaptive norm, static feature, modulation, neighbor grids), the t-SNE and flow-warping analysis give evidence that the alignment actually groups corresponding pixels, and the downstream results (SR, interpolation, inpainting, compression) are more thorough than most video-INR papers. The parameter-efficiency comparison in Table 3 is convincing: NVTM beats NVP by 1.5 dB with 10% fewer params on their selected UVG subset. That is a solid result.\n\nThe soft spots, in proportion. First, the speed comparison in Table 2 is explicitly 'training time', but the abstract says 'fastest encoding speed' without that caveat. Since NVTM needs RAFT flows for every frame in every GOP before training, the end-to-end encoding time includes flow precomputation. The paper doesn't report that cost. It might be small relative to 60 minutes, but at the 1-minute mark it could be the difference between the claimed 3x and parity. This is an experimental-reporting gap, not a flaw in the method.\n\nSecond, Eq. (3) writes log(t - tk) with t = tk giving log(0). The paper never says how the keyframe is handled. Also, if frames are indexed with t as integers, t - tk = 1 on the next frame gives zero flow, which makes the alignment trivially the identity for adjacent frames. They likely normalize time so this isn't as bad as it looks, but the convention should be stated. This is a minor formal gap, but it's exactly the kind that a referee should catch.\n\nThird, the evaluation is on manually selected dynamic sequences, and the paper admits NVTM loses to NVP on the low-motion Beauty sequence. That's honest, and the limitation is discussed in E.3, but the abstract's averages hide it. Readers should know the advantage is motion-dependent. It's fine as a conditional claim; just be upfront in the paper.\n\nOverall, the central argument holds: flow-aligned modulation with 2D latent grids is faster and more parameter-efficient than grid-only or frame-wise INRs on dynamic content. The flaws are addressable.\n\nThis paper deserves peer review. A serious referee should ask for the RAFT timing breakdown, a stated convention for Eq. (3), and a prominent discussion of the low-motion failure case. I'd bring it to a reading group and cite it if I were working on video INR.\n\nRecommendation: send to review; expect a conditional accept after revisions.","headline":"Real contribution on dynamic video INR; speed claim needs a caveat about RAFT precomputation and Eq. (3) has a minor formal gap.","tokens_in":19161,"tokens_out":2948,"would_cite":true,"duration_ms":29583,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NVTM represents a video as flow-aligned 2D grids, letting temporally matching pixels share one modulation code for faster, more parameter-efficient encoding.","keywords":["implicit neural representation","neural video compression","parametric encoding","temporally coherent modulation","optical flow alignment","video reconstruction","video inpainting","frame interpolation"],"falsifier":"Train NVTM on a synthetic dynamic video with known ground-truth optical flow, first with no occlusions and then with large occluded bands that force unrelated pixels into the same aligned grid cell. If the PSNR margin over NVP stays roughly 1.5 dB in both cases, the shared-latent mechanism is robust to broken correspondence; if the margin collapses when occlusions are added, the method's gain depends on flow accuracy exactly where the paper's weakest assumption says it does.","tokens_in":18163,"feed_emoji":"🎬","tokens_out":17744,"duration_ms":162870,"temperature":0.7,"pith_summary":"The paper proposes NVTM, an implicit neural video representation built from a small stack of 2D latent grids plus a learned alignment flow. The flow maps pixels that correspond to the same scene content across frames onto the same grid location, so each group of temporally matching pixels receives one shared modulation code instead of redundant per-frame parameters. This targets two weaknesses of earlier video INRs at once: grid-type methods train fast but waste parameters because they ignore motion, while frame-wise NeRV-style methods are parameter-efficient but slow to fit. On dynamic HD sequences, the paper reports that NVTM reaches the same reconstruction quality about three times faster than NeRV-style methods, and that it beats the grid-based NVP by 1.54 dB PSNR on UVG (Dynamic) with 10% fewer parameters, and by 1.84 dB on MCL-JCV (Dynamic). The consequence is that neural video representation does not have to choose between fast encoding and parameter efficiency.","feed_headline":"Flow-aligned grid codes make neural video encoding 3x faster","feed_subtitle":"The NVTM framework shares one code per matching pixel, cutting training time and beating NVP by up to 1.84 dB on dynamic HD video.","key_machinery":"The central object is the temporally coherent modulation latent: a single vector $z_{xyt}$ that modulates the base network and is shared by pixels the alignment flow judges to be the same scene content. The mechanism that creates it is the alignment flow network $F_H(t)$, a SIREN whose weights are produced by a hypernetwork over time; its output is scaled by $\\log(t-t_k)$ to compensate for the temporal distance to the keyframe of a short group of frames (GOP). Equation (4), $(x_k,y_k) = (x,y) + \\log(t-t_k)F_H(t)(x,y)$, warps each 3D coordinate to the keyframe's 2D coordinate, and Eq. (5) adaptively normalizes those coordinates before they look up the per-GOP 2D latent grid $G_k$. The grid lookup is what lets one latent serve many pixels; the adaptive normalization is what keeps grid cells from being wasted on sparse outlier regions.","core_discovery":"On its own terms, NVTM establishes that the temporal redundancy in video can be removed from a parametric encoding by conditioning the coordinates, not by adding parameters. For each group of frames, an alignment flow network $F_H(t)$ predicts a displacement, and every 3D coordinate $(x,y,t)$ is mapped to a keyframe-time coordinate $(x_k,y_k) = (x,y) + \\log(t-t_k) F_H(t)(x,y)$. The normalized coordinate indexes a 2D latent grid $G_k$, producing the modulation latent $z_{xyt} = G_k(x'_k,y'_k)$ that conditions the base network; because corresponding pixels land on the same grid cell, they share the latent. The reported consequences are a training-speed advantage of over 3 times versus NeRV-style methods and a parameter-efficiency advantage of 1.54 dB and 0.019 PSNR/LPIPS over NVP on UVG (Dynamic) with 10% fewer parameters, plus 1.84 dB and 0.013 on MCL-JCV (Dynamic). The same shared-latent mechanism is then shown to serve super-resolution, frame interpolation, and inpainting, and the GOP grid stack can be compressed with HEVC to reach bitrates competitive with standard codecs.","pith_inferences":["In principle, the same warped-coordinate shared-code mechanism should transfer to other parametric encodings such as hash grids or triplane features, where temporal redundancy is currently absorbed by adding more grid levels; this is an extension, not something the paper tests.","The paper's own GOP-size ablation suggests a motion-energy-based GOP scheduler could widen the gains on mixed-motion footage; the authors note the possibility but do not implement it.","On low-motion or occlusion-heavy footage, a hybrid that keeps a static grid for unreliable-correspondence regions could preserve the advantage; the paper's Beauty result marks the boundary where this would matter.","If the HEVC-on-grid-stack compression result generalizes, the latent grids are temporally coherent in a codec-friendly sense, which could inform future INR codec designs beyond this architecture."],"forward_implications":["At the 5-minute mark on UVG (Dynamic), NVTM already exceeds 29 dB PSNR while every NeRV-style baseline is still below 25 dB at 0.1 bpp, showing that fast encoding does not require giving up quality.","Grid-type video INRs can be made parameter-efficient without losing their training speed: NVTM uses 10% fewer parameters than NVP and still gains 1.54 dB on UVG (Dynamic) and 1.84 dB on MCL-JCV (Dynamic) in PSNR.","Because decoding is coordinate-based, one trained model supports spatial and temporal resampling, which the paper demonstrates as 2x super-resolution and 2x frame interpolation with PSNR and LPIPS gains over NVP and 3D ModSIREN.","Video inpainting follows from the same shared-latent mechanism: masked areas are decoded from temporally corresponding visible pixels, as demonstrated on DAVIS2017 with random box masks.","The per-GOP latent grids form a temporally coherent sequence, so compressing them with HEVC yields bitrate-distortion points comparable to H.264/HEVC and prior INR codecs while keeping the fast-encoding advantage."],"supporting_citations":[{"why":"It is the grid-type video INR baseline NVTM must beat, and it supplies the comparison configuration for the parameter-efficiency experiments.","marker":"[25]"},{"why":"It defines the frame-wise NeRV baseline whose slow encoding motivates the speed comparison.","marker":"[6]"},{"why":"It is the E-NeRV baseline compared in the encoding-speed and compression experiments.","marker":"[27]"},{"why":"It is the HNeRV baseline compared in encoding-speed and compression experiments.","marker":"[5]"},{"why":"It provides the modulated-SIREN scheme used by the base network to consume the shared latent.","marker":"[31]"},{"why":"It supplies the optical flow used to guide the alignment flow network during training.","marker":"[43]"},{"why":"It contributes the hyper-SIREN construction that lets the alignment flow depend on time with few parameters.","marker":"[17]"},{"why":"It is the Instant-NGP grid-type baseline compared on reconstruction quality and parameter efficiency.","marker":"[35]"},{"why":"It provides the HEVC codec used both as a compression baseline and to compress the grid stack.","marker":"[40]"}],"fun_headline_variants":["Temporally coherent modulation makes neural video encoding 3x faster","Flow-aligned 2D grids speed neural video training by over 3x","Shared latents for moving pixels boost video INR speed 3x","NVTM: 3x faster training with 1.54 dB PSNR gain on dynamic video","One latent per temporally matched pixel cuts video encoding time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the alignment flow correctly pairs the same scene point across frames; if it maps unrelated pixels to the same grid cell, one shared code encodes mismatched content, and the speed and parameter gains disappear.","fun_headline_variants_meta":{"raw":{"variants":["Temporally coherent modulation makes neural video encoding 3x faster","Flow-aligned 2D grids speed neural video training by over 3x","Shared latents for moving pixels boost video INR speed 3x","NVTM: 3x faster training with 1.54 dB PSNR gain on dynamic video","One latent per temporally matched pixel cuts video encoding time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1931,"prompt_tokens":1165,"completion_tokens":766,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":781,"completion_tokens_details":{"reasoning_tokens":666}},"tokens_in":781,"tokens_out":766,"duration_ms":6561,"temperature":1.0,"reasoning_tokens":666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:45:09.060353+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train NVTM on a synthetic dynamic video with known ground-truth optical flow, first with no occlusions and then with large occluded bands that force unrelated pixels into the same aligned grid cell. If the PSNR margin over NVP stays roughly 1.5 dB in both cases, the shared-latent mechanism is robust to broken correspondence; if the margin collapses when occlusions are added, the method's gain depends on flow accuracy exactly where the paper's weakest assumption says it does.","supporting_citations":[{"cited_title":"Scalable Neural Video Representations with Learnable Positional Features","cited_arxiv_id":"2210.06823","evidence_quote":"It is the grid-type video INR baseline NVTM must beat, and it supplies the comparison configuration for the parameter-efficiency experiments."},{"cited_title":"Advances in Neural Information Processing Systems 34, 21557–21568 (2021)","cited_arxiv_id":null,"evidence_quote":"It defines the frame-wise NeRV baseline whose slow encoding motivates the speed comparison."},{"cited_title":"In: Com- puter Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, Oc- tober 23–27, 2022, Proceedings, Part XXXV","cited_arxiv_id":null,"evidence_quote":"It is the E-NeRV baseline compared in the encoding-speed and compression experiments."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)","cited_arxiv_id":null,"evidence_quote":"It is the HNeRV baseline compared in encoding-speed and compression experiments."},{"cited_title":"In: Proceedings of the IEEE/CVF International Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"It provides the modulated-SIREN scheme used by the base network to consume the shared latent."},{"cited_title":"In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","cited_arxiv_id":null,"evidence_quote":"It contributes the hyper-SIREN construction that lets the alignment flow depend on time with few parameters."},{"cited_title":"ACM Transactions on Graphics (ToG) 41(4), 1–15 (2022)","cited_arxiv_id":null,"evidence_quote":"It is the Instant-NGP grid-type baseline compared on reconstruction quality and parameter efficiency."}],"review_version":1}