{"id":"75f6e3be-fadd-4092-850a-d803b661fe4b","arxiv_id":"2412.03758","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Alternating semantic-map tokens and RGB tokens during autoregressive video continuation improves long-term consistency and FVD for driving videos.","lead":"The paper presents ARCON, a video-continuation method that makes an autoregressive vision model generate semantic scene sketches and detailed RGB frames in alternation. The authors report this yields longer, structurally consistent driving videos with competitive FVD scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported FVD gains for semantic tokens are computed at t=0.7, a temperature chosen on the same test sets, while the w/o-sem baseline is not given the same temperature search; the comparison may therefore be an artifact of test-set selection.","rationale":"The reader's weakest assumption concerns whether Uniformer/MAGVIT semantic tokens faithfully capture structure. That is a legitimate mechanism concern, but the central quantitative claim rests on Table 4's FVD/FID comparison. The most direct threat to that comparison is that the temperature t=0.7 was selected on the same test sets that produce the headline numbers, and the w/o-sem baseline was not given the same temperature search. Because autoregressive sampling temperature strongly controls diversity and stability (the paper itself shows FVD varies from 75.2 to 150.3 with t on this task), a baseline run at a suboptimal temperature could exaggerate the semantic-token gain. This is not an accusation of dishonesty; it is a missing control that a reader cannot check without code. The paper's own Table 4 provides the evidence for the concern, and the fix is straightforward. The reader's Uniformer concern is secondary: even if semantic maps are imperfect, the FVD comparison could still be valid, whereas an untuned baseline directly undermines the comparison. Thus I agree partially with the reader: we identify different primary weak points, but both point to the need for a conditional verdict with released code and a rigorous evaluation protocol.","tokens_in":14275,"tokens_out":9535,"duration_ms":101890,"concrete_test":"Run the 'w/o sem' model on the same BDD100K test and nuScenes validation sets at t = 0.2, 0.5, 0.7, and 1.0, with at least three random seeds per setting, and report FID/FVD with standard errors. Then compare best-vs-best (each model at its own optimal temperature) and also evaluate the semantic-token baseline at the w/o-sem optimal temperature. If the best w/o-sem FVD is within about 5% of the semantic-token baseline, or overlaps within error bars, the central claim that semantic tokens improve continuation is not established; if it remains clearly worse, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline FVD improvement in the strongest claim (91.6 to 73.6 on BDD100K and 84.2 to 57.6 on nuScenes, Table 4) is reported at inference temperature t=0.7. The same table sweeps t over 0.2, 0.5, 0.7, and 1.0 and retains the best value for the semantic-token model, using the very BDD100K test and nuScenes validation sets on which the final numbers are reported. No temperature sweep or variance estimate is reported for the 'w/o sem' condition. If the RGB-only model has a different optimal temperature, the comparison is not best-vs-best, and the apparent benefit of semantic tokens could be an artifact of choosing t=0.7 after seeing the test metrics. This is a test-set selection issue, not a critique of the idea: it directly affects the quantitative support for Contribution 2. A secondary issue is that the 77.4% semantic-RGB consistency in Fig. 5 is measured with the same Uniformer used to produce training pseudo-labels, which is circular for the consistency sub-claim, but the FVD claim is the load-bearing one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ARCON, an autoregressive large vision model for driving-video continuation. The main methodological idea is to interleave semantic tokens (obtained by running a pre-trained Uniformer segmenter on RGB frames and encoding the resulting maps with the MAGVIT-v2 tokenizer) with RGB tokens during both training and inference, so that the model first predicts semantic structure and then translates it into pixels. A secondary component is a flow-based feature-warping decoder that transfers texture from high-resolution input frames into the decoded low-resolution generated frames. Experiments on BDD100K and nuScenes report FVD improvements for the semantic-interleaved model over an RGB-only variant, reduced optical-flow magnitude decay, high semantic/RGB consistency, and qualitative long-video generation. The paper also compares against several driving-world-model baselines on nuScenes without fine-tuning.","tokens_in":14498,"tokens_out":4729,"duration_ms":48296,"significance":"If the central claim is correct, ARCON provides a practical and conceptually clean way to reduce degeneration in long autoregressive video generation by decomposing the task into semantic continuation and modality translation. The use of external metrics (FVD, RAFT optical-flow magnitude) for the main comparison is a strength, as is the explicit ablation of the interleaving scheme in Table 4 and the flow-warping decoder in Table 3. The paper is, however, primarily an empirical systems paper: the key evidence for Contribution 2 rests on a small number of FVD point estimates whose evaluation protocol is incompletely specified and whose temperature selection is asymmetric between the compared conditions. These issues are fixable, but until they are addressed the quantitative support for the headline claim is not conclusive.","major_comments":[{"comment":"The headline comparison between 'w/o sem' and the semantic-interleaved model is not a best-vs-best comparison as reported. Table 4 gives FVD values of 91.6 vs. 73.6 on BDD100K and 84.2 vs. 57.6 on nuScenes for 'w/o sem' versus the baseline, and then shows an inference-temperature sweep for t = 0.2, 0.5, 0.7, 1.0 whose t = 0.7 row exactly matches the baseline. No analogous temperature sweep is reported for 'w/o sem'. Since t = 0.7 was selected using FVD computed on the same BDD100K test set and nuScenes validation set on which the final numbers are reported, the apparent benefit of semantic tokens could partly be an artifact of asymmetric hyperparameter selection. Please report a temperature sweep for the 'w/o sem' condition, or otherwise justify why t = 0.7 is the appropriate comparison point for both models.","section":"Table 4, §4.5"},{"comment":"The evaluation protocol for the main comparison table is underspecified. Table 2 states only that FVD is computed on the nuScenes validation set; it does not specify the number of generated frames per clip, the number of clips, the generation resolution, the conditioning frames, the sampling temperature, or the FVD implementation/version. Table 3 gives such details for its own ablation (400 samples, 16-frame clips at 224x224), but Table 4 and Table 2 do not state whether the same protocol applies. Without this information the comparison with external baselines cannot be reproduced, and the internal 'w/o sem' versus baseline comparison may depend on clip length. Please state the exact protocol for every FVD/FID table and, ideally, report variance over multiple seeds or bootstrap intervals, since all numbers are point estimates.","section":"§4.3, Table 2"},{"comment":"The reported 77.4% semantic/RGB consistency is computed by re-segmenting generated RGB frames with the same Uniformer model that produced the training pseudo-labels. This makes the consistency sub-claim partially circular: the evaluator is the same function that defined the supervision signal, so high agreement may reflect the model reproducing the pseudo-label distribution rather than genuine structural alignment. I am not asking to remove the figure, but the claim should be corroborated with an independent segmenter or a different evaluation protocol (e.g., human ratings or a second segmentation model), and the circularity should be acknowledged in the text.","section":"Figure 5, §4.2"},{"comment":"The paper assumes that semantic maps encoded by the RGB-trained MAGVIT-v2 tokenizer faithfully preserve the high-level structure needed for continuation, but this is not validated. Section 3.2 says the same tokenizer is used for both modalities without specifying how a semantic map (a single-channel label array) is formatted before encoding (e.g., colorized RGB, repeated channels, or some other projection). Table 1's codebook-coverage statistics suggest the tokenizer does allocate codes to semantic inputs, but they do not show that decoded semantic maps are semantically accurate. Please describe the preprocessing of semantic maps and report reconstruction quality of semantic tokens (e.g., mIoU between original and re-encoded semantic maps), since this is a load-bearing assumption for interpreting the interleaving benefit as a semantic-structure effect.","section":"§3.1–3.2"}],"minor_comments":[{"comment":"There is an internal inconsistency in the number of tokens per frame: Section 3.1 says each 112x112x3 frame is converted into 784 1D tokens, while Section 4.1 says frames are encoded to 392 tokens per image. With a 14x14 spatial grid and two factorized codebooks, 392 is the plausible count, and it also matches the context-length arithmetic: 16,384 tokens would fit about 41 frames at 392 tokens per frame, not 20 frames. Please correct the 784 figure and the related 'no more than 20 images' statement.","section":"§3.1 vs. §4.1"},{"comment":"The paragraph beginning 'As exhibited in Tab. 2...' is duplicated verbatim from Section 4.3. Remove the duplicate or replace it with a discussion specific to the qualitative results.","section":"§4.4"},{"comment":"The row label 'semantic segmentation' is confusing because it heads both the 'w/o sem' and 'baseline' rows; consider renaming the block 'Semantic interleaving' and labeling the rows 'RGB-only' and 'ARCON (semantic + RGB)'.","section":"Table 4"},{"comment":"The paper states that a 20B model is trained, but it is not clear which quantitative results use the 7B model and which use the 20B model. Please state the model size for each table and figure, since the probing experiments in Section 4.2 explicitly concern the 7B model.","section":"§4.1"},{"comment":"The caption reports optical-flow mean magnitude but does not state whether the values are averaged over all pixels, frames, or clips, nor whether the RAFT flows are computed at the generation resolution or a downsampled resolution. Adding these details would make the figure interpretable.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the paper is honest about its limitations, but the quantitative evidence for the main claim currently rests on an asymmetric temperature comparison and under-specified FVD protocols. The internal token-count inconsistency (392 vs. 784) also suggests a need for a careful revision pass. I do not see a fundamental flaw in the approach itself; the issues are fixable within the manuscript's scope, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First the substance: the ARCON idea — alternating semantic tokens and RGB tokens within a single autoregressive pass, with semantic tokens predicted before RGB tokens each frame — is new relative to the cited prior work. CAR and GAIA-1 use auxiliary modalities as conditioning inputs; ARCON makes them sequential targets. The ablation in Table 4 supports the claim: FVD drops from 91.6 to 73.6 on BDD100K and from 84.2 to 57.6 on nuScenes with semantic interleaving, and the RAFT optical-flow magnitude in Fig. 4 independently shows less degeneration. That is a plausible and useful result.\n\nWhat the paper does well: it uses external metrics (FVD, RAFT), builds on open MAGVIT-v2 weights, and the codebook-usage analysis in Table 1 is a nice supporting observation. The flow-based texture-stitching decoder is a practical addition, though not the core contribution. The authors also state their own limitations in the conclusion, which I take as honest.\n\nThe soft spots, in order. First, the temperature issue raised in the stress-test is real. Table 4 sweeps t for the semantic model and appears to pick the best value (0.7) on the same BDD100K test and nuScenes validation sets used for the headline numbers; no sweep is reported for the w/o-sem baseline. That means the 91.6→73.6 comparison is not necessarily best-vs-best. It is not fatal — the gain is large — but it weakens the quantitative support for Contribution 2, and the paper should report the sweep for both conditions or use a held-out validation split. Second, the 77.4% semantic/RGB consistency in Fig. 5 is measured with the same Uniformer used to generate training pseudo-labels. That is circular for the consistency sub-claim; FVD is the load-bearing evidence. Third, Table 3 shows feature warping increases FVD on K700 at 3FPS (63.8→93.4), yet the text says the method \"demonstrated benefits in both in-distribution and out-of-distribution scenarios.\" That overclaim is directly contradicted by their own table. Minor: no error bars, underspecified evaluation protocol, and no code release.\n\nVerdict: the central idea is real and the paper is honestly engaged with prior work. The concerns are fixable with more evaluation, not structural invalidation. Who benefits: researchers working on autoregressive world models and driving simulation. I would send it to peer review, with a request for the temperature sweep, a non-circular consistency check, and error bars.","headline":"A genuinely new interleaving idea with real FVD gains, but the headline comparison is weakened by test-set temperature selection and a circular consistency metric; still worth refereeing.","tokens_in":15060,"tokens_out":2798,"would_cite":true,"duration_ms":26773,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ARCON claims that predicting semantic segmentation tokens before RGB tokens in an autoregressive large vision model makes long driving-video continuation more stable and consistent.","keywords":["auto-regressive video generation","video continuation","semantic tokens","large vision model","world model","autonomous driving","discrete tokenizer","texture stitching"],"falsifier":"Swap the semantic modality for a different structural signal (e.g., depth maps or optical flow) or shuffle the semantic class labels during training and re-measure FVD and optical-flow magnitude on nuScenes; if the long-horizon stability persists, the claimed role of semantic structure is not confirmed.","tokens_in":14056,"feed_emoji":"🚗","tokens_out":5903,"duration_ms":52885,"temperature":0.7,"pith_summary":"The paper sets out to show that an auto-regressive large vision model can generate long, temporally stable driving videos if it is asked to predict a semantic segmentation map before predicting each RGB frame. The proposed ARCON scheme encodes both RGB frames and semantic maps into the same discrete token vocabulary, then interleaves the two token types so that the model continues the scene structure first and translates that structure into pixels. On BDD100K and nuScenes, adding semantic tokens lowers FVD from 91.6 to 73.6 and from 84.2 to 57.6 respectively, and slows the tendency of long rollouts to freeze into near-stationary frames. A separate flow-based texture-stitching step transfers texture from high-resolution input frames to the low-resolution generated tokens, improving visual quality during decoding.","feed_headline":"Semantic tokens stop auto-regressive driving videos from stalling","feed_subtitle":"Predicting a segmentation map before each frame cuts FVD on nuScenes from 84.2 to 57.6.","key_machinery":"The load-bearing mechanism is the interleaved token sequence: semantic tokens from a Uniformer segmentation map and RGB tokens from the same frame are both encoded by the same MAGVIT-v2 tokenizer, and the LVM is trained to predict first the semantic tokens and then the RGB tokens of each future frame. This arrangement decomposes continuation into structure prediction plus structure-to-pixel translation, and it makes the model's next-token objective explicitly aware of high-level scene layout. The flow-based feature warping decoder is supporting machinery that borrows texture from reference frames during decoding rather than improving the tokenizer itself.","core_discovery":"ARCON's central claim is that semantic tokens are a cheap and effective scaffold for auto-regressive video continuation. Instead of predicting only RGB tokens, the model first generates the discrete tokens of a semantic segmentation map for the next frame, then generates that frame's RGB tokens; the two token streams share one MAGVIT-v2 tokenizer and one LLaMA-style transformer. The paper reports that this interleaving reduces degeneration into repeated frames, improves temporal consistency, and produces generated semantic maps that align closely with the generated RGB frames, reaching 77.4% re-segmentation accuracy on nuScenes. The method is a training and inference scheme plus a decoder-side texture transfer, not a new tokenizer or backbone.","pith_inferences":["A direct test the paper leaves open is whether the benefit is specific to semantics or generic to any compact structural modality; depth, optical flow, or lane maps could be swapped in and compared under the same protocol.","The 77.4% re-segmentation accuracy is measured with the same Uniformer that produced the pseudo-labels, so a stricter evaluation would use a different segmenter or human-annotated maps to rule out label-copying.","For very long rollouts the reference frame used by texture stitching recedes into the past, so the method may need periodic re-anchoring to recent high-quality frames to keep texture transfer effective.","If the interleaving works by giving the model an explicit 'what happens next' prediction, it could be combined with text or action-conditioned tokens to make driving world models controllable, a step this paper does not attempt."],"forward_implications":["Any auxiliary modality that is cheap to tokenize and structurally informative could be inserted in the same interleaved pattern, without retraining the tokenizer.","Because the same token vocabulary is used for both modalities, the scheme adds no new codebook or embedding parameters beyond the semantic input stream.","The FVD gains on nuScenes without fine-tuning indicate that the structural scaffold transfers across driving datasets, not just within the training distribution.","The texture-stitching decoder improves FVD on both in-distribution and out-of-distribution clips, so it is a general decode-time enhancement for token-based generators.","The strong RGB-semantic consistency implies the generated semantic maps could serve as a free structural signal for downstream modules such as planning or anomaly detection."],"supporting_citations":[{"why":"MAGVIT-v2 tokenizer that encodes RGB frames and semantic maps into discrete tokens and factorizes the codebook.","marker":"[85]"},{"why":"Uniformer segmenter that produces the semantic maps used as pseudo-labels during training and as the consistency evaluator.","marker":"[33]"},{"why":"Establishes the sequential large vision model paradigm of training on tokenized image and video frame sequences.","marker":"[2]"},{"why":"LLaMA architecture used as the auto-regressive transformer backbone for next-token prediction.","marker":"[59]"},{"why":"Open-source MAGVIT-v2 weights used to keep the encoder frozen and to fine-tune the decoder for texture stitching.","marker":"[41]"},{"why":"BDD100K provides the training videos and the in-distribution evaluation set.","marker":"[83]"},{"why":"nuScenes validation set used to measure cross-dataset generalization without fine-tuning.","marker":"[7]"},{"why":"Fréchet Video Distance is the metric used to compare generated videos against baselines.","marker":"[60]"},{"why":"RAFT optical flow is used to measure motion magnitude and quantify degeneration into stationary videos.","marker":"[57]"},{"why":"CrossNet-style reference-based super-resolution motivation for the flow-based feature warping decoder.","marker":"[90]"}],"fun_headline_variants":["ARCON interleaves semantic tokens to keep driving videos going","Semantic map tokens cut auto-regressive driving video stalls","ARCON: Semantic tokens first, then RGB, for smoother driving videos","Auto-regressive driving videos persist with ARCON's semantic scaffold"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that semantic segmentation maps produced by Uniformer and tokenized by MAGVIT-v2 faithfully capture the structure that matters for continuation, so the interleaving benefit is not an artifact of this particular pseudo-label pipeline.","fun_headline_variants_meta":{"raw":{"variants":["ARCON interleaves semantic tokens to keep driving videos going","Semantic map tokens cut auto-regressive driving video stalls","ARCON: Semantic tokens first, then RGB, for smoother driving videos","Auto-regressive driving videos persist with ARCON's semantic scaffold"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1266,"prompt_tokens":789,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":405}},"tokens_in":405,"tokens_out":477,"duration_ms":4849,"temperature":1.0,"reasoning_tokens":405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:07:03.736794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Swap the semantic modality for a different structural signal (e.g., depth maps or optical flow) or shuffle the semantic class labels during training and re-measure FVD and optical-flow magnitude on nuScenes; if the long-horizon stability persists, the claimed role of semantic structure is not confirmed.","supporting_citations":[{"cited_title":"Language model beats diffusion - tokenizer is key to visual generation","cited_arxiv_id":null,"evidence_quote":"MAGVIT-v2 tokenizer that encodes RGB frames and semantic maps into discrete tokens and factorizes the codebook."},{"cited_title":"Uniformer: Uni- fying convolution and self-attention for visual recognition","cited_arxiv_id":null,"evidence_quote":"Uniformer segmenter that produces the semantic maps used as pseudo-labels during training and as the consistency evaluator."},{"cited_title":"Bdd100k: A diverse driving dataset for heterogeneous multitask learning","cited_arxiv_id":null,"evidence_quote":"BDD100K provides the training videos and the in-distribution evaluation set."},{"cited_title":"Crossnet: An end-to-end reference-based super reso- lution network using cross-scale warping","cited_arxiv_id":null,"evidence_quote":"CrossNet-style reference-based super-resolution motivation for the flow-based feature warping decoder."}],"review_version":1}