{"id":"1b96be75-7016-42f9-b154-ec7d4c65c773","arxiv_id":"2412.19326","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding task-specific heads for tracking, grounding, and segmentation to multimodal LLMs via a three-stage training recipe improves both fine-grained visual tasks and general video understanding benchmarks.","lead":"Multimodal AI models often answer questions about images and video but miss fine details like exact locations, times, or object paths. This paper adds small specialized 'task heads' that teach the model to track, locate, and segment objects, and shows the model also gets better at ordinary visual conversations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 14.6% general-benchmark gain is confounded with 3M extra conversation samples; the 'only conversation data' run in Table 19 is trained but never evaluated, leaving TPO's mechanism unisolated.","rationale":"The reader's weakest assumption identifies both a data-volume confound and a pre-trained-head confound. I agree with the data-volume half and view it as the most load-bearing: the headline 14.6% improvement is measured against the released VideoChat2 checkpoint, which was not trained on the 3M conversation samples TPO introduces. The 'textualized task data' comparison in Table 11 is useful because it holds the data mixture roughly constant while removing the task heads, but it is only shown for MVBench and Charades-STA, not for the other benchmarks used in the headline. More importantly, the paper itself trained an 'only conversation data' model (Table 19, 42 GPU-hours) but never reports its benchmark performance, so the direct data-scaling control exists in the experimental log yet is omitted from the write-up. Making that evaluation public would settle whether the co-training recipe or the added corpus drives the general MLLM gains. If conversation-only reproduces most of the 14.6% gain, the central claim collapses to 'more data helps' and TPO's specific contribution is limited to the modest MVBench margin over the textualized-data baseline. I do not think rejection is warranted: the task-head results and the textualized-data comparison provide some evidence for a real mechanism, but the headline claim is currently unverifiable. For that reason the CONDITIONAL verdict stands, with the conversation-only control added as a required revision. I did not elevate the SAM2/CG-DETR inheritance issue to primary because the same-data conversation control is the single most decisive missing experiment for the paper's main quantitative claim; the head-only baselines would be a second necessary check.","tokens_in":29151,"tokens_out":9597,"duration_ms":86475,"concrete_test":"Evaluate the 'only conversation data' model listed in Table 19 (VideoChat2 trained on the same 3M conversation data used in TPO stage 3, without task data or task heads) on MVBench, VideoMME, MLVU, MMIU, and SEED-Bench2. Compare its average gain over the VideoChat2 baseline with the claimed 14.6% improvement. If the conversation-only run recovers most of that gain (e.g., more than 10 of the 14.6 points), the general MLLM improvement is attributable to added instruction data rather than to TPO's differentiable task-head co-training, and the headline claim should be recomputed with a data-matched baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TPO's three-stage co-training of differentiable visual task heads improves general MLLM benchmarks by 14.6% on average. The supporting comparison is VideoChat-TPO against the released VideoChat2 checkpoint, but TPO adds roughly 3M conversation samples (ShareGPT-4o, LLaVA-Hound-DPO, ShareGPT4V, VideoChat2-IT) plus about 0.77M task samples, per Table 18. The only same-data control reported is 'textualized task data' in Table 11, which retains the extra conversations while converting task labels to text; it already lifts MVBench from 60.4 to 64.7, leaving TPO a 2.1-point edge. That control is not reported for VideoMME, MLVU, MMIU, or SEED-Bench2, the other benchmarks behind the 14.6% headline. Critically, Table 19 lists a run trained with 'only conversation data' (no task data, no heads) but no benchmark scores for it appear anywhere in the paper. Without those numbers, the 14.6% improvement over VideoChat2 cannot be attributed to the TPO training scheme; it may be mostly data scaling. A related but secondary confound affects the zero-shot task tables: the mask head starts from SAM2 and the temporal head uses CG-DETR with InternVideo2 features, and Tables 8-9 include no head-only baseline without MLLM co-training, so part of the expert-comparable task performance may be inherited. The load-bearing missing control is therefore a matched-data comparison without the TPO-specific training signal.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Task Preference Optimization (TPO), a three-stage training scheme that attaches differentiable visual task heads (region, temporal, and mask heads) to a multimodal large language model through learnable task tokens. Stage 1 trains the LLM to emit task-assignment tokens, Stage 2 trains the task heads and tokens on dense visual supervision, and Stage 3 co-trains the MLLM and heads on a mixture of task data and conversation data. The method is instantiated on VideoChat2 and LLaVA-OneVision. Experiments report gains on general video/image understanding benchmarks (MVBench, VideoMME, MLVU, MMIU, SEED-Bench2, NExT-GQA) and on spatial grounding, moment retrieval, highlight detection, tracking, and referring segmentation, with an overall claimed 14.6% average improvement over baselines and synergistic benefits from multi-task co-training.","tokens_in":29528,"tokens_out":6763,"duration_ms":58400,"significance":"If the attribution to the TPO training scheme is established, the paper would make a practical contribution: it offers a scalable way to inject dense visual supervision into MLLMs without the information loss of serializing masks, boxes, and timestamps into text. The strongest evidence in the paper is the matched control in Table 11, where TPO's decoupled heads outperform textualized task data at fixed data, and the data-scaling ablation in Table 15, which isolates the effect of task-data volume. The three-stage recipe is clearly described and the code release is promised. However, as detailed below, the headline 14.6% gain is not yet isolated from the large amount of additional conversation data, and the zero-shot task results are not separated from the pre-trained expert components (SAM2, CG-DETR, InternVideo2) embedded in the task heads. The central mechanism is defensible, but several load-bearing controls are missing.","major_comments":[{"comment":"The reported '14.6% average improvement' is not a well-defined quantity. The underlying deltas are absolute point differences on benchmark-specific scales: MVBench +6.4, VideoMME +9.3, MLVU +10.2, MMIU +5.2, and SEED-Bench2 +40.8/+42.4. Averaging these point deltas gives about 14.6, but that number is neither a percentage improvement nor a meaningful pooled effect, and it is dominated by the SEED-Bench2 scale. In addition, the text in Section 4.1 says VideoChat-TPO achieves a '41.7% improvement on average performance' on SEED-Bench2, while Table 4 shows point gains of 40.8 and 42.4. Please report per-benchmark relative and absolute changes, define the aggregation protocol explicitly, or drop the pooled '14.6%' claim.","section":"Section 4.1, Tables 2 and 4; Abstract"},{"comment":"The central comparison is VideoChat-TPO against the released VideoChat2 checkpoint, but TPO adds roughly 3M conversation samples (VideoChat2-IT, ShareGPT-4o, LLaVA-Hound-DPO, ShareGPT4V) plus about 0.77M task samples. The only same-data control, the 'textualized task data' row of Table 11, is reported only for MVBench and Charades-STA and retains the extra conversation data; it is not reported for VideoMME, MLVU, MMIU, or SEED-Bench2, which are the other benchmarks behind the 14.6% headline. Table 19 lists an 'only conversation data' run but gives no benchmark scores for it anywhere in the paper. Without those scores, the 14.6% improvement over VideoChat2 cannot be attributed to the TPO training signal rather than to the additional conversation data. Please evaluate the only-conversation run on all headline benchmarks and report it beside VideoChat2 and VideoChat-TPO.","section":"Section 4.3, Tables 18 and 19"},{"comment":"The zero-shot task results are confounded by pre-trained expert components. The temporal head is built on CG-DETR with pre-trained InternVideo2 features, and the mask head is initialized from SAM2; only the region head is randomly initialized. Tables 5, 7, 8, and 9 do not include head-only baselines that use the same heads and task data without MLLM co-training. For example, Table 8 reports tracking results but has no SAM2-only baseline row, so the 'zero-shot' tracking success may be largely inherited from SAM2 rather than produced by TPO. Please add head-only baselines (with the same task data, with and without fine-tuning) and report the increment specifically attributable to MLLM co-training.","section":"Section 4.2 and Implementation Details"},{"comment":"The synergy experiment conflates adding tasks with adding data and parameters. Moving from row 4 to row 5 adds the mask head, mask training data, and mask token simultaneously, so the improvement in temporal grounding and referring segmentation could reflect more data or parameters rather than synergistic co-training; row 6 then adds conversation data, which is a separate data-scale effect already acknowledged in the text. An equal-data or total-parameter-matched control is needed before claiming that 'collaborative learning of visual tasks facilitates the transfer of knowledge' and produces synergy. The cleaner data-scaling comparison in Table 15 is a good start and should be extended to the co-training claim.","section":"Section 4.3, Table 12"}],"minor_comments":[{"comment":"There is a typo 'SEED2-Bench2' in the benchmarks paragraph; the correct name is SEED-Bench2. Also, Table 4's column headers 'SEED2I' and 'SEED2M' are not defined in the text, and the text's '41.7% improvement' should be reconciled with the table's point gains.","section":"Section 4.1, 'Benchmarks' paragraph and Table 4"},{"comment":"The caption text 'replace by simple head' is a fragment; please specify which head was replaced and confirm that all other training data and hyperparameters were held fixed.","section":"Table 11 caption"},{"comment":"The notation L_mllm is not defined; please state explicitly that it is the standard next-token cross-entropy loss over the dialogue and clarify how the three loss terms are combined when a training example contains both conversation and task supervision.","section":"Equation (1) and surrounding text"},{"comment":"The figures are visually dense: the distinction between solid and dotted arrows, the meaning of flame symbols, and the P2S/P2E labels are hard to parse at print scale. The captions should define every symbol and state which modules are frozen and unfrozen at each stage.","section":"Figure 2 and Figure 3 captions"},{"comment":"The caption contains a typo, 'Perfermance', and the caption line is duplicated. Please also state whether the LLaVA-OV-TPO row uses the same stage-3 conversation data as VideoChat-TPO.","section":"Table 16 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's main general benchmark, MVBench, was introduced by the same group that developed the VideoChat2 baseline, and the task heads are initialized from the same expert models used as comparison points. These overlaps are not disqualifying, but the authors should state them explicitly and ensure the matched-data controls address them. If the authors provide the missing only-conversation and head-only baselines, the paper could become a solid contribution; in its current form, the headline claims outrun the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a referee's time, but the headline number overstates what the evidence supports. TPO—learnable task tokens that route query embeddings to decoupled region/temporal/mask heads, trained in three stages—is a sensible engineering recipe, and the paper's best result is a controlled one: on MVBench, swapping textualized task labels for the decoupled heads with identical conversation data gives 2.1 points (64.7→66.8). That is a real, if modest, advantage for the representation choice. The fact that the same recipe transfers to LLaVA-OneVision (+8.1 MVBench) suggests the effect is not a VideoChat2 quirk. The training details are complete enough to reproduce, and the limitations section is honest.\n\nBut the 14.6% average improvement across five benchmarks is not defined. It mixes absolute point gains on different scales (SEED-Bench2 jumps ~40 points, MVBench 6.4), and unlike MVBench, VideoMME, MLVU, and MMIU have no same-conversation-data control. The appendix lists a run trained on 'only conversation data' (Table 19) but never reports its benchmark scores. Without that number, most of the 14.6% could be the 3M extra conversation samples, not TPO. That is the load-bearing missing control.\n\nThe synergy table (Table 12) also needs a closer look. Temporal R@0.5 goes 30.2 (T only) → 80.2 (T+R) → 36.7 (T+R+M) → 40.2 (+conversation). The drop when adding the mask head looks like a typo or an inconsistent evaluation setup; as printed, it undermines the 'synergistic gains' claim. And the zero-shot tracking table has no SAM2-only baseline, so the expert-comparable tracking scores are partly inherited from SAM2's weights, not clearly from TPO co-training.\n\nBottom line: the paper is a solid engineering contribution with one well-controlled positive result and several confounded headline claims. The missing 'only conversation data' evaluation is easy to run and would settle the central question. I'd send it to review, but ask for that control and for a cleanup of Table 12.","headline":"Useful recipe and one clean ablation; the headline 14.6% is confounded by extra conversation data, and Table 12 looks internally inconsistent.","tokens_in":30078,"tokens_out":3339,"would_cite":true,"duration_ms":29616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Task Preference Optimization improves multimodal LLMs by co-training differentiable vision-task heads, gaining 14.6% on benchmarks and zero-shot fine-grained skills.","keywords":["task preference optimization","multimodal large language models","visual task alignment","fine-grained visual perception","multi-task co-training","temporal grounding","referring segmentation","zero-shot grounding"],"falsifier":"Run a matched control that trains VideoChat on the same task datasets and the same heads but without TPO's three-stage schedule (for instance, joint training from the start on all task and conversation data), and compare MVBench and Charades-STA. If that control matches or exceeds VideoChat-TPO, then the task-token routing and staged training, rather than the dense supervision itself, would not be the cause of the reported gains.","tokens_in":28985,"feed_emoji":"🎯","tokens_out":7921,"duration_ms":68784,"temperature":0.7,"pith_summary":"Multimodal large language models (MLLMs) describe scenes fluently but routinely miss precise visual facts such as which cup the candy is under, when an action starts, or where an object is in a frame. This paper proposes Task Preference Optimization (TPO), which attaches small task-specific heads to an MLLM and uses dense visual labels to backpropagate a “task preference” signal while the model continues learning dialogue. The authors claim that this improves general image and video understanding by 14.6% on average across multiple benchmarks, and simultaneously gives the MLLM zero-shot fine-grained skills such as tracking, moment retrieval, and referring segmentation that rival supervised expert models. If true, TPO offers a way to add perceptual precision to conversational models without sacrificing their general abilities.","feed_headline":"Adding vision-task heads lifts multimodal LLMs by 14.6%","feed_subtitle":"Co-training a chatbot with box, moment, and mask heads sharpens video perception and adds zero-shot grounding.","key_machinery":"The machinery is a “task preference model”: a bank of learnable task tokens $\\{v_i\\}$ that the MLLM's language model transforms into task embeddings $e_i$, each feeding one of three task heads—a two-layer MLP region head for boxes, a CG-DETR-based temporal head for moment retrieval and highlights, and a SAM2-based mask head for segmentation and tracking. These tokens make task supervision differentiable into the LLM, because gradients from dense annotation losses update the MLLM's hidden representations through the tokens. TPO organizes training into three stages: instruction-tuned task assignment so the model emits the right special token, per-head training on task data, then joint co-training on task and conversation data, so the model first learns to recognize a task, then aligns each head, then lets the heads refine the shared visual backbone.","core_discovery":"The paper's central claim is that an MLLM can be improved for general dialogue by treating fine-grained visual annotations as differentiable task preferences rather than as text to be generated. TPO attaches several task heads—a region head, a temporal head, and a mask head—to hidden embeddings routed through learnable task tokens, and optimizes the MLLM with the sum of the dialogue loss, a task-assignment cross-entropy, and task-specific regression or classification losses. Using a three-stage local-to-global schedule (task assignment, head adaptation, joint co-training), the authors report that VideoChat-TPO and LLaVA-TPO outperform their baselines on image and video benchmarks while also acquiring zero-shot capabilities in spatial grounding, moment retrieval, highlight detection, tracking, and referring segmentation that are comparable to or better than supervised specialist models. The paper further claims that co-training multiple task heads together yields synergistic gains, with each individual task improving beyond its single-task training result.","pith_inferences":["A control that keeps data and heads fixed but replaces the three-stage recipe with one-stage joint training would isolate whether the schedule, rather than raw supervision, drives the gains; the paper's Table 12 cannot fully separate those factors because data and heads are added simultaneously.","If the mechanism is the routing of dense spatial-temporal gradients rather than the specific heads, then substituting depth, optical flow, or audio-visual sync heads should produce similar dialogue gains; this is a cheap empirical check of the “task preference” abstraction.","The reported synergy among heads suggests a design heuristic: choose a complementary set of heads that share underlying geometry (space, time, objects) but emit different output formats, and co-train them; this could turn head selection into a portfolio-optimization problem rather than a per-task choice.","The zero-shot tracking and segmentation results imply that MLLM-TPO could serve as a unified interface for interaction tasks such as pointing, following, and referring in robotic or embodied settings, an extension the authors do not test."],"forward_implications":["On MVBench, VideoChat-TPO scores 66.8 versus 60.4 for the VideoChat2 baseline, a 6.4-point gain with the same 16 input frames.","On VideoMME, the same model improves by 9.3 points overall and by 10 points on short videos without subtitles; on MLVU it gains 10.2 points in mean average.","Applying TPO to a second MLLM family, LLaVA-OneVision, lifts MVBench by 8.1 points and PerceptionTest by 6.9 points, suggesting the recipe transfers across models.","Zero-shot, VideoChat-TPO reaches 40.2 R@1 at IoU=0.5 on Charades-STA moment retrieval and 69.4 success on LaSOT tracking, beating specialist models that were fine-tuned on the target benchmarks.","Replacing textualized task data with TPO heads improves MVBench by 2.1 points while lifting moment retrieval R@0.5 from 18.6 to 40.2, indicating that dense supervision is preserved better through heads than through next-token prediction."],"supporting_citations":[{"why":"Supplies the direct-preference-optimization analogy that TPO adapts into differentiable visual task preferences.","marker":"[66]"},{"why":"Provides the pre-trained image encoder, mask decoder, and memory bank used to build the mask head for segmentation and tracking.","marker":"[68]"},{"why":"Provides the CG-DETR temporal grounding architecture and loss that the temporal head is initialized from.","marker":"[60]"},{"why":"Supplies the pre-trained InternVideo2 features that the temporal head consumes for moment retrieval and highlight detection.","marker":"[85]"},{"why":"Serves as the VideoChat2 baseline MLLM and the source of MVBench, the main video-understanding evaluation.","marker":"[48]"},{"why":"Acts as the second MLLM instantiation (LLaVA-OneVision) used to show TPO transfers across model families.","marker":"[41]"},{"why":"Provides the textualized tracking baseline that TPO's dense head supervision is compared against.","marker":"[97]"}],"fun_headline_variants":["Multimodal LLMs gain 14.6% with task-preference heads","Co-training task heads lifts MLLMs, adds zero-shot skills","TPO: vision tasks as preferences boost MLLMs by 14.6%","Zero-shot grounding from learnable task tokens","Task-preference co-training sharpens multimodal LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains are attributed to the TPO co-training recipe rather than to the extra volume of task data and the pre-trained expert heads; the synergy ablation in Table 12 adds data and heads together, so that separation is not directly tested.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal LLMs gain 14.6% with task-preference heads","Co-training task heads lifts MLLMs, adds zero-shot skills","TPO: vision tasks as preferences boost MLLMs by 14.6%","Zero-shot grounding from learnable task tokens","Task-preference co-training sharpens multimodal LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2212,"prompt_tokens":963,"completion_tokens":1249,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1159}},"tokens_in":579,"tokens_out":1249,"duration_ms":10703,"temperature":1.0,"reasoning_tokens":1159,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:42:18.432413+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a matched control that trains VideoChat on the same task datasets and the same heads but without TPO's three-stage schedule (for instance, joint training from the start on all task and conversation data), and compare MVBench and Charades-STA. If that control matches or exceeds VideoChat-TPO, then the task-token routing and staged training, rather than the dense supervision itself, would not be the cause of the reported gains.","supporting_citations":[],"review_version":1}