{"id":"d2179f0b-3fde-4b25-adcc-f0530d3b9ff2","arxiv_id":"2412.12561","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TellTrack improves referring multi-object tracking by adding collaborative query matching, direct query-level language infusion, and a reordered cross-modal encoder, achieving SOTA HOTA on Refer-KITTI and Refer-KITTI-V2.","lead":"TellTrack is a new system for referring multi-object tracking that follows only the objects described in a text prompt, like \"a person riding a bike,\" across video frames. It reports higher tracking accuracy than previous methods on two autonomous driving benchmarks, with three targeted design changes to the transformer pipeline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed SOTA margins rest on an unspecified threshold-selection split, and the sweep tables do not reproduce the final HOTA, so the headline numbers are not currently checkable.","rationale":"The paper's contribution is an RMOT architecture, and the decisive evidence is the quantitative comparison in Table 1. The most load-bearing threat to that evidence is not the architectural novelty but the evaluation protocol around the two inference thresholds. Tables 4 and 5 sweep beta_ref and beta_obj without specifying whether they use a validation split or the test split, and the HOTA swing across beta_ref is roughly 7 points; that is exactly the scale of the claimed improvement over the prior best. If the thresholds were selected by evaluating the test set, the reported margins would be optimistic in a way that cannot be assessed from the paper. The 37.22 versus 37.67 discrepancy between the sweep tables and Table 1 sharpens the problem: either the sweep is on a different split, which must be stated, or the final number is not reproducible from the described inference configuration. I also considered the ablation inconsistencies around RIQA (Table 2 vs. Table 3) and the abstract/body +3.42 versus +3.22 difference; both are real reporting weaknesses, but they mainly affect attribution of gains and editorial consistency, not the existence of a SOTA-level system. The CQM idea itself is coherent: allowing detection queries to match existing targets in intermediate decoder layers increases detection-query activation frequency, and the loss formulation in Sec. 3.4 is consistent with that description. No internal logical contradiction forces rejection. I therefore keep the reader's CONDITIONAL verdict: the central claim is plausible but not yet verifiable without clarifying the validation/test protocol and reconciling the numerical mismatch.","tokens_in":15541,"tokens_out":6078,"duration_ms":53369,"concrete_test":"Ask the authors to state explicitly whether Tables 4 and 5 were computed on the Refer-KITTI-V2 validation or test split, and to release the split assignments. Then run one decisive check: select beta_ref and beta_obj using only the official validation split, freeze those thresholds, evaluate the full TellTrack model on the test split, and compare with baselines evaluated under the same protocol; if the resulting HOTA does not exceed the baselines by the claimed margins, the headline superiority is not established. As a minimal reproducibility check, re-run the exact configuration beta_ref=0.3, beta_obj=0.7 on the partition used for Table 1 and verify that it yields HOTA=37.67; if it instead yields 37.22, Tables 4/5 and Table 1 are not measuring the same model or split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TellTrack improves HOTA by +3.22 on Refer-KITTI and +1.49 on Refer-KITTI-V2 (Sec. 4.3, Table 1). These numbers depend on the inference thresholds beta_ref=0.3 and beta_obj=0.7 (Sec. 4.1). The paper never states the split on which Tables 4 and 5 are computed: it sweeps beta_ref from 0.2 to 0.8 and beta_obj over {0.3, 0.7, 0.9}, then reports that TellTrack 'achieves the best performance when beta_ref=0.3, beta_obj=0.7' (Sec. 4.4). If the sweep was performed on the same test partition used for Table 1, the comparison is biased because the baselines are quoted at fixed thresholds without equivalent test-set tuning. The sensitivity is large: HOTA ranges from 29.99 to 37.22 across beta_ref in Table 4, so threshold overfitting could plausibly account for a substantial fraction of the claimed margin. Moreover, the sweep tables are internally inconsistent with the final result: the configuration claimed optimal in Tables 4 and 5 gives HOTA=37.22, DetA=23.38, AssA=59.25 on Refer-KITTI-V2, whereas Table 1 reports HOTA=37.67, DetA=24.09, AssA=58.92 for the same dataset. This could be explained if Tables 4/5 were computed on a validation subset and Table 1 on the test subset, but no such split is stated anywhere in Section 4. Until the split is documented and the final configuration is re-run on the exact partition that produced Table 1, the reported superiority cannot be independently verified. A secondary issue is that the RIQA attribution is ambiguous (Table 2's RIQA-only row gives 35.66, while Table 3's in-decoder row gives 35.37 and its pre-decoder both-queries row gives 36.23), but this affects which component earns credit rather than the headline comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TellTrack, an end-to-end transformer-based referring multi-object tracking method with three main components: Collaborative Query Matching (CQM), which lets detection queries match existing targets in auxiliary decoder layers to compensate for the imbalance between newborn and existing targets; Referring-Infused Query Adaptation (RIQA), which injects a sentence embedding into decoder queries either before or inside the decoder; and a Cross-Modal Encoder (CME), which applies deformable attention to the image feature pyramid before cross-attending to word embeddings. Experiments on Refer-KITTI and Refer-KITTI-V2 report HOTA of 55.63 and 37.67, respectively, exceeding the previous best results by 3.22 and 1.49 HOTA points. Ablations in Sec. 4.4 claim that each proposed component contributes positively, and threshold sweeps are reported for the inference thresholds beta_ref and beta_obj.","tokens_in":15827,"tokens_out":6983,"duration_ms":57617,"significance":"If the reported results are reproducible, the work is a useful engineering contribution to RMOT. The CQM idea of increasing detection-query activation during auxiliary training is simple and plausible, and the RIQA and CME components address a real limitation in existing indirect language fusion. The paper has several strengths: it trains end-to-end, requires no post-processing such as NMS, reports a comparable inference speed of 15.08 FPS versus 15.13 FPS for TempRMOT, and provides qualitative examples showing language-conditioned behavior. At the same time, the claimed margins rest on a threshold-selection protocol that is not documented, and the ablation tables contain internal inconsistencies. These issues are fixable, but until they are resolved, the headline improvement cannot be independently verified.","major_comments":[{"comment":"The paper never states whether the beta_ref and beta_obj sweeps in Tables 4 and 5 were performed on a validation split or on the same test partition used for Table 1. The final inference thresholds beta_ref=0.3 and beta_obj=0.7 are selected from these sweeps, and Table 4 shows HOTA varying from 29.99 to 37.22 across beta_ref, so test-set threshold selection could account for a substantial part of the claimed 1.49-point margin on Refer-KITTI-V2. Please specify the split used for the sweeps; if thresholds were tuned on the test set, the comparison should be re-run with thresholds fixed independently of the test partition.","section":"Sec. 4.1, Sec. 4.4, Tables 4-5"},{"comment":"The final configuration is internally inconsistent across tables. Table 1 reports HOTA=37.67, DetA=24.09, AssA=58.92 for TellTrack on Refer-KITTI-V2, whereas both Table 4 (beta_ref=0.3) and Table 5 (beta_obj=0.7) report HOTA=37.22, DetA=23.38, AssA=59.25 for the configuration stated as optimal in Sec. 4.4. The text does not explain this discrepancy. If Tables 4-5 were computed on a validation subset and Table 1 on the test subset, that split must be stated explicitly so the results can be checked.","section":"Table 1 vs. Tables 4-5"},{"comment":"The ablation tables do not agree on the effect of RIQA. Table 2 lists the RIQA-only condition as HOTA=35.66, but Table 3 reports 35.37 for in-decoder RIQA, 35.45 for pre-decoder with detection queries only, 35.88 for pre-decoder with track queries only, and 36.23 for pre-decoder with both; no row in Table 3 equals 35.66. In addition, the text attributes a +3.32 AssA and +8.02 AssPr improvement to RIQA, but those numbers match Table 3's pre-decoder-both row relative to the base row, not the RIQA row in Table 2, whose differences are +2.84 and +0.48. Please state which RIQA variant is used in Table 2 and reconcile the two tables.","section":"Table 2 vs. Table 3, Sec. 4.4"},{"comment":"All reported results, including the headline margins of +3.22 and +1.49 HOTA, come from a single run with no error bars or significance testing. Given the sensitivity of HOTA to threshold choices documented in Table 4, and the fact that the margin over HFF-Tracker on Refer-KITTI-V2 is 1.49 points, the paper should report results over multiple seeds or at least state the evaluation protocol used to produce Table 1 so that the robustness of the claimed improvement can be assessed.","section":"Sec. 4.3, Table 1"}],"minor_comments":[{"comment":"The abstract reports a +3.42% improvement while the introduction, Sec. 4.3, and conclusion report +3.22%; please reconcile the headline number and specify which dataset and configuration each number refers to.","section":"Abstract, Sec. 4.3, Sec. 6"},{"comment":"There are several typographical errors, including 'activited' in Sec. 3.1, 'catenating' around Eq. (12), and inconsistent spacing in 'TellTrack' in Fig. 1; a copyedit would improve readability.","section":"Sec. 3.1, Sec. 3.2.2, Fig. 1"},{"comment":"The indicator in Eq. (17) is written as 'I refers to I{cDet!=empty}'; please define the indicator function formally, as the current notation is ambiguous.","section":"Eq. (17)"},{"comment":"Please consider adding a reproducibility statement, including whether code and trained models will be released, since the paper does not currently state this.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid but incremental engineering contribution, and the core ideas are reasonable for the RMOT task. The main risk is experimental protocol transparency: the undocumented threshold-selection split and the inconsistencies between Tables 1, 4, and 5 mean that the reported gains are not currently checkable. These issues are fixable with additional detail and re-runs, so I would not recommend rejection on the current evidence. The scope fits a computer vision venue, and the novelty disclosure relative to TransRMOT and TempRMOT is adequate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TellTrack is a competent integration of three known transformer ideas into referring multi-object tracking: one-to-many query matching during auxiliary decoder layers, prompt-query fusion at the decoder, and reordered deformable/cross-modal attention in the encoder. The CQM component is a sensible answer to a real imbalance problem—detection queries get far fewer activations than track queries—and the paper deserves credit for identifying that issue and for testing its components in ablations. The experiments cover two public benchmarks and, if the numbers hold, the +3.22/+1.49 HOTA gains over prior SOTA are meaningful in this niche.\n\nThe soft spots are mostly about verification. The threshold sweeps in Tables 4 and 5 do not reproduce the final numbers in Table 1. The claimed optimal configuration (βref=0.3, βobj=0.7) yields HOTA 37.22, DetA 23.38, AssA 59.25 in the sweep, but Table 1 reports 37.67, 24.09, 58.92 for the same dataset. The paper never says the sweeps were on a validation split and Table 1 on a test split. Given HOTA ranges from 29.99 to 37.22 across βref on this dataset, threshold overfitting could account for a meaningful chunk of the claimed margin. This has to be fixed with a stated protocol before the headline claim is checkable.\n\nThere are also smaller issues: no error bars or multi-seed runs; the abstract says +3.42% while the body says +3.22%; the RIQA attribution is ambiguous because Table 2's RIQA row doesn't match either variant in Table 3; and CQM essentially rediscovered the one-to-many matching idea from H-DETR without citing it. None of these is fatal by itself, but they add up.\n\nThis paper is for people working on referring MOT and end-to-end trackers. It deserves a serious referee: the core idea is reasonable, the benchmarks are public, and the evaluation can be made rigorous with a clear split and error bars. The technical novelty is modest but the engineering is honest. I'd want the threshold issue resolved before trusting the SOTA claim.\n\nRecommendation: send to peer review, but require the authors to document the validation/test split used for threshold selection, reconcile Tables 1/4/5, and report variance.","headline":"Solid RMOT engineering integration with SOTA numbers, but the threshold sweeps don't match the final reported numbers, so the margin is not yet verifiable.","tokens_in":16503,"tokens_out":2744,"would_cite":true,"duration_ms":22069,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TellTrack claims that rebalancing detection-query training and injecting language directly into decoder queries raises referring multi-object tracking accuracy, reporting 55.63 HOTA on Refer-KITTI and 37.67 on Refer-KITTI-V2.","keywords":["referring multi-object tracking","language-guided tracking","transformer decoder","query matching","cross-modal fusion","autonomous driving","HOTA","end-to-end tracking"],"falsifier":"Run TellTrack and the strongest baseline on Refer-KITTI-V2 with all thresholds fixed by validation-set selection and report test HOTA; if the gap falls below the claimed 1.49 points, the headline result depends on threshold choice. Separately, replace CQM by simply training detection queries for twice as many gradient updates; if HOTA matches, the matching rule itself is not the active ingredient.","tokens_in":15271,"feed_emoji":"🎯","tokens_out":5716,"duration_ms":47079,"temperature":0.7,"pith_summary":"Referring multi-object tracking (RMOT) asks a tracker to follow only the objects described by a sentence such as \"a person riding a bike\" across a video. The paper argues that transformer-based RMOT models have two fixable weaknesses: detection queries for newborn objects are trained far less often than track queries for already-tracked objects, and language guidance is fused only into image features, never reaching the decoder queries directly. TellTrack responds with three moves: a collaborative query matching strategy that lets detection queries also match existing targets in intermediate decoder layers, a referring-infused query adaptation that adds the sentence embedding to query content before and inside the decoder, and a cross-modal encoder that runs deformable multi-scale self-attention before textual cross-attention. The method reports HOTA of 55.63% on Refer-KITTI and 37.67% on Refer-KITTI-V2, surpassing the previous best by 3.22 and 1.49 percentage points, with inference speed essentially unchanged. The intended contribution is a reusable recipe for stronger end-to-end language-guided tracking.","feed_headline":"TellTrack lifts referring-tracking HOTA by 3.22 points","feed_subtitle":"A new query-matching scheme and direct language infusion into the decoder outrun prior RMOT models on both KITTI benchmarks.","key_machinery":"The load-bearing objects are the two query types in a DETR-style decoder: detection queries, which are supposed to fire when a target first appears, and track queries, which carry identities across frames. The paper's central identity is that a target activates its detection query exactly once but its track query many times, so detection queries are undertrained; CQM rebalances this by extending the auxiliary bipartite matching in intermediate decoder layers to include existing targets, while keeping strict one-to-one matching in the final layer. The other machinery is the sentence embedding added element-wise to the content part of queries (pre-decoder adaptation) or prepended as an extra query that participates in self-attention only (in-decoder adaptation), plus a cross-modal encoder that reorders the deformable-attention and text-cross-attention steps. Together these carry the argument that explicit linguistic guidance has to reach the query level, not just the feature map, and that training signal has to reach detection queries more often.","core_discovery":"The paper's central claim is that the imbalanced activation of detection versus track queries is a measurable bottleneck in referring multi-object tracking, and that this bottleneck can be relieved during training without changing inference. The mechanism is collaborative query matching (CQM): in auxiliary decoder layers, detection queries are matched not only with newborn targets but also with existing targets, so they receive gradients in more frames and learn to localize objects the track queries already know. Around this, the paper adds a referring-infused query adaptation that injects the sentence embedding into the content part of queries before and inside the decoder, and a cross-modal encoder that runs deformable multi-scale self-attention before textual cross-attention so that text interacts with structured rather than raw visual features. The reported result is a HOTA of 55.63% on Refer-KITTI and 37.67% on Refer-KITTI-V2, with each ablated component contributing positively and inference speed essentially unchanged.","pith_inferences":["One extension the paper leaves implicit is that sentence-level embedding may be too coarse for expressions that require distinguishing two objects of the same category by attributes or spatial relations; injecting word-level or phrase-level embeddings into queries would be a direct test of whether finer linguistic control helps.","Because threshold sweeps are reported only for the full model, the components' individual contributions at other operating points are unknown; evaluating each component across the full beta_ref--beta_obj grid would clarify whether CQM improves detection or merely shifts the precision-recall tradeoff.","CQM's logic generalizes beyond referring tasks to any set-prediction tracker where newborn targets are rare; a cheap test is to apply the same auxiliary matching rule to MOTR-style trackers on a standard MOT benchmark and check whether detection recall rises.","The paper compares against published numbers; an apple-to-apple re-run of the strongest baselines with the same backbone, initialization, and training schedule would separate architectural gains from training-length and hyperparameter effects."],"forward_implications":["If CQM's activation-rebalancing explanation is right, any end-to-end MOT model with one-to-one bipartite matching could improve newborn detection by allowing intermediate-layer matches to existing targets, independent of language.","Putting sentence embeddings directly in decoder queries means RMOT no longer depends solely on early feature fusion; the decoder can be treated as a language-conditioned set predictor.","The reported gains come with negligible inference cost (15.08 versus 15.13 FPS), so the improvements are compatible with real-time autonomous-driving settings if thresholds are calibrated.","Because CQM only changes auxiliary training losses, the same pretrained backbone and initialization can be reused, making the recipe easy to graft onto other transformer trackers."],"supporting_citations":[{"why":"Supplies the baseline TransRMOT architecture and the Refer-KITTI dataset that TellTrack builds on and compares against.","marker":"[37]"},{"why":"Supplies the TempRMOT baseline and the Refer-KITTI-V2 dataset, the second benchmark and strongest prior model to beat.","marker":"[48]"},{"why":"Supplies deformable attention, the query design, and COCO-initialized pretrained weights used by the encoder, decoder, and temporal reasoning.","marker":"[50]"},{"why":"Supplies the MOTR-style track-query and temporal reasoning design that TellTrack extends for referring multi-object tracking.","marker":"[44]"},{"why":"Supplies the decoder-prompt-concatenation paradigm that in-decoder RIQA adapts from segmentation to RMOT.","marker":"[19]"},{"why":"Supplies the HOTA metric, the primary evaluation measure for all quantitative comparisons in the paper.","marker":"[26]"},{"why":"Supplies the idea of concatenating a prompt token to queries, which the in-decoder adaptation uses to fuse the sentence embedding.","marker":"[17]"},{"why":"Supplies the DETR set-prediction formulation and auxiliary decoding loss that CQM modifies.","marker":"[9]"},{"why":"Supplies a two-stage referring tracker baseline with oracle boxes that the paper includes for comparison.","marker":"[15]"}],"fun_headline_variants":["Balanced query matching lifts referring multi-object tracking HOTA","Referring tracking gets a boost from collaborative query matching","New query matching reduces imbalance in referring tracking","Language-guided query matching improves referring tracking HOTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported margin assumes the inference thresholds $\\beta_{\\rm ref}=0.3$ and $\\beta_{\\rm obj}=0.7$ were chosen on a validation split rather than by tuning on the test set, because the paper's threshold sweeps do not state which split they used.","fun_headline_variants_meta":{"raw":{"variants":["Balanced query matching lifts referring multi-object tracking HOTA","Referring tracking gets a boost from collaborative query matching","New query matching reduces imbalance in referring tracking","Language-guided query matching improves referring tracking HOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000506,"raw_usage":{"total_tokens":2461,"prompt_tokens":931,"completion_tokens":1530,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1470}},"tokens_in":547,"tokens_out":1530,"duration_ms":11782,"temperature":1.0,"reasoning_tokens":1470,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:57:05.256641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TellTrack and the strongest baseline on Refer-KITTI-V2 with all thresholds fixed by validation-set selection and report test HOTA; if the gap falls below the claimed 1.49 points, the headline result depends on threshold choice. Separately, replace CQM by simply training detection queries for twice as many gradient updates; if HOTA matches, the matching rule itself is not the active ingredient.","supporting_citations":[{"cited_title":"Referring multi- object tracking","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline TransRMOT architecture and the Refer-KITTI dataset that TellTrack builds on and compares against."},{"cited_title":"Motr: End-to-end multiple- object tracking with transformer","cited_arxiv_id":null,"evidence_quote":"Supplies the MOTR-style track-query and temporal reasoning design that TellTrack extends for referring multi-object tracking."},{"cited_title":"Vi- sual prompt tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the idea of concatenating a prompt token to queries, which the in-decoder adaptation uses to fuse the sentence embedding."},{"cited_title":"End-to- end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the DETR set-prediction formulation and auxiliary decoding loss that CQM modifies."},{"cited_title":"ikun: Speak to trackers without retraining","cited_arxiv_id":null,"evidence_quote":"Supplies a two-stage referring tracker baseline with oracle boxes that the paper includes for comparison."}],"review_version":1}