{"id":"a370e0c3-57bb-454d-b125-640146eb9663","arxiv_id":"2412.10028","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Multi-route training with instructive self-attention tokens and a route-aware mixture-of-experts raises detection mAP by 2 to 4 points across several DETR baselines at no inference cost.","lead":"This paper introduces a training scheme for detection transformers that runs a primary one-to-one matching route alongside auxiliary one-to-many routes, then discards the auxiliaries at inference. Across COCO, Objects365, and NuImages, the approach adds 2 to 4 points of mAP over strong baselines without changing the deployed model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies how the three route losses are combined; since the method is a multi-task training scheme, the missing total-loss/weighting equation leaves the central empirical claims under-specified and unreproducible.","rationale":"The reader's conditional verdict is justified. I initially suspected a numerical inconsistency between Table II and Table VIII, but re-reading shows Table VIII uses the 900-query Deformable-DETR++ (baseline 47.6, Mr DETR 50.7, Mr DETR++ 51.8), matching Table II's 900-query rows; that is not a discrepancy. Other issues, such as the Fig. 6 caption swapping G and G' and Eq. (13) reducing to the IoU score alone at the default phi=0, are real but secondary. The main risk to the central claim is the unstated multi-task loss weighting: it affects every reported experiment and cannot be checked without code. I therefore agree with the reader's 'related implementation gap' and regard it as more load-bearing than the COCO-val hyperparameter transfer question, hence partial rather than full agreement.","tokens_in":28349,"tokens_out":24894,"duration_ms":251988,"concrete_test":"Request or release the official code and the exact total loss/weights used for Tables II and VIII. Then retrain the headline configuration (Deformable-DETR++-300q, 12 epochs, COCO val) with the written components and with the total loss as an equal-weight sum of the three route losses, comparing to the reported 49.5 AP (Mr DETR) and 51.0 AP (Mr DETR++). If either number moves by more than about 0.5 AP, the missing loss weighting is a substantive unconstrained degree of freedom; if both reproduce within 0.1 AP, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Three route losses (one-to-one for Route-2, one-to-many for Route-1 and Route-3) are each described, but the paper never writes the total loss. Eq. (1) gives the one-to-one loss; the one-to-many loss is only described as 'calculated as in Eqn. (1)' after Eq. (2). No equation combines L_o2o and the two L_o2m terms, and no weights are reported. This is load-bearing because the method is precisely a multi-task training scheme: with K=6 and alpha=0.3, the one-to-many assignment yields far more positives than the bipartite match, so the auxiliary gradients can dominate or vanish depending on the (unspecified) weights. Table I's 'Share All' collapse (47.6 to 41.6) already shows that the o2o/o2m balance is a sensitive degree of freedom. Without the total-loss/weight equation, the central empirical claims in Tables II-VIII cannot be independently reproduced or attributed to the proposed multi-route mechanism rather than to a particular loss-balancing choice. The lack of released code leaves this gap unresolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-only enhancement for DETR-family detectors. It first treats an auxiliary one-to-many supervision branch as a second task and, through the ablations in Table I, concludes that sharing all decoder components between one-to-one and one-to-many targets degrades the primary route while making any single component independent helps. On this basis it introduces three training routes: Route-2, the primary one-to-one route; Route-1, an auxiliary one-to-many route with an independent FFN; and Route-3, an auxiliary one-to-many route with a novel instructive self-attention that concatenates learnable instruction tokens to the object queries. Mr. DETR++ extends this with a route-aware MoE in the decoder, a scale-aware encoder MoE, and a localization-aware score calibration. The auxiliary routes are discarded at inference, so the architecture and inference cost match the baseline. Experiments on COCO 2017, Objects365, and NuImages report consistent mAP improvements over Deformable-DETR++, DINO, and Align-DETR, with further results on instance and panoptic segmentation.","tokens_in":28638,"tokens_out":8753,"duration_ms":95353,"significance":"The empirical pattern in Table I is a useful and nontrivial contribution: it maps which decoder components can be shared between the two assignment targets, and it directly motivates the multi-route design. The proposed instruction tokens are an inexpensive way to specialize shared self-attention, and the route-aware MoE with separate gating for the primary route is a reasonable way to share experts while limiting route conflict. The evaluation is broad: three datasets, three baselines, two additional tasks, training-cost measurements, and a probing analysis of the mechanism. The paper also states the relation to the preliminary CVPR version [21] explicitly and credits prior work. If the missing training-objective details and single-run results are addressed, the work would be a solid empirical contribution to the DETR training literature. In its current form, the missing total-loss specification prevents the central claimed gains from being independently reproduced.","major_comments":[{"comment":"The total training objective is never written down. Eq. (1) defines the one-to-one loss for the bipartite match, and the text says the one-to-many losses are 'calculated as in Eqn. (1)' after Eq. (2), but no equation or sentence specifies how the Route-2 one-to-one loss and the Route-1/Route-3 one-to-many losses are combined, and no weighting coefficients are reported. This is load-bearing for a multi-task training method: with K=6 and alpha=0.3 the one-to-many assignment yields many more positives than the one-to-one match, so the relative loss weights determine whether the auxiliary gradients dominate or vanish. Table I already shows that the o2o/o2m balance is sensitive, since the 'Share All' configuration collapses the primary route from 47.6 to 41.6. Please add the explicit total-loss equation, report the route weights, and include an ablation over those weights.","section":"III-B, Eq. (1)-(2), Table I"},{"comment":"All free hyperparameters are selected on the COCO 2017 validation set and then applied without re-tuning to Objects365, NuImages, and the segmentation tasks: K=6, alpha=0.3, tau=0.4 (App. D), 10 instruction tokens (App. B), 4 experts with top-2 (App. C), and phi=0 (App. E). Because all reported numbers are single runs without error bars, the claim of 'consistent improvements' across datasets and tasks is hard to evaluate: the cross-domain gains could be partly due to settings that are accidental to COCO-val. Please provide at least three seeds (or equivalent variance estimates) for the main comparisons and a sensitivity analysis of K, alpha, tau on at least one non-COCO benchmark.","section":"IV-A, App. B-E, Tables II-VIII"},{"comment":"The manuscript does not state that code will be released, and the project page link does not provide a repository. For a method whose only evidence is empirical, the missing total-loss equation (Major 1) combined with absent code and single-run results is a substantial reproducibility gap. Please either release the code or provide a complete implementation specification, including the loss combination, route-wise loss masking, MoE auxiliary losses or their absence, and the exact training schedule for each dataset.","section":"IV, Tables II-XI"}],"minor_comments":[{"comment":"The definition after the equation uses 'spred' but the formula uses s_iou; align the notation.","section":"III-D, Eq. (13)"},{"comment":"The typo 'InstrcutSA' should be 'InstructSA'.","section":"III-B, Eq. (6), Fig. 4 caption"},{"comment":"The text says Objects365 models are trained for 4 epochs, while the caption says 'All models are trained for 12 epochs'; correct the inconsistency and state the learning-rate schedule precisely.","section":"IV-B.2 and Table IV caption"},{"comment":"The phrase 'parallel axillary queries' should read 'parallel auxiliary queries'.","section":"II"},{"comment":"The text says 'performance improves with an increased number of instruction tokens,' but the table shows 50.2, 50.1, 50.7, 50.4, and 50.5 for 1, 5, 10, 50, and 100 tokens, which is not monotonic; rephrase the conclusion.","section":"App. B, Table XIIIa"},{"comment":"Specify whether the gating scores c_i are normalized over the selected top-k experts and whether the MoE block includes a residual connection; also state whether any load-balancing loss is used for the route-aware MoE.","section":"III-D, Eq. (11)"},{"comment":"The table and text compare 'instruction by concatenation (ours)' against variants, but the baseline configuration (which routes are active) should be stated explicitly for each row so the reader can connect Table IX to Table VIII.","section":"IV-F, Table IX"},{"comment":"Please add a reproducibility statement covering code availability, random seeds, and hardware details for each table.","section":"IV-A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical paper whose main claim is plausible, but the missing total-loss equation is a genuine gate for a method whose entire contribution is a training scheme. I would like to see the authors provide the total-loss specification and route-weight ablation, plus seed variance for the headline tables, before acceptance. I also note that the incremental contribution over the authors' CVPR 2025 paper [21] is clearly concentrated in the MoE and calibration components; the editor may want to check that the journal version is sufficiently distinct in scope from [21]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a real multi-route training recipe for DETR-family detectors, with consistent gains across COCO, Objects365, NuImages, instance segmentation, and panoptic segmentation, at zero inference overhead. The new mechanisms are instructive self-attention (concatenated learnable instruction tokens that steer shared object queries toward one-to-many targets) and a route-aware MoE with separate gating for the one-to-many FFN route. The component-role study in Table I is genuinely useful: it shows that a single independent component suffices to avoid the one-to-one/one-to-many conflict, and that the \"share all\" baseline collapses by 6 mAP. That empirical mapping is the paper's real contribution.\n\nThe soft spots are real but not fatal. The stress-test criticism is correct: the paper never writes the total loss. It gives Eq. (1) for the one-to-one route, says the one-to-many losses are \"calculated as in Eqn. (1)\", and then stops. No weights, no summation. For a method whose entire point is balancing two auxiliary one-to-many routes against a primary one-to-one route, that is a load-bearing omission. With K=6 and α=0.3, the one-to-many assignments produce far more positives than the bipartite match; the auxiliary gradients can dominate or vanish depending on the unspecified weighting. Table I's 'Share All' row shows the balance is sensitive. This is not a cosmetic gap; it blocks independent reproduction and makes it hard to attribute the gains to the mechanism rather than to a specific loss-balancing choice.\n\nOther concerns are minor-to-moderate. Results are single-run, no error bars; K, α, τ are tuned on COCO validation and assumed to transfer. The cross-dataset consistency mitigates this, but code would settle it. No code is released, which is disappointing given the CVPR predecessor exists and the overhead numbers in Table XI are training-only.\n\nThe paper is coherent, the citations to prior one-to-many training work are appropriate, and the explicit self-citation to Mr. DETR is disclosed and correct. I'd send it to review. Ask for the total-loss equation, the loss weights, and code. If those come back, this is a citable training recipe.","headline":"A genuinely useful multi-route training recipe with consistent gains, but the missing total-loss weighting equation and no code make it currently unreproducible; referee it and demand the fix.","tokens_in":29161,"tokens_out":2524,"would_cite":true,"duration_ms":26600,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training detection transformers with two auxiliary one-to-many routes, discarded at inference, improves mAP by up to 4.0 points on COCO while leaving the deployed model unchanged.","keywords":["object detection","detection transformers","DETR","one-to-many assignment","multi-route training","instructive self-attention","mixture-of-experts","panoptic segmentation"],"falsifier":"Run the 12-epoch Deformable-DETR++ (300-query) setup with only the instructive self-attention route added: the paper's recipe predicts primary-route AP of 50.4, up from 47.6. A reader can check that single number; if it does not reproduce, the instructive self-attention is not carrying the reported gain.","tokens_in":28186,"feed_emoji":"🎯","tokens_out":11042,"duration_ms":106105,"temperature":0.7,"pith_summary":"The paper asks what must be separated, inside a DETR-style transformer decoder, for one-to-one matching and one-to-many auxiliary supervision to coexist during training. Its empirical answer is that any single independent component — self-attention, cross-attention, or feed-forward network — suffices, even with all other components shared; fully sharing every component degrades the primary one-to-one prediction. From this it builds a three-route training scheme with a primary one-to-one route and two auxiliary one-to-many routes, later upgraded to a route-aware mixture-of-experts and an encoder MoE. The auxiliary routes are discarded at inference, so the deployed model keeps the baseline architecture and cost. Reported gains on COCO validation reach 4.0 mAP over Deformable-DETR++, 3.2 over DINO, and 2.0 over Align-DETR in 12-epoch schedules, with further gains on Objects365, NuImages, instance segmentation, and panoptic segmentation.","feed_headline":"Training-only routes lift detection transformers by up to 4 mAP","feed_subtitle":"Two auxiliary routes train a one-to-many matching skill, then vanish at inference, so accuracy rises while runtime does not.","key_machinery":"The load-bearing machinery is the multi-route decoder. Route-2 is the primary one-to-one route and is identical to the baseline; Route-1 is an auxiliary one-to-many route with an independent feed-forward network; Route-3 is an auxiliary one-to-many route whose instructive self-attention concatenates $m$ learnable instruction tokens to the object queries, runs self-attention over the combined sequence, and then discards the token outputs, so the tokens condition the shared queries without surviving to inference. The one-to-many supervision uses the matching score $M_{ij} = \\alpha s_i + (1-\\alpha)\\,\\mathrm{IoU}(b_i,\\bar{b}_j)$, keeping at most $K$ positive candidates above an IoU threshold $\\tau$. In Mr. DETR++ the two FFN roles are replaced by a route-aware MoE: a pool of experts shared by all routes, with Route-1 gated independently and Routes 2 and 3 sharing a gate, using top-$k$ sparse activation. A scale-aware MoE applies gated experts only to low-scale encoder features while all tokens keep a shared FFN, and a localization-aware calibration $s_{\\mathrm{calib}} = s_{\\mathrm{cls}}^{\\phi} s_{\\mathrm{iou}}^{1-\\phi}$ adjusts classification during inference.","core_discovery":"The central claim is that one-to-one and one-to-many objectives are not inherently in conflict inside a detection transformer; they become separable as soon as the decoder gives each objective its own copy of some minimal component. The paper demonstrates this with a systematic variant study: sharing everything drops the one-to-one route by 6.0 AP, whereas making only self-attention, only cross-attention, or only the FFN independent improves it by 2.1, 1.6, and 2.0 AP respectively. The resulting recipe, Mr. DETR++, runs three parallel routes on shared object queries and heads: a primary one-to-one route, an auxiliary route with an independent FFN, and an auxiliary route whose instructive self-attention attaches learnable instruction tokens to guide queries toward one-to-many targets. The two auxiliary routes are supervised by a one-to-many assignment and thrown away at deployment. An added route-aware MoE lets the two FFN variants share experts through separate gates, a scale-aware MoE strengthens low-resolution encoder features, and a localization-aware score calibrates classification at inference; the paper reports consistent mAP gains across baselines and tasks.","pith_inferences":["Editorial inference: if the 'any one independent component suffices' result is general, it suggests a cheap diagnostic for other multi-task transformers: duplicate only the component most coupled to each objective and share the rest, rather than building separate branches.","Editorial inference: instructive self-attention is essentially a soft-prompt mechanism on object queries; the same token-conditioning idea could steer one decoder toward different label assignments, segmentation heads, or multi-dataset objectives with no inference-time cost.","Editorial inference: the scale-aware encoder MoE ablation defines an accuracy-cost frontier (adding x32 and x16 scales costs about 16.6 GFLOPs for +0.1 AP), so the marginal value of more MoE scales is small and probably dataset-dependent.","Editorial note: the paper never writes the total loss built from the three routes, so an exact reproduction requires choosing route weights that the text does not specify."],"forward_implications":["Training-only upgrade: any DETR-family detector can take the three-route recipe without changing its inference graph; the paper verifies this on Deformable-DETR++, DINO, and Align-DETR.","The reported COCO gains are largest in short schedules: +4.0 AP at 12 epochs for Deformable-DETR++ (300 queries), +3.2 for DINO, and +2.0 for Align-DETR, with smaller but positive gains at 24 epochs.","Cross-domain gains follow the same recipe without per-dataset retuning: +4.5 AP on Objects365 and +3.8 AP on NuImages with Deformable-DETR++.","The same multi-route training lifts mask AP by 5.3 points on instance segmentation and PQ by 2.2 points on panoptic segmentation over the respective baselines.","The route-aware MoE visualisation shows Route-2 and the auxiliary routes selecting overlapping but distinct experts, which the paper reads as evidence that shared experts can specialise per route."],"supporting_citations":[{"why":"Supplies the Deformable-DETR++ baseline architecture and the primary detection numbers the method improves on.","marker":"[11]"},{"why":"DINO is the second central baseline; the method's 12-epoch gain of 3.2 AP is measured against it.","marker":"[9]"},{"why":"Align-DETR provides the third detection baseline and its IoU-aware BCE loss is used in some experiments.","marker":"[12]"},{"why":"H-DETR defines the hybrid one-to-one/one-to-many matching line that the paper compares with and extends.","marker":"[8]"},{"why":"DAC-DETR is a one-to-many auxiliary training baseline and a source of the assignment rule in Eq. (2).","marker":"[13]"},{"why":"MS-DETR is a mixed-supervision baseline that uses one-to-many assignment for cross-attention, compared throughout the tables.","marker":"[14]"},{"why":"Provides the straightforward one-to-many assignment used to supervise the auxiliary routes.","marker":"[17]"},{"why":"VarifocalNet's VFL loss and IoU-aware scoring inspire the localization-aware score calibration in Mr. DETR++.","marker":"[18]"},{"why":"Mask DINO is the panoptic segmentation baseline that the method is added to and evaluated against.","marker":"[97]"}],"fun_headline_variants":["Multi-route DETR training: extra paths vanish, mAP rises","Route-aware MoE plus instructive attention lifts DETR","Auxiliary routes teach one-to-many, then disappear at inference","DETR++: separate decoder routes for two matching objectives"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains depend on one hand-set matching rule — up to six positive boxes per target, weighted by classification confidence and an IoU floor of 0.4 — that was tuned on COCO validation and is assumed to transfer to other datasets and tasks without re-tuning; if that rule is COCO-specific, the reported cross-domain improvements could shrink.","fun_headline_variants_meta":{"raw":{"variants":["Multi-route DETR training: extra paths vanish, mAP rises","Route-aware MoE plus instructive attention lifts DETR","Auxiliary routes teach one-to-many, then disappear at inference","DETR++: separate decoder routes for two matching objectives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1498,"prompt_tokens":1066,"completion_tokens":432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":357}},"tokens_in":682,"tokens_out":432,"duration_ms":5665,"temperature":1.0,"reasoning_tokens":357,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:25:20.570020+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 12-epoch Deformable-DETR++ (300-query) setup with only the instructive self-attention route added: the paper's recipe predicts primary-route AP of 50.4, up from 47.6. A reader can check that single number; if it does not reproduce, the instructive self-attention is not carrying the reported gain.","supporting_citations":[{"cited_title":"Mask dino: Towards a unified transformer-based framework for object detection and segmentation,","cited_arxiv_id":null,"evidence_quote":"Mask DINO is the panoptic segmentation baseline that the method is added to and evaluated against."}],"review_version":1}