{"id":"94fb15b8-da8c-416a-b24f-e9e04734bfe3","arxiv_id":"2411.12943","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Combining thermal histogram similarity with motion similarity improves ByteTrack and OCSORT MOT scores by about one to two MOTA points on a new RGB-thermal pedestrian dataset.","lead":"This paper adds a thermal-histogram matching step to standard multi-object trackers and reports small gains in tracking accuracy on a new RGB-thermal urban pedestrian dataset. The dataset itself, 30 synchronized RGB-thermal sequences with tracking annotations, may be the more lasting contribution.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may reflect alpha selection on the evaluation set; without a held-out test, the claim that thermal identity improves association is not established.","rationale":"The paper's novelty is a dataset and a simple association-fusion idea. The strongest empirical claim is that the fusion improves two trackers. For that claim to hold, the comparison must attribute the improvement to the method. The protocol does not: alpha is tuned on the same six sequences used for the final numbers, and no confidence intervals or significance tests are provided. This is a textbook selection-on-the-test-set risk and is more directly load-bearing than the thermal-histogram stability assumption: even if thermal signatures are perfectly discriminative, the reported numbers do not demonstrate it; if they are not, the method would fail regardless. The reader's verdict (CONDITIONAL) and rationale already flag this issue, but the reader's designated 'weakest_assumption' focuses on histogram stability. I agree with the overall conditional assessment but identify the evaluation protocol as the single most load-bearing concern; hence 'partial' agreement. A leave-one-sequence-out alpha-selection test would settle whether the improvement persists out-of-sample.","tokens_in":13140,"tokens_out":5731,"duration_ms":54427,"concrete_test":"Perform a leave-one-sequence-out evaluation on the six thermal validation sequences: for each held-out sequence, select alpha by maximizing MOTA on the remaining five sequences, then report MOTA/IDF1 on the held-out sequence. If the average held-out MOTA/IDF1 gain over the standard ByteTrack/OCSORT baseline is not positive (or is within per-sequence variability), the reported improvement in Tables 3–4 is a selection artifact rather than evidence for the thermal association term.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that adding the thermal-histogram similarity term of Eq. (3) to the box-association step improves MOT in thermal video—rests on Tables 3–4, which compare the standard trackers with the proposed method on six sequences. The weighting factor alpha, however, is selected on exactly those six sequences in Section 5.2 by maximizing MOTA/IDF1 (Figure 2). There is no held-out test set, so the reported gains (ByteTrack: MOTA 65.5→66.4, IDF1 62.6→63.8; OCSORT: MOTA 54.4→56.4, IDF1 57.8→58.6) are the result of choosing the parameter that best fits the evaluation data. With only six sequences and no significance testing, per-sequence results are noisy and even decline in some rows (e.g., Table 4: sequence 2 IDF1 drops from 51.7 to 42.4; sequence 54 MOTA improves only 40.1→42.4, near noise). The text says alpha is 'proven' to be 0.3/0.8, but it is only selected by grid search on the same data. Consequently, the observed improvements may be overfitting noise rather than evidence that thermal identity helps association. The limitation section (5.4) does not acknowledge this, listing only equipment and urban-scene constraints.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a box-association method for two-stage multi-object trackers operating on thermal video. For each track-detection pair, the method computes a normalized histogram of the thermal ROI for each box, measures their similarity with the Bhattacharyya coefficient, and combines this with the standard motion similarity as S_comp = alpha * S_motion + (1 - alpha) * S_thermal (Eq. 3). The alpha weighting is selected per tracker on six validation sequences. The paper also introduces an RGB-Thermal MOT dataset of 30 synchronized RGB/thermal sequences collected at urban intersections, with a 24/6 train/test split, and benchmarks ByteTrack and OC-SORT with and without the proposed association term. The reported results show small aggregate gains on the six validation sequences: ByteTrack MOTA 65.5->66.4 and IDF1 62.6->63.8; OC-SORT MOTA 54.4->56.4 and IDF1 57.8->58.6.","tokens_in":13409,"tokens_out":3256,"duration_ms":39998,"significance":"The dataset release and the idea of exploiting thermal identity in the association step are potentially useful contributions to thermal MOT, which has few public benchmarks. The authors also provide source code and a benchmark of two standard trackers on RGB and thermal data, which is a service to the community. However, the central empirical claim that adding the thermal-histogram term improves tracking accuracy is not established by the current evaluation: alpha is tuned on the same six sequences on which the final metrics are reported, the gains are small and inconsistent per sequence, and there is no significance testing or external benchmark. If the method holds up on a properly held-out test set, the contribution would be modest but sound; with the current protocol it reads as a tuned outcome rather than a validated prediction.","major_comments":[{"comment":"The weighting factor alpha is selected by grid search on the six validation sequences (Section 5.2, Figure 2), and the final MOTA/IDF1 values are then reported on exactly those same six sequences (Tables 3 and 4). This makes the reported improvements a tuned result rather than an unbiased estimate. The text in Section 5.2 says alpha is 'proven' to be 0.3/0.8, but the figures only show that these values maximize metrics on the evaluation set. The central claim requires a held-out test set or nested cross-validation; otherwise the comparison is circular.","section":"Sections 5.2 and 5.3"},{"comment":"The method assumes that a pedestrian's thermal ROI histogram is stable across frames and discriminative between different people, so that the Bhattacharyya coefficient is a reliable identity cue. This assumption is load-bearing but is not tested. The paper provides no measurement of intra-identity histogram stability, inter-identity separability, or sensitivity to pose/distance/background radiation. Without such an analysis or an ablation that isolates the thermal term's contribution, it is unclear whether the small aggregate gains come from genuine identity information or from fitting alpha to noise.","section":"Section 4.2, Eq. (3)"},{"comment":"The per-sequence results are inconsistent with the aggregate claim. For example, in Table 4 sequence 2, IDF1 drops from 51.7% to 42.4% with the proposed method, and in sequence 54 the MOTA gain is only 40.1% to 42.4%. These swings are comparable in magnitude to the reported aggregate improvements and no significance testing is provided. The claim of 'significant improvements' in the abstract and Section 5.3 is therefore not supported. Paired per-sequence statistics and ideally evaluation on an external thermal MOT benchmark are needed.","section":"Tables 3 and 4"},{"comment":"The limitations section lists only equipment and urban-scene constraints and does not acknowledge that alpha selection and the final evaluation share the same six sequences. Given that this is the main threat to the paper's empirical claim, the limitation should be stated explicitly, along with the absence of a held-out test set.","section":"Section 5.4"}],"minor_comments":[{"comment":"The notation is confusing: Eq. (3) uses S_comp and calls it a similarity matrix, but later text refers to a 'comprehensive distance matrix.' Since IoU and the Bhattacharyya coefficient are both similarity measures, the paper should clarify whether the combination is maximized or converted to a cost, and how the sign/direction is handled in the Hungarian matching.","section":"Section 4.2"},{"comment":"The histogram bin size and range used to compute H_ti and H_dj are not specified, and the normalization procedure is described only vaguely. These are free parameters that affect the result and should be reported for reproducibility.","section":"Section 4.2"},{"comment":"There are formatting and naming inconsistencies: 'V al' and 'T able' appear in table headers, 'Bytetrack' and 'ByteTrack' are used interchangeably, and 'OCSort' appears alongside 'OCSORT'. These should be unified.","section":"Tables 1-4"},{"comment":"The axes and legend of Figure 2 are not described in the caption, so the reader cannot verify which curve corresponds to MOTA versus IDF1 or how alpha values were sampled. Please label the axes and include the alpha values on the x-axis.","section":"Figure 2"},{"comment":"The claim that the dataset is 'comparable in size' to MOT17 is not substantiated; the paper should state MOT17's frame/annotation counts alongside the numbers in Table 1 for a direct comparison.","section":"Section 1.2"},{"comment":"The statement that this is the 'world's first large-scale dataset' of RGB and thermal images annotated for MOT should be qualified with respect to the authors' earlier City-Scene dataset (reference [14]) and any other recent thermal MOT datasets, explaining the concrete novelty of the new collection.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The dataset appears to be a continuation of the authors' earlier City-Scene dataset (reference [14]); the relationship and incremental contribution should be clarified during revision. The central issue is the evaluation protocol: without a held-out test set, the reported gains are not convincing. The paper would be much stronger if the authors added an external thermal MOT evaluation or a cross-validation scheme that keeps alpha selection separated from the reported metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know about this paper is that the dataset is the real contribution. The RGB-Thermal MOT dataset—30 sequences, 9000 frames per modality, paired RGB and thermal, with MOT annotations—fills a genuine gap. That part is solid and will be useful to anyone working in thermal tracking. The method, by contrast, is a straightforward extension: fuse the standard motion/IoU cost with a Bhattacharyya coefficient on thermal histograms (Eq. 3) and drop it into ByteTrack and OCSORT. That is textbook appearance-motion fusion, and DeepSORT already does something similar. Applying it to thermal data is legitimate, but the novelty is thin and the writing oversells it.\n\nThe empirical claim is the soft spot. Alpha is chosen on the validation sequences (Section 5.2, Figure 2), and the final numbers in Tables 3–4 come from the same six sequences. That is tuning and reporting on the same data. The gains are small (MOTA +0.9 for ByteTrack, +2.0 for OCSORT), per-sequence results are noisy and sometimes negative—OCSORT sequence 2 IDF1 drops from 51.7 to 42.4, sequence 54 MOTA only goes 40.1 to 42.4—and there is no significance testing or external benchmark. The limitation section (5.4) lists only equipment and urban-scene constraints; it never acknowledges the evaluation overlap. Also, calling alpha \"proven\" by grid search on the validation set is the wrong word, and it exposes the circularity.\n\nThe direction of the result is plausible, and the method may well help in some thermal scenarios, but the evidence here does not support the \"significant improvements\" language. That said, the dataset is a real resource and the authors promise code and data. If they restructure the evaluation—hold out a test split, fix alpha on a separate validation split, or at least report per-sequence variability with some error bars—this could become a useful benchmark paper.\n\nWho is this for? Thermal MOT researchers, and people who want a paired RGB-thermal MOT dataset. The method section will not teach them much. I would send it to peer review because the dataset deserves referee time and the evaluation flaw is fixable.\n\nRecommendation: accept for review with the clear expectation that the authors must address the train/validation/evaluation overlap before acceptance.","headline":"Worth knowing for the new RGB-thermal MOT dataset; the tracking improvement claim is not established because alpha is tuned on the same six sequences used for evaluation.","tokens_in":13915,"tokens_out":1504,"would_cite":true,"duration_ms":18464,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a thermal-histogram similarity term to the box-association step of two-stage trackers improves MOT accuracy on thermal video, with ByteTrack's MOTA rising from 65.5 to 66.4 and OCSORT's from 54.4 to 56.4 on the authors' validation…","keywords":["multi-object tracking","thermal imaging","box association","Bhattacharyya coefficient","thermal identity","RGB-thermal dataset","pedestrian tracking","two-stage tracker"],"falsifier":"Measure the same person's thermal histogram across consecutive frames and at different distances in the new dataset: if the within-person variation over the tracking interval approaches the between-person variation, then the Bhattacharyya term cannot be separating identities, and any MOTA/IDF1 gain would have to come from a different mechanism, such as a smoothing of the association cost.","tokens_in":12909,"feed_emoji":"🔥","tokens_out":6405,"duration_ms":57503,"temperature":0.7,"pith_summary":"This paper claims that the box-association step of two-stage multi-object trackers can be improved in thermal video by adding a thermal-identity term: the pixel-intensity histogram of each detected box, compared between tracklets and detections via the Bhattacharyya coefficient. The authors fuse this thermal similarity with the usual motion-based similarity as $S_{\\mathrm{comp}} = \\alpha S_{\\mathrm{motion}} + (1-\\alpha) S_{\\mathrm{thermal}}$, and show on their six validation sequences that ByteTrack's MOTA rises from 65.5 to 66.4 with IDF1 from 62.6 to 63.8, while OCSORT's MOTA rises from 54.4 to 56.4 with IDF1 from 57.8 to 58.6. The method is plug-in in spirit: it changes only the association cost, not the detector or the tracker's internal motion model. The paper also contributes a new paired RGB-thermal pedestrian MOT dataset of 30 sequences, which it uses to fine-tune detectors and benchmark standard trackers with and without the proposed association.","feed_headline":"Thermal identity cue improves multi-object tracking in infrared video","feed_subtitle":"Adding a histogram-overlap term to box matching raises MOTA and IDF1 for ByteTrack and OCSORT on a new RGB-thermal dataset.","key_machinery":"The load-bearing object is the thermal similarity matrix $S_{\\mathrm{thermal}}$, where each entry is the Bhattacharyya coefficient between the normalized intensity histograms of the bounding-box regions of a tracked box and a candidate detection in the current thermal image. The Bhattacharyya coefficient is a standard histogram-overlap measure ranging from 0 (no overlap) to 1 (identical); here it serves as a 'thermal identity' score. The mechanism is completed by the convex combination $S_{\\mathrm{comp}} = \\alpha S_{\\mathrm{motion}} + (1-\\alpha) S_{\\mathrm{thermal}}$, which is fed into the tracker's association step in place of the pure motion cost. The weight $\\alpha$ is tuned on validation sequences and differs by tracker (0.3 for ByteTrack, 0.8 for OCSORT), reflecting how much each tracker's motion model is trusted. This combination is what carries the reported improvement.","core_discovery":"The central claim is that thermal images carry a sparse but stable per-object identity signal — the distribution of pixel intensities within a detection window — that existing two-stage trackers discard at the association step. The paper formalizes this as a thermal similarity matrix $S_{\\mathrm{thermal}}$ whose entry $(i,j)$ is the Bhattacharyya coefficient between the normalized histograms of tracklet $i$ and detection $j$ extracted from the thermal frame. Combining this with the tracker's motion similarity matrix as $S_{\\mathrm{comp}} = \\alpha S_{\\mathrm{motion}} + (1-\\alpha) S_{\\mathrm{thermal}}$ improves tracking on the validation sequences for both ByteTrack and OCSORT, with optimal $\\alpha = 0.3$ for ByteTrack and $\\alpha = 0.8$ for OCSORT. The paper further observes that a purely thermal association ($\\alpha = 0$) already gives ByteTrack 63.4 MOTA and 55.1 IDF1, which it reads as evidence that thermal identity alone is a substantial association cue in these urban pedestrian scenes.","pith_inferences":["If the thermal histogram is stable over longer intervals, the same identity term could be used for re-identification after occlusion, a case the paper does not test; the reported gains come from near-term association, so extending the term to a tracklet-recovery stage is a natural next experiment.","The fact that a raw intensity histogram helps suggests that learned thermal embeddings, trained on the new dataset, might extract a stronger identity signal; the paper's mechanism provides a baseline that such embeddings would need to beat.","The paired RGB-thermal sequences invite a cross-modal test: whether an RGB-trained re-identification model, or a shared embedding, can improve thermal association when the thermal histogram alone is ambiguous (e.g., similarly heated pedestrians)."],"forward_implications":["Because the change is confined to the association cost, any two-stage tracker in the thermal spectrum could adopt the method without retraining its detector or motion model.","The tracker-specific optimal $\\alpha$ implies that trackers with weaker or noisier motion prediction benefit more from thermal identity; tuning this weight becomes a per-tracker hyperparameter.","The $\\alpha=0$ result for ByteTrack shows that thermal histogram matching alone, without motion, is already competitive, suggesting that a purely appearance-based association is viable in thermal pedestrian scenes.","The new RGB-thermal MOT dataset of 30 paired sequences gives the community a benchmark for developing and comparing thermal and RGB trackers beyond the two models tested here."],"supporting_citations":[{"why":"ByteTrack is the first baseline tracker; its association step is replaced by the proposed thermal-motion cost and its MOTA/IDF1 improvement is the paper's headline result.","marker":"[51]"},{"why":"OCSORT is the second baseline tracker modified and benchmarked, providing the second set of MOTA/IDF1 comparisons.","marker":"[6]"},{"why":"TOOD is the detector fine-tuned on the dataset; it supplies the detection boxes that feed both trackers and the association method.","marker":"[16]"},{"why":"The authors' earlier RGB-thermal dataset and detection/tracking work motivate the new dataset and the thermal-feature approach.","marker":"[1,14]"},{"why":"SORT defines the Kalman-filter motion-association baseline that the proposed method augments with thermal identity.","marker":"[2]"}],"fun_headline_variants":["Thermal histogram cues boost multi-object tracking","Thermal identity + motion similarity sharpen MOT","Thermal box association lifts MOTA and IDF1","Infrared pixel identity improves tracker association","Thermal cues aid tracking in urban scenes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a pedestrian's thermal histogram, extracted from the detected bounding box, is stable enough from frame to frame and discriminative enough between different people that the Bhattacharyya coefficient is a reliable identity cue; if pose, distance, or background radiation makes the histogram unstable, the added term could inject noise rather than signal.","fun_headline_variants_meta":{"raw":{"variants":["Thermal histogram cues boost multi-object tracking","Thermal identity + motion similarity sharpen MOT","Thermal box association lifts MOTA and IDF1","Infrared pixel identity improves tracker association","Thermal cues aid tracking in urban scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1212,"prompt_tokens":930,"completion_tokens":282,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":215}},"tokens_in":546,"tokens_out":282,"duration_ms":3849,"temperature":1.0,"reasoning_tokens":215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:01:17.330912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the same person's thermal histogram across consecutive frames and at different distances in the new dataset: if the within-person variation over the tracking interval approaches the between-person variation, then the Bhattacharyya term cannot be separating identities, and any MOTA/IDF1 gain would have to come from a different mechanism, such as a smoothing of the association cost.","supporting_citations":[{"cited_title":"In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII","cited_arxiv_id":null,"evidence_quote":"ByteTrack is the first baseline tracker; its association step is replaced by the proposed thermal-motion cost and its MOTA/IDF1 improvement is the paper's headline result."},{"cited_title":"In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)","cited_arxiv_id":null,"evidence_quote":"TOOD is the detector fine-tuned on the dataset; it supplies the detection boxes that feed both trackers and the association method."}],"review_version":1}