{"id":"d9d3c291-7b52-4f86-b584-b18fed894c06","arxiv_id":"2505.07446","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"TPT-Bench is a long-term, egocentric, multimodal benchmark that forces target person re-identification after frequent disappearances, and current trackers underperform on it.","lead":"This paper presents TPT-Bench, a new dataset of 48 long robot-egocentric recordings in crowded indoor and outdoor spaces, with 571,982 annotated frames of a target person. It benchmarks 18 existing tracking and re-identification methods and finds they perform far worse on this data than on prior benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth validity is the load-bearing weak point: annotations are bootstrapped from the very MOT+ReID methods evaluated and no validation statistics are provided, so benchmark difficulty and method rankings could be annotation artifacts.","rationale":"The paper's value proposition is a new benchmark that exposes unsolved long-term TPT; every headline statistic and Table 2/3 ranking is computed against the GT. The construction in Sec. 3.2 makes that GT partly dependent on the very class of algorithms being benchmarked. Without a quantitative validation study, one cannot distinguish true task difficulty from artifacts of the chosen tracking prior. This is exactly the reader's weakest assumption. I agree, and the conditional verdict is appropriate. The proposed re-annotation check is feasible and would settle the point: it quantifies GT quality and shows whether rankings survive independent labels. The cross-dataset comparison is secondary and potentially confounded by protocol differences, but the annotation issue is more load-bearing because it affects the benchmark itself, not just the comparison.","tokens_in":21913,"tokens_out":3072,"duration_ms":31781,"concrete_test":"Select ~10,000 frames from 10 sequences stratified by disappearance onset/offset, fast motion, and crowd density. Have two annotators re-annotate from scratch without tracker priors, using the paper's LaSOT-style tight-box/absent rule. Compute inter-annotator IoU and absent-label Cohen's kappa; then replace original GT with the reconciled independent GT and re-run RPF-ReID w/ KPR, RPF-ReID+OCL w/ parts-R18, STARK, and Siam-RCNN under the Sec. 4.1 protocol. Also label a 2-minute clip at full 30 Hz from scratch, bypassing interpolation, and compare interpolated vs. manual boxes. If best-method F-score changes by >2 points, absent-label agreement kappa <0.8, or the best method changes, the benchmark ranking and difficulty claims are not stable; otherwise the annotation assumption is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 3.2 describes a hybrid annotation pipeline: tracking outputs from MOT+Target-ReID methods are generated on every sequence, visually inspected, and the 'best' result is selected as a prior; annotators then refine boxes and correct absent/present labels. This prior is not independent of the methods Table 2 ranks, and no inter-annotator agreement, per-frame manual validation count, or error analysis is reported. The absence of such validation is material because the headline claims—65.3 s mean disappearance, 9.6 distractors during absence, and best F-score only 66.06—all depend on GT absent intervals and box quality. If the selected prior systematically over-marks disappearance (e.g., the Target-ReID component loses the target after occlusions), the benchmark's central difficulty claim is inflated; if it biases ID assignment toward the ByteTrack pipeline, the MOT+ReID family is favored. Linear upsampling from 15 Hz to 30 Hz (interpolated flag) can additionally distort fast-moving boxes, affecting IoU thresholds and AMR. The burden is on the benchmark to establish GT reliability; currently only qualitative statements support it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TPT-Bench, a large-scale, long-term, robot-egocentric dataset for target person tracking (TPT), collected by a human pushing a sensor-equipped cart in crowded indoor and outdoor environments. The dataset includes 48 sequences totaling 5.3 hours, with panoramic, RGB-D, LiDAR, IMU, and odometry data, and 571,982 frames of 2D bounding-box annotations at 30 Hz (interpolated from 15 Hz manual annotations). The authors describe the annotation pipeline, which uses MOT+Target-ReID method outputs as priors and human refinement, and then evaluate a wide range of SOT, MOT+Target-ReID, and Detection+ReID baselines using measures such as F-score and average max recall at 100% precision (AMR). The results show that existing methods perform poorly, with the best method (RPF-ReID with KPR features) reaching an F-score of 66.06 and AMR of 16.26, and the paper claims TPT-Bench is more challenging than LaSOT and Chen's dataset for long-term tracking.","tokens_in":22083,"tokens_out":8461,"duration_ms":75091,"significance":"If the ground-truth annotations are reliable and the evaluation is unbiased, TPT-Bench is a valuable contribution to the HRI, robotics, and computer-vision communities. Its strengths are the large multi-modal dataset, the inclusion of long-term challenges such as target disappearance and re-identification among many distractors, the comprehensive evaluation of current state-of-the-art methods, and the public release of data, code, and development tools. The benchmark fills a clear gap in TPT evaluation, which has been limited to controlled or short-term settings.","major_comments":[{"comment":"The ground-truth labels are generated with a prior from the same family of algorithms that are later evaluated: the text states, \"we apply all MOT + Target-ReID methods to a given sequence and visually inspect the resulting tracking videos to select the best tracking result as a prior annotation.\" This introduces a risk of circularity. If the prior method systematically loses the target or mislabels absent frames, the human annotators may unconsciously inherit those errors, and the evaluated MOT+Target-ReID methods (including the authors' own RPF-ReID, Ye et al. 2024) would be favored. The paper provides no quantitative evidence that the final ground truth is independent of the prior, e.g., inter-annotator agreement, comparison between the prior and final labels, or analysis of annotation error rates. This is load-bearing because the central claims that the benchmark is difficult and that MOT+Target-ReID methods outperform SOT methods depend directly on the validity and unbiasedness of the ground truth.","section":"Sec. 3.2, Annotation and Labels"},{"comment":"The paper states that annotations are made at 15 Hz and then \"upsample the annotations from 15 Hz to 30 Hz using linear interpolation.\" No validation of this interpolation is reported. Since the evaluation metrics in Sec. 4.1, especially AMR in Eq. (3), rely on IoU thresholds, any distortion of interpolated boxes during fast motion could significantly change the measured performance, potentially inflating the apparent difficulty of the benchmark. The authors should provide an interpolation error analysis (e.g., comparing interpolated boxes against independently annotated 30 Hz boxes on a random subset) and report sensitivity results by running the evaluation on the 15 Hz hand-annotated frames alone.","section":"Sec. 3.2, annotation upsampling"},{"comment":"The definition of AMR in Eq. (3) sums over a set of IoU thresholds, but the exact values of this set are not specified anywhere in the paper. Similarly, the method for selecting the primary F-score (\"the highest F-score at an optimal threshold\") needs a precise description (grid, step size, per-sequence vs. pooled). Without these details, the reported numbers cannot be independently reproduced, and the cross-dataset comparisons in Table 3 lose their evidential value.","section":"Sec. 4.1, evaluation protocol"}],"minor_comments":[{"comment":"The phrase \"tightest up-right one\" should be \"tightest upright one\"; this is a typographical error.","section":"Sec. 3.2, Annotation and Labels"},{"comment":"The symbols \"%\" and \"!\" in the \"Robot Egocentric\" column are not explained in the caption or the text; the legend should be made explicit.","section":"Table 1, caption"},{"comment":"The set of IoU thresholds used to compute AMR is unspecified; the authors should list the exact thresholds, e.g., {0.5, 0.75, 0.9} or a range of values.","section":"Sec. 4.1, evaluation protocol"},{"comment":"The statement \"using the initial template\" is ambiguous: it should clarify whether the trackers are initialized with only the first-frame bounding box or also with the textual sequence descriptions provided in the dataset.","section":"Sec. 4.1, evaluation protocol"},{"comment":"The comparisons in Table 3 involve datasets with different sequence counts (48 vs. 20 vs. 11); reporting sequence-level standard deviations or confidence intervals would make the claim of higher difficulty more robust.","section":"Sec. 4.5, comparison with other datasets"},{"comment":"The units \"114.3 pixel 2\" and \"167.4 pixel2\" should be rendered as pixels² (or px²) for clarity.","section":"Sec. 3.3, dataset statistics"}],"recommendation":"major_revision","confidential_remarks":"The authors evaluate a method from their own group (RPF-ReID; Ye et al. 2024) on their own benchmark. While self-evaluation is common in the field, the combination of this fact with the annotation pipeline (which uses MOT+Target-ReID outputs as priors) means that reviewers should specifically scrutinize the independence of the ground truth. I would recommend asking the authors for a direct validation of the annotations before considering the manuscript for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: TPT-Bench is a genuine contribution, not a filler benchmark. It provides 48 long robot-egocentric sequences, 571,982 annotated frames, multi-modal streams (panoramic, RGB-D, LiDAR, IMU, odometry), and—unlike earlier TPT data—long disappearances (mean 65.3 s), many distractors during absence, and repeated re-identification demands. The evaluation of 18 methods is systematic, and the result that the best tracker sits around 66 F-score while most SOT trackers collapse is a meaningful signal. The dataset, code, and video are public; that alone justifies serious engagement.\n\nWhat the paper does well: it targets an under-served task—long-term target-person tracking from a moving robot—and the collection platform is sensible for robot-person following. The failure-case analysis in Sec. 5 is concrete and gives grounded directions. The comparison against LaSOT and Chen's dataset is a reasonable sanity check, although protocol differences mean the numbers shouldn't be read too finely. The references cover the relevant MOT, SOT, and ReID literature, and the metric definitions follow standard long-term tracking practice.\n\nThe soft spot is the annotation validation, and the stress-test note is mostly right. Section 3.2 says the prior annotation comes from MOT + Target-ReID outputs, the same family of methods being benchmarked. There is a human review step on every frame, so this is not pure circularity, but the paper reports no quantitative validation: no inter-annotator agreement, no count of corrected boxes, no error analysis on absent/present labels. Those labels drive the ADL and distractor statistics, which drive the headline difficulty claims. The 15 Hz to 30 Hz linear interpolation is also unvalidated; fast motion could distort IoU-based metrics. These are fixable, but the central claim currently rests on trust rather than evidence.\n\nOne more caveat: the best-performing method is the authors' own RPF-ReID w/ KPR. That is common in benchmark papers and not disqualifying, but it raises the bar for independent reproducibility. The cross-dataset comparison is suggestive, not conclusive.\n\nBottom line: this paper deserves a serious referee and probably a major revision. If the authors add annotation validation (even on a random subset), report interpolation errors, and release a small independent evaluation protocol, TPT-Bench could become a standard resource. I'd cite it if I work on robot person following, and I'd bring it to a reading group.","headline":"TPT-Bench fills a real gap as a public robot-egocentric long-term tracking dataset, but the annotation-validation evidence is too thin to fully trust the difficulty claims.","tokens_in":22678,"tokens_out":3168,"would_cite":true,"duration_ms":30213,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TPT-Bench, a 48-sequence robot-egocentric dataset, shows that current target-person trackers reach only 66.06 F-score and 16.26 zero-false-positive recall in crowded environments.","keywords":["target person tracking","robot egocentric vision","long-term tracking benchmark","person re-identification","robot person following","crowded environments","multi-modal sensor dataset","human-robot interaction"],"falsifier":"Re-annotate a random sample of sequences from scratch without showing annotators any tracker output, then compare the new boxes to the released ones; if mean intersection-over-union falls below about 0.9 on visible frames, or if re-running the top methods on the re-annotated subset changes their F-score ranking, then the benchmark's difficulty claims rest on annotation bias rather than on the tracking task itself.","tokens_in":21683,"feed_emoji":"🤖","tokens_out":8934,"duration_ms":76920,"temperature":0.7,"pith_summary":"TPT-Bench is a new benchmark for tracking a single target person from a mobile robot's egocentric viewpoint in crowded, unstructured environments. The paper's central claim is that long-term target person tracking is far harder than existing benchmarks imply: on its 48 sequences with 571,982 annotated frames, the best evaluated method, RPF-ReID with KPR re-identification features, reaches an F-score of 66.06 and an average max recall at 100% precision of 16.26, compared with up to 99.1 F-score on an older person-following dataset and 70.3 on LaSOT. If the benchmark is sound, it shows that re-identifying a lost target among many pedestrians, rather than short-term occlusion handling, is the unsolved core of robot person following. That matters for assistive robots and embodied agents that must stay with the right person over minutes-long episodes.","feed_headline":"Best tracker scores 66 on crowded robot-following test","feed_subtitle":"Best method manages F-score 66 and zero-false-positive recall 16 on 48 long, crowded sequences.","key_machinery":"The load-bearing object is the dataset's annotation layer: 571,982 frames of panoramic RGB images with tight 2D bounding boxes, absent-indicator labels, and per-sequence language descriptions covering appearance, lighting, scenario, and clothing changes. These labels convert long-term tracking into a measurable re-identification problem, since disappearance intervals are long enough and crowded enough to make target search necessary. The supporting evaluation machinery is the long-term tracking metric pair from Lukežič et al. 2020: F-score, which balances precision and recall over a confidence threshold, and AMR, average max recall at 100% precision, which penalizes any false positive that would make a robot follow the wrong person. The annotation pipeline also supplies a prior: MOT+Target-ReID trackers are run first, the best visual result is selected as a seed, and annotators refine boxes and absent labels, with 15 Hz annotations interpolated to 30 Hz.","core_discovery":"The paper introduces TPT-Bench as the first large-scale, multi-modal, robot-egocentric target-person-tracking benchmark, and demonstrates that state-of-the-art trackers fail on it. The dataset consists of 5.3 hours of recordings across 48 sequences in crowded indoor and outdoor settings, captured by a human pushing a sensor-equipped cart; annotations are frame-level 2D bounding boxes with explicit absent labels, totaling 571,982 frames. Target disappearances are frequent, averaging 45.4 per sequence and 65.3 seconds absent per sequence, and while the target is gone the field of view contains an average of 9.6 distractor pedestrians. Under long-term tracking metrics, the strongest method, RPF-ReID with KPR features, reaches only 66.06 F-score and 16.26 AMR, while the same method class scores 70.3 F-score on LaSOT and 99.1 F-score on Chen's dataset. The authors conclude that no current method reliably re-identifies a target person after long disappearance in crowded public spaces.","pith_inferences":["The large jump from ResNet18 to KPR features suggests that part-aware, occlusion-robust re-identification, rather than better short-term tracking, is the highest-leverage direction; a direct test would be training a part-based ReID model on these sequences and watching whether the 66 F-score ceiling moves.","Because online continual learning adds roughly 12 F-score points to the same method, TPT-Bench is well suited as a stress test for memory-consolidation and replay algorithms under appearance drift, an extension the paper motivates but does not run.","The 2.5D ground-plane tool suggests a concrete fusion baseline: using projected LiDAR depth to gate re-identification candidates could reduce the false positives that AMR punishes, since AMR of the best method is only 16.26.","The dataset includes person-search episodes, so it can be repurposed beyond tracking as an active-search benchmark, measuring how long a robot takes to locate and re-acquire a disappeared target."],"forward_implications":["Long-term re-identification is the main bottleneck: targets disappear on average 45.4 times per sequence, for 65.3 seconds in aggregate, and even the best tracker loses them.","MOT-plus-target-ReID methods with human priors beat generic single-object trackers by a wide margin, with the best SOT method at 40.05 F-score versus 66.06 for the best overall method.","The performance drop relative to older datasets, up to 33 points of F-score versus Chen's dataset, implies that laboratory-style person-following evaluations overstate the readiness of current methods for crowded public spaces.","Zero-false-positive recall is especially low for every method, so a robot using these trackers could confidently follow the wrong person; the safest current method still scores only 16.26 AMR.","The released multi-modal streams, including panoramic, RGB-D, LiDAR, IMU, and odometry data, provide the substrate for moving beyond vision-only tracking, although only panoramic images are annotated in this version."],"supporting_citations":[{"why":"Provides the existing public target-person-tracking dataset used as the low-difficulty comparison, where the same methods score up to 99.1 F-score.","marker":"Chen et al. 2017b"},{"why":"Gives LaSOT person sequences as a second comparison benchmark, where the best evaluated method scores 70.3 F-score.","marker":"Fan et al. 2021"},{"why":"Supplies RPF-ReID and RPF-ReID+OCL, the top-performing baselines whose online ridge-regression classifier and replay-based continual learning carry the main experimental analysis.","marker":"Ye et al. 2024"},{"why":"Provides the KPR partial-re-identification features; swapping them in improves F-score by about 19 points over ResNet18, showing the role of occlusion-robust appearance features.","marker":"Somers et al. 2025"},{"why":"Supplies CARPE-ID, the comparison target-reID method with exponentially moving average feature updating.","marker":"Rollo et al. 2024"},{"why":"Supplies the long-term tracking metrics F-score and AMR, plus the dataset statistics methodology used to quantify disappearances and distractors.","marker":"Lukežič et al. 2020"},{"why":"Provides JRDB, the closest robot-egocentric human-perception benchmark, and the metadata conventions reused for the sequence annotations.","marker":"Martin-Martin et al. 2021"},{"why":"Supplies ByteTrack, the multi-object tracker used as the front-end for the MOT+Target-ReID baselines.","marker":"Zhang et al. 2022"}],"fun_headline_variants":["New robot egocentric tracking benchmark exposes SOTA weaknesses","Robot-view target tracking: SOTA fails in crowded long-term test","New robot-following benchmark trumps trackers, best F-score 66","Robot egocentric tracking dataset: existing methods fail in crowds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground-truth boxes are assumed to be accurate and neutral, but they are seeded by the same family of MOT-plus-target-re-identification trackers the paper evaluates, with no published inter-annotator agreement or manual-validation statistics; if those priors carry tracker-specific bias, the measured difficulty and method rankings could shift.","fun_headline_variants_meta":{"raw":{"variants":["New robot egocentric tracking benchmark exposes SOTA weaknesses","Robot-view target tracking: SOTA fails in crowded long-term test","New robot-following benchmark trumps trackers, best F-score 66","Robot egocentric tracking dataset: existing methods fail in crowds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3224,"prompt_tokens":971,"completion_tokens":2253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2180}},"tokens_in":587,"tokens_out":2253,"duration_ms":14072,"temperature":1.0,"reasoning_tokens":2180,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:15:49.613405+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of sequences from scratch without showing annotators any tracker output, then compare the new boxes to the released ones; if mean intersection-over-union falls below about 0.9 on visible frames, or if re-running the top methods on the re-annotated subset changes their F-score ranking, then the benchmark's difficulty claims rest on annotation bias rather than on the tracking task itself.","supporting_citations":[{"cited_title":"(2021) Lasot: A high-quality large-scale single object tracking benchmark","cited_arxiv_id":null,"evidence_quote":"Gives LaSOT person sequences as a second comparison benchmark, where the best evaluated method scores 70.3 F-score."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides the KPR partial-re-identification features; swapping them in improves F-score by about 19 points over ResNet18, showing the role of occlusion-robust appearance features."},{"cited_title":"IEEE Trans","cited_arxiv_id":null,"evidence_quote":"Provides JRDB, the closest robot-egocentric human-perception benchmark, and the metadata conventions reused for the sequence annotations."}],"review_version":1}