REVIEW 4 major objections 5 minor 64 references
Temporal Coherence for Active Learning in Videos
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Tracking detections across nearby frames and minimizing a graph-cut energy estimates where a video object detector errs, and those estimates select the frames most worth annotating.
desk verdict A plausible idea and a useful new dataset, but the central FP/FN error estimates are never directly validated and the final gains over uncertainty baselines are small enough that the main claim is only weakly supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the temporal-coherence graph and its energy function. Detections are linked across frames by forward and backward tracking (optical flow or a Siamese tracker), and unmatched tracked boxes are clustered into candidates for missed objects. The unary potentials encode the prior that detections are positives and candidates are negatives; the pairwise matrix penalizes connected nodes that receive inconsistent true/false labels. Minimizing the energy with graph cut produces global false-positive and false-negative estimates, and the estimated false-positive count per frame is the acquisition score. This same graph gives the oracle-based upper-bound acquisition function used as a reference.
What would settle it
Compare the method's per-frame estimated error counts with true error counts on a held-out video set with ground truth; the claim collapses if the correlation is near zero, or if temporal-coherence selection fails to beat random sampling on sequences with frequent long occlusions.
Extended reading notes
Core claim
The central claim is that the visual consistency of a video is a usable proxy for annotation value. Detector outputs from each frame become graph nodes; tracked boxes from neighboring frames that match nothing become 'candidates' representing likely missed instances. A four-label energy function with unary costs that initially trust detections and distrust candidates, plus pairwise costs that reward temporally linked nodes for sharing the same true/false status, is minimized by graph cut. The solution labels nodes as estimated true positives, false positives, true negatives, or false negatives, and the per-frame estimated error counts drive the acquisition function. The paper reports that this beats uncertainty-based and random selection and comes close to an oracle that selects frames by true error counts.
Load-bearing premise
The load-bearing premise is that the tracker keeps a reliable lock on objects across neighboring frames, so an unmatched tracked box really is a missed detection rather than a tracking failure; fast motion, occlusion, or objects entering and leaving the frame would turn this evidence into noise.
Editorial extensions
If this is right
- Temporal coherence reaches 95.3 percent of full-data mAP on SYNTHIA-AL with 12 percent of labels and 98.9 percent on ImageNet-VID with 10 percent, outperforming all tested baselines at every cycle.
- Optical-flow tracking performs as well as a learned Siamese tracker, and since flow is computed once it is the cheaper option for active learning.
- Counting estimated false positives in a frame is enough for selection; combining false-positive and false-negative counts did not improve results.
- Temporal representativeness, which avoids selecting frames adjacent to already-labeled ones, improves random sampling and is part of the reported setup.
Reading between the lines
- The success of a false-positive-only score suggests the usable signal is the total amount of temporal disagreement, not the precise error type; a simpler inconsistency count may reproduce the gains.
- Since the estimated errors rank frames well, the same graph labels could generate pseudo-annotations for unselected frames, turning selection into label propagation.
- The main headroom is false-negative recovery: better long-range tracking or interpolation should close more of the gap to the oracle, especially under occlusion.
- Applying the method to sequences with frequent long occlusions or fast camera motion is a natural stress test; if tracking breaks, estimated errors should stop correlating with true errors.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an active learning method for video object detection in which temporal coherence is used to estimate detector errors. Detections from neighboring frames are linked by a tracker, an undirected graph is built on detections and candidate proposals (unmatched tracked detections), and a graph-cut energy minimization assigns each node a label (TP/FP/TN/FN). The estimated number of false positives per frame is then used as the acquisition score. The authors also introduce SYNTHIA-AL, a synthetic road-scene video dataset with unbalanced and redundant content, and report experiments on SYNTHIA-AL and ImageNet-VID showing that the temporal-coherence method outperforms entropy, least confidence, margin sampling, and random baselines, achieving roughly 95–99% of full-data mAP with 10–12% of the labels.
Significance. If the results hold, the paper offers a practical and conceptually simple way to leverage video-specific temporal redundancy for active learning, with a new dataset that can support future work in this direction. The experimental setup is mostly careful: three repeats, consistent training pipeline, and an oracle-based upper bound. The optical-flow version is computationally attractive because the flow is computed once. However, the reported gains over strong uncertainty baselines are small, and the core claim—that the graph-cut FP/FN estimates are accurate enough to drive acquisition—is not directly tested. The significance of the contribution therefore hinges on additional validation that the mechanism genuinely measures annotation value.
major comments (4)
- [Section 3.2, Eqs. (1)–(4); Table 2] The central mechanism of the paper—that graph-cut labels provide accurate estimates of false positives and false negatives—is never directly validated. All evidence is qualitative (Fig. 3) or indirect (the TC acquisition curve versus oracle in Fig. 4c). No precision/recall of the estimated FP/FN labels, no per-frame correlation between estimated and actual error counts, and no ablation isolating the effect of the error estimates from other frame statistics are reported. This is load-bearing because the advantage over entropy is tiny (Table 2: 0.599 vs 0.597 mAP on SYNTHIA-AL, 0.830 vs 0.821 on ImageNet-VID), so even a modest confound (e.g., number of detections per frame) could explain the ordering without the proposed error-estimation story being true. I request a direct evaluation of the FP/FN estimates against ground truth on a subset of frames, and an ablation in which acquisition is based on raw detection counts or mean detection confidence rather than graph-cut labels.
- [Table 2; Figs. 4c and 5] The key quantitative claims are not supported by significance testing or by error bars at the reported budget points. Table 2 gives only means for the final budget, while standard deviations are shown only in the figures for intermediate cycles. The differences between TC and the best uncertainty baseline are within 0.002–0.009 mAP, which could easily be within run-to-run variability given that only three repeats are used. I ask that the authors report mean ± standard deviation for the final budget in Table 2, and perform a paired statistical test (e.g., a paired t-test or Wilcoxon signed-rank test across the three runs) to establish that TC consistently outperforms the baselines at the reported budget.
- [Sections 3.2 and 5.2] Several design choices appear to be selected on the same datasets used for evaluation: the IoU threshold θ for temporal linking, the number of surrounding frames T, the unary/pairwise energy costs in Eqs. (2)–(4), the choice of the FP-only acquisition variant, and the representativeness neighborhood k. Without a separate validation split or a sensitivity analysis, it is difficult to rule out overfitting of the acquisition function to the two benchmarks. Please provide a sensitivity study in which θ, T, and the FP/FN combination are varied, and state which parameters were fixed a priori and which were tuned on held-out data.
- [Section 3.2, tracking discussion] The reliability of the FP/FN estimates depends critically on the tracker maintaining correspondence across frames; fast motion, occlusion, and sequence boundaries are obvious failure cases. The paper does not analyze how often tracking fails or how these failures corrupt the graph structure and the resulting error estimates. Given that this is the central assumption behind the edge construction, I request a quantitative analysis of tracking reliability (e.g., tracking failure rates on sequences with and without occlusion) and at minimum a discussion of when the method can be expected to break down.
minor comments (5)
- [Section 3.1] The sentence introducing false negatives reads "False Negatives (FP)"; the abbreviation should be (FN).
- [Section 1 and Related Work] The claim to be "the first to consider" active learning for video object detection is too strong given the cited work [29] (Karasev et al., CVPR 2014), which explicitly addresses active frame, location, and detector selection for video annotation. Please qualify the novelty claim relative to [29] and discuss how the objective differs.
- [Section 3.2, Eq. (4)] The pairwise cost matrix is presented without an intuitive explanation of the rationale for each allowed/disallowed label transition (e.g., why TP connects to FN but not to FP). Adding a short explanation would make the model more accessible.
- [Figure 4] The three panels of Figure 4 are not labeled (a), (b), (c) in the caption, although the text refers to them as Fig. 4a, Fig. 4b, and Fig. 4c. Please add explicit subfigure labels.
- [Table 2] Oracle results are reported only for SYNTHIA-AL; please state explicitly that oracle-based acquisition was not evaluated on ImageNet-VID, and explain why (e.g., label noise or annotation cost).
Circularity Check
No significant circularity: temporal-coherence error estimates are a hand-designed model, and the oracle functions are used only as comparison.
full rationale
The paper's derivation chain is self-contained: Section 3.2 defines a graph over tracked detections and minimizes a hand-designed energy function (Eqs. 1-4) to obtain FP/FN estimates; these estimates feed an acquisition function that selects frames for annotation. No parameter is fitted to ground-truth labels, and the resulting selection is evaluated by measured mAP on SYNTHIA-AL and ImageNet-VID against external baselines. The oracle-based acquisition in Section 3.1 is explicitly presented as an upper-bound comparison, not as a training target or fitted input, so it does not make the TC predictions circular. The only self-citation is the SYNTHIA dataset (Ref. [42]), which is a data resource and is not load-bearing for the method. Choices such as using the FP-only variant, the IoU threshold, and the tracking window are empirical hyperparameter selections rather than reductions of the prediction to its inputs. Thus no circular step can be exhibited.
Assumptions & free parameters
free parameters (5)
- IoU threshold for temporal linking (theta)
- Number of surrounding frames tracked (T) =
3
- Unary and pairwise energy costs =
0, 1, infinity; 0/1 pairwise matrix
- Acquisition variant (FP vs FN vs both) =
FP only
- Temporal representativeness neighborhood k =
1 for SYNTHIA-AL, 3 for ImageNet-VID
assumptions (4)
- domain assumption Temporal coherence: nearby frames contain the same object instances in nearby locations.
- ad hoc to paper Detector outputs are initially trusted: detections cannot be labeled as true negatives or false negatives a priori.
- standard math The energy function in Eq. (1) can be globally minimized via graph cut.
- domain assumption Unmatched tracked detections indicate missed instances (false negatives).
invented entities (2)
-
SYNTHIA-AL dataset
independent evidence
-
Detection candidates
Cite this review
Pith. "Pith review of Temporal Coherence for Active Learning in Videos." pith.science (2026). https://pith.science/paper/LDMP2OJ2
@misc{pith2026190811757,
author = {Pith},
title = {Pith review of: Temporal Coherence for Active Learning in Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/LDMP2OJ2}},
note = {Machine review of arXiv:1908.11757}
}
read the original abstract
Autonomous driving systems require huge amounts of data to train. Manual annotation of this data is time-consuming and prohibitively expensive since it involves human resources. Therefore, active learning emerged as an alternative to ease this effort and to make data annotation more manageable. In this paper, we introduce a novel active learning approach for object detection in videos by exploiting temporal coherence. Our active learning criterion is based on the estimated number of errors in terms of false positives and false negatives. The detections obtained by the object detector are used to define the nodes of a graph and tracked forward and backward to temporally link the nodes. Minimizing an energy function defined on this graphical model provides estimates of both false positives and false negatives. Additionally, we introduce a synthetic video dataset, called SYNTHIA-AL, specially designed to evaluate active learning for video object detection in road scenes. Finally, we show that our approach outperforms active learning baselines tested on two datasets.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr. Fully-convolutional siamese networks for object tracking. In ECCV, pages 850–865, 2016. 5, 7
work page 2016
-
[2]
Y . Boykov and V . Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. on PAMI, 26(9):1124–1137, 2004. 5
work page 2004
- [3]
-
[4]
W. Cai, Y . Zhang, S. Zhou, W. Wang, C. Ding, and X. Gu. Active learning for support vector machines with maximum model change. In Machine Learning and Knowledge Dis- covery in Databases, pages 211–226. Springer, 2014. 2
work page 2014
- [5]
- [6]
-
[7]
I. Dagan and S. P. Engelson. Committee-based sampling for training probabilistic classifiers. In Machine Learning Pro- ceedings 1995, pages 150–157. Elsevier, 1995. 7
work page 1995
-
[8]
C. Deng, X. Liu, C. Li, and D. Tao. Active multi-kernel do- main adaptation for hyperspectral image classification. Pat- tern Recognition, 77:306–315, 2018. 1, 2
work page 2018
Show all 64 references
-
[9]
Everingham, L
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) chal- lenge. IJCV, 88(2):303–338, 2010. 3, 4, 5
2010
-
[10]
P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ra- manan. Object detection with discriminatively trained part- based models. IEEE Trans. on PAMI , 32(9):1627–1645,
-
[11]
Freytag, E
A. Freytag, E. Rodner, and J. Denzler. Selecting influen- tial examples: Active learning with expected model output changes. In ECCV, pages 562–577, 2014. 2
2014
-
[12]
W. Fu, M. Wang, S. Hao, and X. Wu. Scalable active learning by approximated error reduction. InKDD, pages 1396–1405,
-
[13]
Y . Gal, R. Islam, and Z. Ghahramani. Deep bayesian active learning with image data. In ICML, pages 1183–1192, 2017. 2, 5, 7
2017
-
[14]
Gavves, T
E. Gavves, T. E. J. Mensink, T. Tommasi, and T. Snoek, C. G. M.and Tuytelaars. Active transfer learning with zero-shot priors: Reusing past datasets for future tasks. InICCV, pages 1–9, 2015. 1, 2
2015
-
[15]
Geiger, P
A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. IJRR, 32(11):1231–1237, 2013. 1, 5
2013
-
[16]
Q. Gu, T. Z. Zhang, C. Ding, and J. Han. Selective labeling via error bound minimization. In NIPS, pages 1–9, 2012. 2
2012
-
[17]
Y . Guo. Active instance sampling via matrix partition. In NIPS, pages 1–9, 2010. 2
2010
-
[18]
W. Han, P. Khorrami, T. L. Paine, P. Ramachandran, M. Babaeizadeh, H. Shi, J. Li, S. Yan, and T. S. Huang. Seq-nms for video object detection. arXiv preprint arXiv:1602.08465, 2016. 2
2016 arXiv
-
[19]
F. C. Heilbron, J.-Y . Lee, H. Jin, and B. Ghanem. What do i annotate next? an empirical study of active learning for action localization. In ECCV, pages 212–229, 2018. 2
2018
-
[20]
Hoffman, S
J. Hoffman, S. Guadarrama, E. Tzeng, J. Donahue, R. B. Gir- shick, T. Darrell, and K. Saenko. Lsda: Large scale detection through adaptation. In NIPS, pages 1–9, 2014. 2
2014
-
[21]
Huang, V
J. Huang, V . Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y . Song, S. Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In CVPR, pages 7310–7311, 2017. 6
2017
-
[22]
Huang, R
S.-J. Huang, R. Jin, and Z.-H. Zhou. Active learning by querying informative and representative examples. IEEE Trans. on PAMI, 10(36):1936–1949, 2014. 2
1936
-
[23]
S. Jin, A. RoyChowdhury, H. Jiang, A. Singh, A. Prasad, D. Chakraborty, and E. Learned-Miller. Unsupervised hard example mining from videos for improved object detection. In ECCV, pages 307–324, 2018. 1
2018
-
[24]
A. J. Joshi, F. Porikli, and N. P. Papanikolopoulos. Scal- able active learning for multiclass image classification.IEEE Trans. on PAMI, 34(11):2259–2273, 2012. 1, 2
2012
-
[25]
K ¨ading, E
C. K ¨ading, E. Rodner, A. Freytag, O. Mothes, B. Barz, and J. Denzler. Active learning for regression tasks with expected model output changes. In BMVC, pages 1–15, 2018. 2
2018
-
[26]
K. Kang, H. Li, T. Xiao, W. Ouyang, J. Yan, X. Liu, and X. Wang. Object detection in videos with tubelet proposal networks. In CVPR, pages 727–735, 2017. 1, 2
2017
-
[27]
K. Kang, H. Li, J. Yan, X. Zeng, B. Yang, T. Xiao, C. Zhang, Z. Wang, R. Wang, X. Wang, et al. T-cnn: Tubelets with con- volutional neural networks for object detection from videos. IEEE TCSVT, 28(10):2896–2907, 2018. 1, 2
2018
-
[28]
Kapoor, K
A. Kapoor, K. Grauman, R. Urtasun, and T. Darrell. Active learning with gaussian processes for object categorization. In CVPR, pages 1–8, 2007. 2
2007
-
[29]
Karasev, A
V . Karasev, A. Ravichandran, and S. Soatto. Active frame, location, and detector selection for automated and manual video annotation. In CVPR, pages 2131–2138, 2014. 1
2014
-
[30]
Kolmogorov and R
V . Kolmogorov and R. Zabih. What energy functions can be minimizedvia graph cuts? IEEE Trans. on PAMI, 26(2):147– 159, 2004. 5
2004
-
[31]
Krizhevsky and G
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 5
2009
-
[32]
Y . LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998. 5
1998
-
[33]
D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In SIGIR, pages 3–12, 1994. 6
1994
-
[34]
Li and Y
X. Li and Y . Guo. Adaptive active learning for image classi- fication. In cvpr, pages 860–866, 2013. 1
2013
-
[35]
Li and Y
X. Li and Y . Guo. Multi-level adaptive active learning for scene classification. In ECCV, pages 234–249, 2014. 1, 2
2014
-
[36]
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Doll´ar, and C. L. Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, pages 740–755, 2014. 3, 6
2014
-
[37]
Lin and D
X. Lin and D. Parikh. Active learning for visual ques- tion answering: An empirical study. arXiv preprint arXiv:1711.01732, 2017. 2
2017 arXiv
-
[38]
Liu and M
M. Liu and M. Zhu. Mobile video object detection with temporally-aware feature maps. In CVPR, pages 5686–5695,
-
[39]
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y . Fu, and A. C. Berg. Ssd: Single shot multibox detector. In ECCV, pages 21–37, 2016. 2
2016
-
[40]
Madhavan and T
V . Madhavan and T. Darrell. The bdd-nexar collective: A large-scale, crowsourced, dataset of driving scenes. Master’s thesis, EECS Department, University of California, Berke- ley, May 2017. 1
2017
-
[41]
S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91–99, 2015. 2, 4, 6
2015
-
[42]
G. Ros, L. Sellart, J. Materzyska, D. V ´azquez, and A. L´opez. The SYNTHIA dataset: a large collection of synthetic im- ages for semantic segmentation of urban scenes. In CVPR, pages 3234–3243, 2016. 5
2016
-
[43]
Rosenfeld, R
A. Rosenfeld, R. Zemel, and J. K. Tsotsos. The elephant in the room. arXiv preprint arXiv:1808.03305, 2018. 1
2018 arXiv
-
[44]
S. Roy, A. Unmesh, and V . P. Namboodiri. Deep active learn- ing for object detection. In BMVC, pages 1–12, 2018. 1, 2, 5, 6
2018
-
[45]
Russakovsky, J
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recog- nition challenge. IJCV, 115(3):211–252, 2015. 1, 6, 7, 8
2015
-
[46]
Saito, C
P. Saito, C. Suzuki, J. Gomes, P. de Rezende, and A. Falc ˜ao. Robust active learning for the diagnosis of parasites. Pattern Recognition, 48(11):3572–3583, 2015. 1, 2
2015
-
[47]
A. I. Schein and L. H. Ungar. Active learning for logistic regression: an evaluation. Machine Learning, 68(3):235– 265, 2007. 2
2007
-
[48]
Sener and S
O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, pages 1–13,
-
[49]
B. Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009. 1, 2, 3, 5, 7
2009
-
[50]
Richter, Z
Stephan R. Richter, Z. Hayder, and V . Koltun. Playing for benchmarks. In ICCV, pages 2213–2222, 2017. 5
2017
-
[51]
D. Sun, X. Yang, M.-Y . Liu, and J. Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. June
-
[52]
Vezhnevets, J
A. Vezhnevets, J. M. Buhmann, and V . Ferrari. Active learn- ing for semantic segmentation with expected change. In CVPR, pages 3162–3169, 2012. 2
2012
-
[53]
Vijayanarasimhan and K
S. Vijayanarasimhan and K. Grauman. Large-scale live ac- tive learning: Training object detectors with crawled data and crowds. IJCV, 108(1–2):97–114, 2014. 1, 2, 5
2014
-
[54]
S. Wang, Y . Zhou, J. Yan, and Z. Deng. Fully motion-aware network for video object detection. In ECCV, pages 542– 557, 2018. 1, 2
2018
-
[55]
Y . Yan, F. Nie, W. Li, C. Gao, Y . Yang, and D. Xu. Image classification by cross-media active learning with privileged information. IEEE Trans. on Multimedia , 18(12):2494– 2502, 2016. 1
2016
-
[56]
Yang and M
Y . Yang and M. Loog. A benchmark and comparison of active learning for logistic regression. Pattern Recognition, 83:401–415, 2018. 2
2018
-
[57]
Yang and M
Y . Yang and M. Loog. A variance maximization criterion for active learning. Pattern Recognition, 78:358–370, 2018. 2
2018
-
[58]
Y . Yang, Z. Ma, F. Nie, X. Chang, and A. G. Hauptmann. Multi-class active learning by uncertainty sampling with di- versity maximization. IJCV, 113(2):113–127, 2015. 2
2015
-
[59]
A. Yao, J. G. Gall, C. Leistner, and L. Van Gool. Interactive object detection. In CVPR, pages 3242–3249, 2012. 1, 2
2012
-
[60]
Yoo and I
D. Yoo and I. S. Kweon. Learning loss for active learning. In CVPR, pages 93–102, 2019. 2, 5
2019
-
[61]
F. Yu, W. Xian, Y . Chen, F. Liu, M. Liao, V . Madha- van, and T. Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint arXiv:1805.04687, 2018. 1, 5
2018 arXiv
-
[62]
Zhang, F
D. Zhang, F. Wang, Z. Shi, and C. Zhang. Interactive lo- calized content based image retrieval with multiple-instance active learning. Pattern Recognition, 43(2):478–484, 2010. 2
2010
-
[63]
X. Zhu, Y . Wang, J. Dai, L. Yuan, and Y . Wei. Flow-guided feature aggregation for video object detection. In ICCV, pages 408–417, 2017. 1, 2
2017
-
[64]
X. Zhu, Y . Xiong, J. Dai, L. Yuan, and Y . Wei. Deep fea- ture flow for video recognition. In CVPR, pages 2349–2358,
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.