REVIEW 5 major objections 5 minor 1 cited by
Neural Spatial-Temporal Tensor Representation for Infrared Small Target Detection
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read NeurSTT represents an infrared video's background as a low-rank neural tensor function, so the residual contains only sparse moving targets, and detects them without any labeled training data.
desk verdict A genuinely new unsupervised INR-based ISTD method, but the missing binarization threshold makes the headline IoU/F1 numbers uncheckable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the neural-represented low-rank background tensor: three sinusoidal MLPs map coordinate vectors along height, width, and time into factor matrices that multiply a learnable core tensor, so the Tucker rank is set by the downsampling rate $r_d = 1/4$. The argument is carried by the interplay of three loss terms: a nuclear-norm loss on each background frame that enforces low rank, a neural 3-D total variation loss computed from the MLP derivatives that enforces local smoothness in space and time, and an $\ell^1$ loss on the soft-thresholded residual that preserves sparse targets. This machinery lets a deep optimizer do the work that classical tensor RPCA solvers did with singular-value thresholding.
What would settle it
Run NeurSTT on a sequence with strong camera pan or vigorous moving clutter, such as waving trees, and measure whether the residual $D - B^{NLR}$ remains sparse: if residual energy concentrates in large moving background structures rather than in the target footprint, the background model is inadequate. A more quantitative version is to sweep $r_d$ from 1/8 to 1 on such a sequence; the method should show a clear performance peak only if the fixed rank assumption matches the sequence's background complexity.
Extended reading notes
Core claim
The central discovery is that the classical low-rank-plus-sparse decomposition of infrared sequences can be neuralized by writing the background as a continuous tensor function $B^{NLR}_{ijk} = [\mathcal{G}; f_{\theta_h}, f_{\theta_w}, f_{\theta_t}](i,j,k)$ with a small Tucker-style core $\mathcal{G}$ and three shallow SIREN MLPs, then unrolling the optimization: a nuclear-norm loss on each temporal slice, a neural 3-D total variation loss on the MLP derivatives, an $\ell^1$ target loss, and a soft-thresholding update $T = ST_{\lambda/2}(D - B^{NLR})$. This replaces the heavy tensor solvers of prior methods with Adam updates on network parameters while keeping the domain priors explicit. The paper argues that the nonlinear representation captures spatial-temporal correlations that linear Tucker or tensor operations miss, yielding better background suppression and fewer false alarms.
Load-bearing premise
The load-bearing premise is that real infrared video backgrounds are well approximated by the specific low-rank neural function with Tucker ranks set to one quarter of the sequence dimensions, so that after subtracting it the residual is mostly sparse target signal rather than structured background.
Editorial extensions
If this is right
- Infrared small-target detection can be done with zero labeled training data, removing the labeling bottleneck for video datasets.
- The same model can process both 256x256 and 720x480 sequences, with parameter counts scaling more gently than 3D-CNN baselines.
- Adding neural temporal total variation suppresses moving target-like clusters and improves IoU by about 1.28 points over spatial-only TV in ablations.
- The reported averages across nine sequences include large gains on hard 720x480 scenes where several tensor baselines fail entirely.
- On 256x256 inputs the method is 16.6x lighter and 48.9% faster than 3DSTPM, making the unsupervised scheme practical as a baseline.
Reading between the lines
- An implication the authors leave implicit: the same neural low-rank-plus-sparse decomposition could be applied to other spatiotemporal inverse problems, such as video background subtraction or moving-object detection in other spectra, wherever a slowly varying background is separated from sparse events.
- The rank-setting rule $r_d=1/4$ was selected by ablation on the evaluation sequences; a data-driven or adaptive rank selection would be a natural test of whether the method generalizes to backgrounds with higher effective complexity.
- A testable extension: replacing the SIREN factor MLPs with coordinate-based networks that incorporate temporal motion compensation could extend the method to scenes with global camera motion, which the current static-coordinate representation does not model.
- The soft-thresholding target update suggests the framework could be extended to jointly estimate target motion or track multiple targets by adding a temporal continuity prior on the target tensor.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes NeurSTT, an unsupervised neural-represented spatial-temporal tensor model for infrared small target detection. The background of a stacked sequence is approximated by a low-rank Tucker tensor function implemented with three SIREN-based MLPs, a neural 3D total variation regularizer is added through derivatives of those networks, and the target tensor is updated by soft-thresholding the residual. All parameters are optimized with Adam using a loss composed of a sum of per-frame nuclear norms, an l1 target term, and the Neur3DTV term. The authors evaluate on nine sequences from two public datasets, compare against twelve single- and multi-frame baselines, and report average IoU 82.82% and F1 90.52%, with 16.6x fewer parameters and 48.9% less runtime than 3DSTPM on 256x256 inputs.
Significance. If the reported numbers are trustworthy, the contribution is useful: it is an unsupervised, label-free method that combines low-rank and sparsity priors with implicit neural representation, and it is systematically compared with a broad set of baselines. The architectural idea of representing the background as a low-rank tensor function and regularizing it through neural derivatives is interesting and reasonably well motivated. The paper also provides ablation studies for the main modules and a promised public code link. However, the central empirical claims are currently weakened by an unspecified binarization threshold, hyperparameter selection on the same test sequences, and an inconsistency between the stated objective and the implemented loss; these issues must be resolved before the quantitative conclusions can be accepted.
major comments (5)
- [Section 4.4 and Section 5.2] The adaptive threshold procedure used to binarize the NeurSTT target map is not specified. Section 4.4 states that after solving Eq. (22) the result tensor is post-processed with a 'general tensor-to-sequence operation' and 'an adaptive threshold procedure,' but no formula, criterion, or implementation detail is given, and Section 5.2 only lists thresholds for baseline methods. Because every pixel-level IoU and F1 value in Table 9 is computed on binary maps, the headline average IoU 82.82% and F1 90.52% cannot be reproduced or checked; the paper must state the binarization rule and confirm that it does not use ground-truth information.
- [Section 5.3 and Table 9] Hyperparameters L, lambda, phi, kappa, and kmax are selected by varying one parameter at a time on the same six sequences from [42] that are then used in the final evaluation of Table 9. This selection-on-test protocol makes the reported average IoU/F1 on those six sequences partly an artifact of tuning rather than an unbiased estimate of performance; a validation split, cross-validation, or separate development set is needed before claiming superiority over the baselines.
- [Eqs. (17), (19), and (24)] The optimization problem stated in Eq. (17) contains the tensor nuclear norm ||BNLR||_*, but the loss actually minimized, Eq. (24), uses LNuc defined in Eq. (19) as the sum of per-frame matrix nuclear norms \sum_k ||B_NLR_k||_*. These are different regularizers for a third-order tensor, so the paper does not optimize the objective it derives; the text should either replace ||BNLR||_* in Eqs. (17) and (18) with the per-frame sum or justify the discrepancy.
- [Section 5.3, Config. III, and Table 2] The ablation text for phi says that phi=1e-5 performs best and is selected, but Table 3 shows 5e-5 achieves the highest IoU (81.44) and Table 2 lists the final setting as phi=5e-5. This internal contradiction must be resolved.
- [Section 5.3 and Algorithm 1] No multiple runs or random seeds are reported. Since SIREN weights are randomly initialized and optimization is stochastic Adam-based, the reported IoU/F1 values and the hyperparameter choices in Tables 3 and 4 may depend on initialization; providing means and standard deviations over several runs is necessary to support the quantitative claims.
minor comments (5)
- [Table 4] The F1 value for NeurSTT (Full) is printed as 6094; this appears to be a typo for 60.94.
- [Abstract] The abstract states 'outperforms detection challenges' and 'averaged 19.19% higher in IoU'; the first phrase is ungrammatical and the second should specify that the comparison is with 3DSTPM on the 256x256 sequences and whether the increase is absolute or relative.
- [Eq. (13)] The temporal difference operator D_t has an extra closing parenthesis: 'B(i,j,k−1))' should read 'B(i,j,k−1)'.
- [Fig. 1] The caption contains the typo 'physic-informed'; it should be 'physics-informed'.
- [Section 5.2] The precision formula is written as 'Precision = TP/TP+FP'; parentheses are needed: Precision = TP/(TP+FP).
Circularity Check
Partial circularity: hyperparameters are tuned on the same six sequences that dominate the headline nine-sequence average; adaptive-threshold step is underspecified.
-
fitted input called prediction
[Section 5.3, Ablation Studies (Tables 3, 4, and 9)]
"Notably, we use the average values of two pixel-level metrics, IoU and F1, across six sequences from [42] for this analysis."
Table 4 chooses rd by maximizing the averaged IoU/F1 on the six [42] sequences, and Table 3 does the same for L, λ, φ, κ, and kmax. The final comparison in Table 9 then reports average IoU/F1 over Seqs. 1-9, where Seqs. 1-6 are exactly those same six [42] sequences. The headline 82.82 IoU / 90.52 F1 is therefore, for six of nine sequences, the tuning objective itself rather than an independent prediction. The reported advantage over 3DSTPM on those sequences is partly a test-set selection effect, not a consequence of the model equations.
full rationale
Modeling-side derivation is not circular: B_NLR (Eq. 11) follows from the tensor-function factorization (Eqs. 8-9), Neur3DTV (Eq. 16) is an explicit derivative-based regularizer, and the target update (Eq. 22) is the standard soft-thresholding step of Eq. (21). These equations do not encode the final IoU/F1 numbers, and the priors borrowed from [36], [37], and [40] are external rather than self-citations. The circularity found is evaluation-side: Section 5.3 tunes L, λ, φ, κ, kmax, and rd on the average IoU/F1 of the six [42] sequences, and Table 9 includes those same six sequences in the reported nine-sequence average (82.82 IoU / 90.52 F1), so the headline performance is partly a fitted quantity. Seqs. 7-9 from [22] were not in the ablations and provide some independent support, so the architecture itself is not predetermined. One missing specification is also noted and weighed: Section 4.4 says only that post-processing uses 'an adaptive threshold procedure,' without giving a formula or fitting rule, and Section 5.2 prescribes thresholds only for other methods. This is a reproducibility gap rather than a demonstrated circular step, but it leaves the binary-metric comparison only partially checkable.
Assumptions & free parameters
free parameters (9)
- Temporal index L =
80
- Sparsity weight lambda =
0.20
- Neur3DTV weight phi =
5e-5
- Temporal derivative weight kappa =
100
- Max iterations kmax =
2000
- Downsampling rate rd =
1/4
- MLP channel size ci =
ci=ni
- SIREN initialization std =
5
- Learning rate and weight decay =
0.0005, 0.01
assumptions (6)
- domain assumption Observed tensor is exactly background plus target (D=B+T).
- domain assumption Infrared backgrounds are low-rank and targets are sparse.
- standard math Low F-rank tensor functions admit factorization G x1 f_x(.) x2 f_y(.) x3 f_z(.).
- domain assumption SIREN sine activation and its cosine derivative capture complex structure and remain differentiable.
- ad hoc to paper Derivatives of factor MLPs with respect to coordinate inputs act as spatial and temporal total variation of the reconstructed background.
- ad hoc to paper Sum of per-frame nuclear norms is a valid low-rank background regularizer.
Cite this review
Pith. "Pith review of Neural Spatial-Temporal Tensor Representation for Infrared Small Target Detection." pith.science (2026). https://pith.science/paper/TRTZPVJX
@misc{pith2026241217302,
author = {Pith},
title = {Pith review of: Neural Spatial-Temporal Tensor Representation for Infrared Small Target Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/TRTZPVJX}},
note = {Machine review of arXiv:2412.17302}
}
abstract
Optimization-based approaches dominate infrared small target detection as they leverage infrared imagery's intrinsic low-rankness and sparsity. While effective for single-frame images, they struggle with dynamic changes in multi-frame scenarios as traditional spatial-temporal representations often fail to adapt. To address these challenges, we introduce a Neural-represented Spatial-Temporal Tensor (NeurSTT) model. This framework employs nonlinear networks to enhance spatial-temporal feature correlations in background approximation, thereby supporting target detection in an unsupervised manner. Specifically, we employ neural layers to approximate sequential backgrounds within a low-rank informed deep scheme. A neural three-dimensional total variation is developed to refine background smoothness while reducing static target-like clusters in sequences. Traditional sparsity constraints are incorporated into the loss functions to preserve potential targets. By replacing complex solvers with a deep updating strategy, NeurSTT simplifies the optimization process in a domain-awareness way. Visual and numerical results across various datasets demonstrate that our method outperforms detection challenges. Notably, it has 16.6$\times$ fewer parameters and averaged 19.19\% higher in $IoU$ compared to the suboptimal method on $256 \times 256$ sequences.
Figures
Figures from the paper (9 more)
Forward citations
Cited by 1 Pith paper
-
Continuous Representation Methods, Theories, and Applications: An Overview and Perspectives
A survey organizing continuous representation methods into parametric models, structural modeling, theory, and applications, with a curated open-source reference repository.
Reference graph
Works this paper leans on
-
[42]
B. Hui, Z. Song, H. Fan, P. Zhong, W. Hu, X. Zhang, J. Lin, H. Su, W. Jin, Y . Zhang, et al., A dataset for infrared image dim-small aircraft target detection and tracking underground/air background, Sci. Data Bank 5 (12) (2019) 4
work page 2019
-
[1]
R. Kou, C. Wang, Z. Peng, Z. Zhao, Y . Chen, J. Han, F. Huang, Y . Yu, Q. Fu, Infrared small target segmentation networks: A survey, Pattern Recognit. 143 (2023) 109788
work page 2023
-
[2]
M. Zhao, W. Li, L. Li, J. Hu, P. Ma, R. Tao, Single-frame infrared small-target detection: A survey, IEEE Geosci. Remote Sens. Mag. 10 (2) (2022) 87–119
work page 2022
- [3]
-
[4]
C. Gao, D. Meng, Y . Yang, Y . Wang, X. Zhou, A. G. Hauptmann, Infrared patch- image model for small target detection in a single image, IEEE Trans. Image Process. 22 (12) (2013) 4996–5009. 31
work page 2013
-
[5]
Y . Dai, Y . Wu, Reweighted infrared patch-tensor model with both nonlocal and local priors for single-frame small target detection, IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 10 (8) (2017) 3752–3767
work page 2017
-
[6]
F. Wu, T. Zhang, L. Li, Y . Huang, Z. Peng, Rpcanet: Deep unfolding rpca based infrared small target detection, in: Proc. IEEE Winter Conf. Appl. Comput. Vis. (W ACV), 2024, pp. 4809–4818
work page 2024
-
[7]
C. Kwan, B. Budavari, Enhancing small moving target detection performance in low-quality and long-range infrared videos using optical flow techniques, Remote Sens. 12 (24) (2020) 4024
work page 2020
Show all 43 references
-
[8]
T. Liu, J. Yang, B. Li, C. Xiao, Y . Sun, Y . Wang, W. An, Nonconvex tensor low-rank approximation for infrared small target detection, IEEE Trans. Geosci. Remote Sens. 60 (2021) 1–18
2021
-
[9]
S. Chen, L. Ji, J. Zhu, M. Ye, X. Yao, Sstnet: Sliced spatio-temporal network with cross-slice convlstm for moving infrared dim-small target detection, IEEE Trans. Geosci. Remote Sens. (2024)
2024
-
[10]
Y . Liu, J. Liu, Z. Long, C. Zhu, Y . Liu, J. Liu, Z. Long, C. Zhu, Deep networks for tensor approximation, Tensor Computation for Data Analysis (2022) 265–284
2022
-
[11]
T. Liu, Q. Yin, J. Yang, Y . Wang, W. An, Combining deep denoiser and low-rank priors for infrared small target detection, Pattern Recognit. 135 (2023) 109184
2023
-
[12]
Miao, X.-L
Y .-C. Miao, X.-L. Zhao, X. Fu, J.-L. Wang, Y .-B. Zheng, Hyperspectral denoising using unsupervised disentangled spatiospectral deep priors., IEEE Trans. Geosci. Remote. Sens. 60 (2022) 1–16
2022
-
[13]
Zhang, P
Z. Zhang, P. Gao, S. Ji, X. Wang, P. Zhang, Infrared small target detection com- bining deep spatial-temporal prior with traditional priors, IEEE Trans. Geosci. Remote Sens. (2023)
2023
-
[14]
Molaei, A
A. Molaei, A. Aminimehr, A. Tavakoli, A. Kazerouni, B. Azad, R. Azad, D. Mer- hof, Implicit neural representation in medical imaging: A comparative survey, in: Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023, pp. 2381–2391. 32
2023
-
[15]
D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014)
2014 arXiv
-
[16]
X. Bai, F. Zhou, Analysis of new top-hat transformation and the application for infrared dim small target detection, Pattern Recognit. 43 (6) (2010) 2145–2156
2010
-
[17]
Y . Wei, X. You, H. Li, Multiscale patch-based contrast measure for small infrared target detection, Pattern Recognit. 58 (2016) 216–226
2016
-
[18]
Zhang, Z
L. Zhang, Z. Peng, Infrared small target detection based on partial sum of the tensor nuclear norm, Remote Sens. 11 (4) (2019) 382
2019
-
[19]
X. Kong, C. Yang, S. Cao, C. Li, Z. Peng, Infrared small target detection via nonconvex tensor fibered rank approximation, IEEE Trans. Geosci. Remote Sens. 60 (2021) 1–21
2021
-
[20]
B. Li, C. Xiao, L. Wang, Y . Wang, Z. Lin, M. Li, W. An, Y . Guo, Dense nested attention network for infrared small target detection, IEEE Trans. Image Process. 32 (2022) 1745–1758
2022
-
[21]
Zhang, L
T. Zhang, L. Li, S. Cao, T. Pu, Z. Peng, Attention-guided pyramid context net- works for detecting infrared small target under complex background, IEEE Trans. Aerosp. Electron. Syst. 59 (4) (2023) 4250–4261
2023
-
[22]
H. Sun, J. Bai, F. Yang, X. Bai, Receptive-field and direction induced attention network for infrared dim small target detection with a large-scale dataset irdst, IEEE Trans. Geosci. Remote Sens. 61 (2023) 1–13
2023
-
[23]
Y . Dai, Y . Wu, F. Zhou, K. Barnard, Attentional local contrast networks for in- frared small target detection, IEEE Trans. Geosci. Remote Sens. 59 (11) (2021) 9813–9824
2021
-
[24]
Zhang, R
M. Zhang, R. Zhang, Y . Yang, H. Bai, J. Zhang, J. Guo, Isnet: Shape matters for infrared small target detection, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, pp. 877–886. 33
2022
-
[25]
P. Du, A. Hamdulla, Infrared moving small-target detection using spatial– temporal local di fference measure, IEEE Geosci. Remote Sens. Lett. 17 (10) (2019) 1817–1821
2019
-
[26]
Y . Luo, X. Li, J. Wang, S. Chen, Clustering and tracking-guided infrared spatial- temporal small target detection, IEEE Trans. Geosci. Remote Sens. (2024)
2024
-
[27]
H.-K. Liu, L. Zhang, H. Huang, Small target detection in infrared videos based on spatio-temporal tensor model, IEEE Trans. Geosci. Remote Sens. 58 (12) (2020) 8689–8700
2020
-
[28]
G. Wang, B. Tao, X. Kong, Z. Peng, Infrared small target detection using nonover- lapping patch spatial–temporal tensor factorization with capped nuclear norm reg- ularization, IEEE Trans. Geosci. Remote Sens. 60 (2021) 1–17
2021
-
[29]
F. Wu, H. Yu, A. Liu, J. Luo, Z. Peng, Infrared small target detection using spatiotemporal 4-d tensor train and ring unfolding, IEEE Trans. Geosci. Remote Sens. 61 (2023) 1–22
2023
-
[30]
P. Yan, R. Hou, X. Duan, C. Yue, X. Wang, X. Cao, Stdmanet: Spatio-temporal differential multiscale attention network for small moving infrared target detec- tion, IEEE Trans. Geosci. Remote Sens. 61 (2023) 1–16
2023
-
[31]
R. Li, W. An, C. Xiao, B. Li, Y . Wang, M. Li, Y . Guo, Direction-coded temporal u- shape module for multiframe infrared small target detection, IEEE Trans. Neural Networks Learn. Syst. (2023)
2023
-
[32]
H. Deng, Y . Zhang, Y . Li, K. Cheng, Z. Chen, Bemst: Multi-frame infrared small- dim target detection using probabilistic estimation of sequential backgrounds, IEEE Trans. Geosci. Remote Sens. (2024)
2024
-
[33]
Heide, M
F. Heide, M. Steinberger, Y .-T. Tsai, M. Rouf, D. Paj˛ ak, D. Reddy, O. Gallo, J. Liu, W. Heidrich, K. Egiazarian, et al., Flexisp: A flexible camera image pro- cessing framework, ACM Trans. Graph. 33 (6) (2014) 1–13. 34
2014
-
[34]
Zhang, W
K. Zhang, W. Zuo, L. Zhang, Ffdnet: Toward a fast and flexible solution for cnn- based image denoising, IEEE Trans. Image Process. 27 (9) (2018) 4608–4622
2018
-
[35]
Luo, X.-L
Y .-S. Luo, X.-L. Zhao, T.-X. Jiang, Y . Chang, M. K. Ng, C. Li, Self-supervised nonlinear transform-based tensor nuclear norm for multi-dimensional image re- covery, IEEE Trans. Image Process. 31 (2022) 3793–3808
2022
-
[36]
Y . Luo, X. Zhao, Z. Li, M. K. Ng, D. Meng, Low-rank tensor function representa- tion for multi-dimensional data recovery, IEEE Trans. Pattern Anal. Mach. Intell. (2023)
2023
-
[37]
Y . Luo, X. Zhao, K. Ye, D. Meng, Neurtv: Total variation on the neural domain, arXiv preprint arXiv:2405.17241 (2024)
2024 arXiv
-
[38]
T. G. Kolda, B. W. Bader, Tensor decompositions and applications, SIAM review 51 (3) (2009) 455–500
2009
-
[39]
T. Liu, J. Yang, B. Li, Y . Wang, W. An, Infrared small target detection via noncon- vex tensor tucker decomposition with factor prior, IEEE Trans. Geosci. Remote Sens. 61 (2023) 1–17
2023
-
[40]
Sitzmann, J
V . Sitzmann, J. Martel, A. Bergman, D. Lindell, G. Wetzstein, Implicit neural rep- resentations with periodic activation functions, in: Proc. Adv. Neural Inf. Process. Syst. (NIPS), V ol. 33, 2020, pp. 7462–7473
2020
-
[41]
S. Ma, D. Goldfarb, L. Chen, Fixed point and bregman iterative methods for matrix rank minimization, Math. Program. 128 (1) (2011) 321–353
2011
-
[43]
Chang, An e ffective evaluation tool for hyperspectral target detection: 3d re- ceiver operating characteristic curve analysis, IEEE Trans
C.-I. Chang, An e ffective evaluation tool for hyperspectral target detection: 3d re- ceiver operating characteristic curve analysis, IEEE Trans. Geosci. Remote Sens. 59 (6) (2020) 5131–5153. 35
2020
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.