Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Selective retrieval of stored fusion-layer parameters makes source-free audio-visual continual test-time adaptation practical, cutting catastrophic forgetting to a few percent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:03 UTC pith:QUJCZDAL

load-bearing objection AV-CTTA is a solid, well-engineered contribution to an underexplored problem—the selective retrieval idea is genuinely useful—but the retrieval criterion itself is under-validated, and the paper needs variance estimates and a sharper evaluation protocol before the headline numbers are fully trustworthy. the 5 major comments →

arxiv 2602.18528 v4 pith:QUJCZDAL submitted 2026-02-20 cs.LG cs.SD

Audio-Visual Continual Test-Time Adaptation without Forgetting

Sarthak Kumar Maharana , Akshay Mehra , Bhavya Ramakrishna , Yunhui Guo , Guan-Ming Su This is my paper
classification cs.LG cs.SD
keywords audio-visual test-time adaptationcontinual test-time adaptationcatastrophic forgettingparameter retrievalmodality fusioncorruption robustnesssource-free adaptationKL divergence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Audio-visual models deployed in the wild face streams of shifting domains—snow, rain, crowd noise, and more—often affecting sound and vision together. The authors show that repeatedly adapting the model to each new domain makes it forget the original source distribution, sometimes performing worse than the frozen source model. Their method, AV-CTTA, avoids this by keeping a small buffer of previously adapted fusion-layer parameters, each tagged by the raw statistics of the inputs it was adapted to. At test time, a new batch is matched to the most statistically similar snapshot; those parameters are plugged in, refined with a single gradient step, and written back. On bimodal corruption benchmarks, the approach outperforms prior methods and limits the drop in source accuracy to about three percent, versus more than twenty for earlier continual adaptation.

Core claim

The paper's central discovery is that the attention fusion layer of an audio-visual transformer (the query, key, and value projection matrices of the joint encoder) exhibits strong intra- and cross-category transferability: adapting these weights on one corruption type—say Gaussian noise—improves accuracy on other corruption types that share low-level statistics, and often beats the source model on unrelated categories. This transferability suggests a remedy for catastrophic forgetting in continual test-time adaptation: instead of overwriting shared parameters as each new domain arrives, the model should store parameter snapshots together with modality-specific input statistics, and at each

What carries the argument

The central mechanism is a shared buffer of (statistics, parameters) pairs. Each element stores the channel-wise means and variances of the visual frames, the frequency-wise means and variances of the audio spectrogram, and the corresponding fusion-layer attention weights (WQ, WK, WV). At each time step, the current batch's statistics are modeled as diagonal Gaussian distributions, and the KL divergence between those and each stored element's distribution is computed in closed form (Eq. 16); the element with the smallest combined audio-visual KL distance below a threshold tau is retrieved. The retrieved parameters are re-inserted into the fusion layer, adapted with a confidence and negative-

Load-bearing premise

The method assumes that the raw first- and second-order statistics of the inputs—per-color-channel means and variances of frames and per-frequency-bin means and variances of spectrograms—are a reliable fingerprint of the current domain, so that a close statistical match selects the right fusion-layer parameters; if two distinct corruptions share similar low-order statistics, retrieval will pick the wrong starting point.

What would settle it

Compute the closed-form KL distance (Eq. 16) between the batch statistics of every pair of corruptions in Kinetics50-2C at severity 5. If any distinct pair has a combined distance below tau=0.01, run AV-CTTA while forcing retrieval of the partner's snapshot; a notable accuracy drop relative to using the correct snapshot would indicate the statistical proxy conflates distinct domains.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A deployed audio-visual model can adapt to an unpredictable sequence of corruptions without any source data, using only a buffer of parameter snapshots and input statistics.
  • Adapting just the fusion layer's attention weights is enough to improve on the current domain and to transfer to future domains, so no normalization-layer updates or source-sample rehearsal are needed.
  • The buffer can be small (e.g., 50 elements) without losing performance, because statistically redundant snapshots are merged.
  • The method is robust to task order, batch size, and unimodal vs. bimodal corruption, holding up when only one modality is corrupted or when both shift together.
  • On standard audio-visual benchmarks, mean accuracy rises above all compared continual adaptation baselines, and source-knowledge forgetting drops to roughly a tenth of what prior methods incur.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The success of low-order statistic matching suggests a general recipe: any architectural bottleneck that concentrates domain-specific knowledge (e.g., adapters, prompts) could be paired with a similar retrieval memory. A direct test would be to apply the same buffer scheme to other fusion designs where the statistics are computed at the feature level rather than the raw input.
  • The method's reliance on channel-wise and frequency-wise moments raises a falsifiable hypothesis: corruptions that share first and second moments but differ in higher-order statistics (e.g., texture vs. sharpening) should be conflated by the retrieval and hurt accuracy. An oracle-label experiment would quantify how much headroom remains.
  • The pair-merging strategy averages parameter snapshots, implicitly assuming the parameter space is locally convex. If parameter interpolations are harmful (as can happen with sharp minima), a distilling or functional-interpolation alternative might be needed for long streams.
  • Because the buffer never stores raw samples, the scheme fits privacy-constrained deployment; a natural extension is to combine retrieval with forward-only (no-backprop) adaptation for very low-power settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies source-free audio-visual continual test-time adaptation (CTTA), where a pre-trained CAV-MAE model encounters a sequence of corrupted audio-visual tasks and must adapt online without source data or task boundaries. The authors first show that adapting only the fusion-layer attention projections {W_Q, W_K, W_V} on one corruption transfers to other corruptions within and across categories. They then propose AV-CTTA, which maintains a shared buffer of fusion-layer parameter snapshots indexed by per-modality raw input statistics (means and diagonal variances). At each batch, the method computes KL divergences between current raw statistics and stored statistics, retrieves the closest snapshot if the divergence is below a threshold τ, performs an EMA update, and continues adaptation. If no snapshot is close, a new element is added; merging keeps the buffer within a budget η. Experiments on Kinetics50-C/VGGSound-C (unimodal corruptions) and Kinetics50-2C/VGGSound-2C (bimodal corruptions) report state-of-the-art mean accuracy and substantially reduced forgetting relative to READ (e.g., 2.9% vs. 27.9% source accuracy drop on VGGSound-2C).

Significance. If the claims hold after revision, this is a useful contribution to the underexplored area of audio-visual CTTA. The paper has concrete strengths: broad benchmark coverage including bimodal corruptions, a clear catastrophic-forgetting measurement, ablations over threshold, buffer size, batch size, distance metric, and task order, and a memory-bounded buffer that avoids storing raw data. The transferability observation for fusion-layer parameters is itself interesting and worth reporting. However, the central mechanism—selective retrieval based on raw-statistic KL divergence—is not yet causally isolated from the EMA updates, buffer merging, and resets, and several empirical ambiguities (loss sign, transductive evaluation, lack of error bars) currently make the SOTA and forgetting claims provisional.

major comments (5)
  1. [§4.2, Eq. (16); Table 3; Fig. 7] The load-bearing claim is that selective retrieval of stored W_Q/W_K/W_V snapshots, selected by raw-statistic KL divergence, drives the accuracy and forgetting gains. This is not isolated. AV-CTTA differs from READ by retrieval, EMA updates (Eqs. 6–8), buffer expansion, and merging, so a comparison to READ alone cannot attribute the gains to the retrieval criterion. Please add ablations: (a) random buffer selection, (b) oracle corruption labels as the retrieval key, and (c) always continuing from current parameters with EMA/merging unchanged. Also report retrieval accuracy (fraction of times n* matches the true corruption category) or a 15×15 retrieval confusion matrix. Table 3 only compares KL to a norm distance on the same raw statistics, so it does not validate that the proxy identifies transferable snapshots.
  2. [§5.1, Tables 1–2, Figs. 5 and 7] No error bars, standard deviations, or number of runs are reported. Several headline margins are small: Kinetics50-2C 50.29 vs. READ 48.81, VGGSound-C audio 30.71 vs. PTA 29.26, and VGGSound-C visual 56.20 vs. SuMi 55.28. Since test-batch order and optimizer noise can move these numbers, the SOTA claim needs repeated runs (or batch-order permutations) with means and variances. The forgetting reductions in Fig. 7 should also be reported with the same uncertainty information.
  3. [§4.4, Eq. (11); Algorithm 1, line 23] The adaptation loss is internally inconsistent. §4.4 defines L_ne = Σ p log p as a negative entropy loss and states L = L_conf + L_ne, while Algorithm 1 uses L = L_conf − L_ne. These are opposite objectives with respect to prediction entropy. Since this loss is the only update signal for W_Q/W_K/W_V, every experimental number depends on which sign is correct. Please state the exact loss used in the experiments, make the main text and pseudocode agree, and clarify the relationship to READ's original loss.
  4. [Section 3; Algorithm 1, lines 23–25] The evaluation protocol appears transductive. The problem statement says each batch is available once for a single forward pass, but Algorithm 1 computes a gradient step on x_t (line 24) and then returns y_t = f_θ(x_t) in line 25, i.e., current-batch accuracy is measured with parameters already updated on that same batch. This can inflate accuracy relative to a protocol where predictions are made before adaptation. Please report both pre- and post-adaptation predictions, clarify which protocol the reported numbers use, and confirm that baselines are evaluated identically.
  5. [§5.1, Fig. 5 vs. Table 5; Appendix D] The method depends on per-dataset hyperparameters τ and η, with no validation mechanism in a source-free online setting. τ is set to 0.005 for unimodal and 0.01 for bimodal corruptions, and the reported η varies across tables (e.g., Fig. 5 uses η=∞, but Table 5 shows VGGSound-2C improves from 43.51 at η=∞ to 43.90 at η=200, while VGGSound-C audio improves monotonically from 22.52 at η=50 to 30.71 at η=∞). Appendix D acknowledges that τ tuning is costly in deployment. Please provide a principled selection rule or evidence that a single fixed hyperparameter set works comparably across all datasets.
minor comments (6)
  1. [Abstract] The abstract names the proposed method AVReCAP, while the title, introduction, and body use AV-CTTA. Reconcile the naming in the final version.
  2. [Eqs. (3)–(4)] The audio statistic equations omit the frequency dimension: x_t_a is B×T×F, but μ_t_a and Σ_t_a are defined by summing over b and e without a frequency index, making it unclear how R^F is obtained. Spell out the summation over the frequency axis and the resulting dimensions.
  3. [Algorithm 1, lines 6–9] In the K=∅ branch, “continue” appears to skip the adaptation step and prediction for the first time step, whereas the main text says the first buffer element is stored after adapting at t=1. Clarify the intended flow so implementation matches the text.
  4. [§5.1, Fig. 5a] The text says AV-CTTA obtains a 1.25% improvement over READ on Kinetics50-2C, but Fig. 5a shows 50.29 vs. 48.81, a difference of 1.48 percentage points. Check and correct.
  5. [Appendix D] Typo: “scalibility” should be “scalability.”
  6. [Title and Fig. 7] The title claims adaptation “without forgetting,” but Fig. 7 reports a 2.9% source-accuracy drop for the proposed method. The claim is technically defined in the paper, but “minimizing forgetting” or “reduced forgetting” would be a more accurate title-level statement.

Circularity Check

0 steps flagged

No significant circularity: AV-CTTA's gains are empirically benchmarked; only minor non-load-bearing self-citations appear.

full rationale

The paper's central claim is that selective retrieval of stored fusion-layer (WQ/WK/WV) snapshots based on raw-statistic KL divergence (Eq. 16) improves continual test-time accuracy and mitigates catastrophic forgetting. This claim is supported by external benchmark experiments, not derived from its inputs. Eq. 16 is a standard closed-form KL divergence between diagonal Gaussians; it is not fitted to the reported accuracy numbers, and the reported accuracies are not defined in terms of the retrieval criterion. The buffer is populated online from test batches, and the evaluation compares against CAV-MAE source checkpoints and corruption benchmarks (Kinetics50-C/VGGSound-C, Kinetics50-2C/VGGSound-2C). No equation in Sections 4.1-4.4 reduces a predicted quantity to a fitted constant, and no uniqueness theorem or ansatz is imported from the authors' prior work to make the method forced. The self-citations to Maharana et al. 2025a (AVRobustBench, the READ modality-imbalance observation) and Maharana et al. 2025c (Appendix D, hyperparameter tuning cost) are motivational or benchmark-defining and are not load-bearing for the empirical results. Appendix D's acknowledged limitations (O(N) retrieval latency, dependence on threshold tau, architectural assumptions) are correctness/generality concerns, not circularity. The absence of an oracle-domain or random-retrieval ablation weakens causal attribution of the gains to the retrieval mechanism specifically, but a missing ablation is not a self-referential derivation. Therefore no circular step is exhibited; the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The method is an empirical pipeline, so the ledger contains no invented physical entities. The free quantities are hyperparameters: retrieval threshold τ, buffer budget η, and EMA factor β; τ and η are explicitly tuned per dataset/setting, and the ablation shows performance is sensitive to τ. The axioms are domain assumptions about how raw statistics relate to domain shift and how fusion parameters transfer; none are proven formally. The evaluation is benchmark-based.

free parameters (3)
  • Retrieval threshold τ = 0.005 (unimodal), 0.01 (bimodal)
    Controls when to retrieve a buffer element vs create a new one; chosen separately for unimodal/bimodal settings and ablated in Fig. 6. Performance degrades with larger τ, so main results depend on this tuning.
  • Buffer size η = 50/100/200/300/400/∞ per dataset
    Memory budget at which elements are merged; results vary nontrivially with η (e.g., VGGSound-C audio mean accuracy 22.52 at η=50 vs 30.71 at η=∞), so it is a free design parameter.
  • EMA smoothing factor β = 0.99
    Used in Eqns (6)-(8) to merge retrieved parameters/statistics; hand-set and not ablated, though standard. Included for completeness.
axioms (5)
  • domain assumption Label spaces and label distributions are constant across source and target tasks: p(y^S)=p(y^t).
    Sec 3 problem setting; required for using the classifier head without task-specific adaptation.
  • domain assumption Raw modality means and diagonal variances are sufficient to represent distribution shift for retrieval.
    Sec 4.2 Eqns (1)-(8); if false, retrieval selects mismatched parameters. Not validated against oracle domain labels.
  • domain assumption Adapted fusion attention parameters transfer within and across corruption categories, so reusing past snapshots helps rather than hurts.
    Sec 4.1 Figure 3; empirical premise of the buffer-based method.
  • domain assumption The target architecture is a late-fusion model (unimodal encoders plus a joint encoder with WQ/WK/WV), e.g., CAV-MAE.
    Sec 3 and Limitations D; applicability to other fusion designs is not guaranteed.
  • standard math Diagonal-covariance Gaussian KL divergence is a valid distance metric for this retrieval problem.
    Appendix A.1, Eqns (15)-(16); standard closed-form identities, but the diagonal independence assumption is a modeling choice.

pith-pipeline@v1.3.0-alltime-deepseek · 28202 in / 15897 out tokens · 177276 ms · 2026-08-02T22:03:15.739987+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Audio-Visual Continual Test-Time Adaptation without Forgetting." pith.science (2026). https://pith.science/paper/QUJCZDAL

@misc{pith2026260218528,
  author       = {Pith},
  title        = {Pith review of: Audio-Visual Continual Test-Time Adaptation without Forgetting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUJCZDAL}},
  note         = {Machine review of arXiv:2602.18528}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Audio-visual continual test-time adaptation involves continually adapting a source audio-visual model at test-time, to unlabeled non-stationary domains, where either or both modalities can be distributionally shifted, which hampers online cross-modal learning and eventually leads to poor accuracy. While previous works have tackled this problem, we find that SOTA methods suffer from catastrophic forgetting where the model's performance drops well below even the source model due to continual parameter updates at test-time. In this work, we first show that adapting only the modality fusion layer to a target domain not only improves performance on that domain but can also enhance performance on subsequent domains. Based on this strong cross-task transferability of the fusion layer's parameters, we propose a method, $\texttt{AVReCAP}$, that improves test-time performance of the models without access to any source data. Our approach works by using a selective parameter retrieval mechanism that dynamically retrieves the best fusion layer parameters from a buffer using only a small batch of test data. These parameters are then integrated into the model, adapted to the current test distribution, and saved back for future use. Extensive experiments on benchmark datasets involving unimodal and bimodal corruptions show our proposed $\texttt{AVReCAP}$ significantly outperforms existing methods while minimizing catastrophic forgetting.

Figures

Figures reproduced from arXiv: 2602.18528 by Akshay Mehra, Bhavya Ramakrishna, Guan-Ming Su, Sarthak Kumar Maharana, Yunhui Guo.

Figure 1
Figure 1. Figure 1: We illustrate audio-visual continual test-time adap￾tation using an example of a deployed agent with audio-visual sensors for scene understanding. Starting from a source model parameterized by θ S , the agent encounters a sequence of evolv￾ing target environments where distributional shifts may affect the audio modality, the visual modality, or both, motivating continual adaptation at test-time. The goal i… view at source ↗
Figure 2
Figure 2. Figure 2: ). In Section 4.2, we leverage this core insight and propose a selective parameter retrieval scheme to obtain the most relevant parameter state for adaptation. We intro￾duce a shared buffer to store “snapshots” of the attention Gaussian Impulse Shot Speckle Compression Snow Frost SpatterWindRain Underwater Concert Smoke Crowd Interference 0 20 40 Accuracy Results on VGGSound-2C SOURCE EATA TENT SAR READ Su… view at source ↗
Figure 3
Figure 3. Figure 3: Attention fusion layer adapted on a single domain successfully transfers, achieving performance exceeding or comparable to the source model, motivating us to store parameter snapshots in a buffer that can be reused during audio-visual CTTA. We adapt the projection matrices {WQ, WK, WV } of the joint encoder fj of a pre-trained CAV-MAE (SOURCE) (Gong et al., 2022c) on the first unseen domain of each corrupt… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of AV-CTTA. At time-step t, audio-visual inputs via µ t a, Σ t a, µt v, Σ t v are modeled as Gaussian distributions. The selection retrieval stage uses KL divergence g(·) to compare current statistics against all elements in the shared buffer K (M is the current set of indices). For the best match within threshold τ , stored parameters (WQ, WK, WV ) are retrieved for adaptation at time-step t.… view at source ↗
Figure 5
Figure 5. Figure 5: AV-CTTA achieves SOTA results on Kinetics50-2C (left) and VGGSound-2C (right). We report mean accuracy (%) at a severity level of 5 in the continual setting. Here, buffer size η = ∞. recommended hyperparameters. We expand on the details in Appendix B.3, B.4. All experiments are conducted on an NVIDIA RTX A5000 GPU. 5.1 Main Results Results on the unimodal corruption setting. In Tables 1 and 2, we present r… view at source ↗
Figure 6
Figure 6. Figure 6: Ablation on τ . Results on Kinetics50 (left) under audio, visual, and bimodal corruptions, and on VGGSound (right) under visual, audio, and bimodal corruptions. Catastrophic forgetting. We evaluate the final adapted parameters back on the source test set. We continually adapt on Kinetics50-2C and VGGSound-2C, and illustrate the results in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: AV-CTTA maintains stable performance under small batch sizes. The x-axis shows batch size, and the y-axis reports mean accuracy for AV-CTTA and baselines. The effect of batch size can be critical in our setting. In Eqns. (1), (2), (3), and (4), the mean and covariance are computed along the batch axis, where a smaller batch size can po￾tentially introduce noise into these statistics. We demon￾strate result… view at source ↗
Figure 8
Figure 8. Figure 8: Effect of buffer size η. Mean accuracy (%) on VGGSound-C (left) and VGGSound-2C (right) [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: AV-CTTA is robust to variations in task ordering. The top row reports results on Kinetics50-2C, while the bottom row shows results on VGGSound-2C. In each plot, the x-axis denotes the task order in the continual setting, and the y-axis reports task-wise accuracy. The legend summarizes the mean accuracy achieved by each baseline and by our method. C.4 Sensitivity to Task Orders In real-world continual lear… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

    cs.CV 2026-07 accept novelty 5.0

    CTTA methods fall into optimization-based, parameter-efficient, and architecture-based families that adapt pretrained vision models online under continual unlabeled shifts while fighting forgetting and error accumulation.

  2. Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

    cs.CV 2026-07 accept novelty 5.0

    A hierarchical taxonomy and comparative review of continual test-time adaptation methods that adapt vision models online to non-stationary unlabeled streams while fighting forgetting and error accumulation.

Reference graph

Works this paper leans on

23 extracted references · 14 linked inside Pith · cited by 1 Pith paper

  1. [5]

    Botta: Bench- marking on-device test time adaptation.arXiv preprint arXiv:2504.10149,

    Danilowski, M., Chatterjee, S., and Ghosh, A. Botta: Bench- marking on-device test time adaptation.arXiv preprint arXiv:2504.10149,

    arXiv

  2. [7]

    and Jin, T

    Guo, Z. and Jin, T. Smoothing the shift: Towards stable test-time adaptation under complex multimodal noises. arXiv preprint arXiv:2503.02616,

    Pith/arXiv arXiv

  3. [8]

    and Dietterich, T

    Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions.arXiv preprint arXiv:1903.12261,

    Pith/arXiv arXiv 1903

  4. [9]

    Kim, J., Lee, H., Rho, K., Kim, J., and Chung, J. S. Equiav: Leveraging equivariance for audio-visual con- trastive learning.arXiv preprint arXiv:2403.09502,

    Pith/arXiv arXiv

  5. [10]

    and Feng, S

    Li, J. and Feng, S. Bridging modalities via progressive re-alignment for multimodal test-time adaptation.arXiv preprint arXiv:2511.22862,

    arXiv

  6. [11]

    Polyvit: Co- training vision transformers on images, videos and audio

    Likhosherstov, V ., Arnab, A., Choromanski, K., Lucic, M., Tay, Y ., Weller, A., and Dehghani, M. Polyvit: Co- training vision transformers on images, videos and audio. arXiv preprint arXiv:2111.12993,

    Pith/arXiv arXiv

  7. [13]

    Liu, Z., Xu, Y ., Xu, Y ., Qian, Q., Li, H., Jin, R., Ji, X., and Chan, A. B. An empirical study on distribution shift robustness from the perspective of pre-training and data augmentation.arXiv preprint arXiv:2205.12753,

    Pith/arXiv arXiv

  8. [14]

    Test- time model adaptation with only forward passes.arXiv preprint arXiv:2404.01650,

    Niu, S., Miao, C., Chen, G., Wu, P., and Zhao, P. Test- time model adaptation with only forward passes.arXiv preprint arXiv:2404.01650,

    Pith/arXiv arXiv

  9. [17]

    C., and Xing, E

    Wang, H., He, Z., Lipton, Z. C., and Xing, E. P. Learning robust representations by projecting superficial statistics out.arXiv preprint arXiv:1903.06256,

    Pith/arXiv arXiv 1903

  10. [18]

    Memo: Test time robustness via adaptation and augmentation.Advances in neural information processing systems, 35:38629–38642, 2022a

    Zhang, M., Levine, S., and Finn, C. Memo: Test time robustness via adaptation and augmentation.Advances in neural information processing systems, 35:38629–38642, 2022a. Zhang, X., Zhou, L., Xu, R., Cui, P., Shen, Z., and Liu, H. Towards unsupervised domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    Pith/arXiv arXiv

  11. [19]

    (σt u,i)2 (σn u,i)2 + (µt u,i −µ n u,i)2 (σn u,i)2 −1 + ln σn u,i σt u,i # = rX i=1

    12 AV-CTTA Appendix A Additional Details ofAV-CTTA A.1 KL Divergence Between Modality-Specific Gaussian Distributions In Section 4.2, we utilize the KL divergence as a distance metric between the current modality-specific statistics and the stored buffer elements. We define the distributions at time-step t and for the nth buffer element as P t u =N(µ t u,...

    2017

  12. [20]

    B.2 Source Model Architecture Following all prior A V-TTA methods, we use CA V-MAE (Gong et al., 2022c) as the default audio-visual recognition model

    are visual-dominant and audio-dominant, respectively.This means that task-specific information is in the visual and audio modalities, respectively. B.2 Source Model Architecture Following all prior A V-TTA methods, we use CA V-MAE (Gong et al., 2022c) as the default audio-visual recognition model. CA V-MAE consists of 11-layer transformer encoders for bot...

    2024

  13. [21]

    We use pre-trained weights of Kinetics50 and VGGSound, as released by (Yang et al., 2024)

    and adopt CA V-MAE (Gong et al., 2022c) as the source model, which is used for inference. We use pre-trained weights of Kinetics50 and VGGSound, as released by (Yang et al., 2024). TENT2 (Wang et al., 2020): We adapt all the affine parameters of all LayerNorms and minimize the entropy of model predictions with a learning rate of1×10 −4 and an Adam optimiz...

    2024

  14. [23]

    On average, AV-CTTA demonstrates strong robustness to task order variations compared to baselines, particularly in the difficult bimodal setting. Although a performance drop is observed during the initial tasks, continual adaptation allows the model to stabilize over time and converge to comparable accuracy across different task orderings. Overall, the se...

    2023

  15. [400]

    In Kinetics50-C, the video is corrupted, whereas in VGGSound, the audio is corrupted

    39.52 28.43 24.32 29.37 35.65 26.38 30.61 AV-CTTA(η=∞) 39.52 28.43 24.31 29.44 35.52 27.05 30.71 C Additional Results C.1 Results on a fixed buffer budgetη In Tables 4 and 6, we report additional results on Kinetics50-C and VGGSound, respectively. In Kinetics50-C, the video is corrupted, whereas in VGGSound, the audio is corrupted. As a reminder, the task...

    2024

  16. [2017]

    Efficient lifelong learning with a-gem.arXiv preprint arXiv:1812.00420,

    Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem.arXiv preprint arXiv:1812.00420,

    Pith/arXiv arXiv

  17. [2018]

    K., Torr, P

    Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M. On tiny episodic memories in continual learning.arXiv preprint arXiv:1902.10486,

    Pith/arXiv arXiv 1902

  18. [2020]

    Tent: Fully test-time adaptation by entropy minimiza- tion.arXiv preprint arXiv:2006.10726,

    Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion.arXiv preprint arXiv:2006.10726,

    Pith/arXiv arXiv 2006

  19. [2021]

    Vida: Homeostatic visual domain adapter for continual test time adaptation.arXiv preprint arXiv:2306.04344,

    Liu, J., Yang, S., Jia, P., Zhang, R., Lu, M., Guo, Y ., Xue, W., and Zhang, S. Vida: Homeostatic visual domain adapter for continual test time adaptation.arXiv preprint arXiv:2306.04344,

    Pith/arXiv arXiv

  20. [2022]

    Vg- gsound: A large-scale audio-visual dataset

    Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vg- gsound: A large-scale audio-visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721–725. IEEE,

    2020

  21. [2023]

    Note: Robust continual test-time adaptation against temporal correlation.Advances in Neural Information Processing Systems, 35:27253–27266, 2022a

    Gong, T., Jeong, J., Kim, T., Kim, Y ., Shin, J., and Lee, S.-J. Note: Robust continual test-time adaptation against temporal correlation.Advances in Neural Information Processing Systems, 35:27253–27266, 2022a. Gong, Y ., Liu, A. H., Rouditchenko, A., and Glass, J. Uavm: Towards unifying audio and visual models.IEEE Signal Processing Letters, 29:2437–244...

    Pith/arXiv arXiv

  22. [2024]

    Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001–2010,

    2001

  23. [2025]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450,

    Pith/arXiv arXiv