{"id":"b960c195-93f8-481c-999e-c00f68b3f968","arxiv_id":"2412.15813","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SONO combines second-order neural ODE feature refinement with text-initialized classifiers and text-as-image augmentation, reporting few-shot accuracy gains over existing CLIP adaptation methods.","lead":"This paper introduces SONO, a few-shot image classifier that runs CLIP features through a second-order neural ODE and adds text-derived features as extra training examples. It reports consistent accuracy gains over existing CLIP fine-tuning methods on 11 datasets and on ImageNet distribution-shift benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No experiment isolates the second-order NODE component: the first-order NODE-Adapter baseline is cited but never benchmarked, so the central claim that second-order dynamics drive the gains is unsupported.","rationale":"The reader's weakest_assumption identifies the same load-bearing gap: no first-order NODE baseline and no ablation replacing the NODE module with a first-order counterpart. This is the single most important issue because the paper's contribution is specifically second-order NODEs; if a first-order NODE performs equally, the method reduces to a residual adapter with text augmentation, and the claimed expressive-power advantage is not demonstrated. The paper's own evidence actually makes this concern concrete: Table 2 shows removing SNM entirely costs 3.11%, but the relevant comparison is to a first-order NODE, not to no NODE. The NODE-Adapter paper by the same authors provides a ready-made baseline, yet it is absent from all experiments. The eta tuning on the 16-shot ImageNet test set is a secondary but real methodological flaw that can inflate the reported margin; I would not elevate it to the primary attack because the core scientific claim fails even before that issue. The empirical benchmark still supports SONO as a working method, so REJECT is too strong; the paper merits conditional acceptance with a required first-order baseline and proper validation-based hyperparameter selection. No ad hominem is intended; the critique is about experimental design and the logical link between the method's novelty and its evidence.","tokens_in":12775,"tokens_out":3726,"duration_ms":35143,"concrete_test":"Re-run the 16-shot benchmark on all 11 datasets plus ImageNet-A with SONO's SNM replaced by a first-order NODE of the same parameter count and architecture (e.g., the authors' NODE-Adapter), keeping TAI and the text-initialized classifier identical, with at least 5 seeds and reported mean/std. If the first-order variant matches or beats SONO, the claimed second-order advantage is not supported. Additionally, select eta on a held-out validation split of ImageNet rather than the test set and report whether the chosen value changes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's novelty and central claim rest on Second-Order NODEs (Eq. 2) being more expressive and less prone to overfitting. Yet the ablations in Table 2 remove the NODE module entirely (16-shot accuracy drops from 66.37 to 63.26) and never replace it with a first-order NODE. The authors' own first-order method NODE-Adapter (Zhang et al. 2024b) is cited in Related Work but never benchmarked, and no first-order counterpart of S_theta is tested. Consequently, the observed gains over Tip-Adapter-F and GraphAdapter could come from the residual adapter, the text-initialized classifier, or the TAI augmentation rather than from second-order dynamics. The theoretical appeal to universal approximation is not empirically connected to the results. A second, compounding issue is that Table 4 selects the residual ratio eta on the 16-shot ImageNet evaluation set, choosing eta=0.6 by test accuracy; this introduces selection-on-the-benchmark optimism. No error bars or multiple-seed variance are reported, so the 1-3% differences in Figure 3 and Table 1 may be within run-to-run noise. Together these gaps leave the central claim 'consistently and substantially outperforms' without a clean causal test of its key component.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SONO, a few-shot learning method for CLIP that refines visual features with a second-order neural ODE, initializes the classifier from text embeddings, and augments training features with text prompts. The authors evaluate on 11 recognition datasets at 1-16 shots and on four ImageNet distribution-shift benchmarks, reporting that SONO outperforms prompt-based and adapter-based baselines while using modest parameters and training time.","tokens_in":13032,"tokens_out":6315,"duration_ms":50241,"significance":"If the reported gains are real and reproducible, the method is noteworthy for its simplicity and efficiency: it trains in minutes on a single GPU, adds only 1.54M parameters, and the text-as-image augmentation idea is a cheap way to exploit CLIP's shared embedding space. However, the central attribution of the gains to the second-order ODE formulation is not supported by the current experiments, and the paper's theoretical motivation for second-order versus first-order dynamics is not rigorous. The reported margins over strong baselines are small (0.6-3%), making the absence of multiple seeds a material issue.","major_comments":[{"comment":"The ablation labeled SNM '✗' removes the entire ODE module rather than replacing it with a first-order NODE. Since the paper's core claim is that second-order dynamics provide the expressiveness and regularization benefits, the reader cannot tell whether the 3.11% drop is due to the second-order formulation or simply to any residual feature transformation. The first-order predecessor NODE-Adapter (Zhang et al. 2024b) is cited but never benchmarked. Please add a first-order NODE baseline under the identical experimental setup, including the same adapter architecture and text-initialized classifier, to isolate the contribution of the second-order dynamics.","section":"Ablation Studies, Table 2"},{"comment":"The method section never defines how η enters the feature update; Eq. (2) and Eq. (5) contain no η, so the residual combination described in the text is missing from the formulation. Moreover, Table 4 selects η on the same 16-shot ImageNet configuration used for the headline results and for the domain-generalization source model, which risks selection-on-the-evaluation-set optimism. Please state the exact residual formula (e.g., f_final = η S_θ(f) + (1-η) f) and report η selected on a held-out validation split or show that the reported results are stable across a range of η and multiple seeds.","section":"Residual Ratio η, Table 4"},{"comment":"No error bars or multiple-seed results are reported anywhere. In few-shot protocols the specific randomly chosen training examples can change accuracy by more than a percentage point, so the 0.6-3% margins claimed as 'consistently and substantially outperform' may not be statistically meaningful. Please provide mean and standard deviation over at least three seeds for the main comparisons, or explicitly report the seed set used and demonstrate that performance is stable.","section":"Experimental Results, Figure 3 and Table 1"},{"comment":"The claim of consistent improvement is contradicted by Table 1 on the ResNet-50 backbone: on ImageNet-R, SONO achieves 60.75% whereas GraphAdapter achieves 60.86%, a 0.11% deficit. Also, the ViT-B/16 source ImageNet accuracy differs between Table 1 (74.92%) and Table 3 (74.86%) for the same configuration. Please reconcile the numbers and qualify the claim to acknowledge individual dataset exceptions.","section":"Domain Generalization, Table 1"},{"comment":"The motivation states that first-order NODEs cannot approximate a broad class of functions because their flows do not intersect, whereas second-order NODEs are universal approximators. This is not a precise contrast: any second-order ODE system can be rewritten as a first-order system with a doubled state dimension, so the expressiveness argument relies on state augmentation rather than order. The paper should either provide a precise theoretical statement (e.g., the parameter-efficiency results of Massaroli et al. 2020, already cited) or explicitly treat the second-order choice as an empirical design decision validated by the requested first-order baseline.","section":"Introduction and Related Work, motivation for second-order NODEs"}],"minor_comments":[{"comment":"The acronym for text-as-image augmentation is written 'TAI' in the text and 'TIA' in the table header; please standardize.","section":"Ablation Studies, Table 2"},{"comment":"The loss in Eq. (4) uses the symbol 'n' in the summation without defining it; it should be the total number of augmented training features N+L for the class in question.","section":"Method, Eq. (4)"},{"comment":"In the text the baseline is sometimes called 'Tip-Adapter' while the tables list 'Tip-Adapter-F'; the distinction should be consistent.","section":"Experimental Results, performance comparison"},{"comment":"The optimization is attributed to AdamW with a citation to Kingma and Ba (2015), which is the Adam paper; please add the correct AdamW reference (Loshchilov and Hutter, 2019) if AdamW is intended.","section":"Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are promising, but the central claim is under-tested because the key ablation does not include a first-order NODE baseline and the hyperparameter selection protocol is not clean. Given the small reported margins, the lack of seed variance is also a concern. I would ask for the first-order baseline, proper η selection, and multi-seed results before accepting; these are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv:2412.15813. The useful thing here is the combination: a second-order NODE feature adapter on frozen CLIP, a classifier initialized from text embeddings, and text-as-image augmentation. That specific package appears to be new, and the efficiency numbers are the real story—66.37% on 16-shot ImageNet in 3.5 minutes with 1.54M parameters is a practical result, and the ablations show each component pulls its weight. The paper deserves credit for reporting training time and parameter counts, which most few-shot CLIP papers bury.\n\nThe soft spots are mostly about what the experiments do not isolate. The central claim is that second-order dynamics are what make the method work, but no experiment compares against a first-order NODE. The authors' own NODE-Adapter is cited but never benchmarked, and the ablation removes the NODE module entirely instead of swapping in a first-order version. So the expressive-power story is plausible but untested. The residual ratio eta is tuned on the same 16-shot ImageNet setting used for headline results, which adds selection-on-the-benchmark optimism. And there are no error bars or multiple seeds, so the 1–3% margins in Figure 3 and Table 1 could be within run-to-run noise. These are not fatal—the method could still be a useful practical recipe—but they do mean \"consistently and substantially outperforms\" is stronger than the evidence supports.\n\nI would push back gently on one part of the stress-test note: the absence of a first-order baseline is the real hole, and the missing seeds are a genuine concern. The eta tuning issue is real but minor, since eta=0.6 is not a knife-edge choice in Table 4 (accuracy is 65.9–66.4 across 0.2–1.0), so the benchmark-selection effect is likely small. The paper is not circular; it is just an empirical combination without a causal test of its headline component.\n\nWho is this for? People working on parameter-efficient CLIP adaptation will want to know about it, and the efficiency table is worth taking seriously. I would send it to review—the method is on a well-trodden problem with a clean empirical setup—but I would ask for the first-order NODE baseline, seeds/error bars, and code before accepting anything. A serious referee can get the paper to a solid state; as submitted, it is a conditional accept at best.","headline":"A competent incremental combination of second-order NODEs, text-initialized classifiers, and text-as-image augmentation for CLIP few-shot adaptation, with solid but incompletely isolated empirical gains.","tokens_in":13587,"tokens_out":1982,"would_cite":true,"duration_ms":17856,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SONO routes CLIP features through a second-order neural ODE and reports consistent few-shot and out-of-distribution gains over prompt- and adapter-based fine-tuning.","keywords":["few-shot learning","cross-modal learning","second-order neural ODEs","CLIP fine-tuning","domain generalization","text-based augmentation","vision-language models","feature refinement"],"falsifier":"Train the same pipeline with a first-order ODE in place of the second-order one and compare on all 11 datasets; if the first-order version matches SONO within noise, the central second-order claim is falsified.","tokens_in":12588,"feed_emoji":"📈","tokens_out":8531,"duration_ms":67253,"temperature":0.7,"pith_summary":"SONO is a method for fine-tuning CLIP on few-shot image classification: it pushes each visual feature through a second-order neural ordinary differential equation and then through a linear classifier whose weights start as text embeddings. The paper's central claim is that this combination consistently beats existing prompt- and adapter-based CLIP fine-tuning methods, reaching 77.86% average accuracy over 11 datasets at 16 shots and improving out-of-distribution accuracy on ImageNet variants. The motivation is that second-order ODE flows can approximate a wider class of feature transformations than first-order flows, which the authors argue gives smoother, less overfit features when training examples are scarce. The method also augments the N-shot training set with text-prompt features and keeps CLIP frozen, so it adds only 1.54M trainable parameters and trains in minutes.","feed_headline":"Second-order ODE module beats CLIP fine-tuning at few shots","feed_subtitle":"A 1.54M-parameter refiner lifts 16-shot accuracy to 77.86% average and ImageNet-A by 3.95%.","key_machinery":"The central object is the Second-Order NODE model $S^{(a)}_{\\theta}$, defined by the initial-value problem $x''(t) = S^{(a)}_{\\theta}(x(t), x'(t), t)$, $x(t_0) = x_0$, $x'(t_0) = g_{\\theta}(x(t_0))$. Because standard ODE solvers handle only first-order systems, the paper rewrites the state as $z = [x, x']^{\\mathsf{T}}$ and solves $z'(t) = [x', S^{(a)}_{\\theta}]^{\\mathsf{T}}$ with the fourth-order Runge-Kutta solver, using the first-order adjoint method for backpropagation. This object carries the argument by refining visual features continuously rather than through discrete adapter layers; the paper leans on the universal-approximation property of second-order NODEs to argue that the flow can represent a broader class of transformations. Its two complements are the cross-modal classifier, initialized with text embeddings so the text encoder is not rerun each iteration, and the text-as-image augmentation that adds L class-relevant prompt features to the N-shot training set.","core_discovery":"On the paper's own terms, the discovery is that a Second-Order Neural ODE module, slotted between CLIP's frozen image encoder and a text-initialized classifier, is a better few-shot adapter than the residual adapters, cache models, and prompt learners it is compared with. The feature update is modeled as $x''(t) = S^{(a)}_{\\theta}(x(t), x'(t), t)$ with initial conditions set by the image feature and a learned velocity term; the system is solved with RK4 in the forward pass and trained with the first-order adjoint method. Text-as-image augmentation selects the L prompt features most similar to each class prototype, and the classifier is initialized from text embeddings so the text encoder runs once. The result is 66.37% on 16-shot ImageNet, 77.86% average across 11 datasets, and OOD average gains of 1.58–1.80 points over GraphAdapter, with a 3.95-point gain on ImageNet-A.","pith_inferences":["The paper never benchmarks its own first-order NODE variant, so the second-order dynamics are not isolated; a first-order ODE with the same residual design might account for much of the gain.","The residual ratio $\\eta$ is tuned on the same 16-shot ImageNet configuration used for headline results; fixing $\\eta$ on a validation split before touching the test sets would tell whether the reported advantage is partly a selection effect.","Text-as-image augmentation depends on CLIP's own text-image similarity to pick prompts, which risks circularity on datasets whose class names already determine the easy features; its usefulness should be tested on domains where CLIP's zero-shot alignment is weak.","If the universal-approximation argument is what carries the method, a natural test is to vary solver accuracy and integration horizon: the gain should shrink as the solver is coarsened if expressivity is the active ingredient."],"forward_implications":["If the reported averages hold, SONO at 16 shots (77.86%) becomes the best of the compared CLIP fine-tuning methods by 1.63 points over GraphAdapter and 2.75 points over Tip-Adapter-F.","The ablation attributes 3.11 points of ImageNet accuracy to the ODE module and 0.71 points to text-as-image augmentation, so the ODE refinement is the main contributor.","At 1.54M parameters and 3.5 minutes on one RTX 3090, the method is much cheaper than CoOp's 15-hour training and uses about a tenth of Tip-Adapter-F's parameters.","Domain generalization improves across four ImageNet variants, with the OOD average rising 1.58 points on ResNet-50 and 1.80 points on ViT-B/16 over GraphAdapter, and the largest single gain being 3.95 points on ImageNet-A.","Consistent gains over Tip-Adapter-F across ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16 suggest the benefit is not tied to one visual backbone."],"supporting_citations":[{"why":"Provides CLIP, the frozen image/text encoders and shared embedding space that SONO adapts; its prompt templates are also part of the augmentation codebook.","marker":"Radford et al. 2021"},{"why":"CoOp is a primary prompt-learning baseline, and its protocol and data preprocessing are adopted; SONO's text-initialized classifier is described as its efficiency counterpart.","marker":"Zhou et al. 2022b"},{"why":"Tip-Adapter-F is the cache-based adapter baseline; SONO compares against it across shots and backbones and reports beating it by 2.75% at 16 shots.","marker":"Zhang et al. 2022"},{"why":"GraphAdapter is the strongest compared method; SONO's few-shot and OOD claims are measured against it as the second-best baseline.","marker":"Li et al. 2024"},{"why":"Supplies the second-order NODE formulation and the first-order adjoint efficiency argument that SONO relies on.","marker":"Norcliffe et al. 2020"},{"why":"Supports the universal-approximation statement that second-order NODEs can model a broader function class than first-order flows.","marker":"Kidger 2022"},{"why":"Introduces neural ODEs and the adjoint sensitivity method used for backpropagation.","marker":"Chen et al. 2018"},{"why":"CuPL prompts are one of the two sources for the text-as-image augmentation codebook.","marker":"Pratt et al. 2023"}],"fun_headline_variants":["SONO: second-order ODE refiner lifts few-shot average to 77.86%","1.54M-param second-order ODE outdoes CLIP adapters","Second-order ODEs add 3.95% on ImageNet-A few-shot","Text-augmented NODE beats prompt learners in few-shot","Second-order ODE beats CLIP fine-tuning with 77.86% average"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that refining features with a second-order ODE flow, rather than a simpler first-order flow, is what produces the accuracy gains; the paper never directly tests that by swapping in a first-order ODE.","fun_headline_variants_meta":{"raw":{"variants":["SONO: second-order ODE refiner lifts few-shot average to 77.86%","1.54M-param second-order ODE outdoes CLIP adapters","Second-order ODEs add 3.95% on ImageNet-A few-shot","Text-augmented NODE beats prompt learners in few-shot","Second-order ODE beats CLIP fine-tuning with 77.86% average"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1737,"prompt_tokens":914,"completion_tokens":823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":717}},"tokens_in":530,"tokens_out":823,"duration_ms":6971,"temperature":1.0,"reasoning_tokens":717,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:04:02.415892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same pipeline with a first-order ODE in place of the second-order one and compare on all 11 datasets; if the first-order version matches SONO within noise, the central second-order claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Tip-Adapter-F is the cache-based adapter baseline; SONO compares against it across shots and backbones and reports beating it by 2.75% at 16 shots."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the second-order NODE formulation and the first-order adjoint efficiency argument that SONO relies on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CuPL prompts are one of the two sources for the text-as-image augmentation codebook."}],"review_version":1}