REVIEW 3 major objections 6 minor 29 references
Geometric Mean Improves Loss For Few-Shot Learning
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Replacing the arithmetic mean in a softmax few-shot loss with a geometric mean yields consistently better feature metrics.
desk verdict A genuinely neat loss with correct theory, but the headline empirical claim is confounded with the L1 distance choice; needs an ablation before the numbers are taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the geometric mean of softmax attention weights over samples of the query's class, $\left[\prod_{i:y_i=y_q}a_S(x_q,x_i)\right]^{1/n_{y_q}}$. This is the operation that turns the NCA loss's arithmetic aggregation into the proposed loss. A geometric mean is zero whenever any factor is zero, so minimizing it pushes every in-class attention weight away from zero; the gradient analysis makes this explicit by weighting each sample-wise gradient term by $1/a_S(x_q,x_i)$, which assigns larger updates to far-away samples. The simplified form separates an average in-class distance from a log-sum-exp over all support samples, connecting the loss both to prototypical-net center learning and to NCA neighborhood weighting.
What would settle it
Train the proposed loss and the NCA loss under all four combinations of distance $p=1$ versus $p=2$ and arithmetic- versus geometric-mean aggregation on miniImageNet 1-shot. The attribution to the geometric mean is falsified if the geometric-mean loss does not beat NCA when both use $p=2$, or if NCA with $p=1$ matches the geometric-mean loss.
Extended reading notes
Core claim
The central claim is that aggregating softmax-based attention weights, $a_S(x_q,x_i)=\exp(-d(x_q,x_i))/\sum_j\exp(-d(x_q,x_j))$, by geometric mean rather than arithmetic mean produces a feature metric that generalizes better in few-shot classification. The proposed loss is $\ell_{\mathrm{ours}}=-\log\left[\prod_{i:y_i=y_q}a_S(x_q,x_i)\right]^{1/n_{y_q}}$, which simplifies to $\frac{1}{n_{y_q}}\sum_{i:y_i=y_q}d(x_q,x_i)+\log\sum_j\exp(-d(x_q,x_j))$. The paper argues that this drives a query toward the medoid of its class under the chosen distance, prevents sparse attention from silencing far-away in-class samples, and is equivalent to a multi-label softmax over the support set. In the reported comparisons, the losses use the $L_1$-based distance $d_1(x,z)=\sum_{r}|x_r-z_r|$, which the paper selects after evaluating several $L_p$ variants.
Load-bearing premise
The paper assumes that preventing any in-class attention weight from collapsing to zero is the mechanism behind the gains, but this mechanism is never ablated, and the reported improvement could partly come from the L1 distance chosen in Figure 1 rather than from the geometric mean itself.
Editorial extensions
If this is right
- The geometric-mean loss can replace the NCA loss in an existing few-shot training loop at no extra computational cost, since both use distance terms plus a log-sum-exp.
- Because it upper-bounds the NCA loss, the proposed loss keeps producing gradients in regimes where NCA is already saturated.
- The medoid interpretation suggests the loss is tolerant of outlier support samples, especially when combined with the L1 distance.
- On the three benchmarks, the loss alone is competitive with recent few-shot methods that add architectural components, despite changing only the training objective.
Reading between the lines
- The paper does not isolate the geometric mean from the L1-distance choice; a direct ablation of the two factors would settle which ingredient drives the reported gains.
- The inverse-attention weighting in the gradient resembles an automatic focal-like reweighting, and a soft focal-loss variant might reproduce the same effect without a geometric mean.
- The medoid-based interpretation could transfer to other metric-learning settings, such as retrieval or clustering, where a tolerant aggregate of pairwise similarities is desirable.
- Because training uses a leave-one-out expectation over mini-batches, the loss's behavior in very small support sets might be sensitive to mini-batch composition; varying support size during training could test the robustness of the gains.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a few-shot metric-learning loss that replaces the arithmetic mean of softmax-based attention weights in the NCA loss with a geometric mean (Eq. 6). The loss simplifies to an average distance plus a log-sum-exp term (Eq. 7). The authors provide theoretical analysis relating the loss to the NCA loss (upper bound and gradient weighting), to the PN loss (a within-class variance decomposition for squared L2 distance), and to multi-label classification. Experiments on miniImageNet, CIFAR-FS, and tieredImageNet with a ResNet12 backbone report consistent gains over the PN and NCA losses, along with comparisons to other few-shot methods.
Significance. The proposal is simple, parameter-free apart from the distance metric order p, and theoretically transparent; the derivations are easy to check and the computational overhead over the NCA loss is negligible. If the reported gains are robust and attributable to the geometric-mean aggregation, the loss would be a useful drop-in component for metric-based few-shot learning. The paper also correctly emphasizes the anti-sparsity behavior of the geometric mean. However, the central empirical claim is currently confounded with the choice of distance metric p=1, and part of the theoretical motivation is written for squared L2 distances while the headline experiments use L1 distances. The scientific value hinges on additional ablations that isolate the geometric-mean mechanism.
major comments (3)
- [Sec. 3.2, Table 2] The central comparison to PN and NCA is run entirely with dp=1, a value selected from Fig. 1, which evaluates the distance metric only for the proposed loss. There is no analogous p-sweep for the PN and NCA baselines, and no experiment that varies the aggregation function (arithmetic vs. geometric mean) while holding the distance metric fixed. It is therefore possible that part of the reported gain (e.g., 65.51% vs. 62.68% on miniImageNet 1-shot) comes from imposing a suboptimal metric on the baselines rather than from the geometric-mean term. Please report Ours, PN, and NCA under both dp=1 and dp=2 under an otherwise identical training protocol, and preferably also include an ablation where the arithmetic-mean baseline is replaced by the geometric-mean term with the same p.
- [Sec. 2.3.2, Eq. (11)] The variance-decomposition argument that the proposed loss additionally minimizes within-class variance is derived specifically for d(x,z)=||x-z||^2. The experimental section, however, adopts dp=1 for all headline results in Table 2, for which Eq. (11) does not hold. Thus the theoretical mechanism stated in Sec. 2.3.2 does not directly support the empirical configuration used in the main comparison. The paper should either supply a p=1 analogue of the variance/medoid analysis and connect it to the experiments, or run the headline comparison with p=2 to align theory and experiment.
- [Sec. 2.2 / Sec. 2.3.1] The claimed advantage of avoiding sparse attention weights is motivated analytically and by analogy to focal loss, but it is never empirically isolated. A direct test could compare the proposed loss against an arithmetic-mean NCA loss augmented with a simple anti-sparsity regularizer (e.g., adding -log a_i for in-class samples or an entropy term), under the same distance metric. Without such an ablation, the reader cannot tell whether the gains are due to the geometric-mean aggregation itself or to some other consequence of the loss form.
minor comments (6)
- [Sec. 2.3.1, Eq. (8)] The inequality is attributed to the Cauchy-Schwarz inequality, but the direct justification is the AM-GM inequality (or Jensen's inequality on -log). The inequality itself is correct; the attribution should be fixed.
- [Sec. 3.2, Fig. 1] Figure 1 shows performance versus p but does not report whether error bars or multiple runs are included. Please describe the protocol used to generate Fig. 1, including the number of trials and whether validation or test performance is plotted.
- [Sec. 3.2, Table 2] The text states that 'all the methods are equipped with dp=1' but does not explicitly say whether the PN and NCA baselines were re-trained under exactly the same leave-one-out mini-batch protocol, number of epochs, and learning-rate schedule as the proposed method. Please state this explicitly to confirm that the comparison is controlled.
- [Sec. 3.3, Table 3] The comparison methods [15-18] are cited from their respective papers, which may use different backbones, training protocols, or evaluation details; the underline criterion is also not fully defined. This is acceptable for context, but the text should acknowledge that Table 3 is not a controlled comparison.
- [Throughout] There are several typographical errors: 'prose' should be 'propose' in Sec. 1, 'contraast' in Sec. 3.2, and reference [19] should use proper diacritics for 'João'. A careful proofread is recommended.
- [Sec. 2.4] The statement that p=1 'provides a medoid robust to outliers' is reasonable, but for p=1 the minimizer of the sum of distances is a geometric median, not necessarily a discrete medoid. The wording could be refined to avoid conflating the two notions.
Circularity Check
No significant circularity: the loss is defined directly and the theoretical analysis is self-contained.
full rationale
The paper's proposed loss is defined directly in Eq. (6) as the geometric mean of softmax attention weights, involving no fitted parameters and no dependence on the experimental results. The subsequent theoretical analysis—upper bound to the NCA loss via Cauchy-Schwarz, gradient weighting comparison, variance decomposition for L2 distance, and multi-label softmax interpretation—all follow from the definition by standard algebra and do not assume the target claim. There are no self-citations used as load-bearing evidence, and no uniqueness theorem is imported from the authors' prior work. The only potential concern is the selection of the L1 distance metric based on Fig. 1 for the proposed loss and then applying that same metric to all baselines in Table 2; however, this is a hyperparameter choice common to all compared methods and does not make the geometric-mean loss's superiority an artifact of the loss definition itself. The central empirical comparison is a genuine experimental outcome, and the theoretical claims are self-contained. Thus no circular step can be identified.
Assumptions & free parameters
free parameters (1)
- Distance metric order p =
1
assumptions (4)
- standard math Geometric mean of nonnegative numbers is no greater than arithmetic mean (AM-GM inequality).
- standard math The softmax attention weights sum to 1 over all samples.
- domain assumption Minimizing within-class variance in the feature space improves discriminative metric for few-shot learning.
- domain assumption The 'saturation' problem of NCA loss is harmful, and preventing attention weights from becoming zero improves metric learning.
Cite this review
Pith. "Pith review of Geometric Mean Improves Loss For Few-Shot Learning." pith.science (2026). https://pith.science/paper/FGPV2PUM
@misc{pith2026250114593,
author = {Pith},
title = {Pith review of: Geometric Mean Improves Loss For Few-Shot Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/FGPV2PUM}},
note = {Machine review of arXiv:2501.14593}
}
read the original abstract
Few-shot learning (FSL) is a challenging task in machine learning, demanding a model to render discriminative classification by using only a few labeled samples. In the literature of FSL, deep models are trained in a manner of metric learning to provide metric in a feature space which is well generalizable to classify samples of novel classes; in the space, even a few amount of labeled training examples can construct an effective classifier. In this paper, we propose a novel FSL loss based on \emph{geometric mean} to embed discriminative metric into deep features. In contrast to the other losses such as utilizing arithmetic mean in softmax-based formulation, the proposed method leverages geometric mean to aggregate pair-wise relationships among samples for enhancing discriminative metric across class categories. The proposed loss is not only formulated in a simple form but also is thoroughly analyzed in theoretical ways to reveal its favorable characteristics which are favorable for learning feature metric in FSL. In the experiments on few-shot image classification tasks, the method produces competitive performance in comparison to the other losses.
Reference graph
Works this paper leans on
-
[1]
INTRODUCTION Few-shot learning (FSL) draws inspiration from the remark- able human ability of robust reasoning and analysis, particu- larly in scenarios where limited information is available. This paradigm has gained significant traction in various applica- tions, e.g., autonomous vehicles and medical analysis, where resource constraints necessitate effi...
work page Pith review arXiv 2025
-
[2]
METHOD We start with briefly reviewing two representative loss func- tions for FSL, PN Loss [5] and NCA loss [6], then formulate our proposed loss. Notations. Suppose we have a support set S = {(xi, yi)}n i=1 and a target query sample (xq, yq) for constructing a loss; an input image I is embedded into a D-dimensional feature vector x ∈ RD via a deep model...
-
[3]
RESULT We empirically evaluate and analyze the proposed loss (6) on few-shot image classification tasks; we primarily focus on performance in terms of loss functions on an FSL framework. miniImageNet CIFAR-FS Fig. 1. Performance analysis of various distance metric dp. 3.1. Experimental settings Datasets. We employ three FSL benchmark datasets. The miniIma...
-
[4]
CONCLUSION We have proposed a FSL loss based on geometric mean of softmax-based sample-wise attention weights. While it is for- mulated in a simple form, our theoretical analysis reveals that the method renders various favorable characteristics to metric learning for FSL in comparison to the other FSL losses. The experimental results on few-shot image cla...
-
[5]
Generalizing from a few examples: A sur- vey on few-shot learning,
Yaqing Wang, Quanming Yao, James T Kwok, and Li- onel M Ni, “Generalizing from a few examples: A sur- vey on few-shot learning,”ACM computing surveys, vol. 53, no. 3, pp. 1–34, 2020
work page 2020
-
[6]
Model-agnostic meta-learning for fast adaptation of deep networks,
Chelsea Finn, Pieter Abbeel, and Sergey Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in ICML, 2017, pp. 1126–1135
work page 2017
-
[7]
On first-order meta- learning algorithms,
Alex Nichol and John Schulman, “On first-order meta- learning algorithms,” arXiv:1803.02999, 2018
arXiv 2018
-
[8]
Siamese neural networks for one-shot image recognition,
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al., “Siamese neural networks for one-shot image recognition,” in ICML Deep Learning Workshop, 2015
work page 2015
Show all 29 references
-
[9]
Pro- totypical networks for few-shot learning,
Jake Snell, Kevin Swersky, and Richard Zemel, “Pro- totypical networks for few-shot learning,” in NeurIPS, 2017, pp. 4080–4090
2017
-
[10]
On episodes, pro- totypical networks, and few-shot learning,
Steinar Laenen and Luca Bertinetto, “On episodes, pro- totypical networks, and few-shot learning,” in NeurIPS, 2021, pp. 24581–24592
2021
-
[11]
Matching net- works for one shot learning,
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Ko- ray Kavukcuoglu, and Daan Wierstra, “Matching net- works for one shot learning,” in NeurIPS, 2016
2016
-
[12]
A discriminative feature learning approach for deep face recognition,
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao, “A discriminative feature learning approach for deep face recognition,” in ECCV, 2016
2016
-
[13]
Neighbourhood components analy- sis,
Jacob Goldberger, Sam Roweis, Geoff Hinton, and Rus- lan Salakhutdinov, “Neighbourhood components analy- sis,” in NeurIPS, 2005
2005
-
[14]
Focal loss for dense object detec- tion,
Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Doll´ar, “Focal loss for dense object detec- tion,” in ICCV, 2017, pp. 2980–2988
2017
-
[15]
Mean shift: A robust approach toward feature space analysis,
Dorin Comaniciu and Peter Meer, “Mean shift: A robust approach toward feature space analysis,” TPAMI, vol. 24, no. 5, pp. 603–619, 2002
2002
-
[16]
Duda, Peter E
Richard O. Duda, Peter E. Hart, and David G. Stork, Pattern Classification, Wiley, New York, 2001
2001
-
[17]
Multi-label classification: An overview,
Grigorios Tsoumakas and Ioannis Katakis, “Multi-label classification: An overview,” International Journal of Data Warehousing and Mining, vol. 3, no. 3, pp. 1–13, 2007
2007
-
[18]
Asymmetric loss for multi-label classification,
Emanuel Ben-Baruch, Tal Ridnik, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik- Manor, “Asymmetric loss for multi-label classification,” in ICCV, 2021, pp. 82–91
2021
-
[19]
Revisiting local descriptor based image-to-class measure for few-shot learning,
Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo, “Revisiting local descriptor based image-to-class measure for few-shot learning,” in CVPR, 2019, pp. 7260–7268
2019
-
[20]
Cross attention network for few- shot classification,
Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen, “Cross attention network for few- shot classification,” in NeurIPS, 2019
2019
-
[21]
Meta-baseline: Exploring simple meta-learning for few-shot learning,
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang, “Meta-baseline: Exploring simple meta-learning for few-shot learning,” in ICCV, 2021
2021
-
[22]
Rethinking few-shot im- age classification: a good embedding is all you need?,
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola, “Rethinking few-shot im- age classification: a good embedding is all you need?,” in ECCV, 2020, pp. 266–282
2020
-
[23]
Meta-learning with differentiable closed-form solvers,
Luca Bertinetto, Jo ao Henriques, Philip H.S. Torr, and Andrea Vedaldi, “Meta-learning with differentiable closed-form solvers,” in ICLR, 2019
2019
-
[24]
Learning multiple layers of features from tiny images,
Alex Krizhevsky and Geoffrey E. Hinton, “Learning multiple layers of features from tiny images,” Technical report, University of Toronto, 2009
2009
-
[25]
Optimization as a model for few-shot learning,
Sachin Ravi and Hugo Larochelle, “Optimization as a model for few-shot learning,” in ICLR, 2016
2016
-
[26]
Meta-learning for semi-supervised few-shot classification,
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel, “Meta-learning for semi-supervised few-shot classification,” inICLR, 2018
2018
-
[27]
Simpleshot: Revisiting nearest-neighbor classification for few-shot learning,
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens Van Der Maaten, “Simpleshot: Revisiting nearest-neighbor classification for few-shot learning,” arXiv:1911.04623, 2019
1911 arXiv
-
[28]
Meta-learning with latent embedding opti- mization,
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell, “Meta-learning with latent embedding opti- mization,” arXiv:1807.05960, 2018
2018 arXiv
-
[29]
Meta-learning with differ- entiable convex optimization,
Kwonjoon Lee, Subhransu Maji, Avinash Ravichan- dran, and Stefano Soatto, “Meta-learning with differ- entiable convex optimization,” in CVPR, 2019
2019
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.