Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Efficient Automatic Meta Optimization Search for Few-Shot Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Neural architecture search and Reptile can be jointly optimized to discover a few-shot meta-learner, reaching 74.20% on Mini-ImageNet 5-shot 5-way transductive classification while search costs 1-2 GPU days.

desk verdict First NAS-on-Reptile combination, but Table 1's Train shots=15 makes the headline few-shot numbers apples-to-oranges versus K-shot baselines. read the letter →

arxiv 1909.03817 v1 pith:PS5KUGVQ submitted 2019-09-06 cs.LG cs.CVcs.NE

Xinyue Zheng , Peng Wang , Qigang Wang , Zhongchao shi , Feiyu Xu This is my paper · ORCID
classification cs.LGcs.CVcs.NE
keywords meta-learningfew-shotlearningneuralarchitecturesearchReptileMini-ImageNetOmniglotexperiencereplayparametersharing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Few-shot meta-learning has traditionally paired hand-designed network structures with specialized learning rules. This paper proposes making the architecture itself part of the meta-learning search: a neural-architecture-search controller generates candidate networks, and Reptile trains each candidate on sampled few-shot tasks, with the candidate's validation accuracy fed back to improve the controller. The scheme is meant to be model-agnostic and cheap, with parameter sharing and experience replay bringing the search to one or two GPU days. On Mini-ImageNet the automatically discovered 5-shot 5-way model reaches 74.20% transductive accuracy, and the same architecture transfers to Omniglot when retrained from scratch. The paper's intended contribution is to replace manual architecture engineering in few-shot learning with an automatic, transferable search process.

What carries the argument

The load-bearing object is the two-level objective $\min_{\theta,\varphi} \sum_\tau L(D'_\tau, T(D_\tau, \theta))$ in which the architecture $A$ is folded into the controller policy $\varphi$ instead of being fixed. The controller is a one-layer LSTM that emits a variable-length string of operations and skip connections; Reptile, a first-order gradient meta-learning algorithm that seeks initial parameters adaptable in a few steps, supplies the inner training procedure $T$ that adapts $\theta$ on each task. The child model's validation accuracy $R$ enters the controller's policy-gradient update through an advantage $R - bl$, with $bl$ an exponential moving average of past rewards, and experience replay plus shared child-model parameters are the accelerants that make the search affordable.

What would settle it

Retrain the searched architecture from scratch with exactly 5 and exactly 1 training examples per class in each task, keeping all other retraining choices fixed; if transductive 5-shot accuracy falls toward the Reptile baseline, the reported advantage is a training-shot artifact rather than evidence that the searched architecture is better.

Watch

Extended reading notes

Core claim

The paper's central claim is that a few-shot meta-learner's architecture can be discovered automatically rather than fixed by hand, by optimizing the architecture-generation policy and the meta-learning parameters in one loop. Starting from the usual meta-objective, where task loss is minimized after adapting on each task's training set, the paper lets a recurrent controller sample an architecture string (per-layer operations from a small convolution/pooling library plus skip connections) and trains the resulting child network with Reptile. The child's validation accuracy is passed back as a reward to the controller through a policy-gradient update with a moving-average baseline, while parameter sharing among child models and experience replay in the controller keep the search to one or two GPU days. On Mini-ImageNet the searched model reaches 74.20% ± 0.32% for 5-shot 5-way transductive classification and 67.10% ± 0.90% without transduction; the same architecture retrained for 1-shot 5-way gives 51.62% ± 0.43% transductive. On Omniglot the transferred architecture reaches 98.97% ± 0.12% for 5-shot 20-way, above the non-transductive Reptile baseline, which the paper interprets as cross-task transferability of the discovered architecture.

Load-bearing premise

The empirical comparison assumes the final retraining and evaluation use the same K-shot protocol as the baselines, but Table 1 lists 15 training shots for both the 5-shot and 1-shot settings, so a training-shot mismatch could account for part of the reported margin.

Editorial extensions

If this is right

  • The same controller-plus-Reptile loop can be wrapped around other model-agnostic meta-learning algorithms, so architecture search no longer needs to be redesigned for each few-shot setting.
  • A single search on Mini-ImageNet produces architectures reusable for 1-shot 5-way and for 20-way Omniglot, so the search cost amortizes across benchmarks and tasks.
  • Parameter sharing and experience replay cut search to 32-48 GPU hours, making architecture search feasible on one GPU rather than on tens of thousands of GPU-hours.
  • Retraining more than the top-3 architectures found by the controller is a direct route to higher accuracy, which the paper notes as remaining room for improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the training-shot count is matched to the evaluation setting, the accuracy gap against K-shot baselines may shrink; the paper's Table 1 lists 15 training shots for both the 5-shot and 1-shot retraining runs, so a like-for-like replay is needed before attributing the entire gain to architecture.
  • If the same search loop is run with a meta-learning algorithm other than Reptile, the discovered architectures would likely differ; such an experiment would show whether the controller is learning algorithm-specific inductive biases or simply generic small convnets.
  • If the search-cost claim is the main practical contribution, a fair baseline is a random or grid sample of hand-designed architectures at equal GPU budget; the paper does not compare against that, so the value of the controller's exploration relative to cheaper baselines remains an open question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a joint optimization framework that combines neural architecture search (NAS) with the Reptile meta-learning algorithm. An LSTM controller generates child-model architectures via policy gradient, using validation accuracy as reward; parameter sharing among child models and experience replay are introduced to reduce search cost. The method is evaluated on Mini-ImageNet and Omniglot, reporting 74.20% accuracy for 5-shot 5-way (transductive) and 52.43% for 1-shot 5-way (transductive) on Mini-ImageNet, with search taking 1-2 GPU days. The best Mini-ImageNet architecture is also transferred to Omniglot, showing cross-dataset architecture transferability.

Significance. If the reported results are reproducible, the paper presents a useful and conceptually simple combination of NAS and meta-learning, with a notable efficiency claim (1-2 GPU days) and an interesting demonstration of architecture transfer across datasets. The method is clearly described and builds on established components (ENAS-style parameter sharing and Reptile), making it a plausible baseline for future work. However, the central accuracy claims currently rest on a protocol inconsistency that must be resolved, and the efficiency claim lacks a quantitative ablation, so the significance is conditional on correction and re-evaluation.

major comments (4)
  1. [Section 4, Table 1] Table 1 sets 'Train shots' to 15 for both the 5-shot and 1-shot settings, but Section 4 defines a K-shot task as having K training examples per class in Dtrain. Meta-training on 15-shot tasks while evaluating on 5-shot or 1-shot tasks changes the task distribution and gives the meta-learner more supervision per task than the baselines receive. This makes the comparisons in Table 2 not apples-to-apples for the few-shot claim. The paper neither justifies this choice nor explains it as a typo; the authors must either correct the table or rerun the experiments under the standard K-shot protocol and report the resulting numbers.
  2. [Table 2, Abstract, Section 1] The headline results (74.20% and 52.43%) are obtained in transductive mode, yet the abstract and the contribution list in Section 1 report these numbers without qualification. The non-transductive results (67.10% and 48.00%) are lower than the non-transductive SNAIL baseline (68.88% and 55.71%). The paper should clearly separate transductive from non-transductive comparisons in all claims and provide a fair comparison table that matches settings across methods.
  3. [Section 3.2/3.3, Figure 3] The paper claims that parameter sharing and experience replay greatly reduce search cost, but the only supporting evidence is Figure 3, which shows moving-average curves for replay versus non-replay. There is no quantitative comparison of search time, total controller updates, or final architecture accuracy with and without each mechanism. An ablation study that isolates the contribution of each technique is needed to substantiate the efficiency claim that is central to the paper's contribution.
  4. [Algorithms 1 and 2, Section 4.2] The search procedure lacks several implementation details needed for reproducibility: the number of architectures sampled per controller update (m in Eq. 3-5), the reward baseline update rule, the experience replay buffer capacity and sampling schedule, and the exact schedule of alternate training of controller and child models. Section 4.2 states '1-2 days' and '48 GPU hours' in different places without precise specification. Providing code or a full hyperparameter table would address this concern; as written, the experimental section does not contain enough information to replicate the search.
minor comments (5)
  1. [Section 1] The claim 'we are the first to propose an automatic meta-optimization system by applying neural architecture search technique to meta-learning' is stated without discussion of closely related work; the authors should qualify this claim or survey concurrent approaches.
  2. [Table 1] The row 'Adam learning rate' is ambiguous because Reptile, as described in Algorithm 2, uses Adam for the outer update; clarify whether this is the outer-loop optimizer's learning rate or a separate Adam for the inner loop.
  3. [Figure 3] The x-axis label 'Iterations' with a range of 0-7000 does not specify whether it refers to controller updates, Reptile iterations, or total gradient steps; clarify to match the text that says search takes 1-2 days.
  4. [Section 4.2, Section 1] The paper gives inconsistent search costs: Section 4.2 says '1-2 days' and the contribution list says '5-shot with 48 GPU hours; 1-shot with 32 GPU hours.' These should be reconciled (e.g., specify GPU type and whether the values include retraining).
  5. [Table 2, Section 4.3] The paper says only 'top3 searched architectures' are retrained, but does not state how the top-3 are selected or whether the reported numbers are the best, average, or representative of those three; this should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: final accuracy is measured on held-out meta-test data, while the search reward is validation accuracy, so the central claim does not reduce to its inputs.

full rationale

The paper's derivation chain is: the controller samples an architecture; the child model is trained by Reptile on meta-training tasks; validation accuracy is returned as the reward to update the controller; after search, top architectures are retrained from scratch; final accuracy is measured on the Mini-ImageNet meta-test split and on Omniglot with weights retrained from scratch. The reward signal is meta-validation accuracy, and the reported headline numbers are meta-test accuracies, so the claimed result is not the same quantity as the search objective by construction. The optimization steps are policy gradient for the controller (Eq. 5), Reptile updates for the child model (Algorithm 2), and final retraining; none of these solves for the reported accuracy as a fitted value. There are no load-bearing self-citations: the cited ENAS, Reptile, MAML, and related works are external prior art, and the paper does not invoke a uniqueness theorem or rename a known result. The Table 1 'Train shots 15' discrepancy is an internal inconsistency that challenges apples-to-apples comparability of some baselines, but that is an empirical-validity concern rather than a circularity in which a prediction reduces to its input. Therefore no significant circularity is present.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central empirical results depend on multiple hand-selected hyperparameters and on two domain assumptions about how well a briefly trained, weight-shared child model predicts final retrained performance. No invented physical or conceptual entities are introduced.

free parameters (6)
  • Train shots = 15
    Table 1 sets 15 shots per class during training for both 5-shot and 1-shot evaluations, contradicting Section 4's K-shot description; this is an unexplained and potentially consequential choice.
  • Reptile Adam learning rate = 0.005 (5-shot), 0.003 (1-shot)
    Chosen per setting in Table 1 with no sensitivity analysis; directly affects final accuracy.
  • Child model depth = 8 layers
    The controller only searches 8-layer chains, a hand-selected architectural budget.
  • Controller LSTM hidden units = 100
    Controller capacity is fixed at 100 hidden units in Section 4.2.
  • Dropout rate = 0.25
    Dropout after each layer is fixed at 0.25 in Section 4.2 to reduce overfitting.
  • Reptile meta-batch size and inner iterations = 5 tasks, 8 inner steps
    Table 1 fixes the Reptile training schedule; no ablation is provided.
assumptions (3)
  • domain assumption Child-model validation accuracy under partial Reptile training and ENAS weight sharing is a valid proxy for the quality of an independently retrained architecture.
    The controller ranks architectures by validation accuracy after brief Reptile training (Sections 3.2 and 3.3), and the top-3 are retrained from scratch (Section 4.2). No correlation analysis is provided.
  • ad hoc to paper The fixed 8-layer convolutional search space contains competitive few-shot meta-learners.
    The controller only samples 8-layer chains from a hand-selected operation list (Sections 3.1 and 4.2); there is no evidence this space is sufficient.
  • domain assumption Reptile's first-order gradient-based adaptation is an appropriate meta-optimizer for all searched architectures.
    Algorithm 2 applies Reptile uniformly to every sampled child model; the paper does not test other meta-learning algorithms despite claiming universality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Automatic Meta Optimization Search for Few-Shot Learning." pith.science (2026). https://pith.science/paper/PS5KUGVQ

@misc{pith2026190903817,
  author       = {Pith},
  title        = {Pith review of: Efficient Automatic Meta Optimization Search for Few-Shot Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PS5KUGVQ}},
  note         = {Machine review of arXiv:1909.03817}
}
read the original abstract

Previous works on meta-learning either relied on elaborately hand-designed network structures or adopted specialized learning rules to a particular domain. We propose a universal framework to optimize the meta-learning process automatically by adopting neural architecture search technique (NAS). NAS automatically generates and evaluates meta-learner's architecture for few-shot learning problems, while the meta-learner uses meta-learning algorithm to optimize its parameters based on the distribution of learning tasks. Parameter sharing and experience replay are adopted to accelerate the architectures searching process, so it takes only 1-2 GPU days to find good architectures. Extensive experiments on Mini-ImageNet and Omniglot show that our algorithm excels in few-shot learning tasks. The best architecture found on Mini-ImageNet achieves competitive results when transferred to Omniglot, which shows the high transferability of architectures among different computer vision problems.

Figures

Figures reproduced from arXiv: 1909.03817 by the authors.

Figure 1
Figure 1. An overview of efficient automatic meta-learning method experimental comparisons. Therefore, we propose a novel joint optimization scheme which combines model-agnostic meta-learning algorithm and automatic architecture design to improve the few-shot learning. As is shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: The prediction string made by controller. Right: Connect prediction string to build a complete network. Since the discrete domain search of A can be transformed into the continuous domain optimization of controller network in our method, the formula (1) can be rewrite as a differential form which can be optimized with end-to-end training: min θ,φ X τ L(D 0 τ , θ 0 τ ) = X τ L(D 0 τ , T(Dτ , θ)) (2) 3.1 Generat… view at source ↗
Figure 3
Figure 3. shows the experience replay contributes to the understanding of [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Training curves for the architecture search procedure: exponential moving average architecture accuracy over 7K iterations. conv 5x5 Image conv 5x5 Softmax max 3x3 avg 3x3 avg 3x3 conv 3x3 sep 5x5 sep 5x5 sep 7x7 Image conv 7x7 Softmax sep 7x7 conv 5x5 avg 3x3 avg 3x3 …
Figure 4
Figure 4. Figure 4: High accuracy architectures searched by 5-shot, 5-way classification on Mini￾ImageNet, and can be transferred to other classification scenes. References 1. Andrychowicz, M., Denil, M., Colmenarejo, S.G., Hoffman, M.W., Pfau, D., Schaul, T., de Freitas, N.: Learning to …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 4 canonical work pages

  1. [1]

    CoRR 12 Xinyue Zheng et al

    Andrychowicz, M., Denil, M., Colmenarejo, S.G., Hoffman, M.W., Pfau, D., Schaul, T., de Freitas, N.: Learning to learn by gradient descent by gradient descent. CoRR 12 Xinyue Zheng et al. abs/1606.04474 (2016), http://arxiv.org/abs/1606.04474

    arXiv 2016

  2. [2]

    arXiv preprint arXiv:1703.03400 (2017)

    Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 (2017)

    arXiv 2017

  3. [3]

    CoRR abs/1410.5401 (2014), http://arxiv.org/abs/1410.5401

    Graves, A., Wayne, G., Danihelka, I.: Neural turing machines. CoRR abs/1410.5401 (2014), http://arxiv.org/abs/1410.5401

    arXiv 2014

  4. [4]

    arXiv preprint arXiv:1706.00764 (2017)

    Hazan, E., Klivans, A., Yuan, Y.: Hyperparameter optimization: A spectral approach. arXiv preprint arXiv:1706.00764 (2017)

    arXiv 2017

  5. [5]

    Science 350(6266), 1332–1338 (2015)

    Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B.: Human-level concept learning through probabilistic program induction. Science 350(6266), 1332–1338 (2015)

    2015

  6. [6]

    Larochelle, S.R.H.: Optimization as a model for few-shot learning

    work page

  7. [7]

    arXiv preprint arXiv:1603.06560 (2016)

    Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560 (2016)

    arXiv 2016

  8. [8]

    Object-Level Representation Learning for Few-Shot Image Classification

    Long, L., Wang, W., Wen, J., Zhang, M., Lin, Q., Ooi, B.C.: Object-level represen- tation learning for few-shot image classification. CoRR abs/1805.10777 (2018), http://arxiv.org/abs/1805.10777

    work page Pith review arXiv 2018

Show all 24 references
  1. [9]

    arXiv preprint arXiv:1604.07269 (2016)

    Loshchilov, I., Hutter, F.: Cma-es for hyperparameter optimization of deep neural networks. arXiv preprint arXiv:1604.07269 (2016)

    2016 arXiv

  2. [10]

    CoRR abs/1707.03141 (2017), http://arxiv.org/abs/1707.03141

    Mishra, N., Rohaninejad, M., Chen, X., Abbeel, P.: Meta-learning with temporal convolutions. CoRR abs/1707.03141 (2017), http://arxiv.org/abs/1707.03141

    2017 arXiv

  3. [11]

    CoRR abs/1703.00837 (2017), http: //arxiv.org/abs/1703.00837

    Munkhdalai, T., Yu, H.: Meta networks. CoRR abs/1703.00837 (2017), http: //arxiv.org/abs/1703.00837

    2017 arXiv

  4. [12]

    arXiv preprint arXiv:1803.02999 (2018)

    Nichol, A., Schulman, J.: Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 (2018)

    2018 arXiv

  5. [13]

    arXiv preprint arXiv:1802.03268 (2018)

    Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018)

    2018 arXiv

  6. [14]

    arXiv preprint arXiv:1802.01548 (2018)

    Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548 (2018)

    2018 arXiv

  7. [15]

    arXiv preprint arXiv:1703.01041 (2017)

    Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q., Kurakin, A.: Large-scale evolution of image classifiers. arXiv preprint arXiv:1703.01041 (2017)

    2017 arXiv

  8. [16]

    CoRR (2016), http://arxiv.org/abs/1605.06065

    Santoro, A., Bartunov, S., Botvinick, M.: One-shot learning with memory-augmented neural networks. CoRR (2016), http://arxiv.org/abs/1605.06065

    2016 arXiv

  9. [17]

    CoRR abs/1511.05952 (2015), http://arxiv.org/abs/1511.05952

    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. CoRR abs/1511.05952 (2015), http://arxiv.org/abs/1511.05952

    2015 arXiv

  10. [18]

    Shin, R., Packer, C., Song, D.: Differentiable neural network architecture search (2018)

    2018

  11. [19]

    In: Advances in neural information processing systems

    Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems. pp. 2951–2959 (2012)

    2012

  12. [20]

    arXiv preprint arXiv:1711.06025 (2017)

    Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales, T.M.: Learning to compare: Relation network for few-shot learning. arXiv preprint arXiv:1711.06025 (2017)

    2017 arXiv

  13. [21]

    Submitted to Advances in Neural Information Processing Systems 12, 1057–1063 (1999)

    Sutton, R.S.: Policy gradient methods for reinforcement learning with function approximation. Submitted to Advances in Neural Information Processing Systems 12, 1057–1063 (1999)

    1999

  14. [22]

    CoRR abs/1606.04080 (2016), http://arxiv.org/abs/1606.04080

    Vinyals, O., Blundell, C., Lillicrap, T.P.: Matching networks for one shot learning. CoRR abs/1606.04080 (2016), http://arxiv.org/abs/1606.04080

    2016 arXiv

  15. [23]

    arXiv preprint arXiv:1611.01578 (2016)

    Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)

    2016 arXiv

  16. [24]

    arXiv preprint arXiv:1707.07012 (2017)

    Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. arXiv preprint arXiv:1707.07012 (2017)

    2017 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.