REVIEW 3 major objections 4 minor 35 references
Position: AI Scaling: From Up to Down and Out
T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper argues that AI scaling's future lies in Scaling Down to efficient core models and Scaling Out to distributed ecosystems of specialized interfaces, not in ever-larger models alone.
desk verdict A clear, well-written position framework for scaling down and out, but the central forecast overreaches its own evidence and the synthetic-data tension is real. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the three-paradigm scaling framework, a progression from a large model to a small core model to an ecosystem of specialized interfaces. Scaling Down is carried by a toolkit of concrete techniques — pruning, low-rank approximation (LoRA), quantization, knowledge distillation, dataset pruning, speculative decoding, key-value cache management, mixture-of-experts routing, and high-quality data curation — that shrink models or reduce their computation. Scaling Out is carried by parameter-efficient fine-tuning, condition control, federated learning, and standard communication protocols, which turn a core model into many task-specific agents that interact through APIs and interfaces. The framework's work is to give a single structure to otherwise scattered efficiency and deployment research and to ground the prediction that Down and Out, not Up, will be the main source of future AI progress.
What would settle it
A controlled scaling study that trains a model several times larger than the current frontier on a freshly curated or synthetic dataset and finds loss still decreasing at the rate predicted by established scaling laws, with no plateau from data or compute limits, would directly contradict the 'inherent bottleneck' premise.
Extended reading notes
Core claim
The paper's central claim is that AI scaling is not one thing but three, and that the weight of future progress moves from the first to the second and third: Scaling Up, building larger and more capable foundation models; Scaling Down, distilling the essential functions of those models into small, efficient core modules; and Scaling Out, deriving many task-specific interfaces from a core model and interconnecting them into AI ecosystems. The authors contend that Scaling Up is approaching practical limits, because most high-quality public data has already been mined, training and inference costs grow unsustainably, and marginal returns are diminishing. They present Scaling Down and Scaling Out not as a rejection of Scaling Up but as a dependent progression: Scaling Up sets the benchmarks that define what good performance looks like, Scaling Down operationalizes those benchmarks at affordable cost, and Scaling Out amplifies them through distributed specialization and collaboration.
Load-bearing premise
The load-bearing premise is that Scaling Up has reached practical limits because high-quality public data is nearly exhausted, compute and energy costs are unsustainable, and marginal returns have diminished; if synthetic data, algorithmic advances, or cheaper hardware keep the scaling law alive, the shift to Down and Out is not necessary.
Editorial extensions
If this is right
- If the prediction is correct, research investment in model compression, quantization, and distillation will yield more practical capability gains per dollar than further increases in parameter count.
- Deployment of capable AI shifts from centralized data centers to edge devices and on-device inference, reducing energy use and making AI accessible where internet and compute are scarce.
- The dominant AI architecture becomes a small core model surrounded by many specialized, fine-tuned interfaces that communicate through shared protocols, rather than a single monolithic foundation model.
- Benchmarks and scaling laws will need to be rewritten: instead of loss versus model size, the field will track performance-per-watt, cost-per-inference, and ecosystem-level measures such as the diversity and robustness of fine-tuned variants.
- Scaling Up does not disappear; it continues as the seedbed for Down and Out, since one large foundation model can give rise to hundreds of specialized variants that form the substrate of the ecosystem.
Reading between the lines
- Inference: The framework predicts a measurable shift in where state-of-the-art capability appears: if small models with external tools or test-time compute close much of the gap to frontier models, the size of the best-performing model per fixed compute budget should shrink over time; if that size stops shrinking or starts growing, the Down phase would be losing its force.
- Inference: Scaling Out presumes that interface and protocol standards will diffuse; a testable corollary is that the number and variety of interoperable agents on open platforms will grow, and that parameter-efficient adaptations will increasingly dominate full fine-tunes in open model hubs.
- Inference: If the authors are right, AGI-like capability is more likely to emerge from orchestration of many specialized systems than from a single giant model, which would redirect safety and governance work toward distributed coordination failures rather than monolithic alignment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a framework for AI scaling with three paradigms: Scaling Up (larger models), Scaling Down (compression, distillation, efficiency), and Scaling Out (distributed ecosystems of specialized models). It argues that Scaling Up is hitting inherent bottlenecks—data exhaustion, compute cost, environmental impact—and that the future trajectory of AI progress will shift to Scaling Down and Scaling Out. It surveys supporting techniques and applications, discusses challenges, and explicitly acknowledges alternative views in Section 7. The contribution is a conceptual synthesis and literature survey, not a new empirical or theoretical result.
Significance. As a position paper, the work has the potential to reframe AI scaling discussions toward efficiency, accessibility, and distribution. It gathers a wide range of existing techniques (pruning, quantization, distillation, MoE, PEFT, federated learning, agent protocols) into a unified narrative, which is genuinely useful for orienting new researchers and for structuring debate. The explicit "Alternative Views" section is a strength, as is the paper's honesty about the limitations of its own evidence. However, the paper's significance is tempered by the absence of any quantitative evidence for the central prediction and by internal inconsistencies in how the three paradigms are characterized. The paper is best read as an informed opinion piece rather than a rigorously supported thesis.
major comments (3)
- [Section 2.2 vs Section 3.4] The central justification for abandoning Scaling Up rests on the claim that high-quality public data is nearly exhausted and that the remaining data is "low-quality or consists of AI-generated content, which risks model degradation" (citing Shumailov et al., 2024). Yet Section 3.4 praises the Phi family for being trained on "curated, synthetically generated textbook-style data" that "enhances efficiency and mitigates common issues such as hallucination and bias." These two passages are in direct tension. If synthetic data can be curated into high-quality training data, as the Phi discussion demonstrates, then the data scarcity argument in Section 2.2 is not an inherent limit of Scaling Up but rather a data-curation problem that can be addressed with the Scaling Up paradigm itself. The paper offers no criterion that separates harmful, model-collapse-inducing AI-generated content from beneficial, curated synthetic data. This is load-bearing because the "inherent bottleneck" premise is the paper's main argument for shifting to Scaling Down and Scaling Out. The paper should either resolve this contradiction by specifying the distinguishing criterion, or soften the claim in the Abstract and Section 1 that Scaling Up faces "inherent bottlenecks."
- [Section 2.1 vs Section 3.4] The taxonomy of Scaling Up and Scaling Down is inconsistent with respect to sparse mixture-of-experts (MoE) models. In Section 2.1, GLaM is presented as a Scaling Up example: "GLaM scaled to 1.2 trillion parameters using a mixture of experts, activating only a subset of parameters per task, which reduced computational costs while maintaining high performance." In Section 3.4, the paper describes DeepSeek-V2 and DeepSeek-V3 under the heading "More Parameters but Less Activation" as Scaling Down innovations, celebrating parameter counts of 236B and 671B. Sparse MoE is thus simultaneously treated as Scaling Up and Scaling Down. This is a conceptual inconsistency in the paper's central framework. The paper should clarify whether a model is classified by total parameter count, by activated parameter count, or by the training/inference compute per token, and then apply that criterion consistently.
- [Abstract, Section 1, and Section 7] The paper's central claim is stronger than its own caveats. The Abstract and Section 1 assert that "the future trajectory of AI scaling lies in Scaling Down and Scaling Out," while Section 7 concedes that synthetic data, novel hardware (quantum, optical, neuromorphic), and algorithmic advances could overcome the Scaling Up bottlenecks. The paper does not provide an argument for why these alternative escape routes are less likely than the proposed Down and Out path. For a position paper, it is acceptable to make a speculative forecast, but the wording should match the degree of support. I recommend hedging the central claim, for example by saying that Scaling Down and Scaling Out will be 'increasingly important' or 'likely central' rather than 'the future trajectory lies in', and by explicitly discussing what evidence would falsify the stronger claim.
minor comments (4)
- [References] The reference "Langley, P. Crafting papers on machine learning" appears in the bibliography but is never cited in the text. Please either cite it or remove it.
- [Section 3, intro paragraph] There is a typo in the sentence "Scaling Down can be approached in two distinct ways ." with an extra space before the period. Please correct this and check the manuscript for similar spacing issues.
- [Section 4.3] The paragraph titled "Blockchain" begins with an analogy to app stores and an AI model store, but the connection to blockchain is only made in the second sentence. Consider restructuring so that the heading and content are aligned, or renaming the subsection to reflect the app-store idea as the primary topic.
- [Section 2.1, Data Size] The phrase "large-scale multimodal Corpus" is capitalized mid-sentence and appears to be a placeholder. It should be rewritten, for example as "large-scale multimodal corpora."
Circularity Check
No circularity: position paper with no fitted inputs, no derived predictions, and no load-bearing self-citations.
full rationale
This is a position paper, not a derivation. The central claim that the future of AI scaling lies in Scaling Down and Scaling Out is an argumentative prediction supported by external, independently published evidence. Section 2.2's data-bottleneck premise cites Shumailov et al. (2024) and Wu et al. (2024); Section 3 surveys externally published pruning, quantization, distillation, and MoE techniques; Section 4's ecosystem claims cite PEFT, ControlNet, FLAN-T5, federated learning, and open protocols. No equation is introduced, no parameter is fitted, and no quantity called a prediction is computed from an input fitted by the authors. The only possible self-citation is the TDDS dataset-pruning work mentioned in Section 3.2 as an example of a trend toward dynamic data selection; even if it is authored by the present authors, it is illustrative rather than load-bearing, and the central trajectory claim does not depend on it. The paper's Section 7 explicitly concedes that Scaling Up could remain viable through synthetic data, new hardware, and algorithmic advances, showing the central claim is a position with stated caveats rather than a result forced by definition. The apparent tension between Section 2.2 (AI-generated data risks degradation) and Section 3.4 (curated synthetic data succeeds) is an internal-consistency concern, not circularity, because the paper never defines its conclusion in terms of that premise. Accordingly, no circular step can be exhibited under the required standard.
Assumptions & free parameters
assumptions (3)
- domain assumption Scaling Up faces inherent bottlenecks: high-quality public data is largely exhausted, compute and energy costs are unsustainable, and returns from adding parameters are diminishing.
- domain assumption Scaling Down can reduce model size and cost while retaining or improving task performance.
- domain assumption Scaling Out, as an ecosystem of specialized models communicating through interfaces, yields more capable and adaptable collective intelligence than monolithic models.
Cite this review
Pith. "Pith review of Position: AI Scaling: From Up to Down and Out." pith.science (2026). https://pith.science/paper/PQ3OTHZX
@misc{pith2026250201677,
author = {Pith},
title = {Pith review of: Position: AI Scaling: From Up to Down and Out},
year = {2026},
howpublished = {\url{https://pith.science/paper/PQ3OTHZX}},
note = {Machine review of arXiv:2502.01677}
}
read the original abstract
AI Scaling has traditionally been synonymous with Scaling Up, which builds larger and more powerful models. However, the growing demand for efficiency, adaptability, and collaboration across diverse applications necessitates a broader perspective. This position paper presents a holistic framework for AI scaling, encompassing Scaling Up, Scaling Down, and Scaling Out. It argues that while Scaling Up of models faces inherent bottlenecks, the future trajectory of AI scaling lies in Scaling Down and Scaling Out. These paradigms address critical technical and societal challenges, such as reducing carbon footprint, ensuring equitable access, and enhancing cross-domain collaboration. We explore transformative applications in healthcare, smart manufacturing, and content creation, demonstrating how AI Scaling can enable breakthroughs in efficiency, personalization, and global connectivity. Additionally, we highlight key challenges, including balancing model complexity with interpretability, managing resource constraints, and fostering ethical development. By synthesizing these approaches, we propose a unified roadmap that redefines the future of AI research and application, paving the way for advancements toward Artificial General Intelligence (AGI).
Figures
Reference graph
Works this paper leans on
-
[1]
A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404.14219,
-
[8]
Bert: Pre-training of deep bidirectional transformers for lan- guage understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186,
work page 2019
-
[11]
Gptq: Accurate post-training quantization for generative pre- trained transformers
10 Position: AI Scaling: From Up to Down and Out Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323,
-
[13]
Han, S., Mao, H., and Dally, W. J. Deep compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149,
-
[14]
Distilling the knowledge in a neural network
Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,
-
[16]
Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825,
-
[17]
Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088,
-
[19]
Crafting papers on machine learning
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,
2000
Show all 35 references
-
[21]
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190,
-
[23]
Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 , 2024a. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu...
-
[24]
Pruning convolutional neural networks for resource efficient inference
Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 ,
-
[26]
vattention: Dynamic memory management for serving llms without pagedattention
Prabhu, R., Nayak, A., Mohan, J., Ramjee, R., and Pan- war, A. vattention: Dynamic memory management for serving llms without pagedattention. arXiv preprint arXiv:2405.04437,
-
[27]
N., Kingsbury, B., Sindhwani, V ., Arisoy, E., and Ramabhadran, B
Sainath, T. N., Kingsbury, B., Sindhwani, V ., Arisoy, E., and Ramabhadran, B. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655–6659. IEEE,
2013
-
[28]
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,
-
[29]
Scaling llm test- time compute optimally can be more effective than scal- ing model parameters
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.03314,
-
[30]
Llama 2: Open foundation and fine- tuned chat models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 ,
-
[31]
Neural codec language models are zero-shot text to speech synthesizers
Wang, C., Chen, S., Wu, Y ., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y ., Wang, H., Li, J., et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111,
-
[32]
Z., Khabsa, M., Fang, H., and Ma, H
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768,
2006 arXiv
-
[33]
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115,
-
[34]
B., Shu, T., and Gan, C
Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y ., Tenenbaum, J. B., Shu, T., and Gan, C. Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.02485, 2023a. Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-...
-
[35]
H2o: Heavy-hitter oracle for efficient generative inference of large language models
Zhang, Z., Sheng, Y ., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y ., R´e, C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661–34710, 2023c. Zhu,...
2024
-
[1989]
Gshard: Scaling giant models with conditional computation and automatic sharding
Lepikhin, D., Lee, H., Xu, Y ., Chen, D., Firat, O., Huang, Y ., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668,
2006 arXiv
-
[1994]
B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,
2001 arXiv
-
[2006]
Gunasekar, S., Zhang, Y ., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644,
-
[2009]
Deepseek llm: Scaling open-source language models with longtermism
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954,
-
[2014]
D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,
1901
-
[2015]
J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,
-
[2016]
L., Bari, M
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Bi- derman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786,
-
[2017]
Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts lan- guage models
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y ., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts lan- guage models. arXiv preprint arXiv:2401.06066, 2024a. Dai, Y ., Dharamsi, T., Hsu, P.-L., Song, T., ...
-
[2019]
Towards artificial general intelligence (agi) in the internet of things (iot): Opportu- nities and challenges
Dou, F., Ye, J., Yuan, G., et al. Towards artificial general intelligence (agi) in the internet of things (iot): Opportu- nities and challenges. arXiv preprint arXiv:2309.07438,
-
[2020]
Accelerating large language model decoding with speculative sampling
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318,
-
[2021]
Li, Y ., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y . T. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463,
-
[2022]
A review of sparse expert models in deep learning
Fedus, W., Dean, J., and Zoph, B. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667,
-
[2023]
Gqa: Training generalized multi-query transformer models from multi-head check- points
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebron, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing , pp. 4895–4901,
2023
-
[2024]
L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.