Large-scale semantic mapping of learner agency and autonomy reveals what measurement and generative AI research overlook
Pith reviewed 2026-06-27 13:07 UTC · model grok-4.3
The pith
Semantic mapping of 8,954 definitions shows learner agency and autonomy split into task regulation, personal motivation, and sociocultural action.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Treating meaning as constituted through linguistic use, the analysis of 8,954 definitions and 2,700 scale items shows the definitional landscape of learner agency and autonomy resolves into three dimensions: regulation and control of learning (task), intrinsic motivation and internal decision-making (person), and social-relational action (sociocultural). This mapping empirically quantifies the jingle-jangle fallacy. Existing scales systematically underrepresent the sociocultural dimension, and current generative AI research concentrates on learning regulation and control, narrowing the behavioral repertoire that AI-mediated environments are designed to support.
What carries the argument
The semantic analysis pipeline that clusters extracted definitions and scale items from a 14,000-publication corpus to recover three latent dimensions.
If this is right
- Measurement instruments for agency and autonomy must be expanded to include sociocultural items if they are to capture the full construct.
- Generative AI tools for education should be evaluated on their capacity to support social-relational learner actions in addition to task regulation.
- Conceptual frameworks in educational research should treat agency and autonomy as multidimensional rather than as two distinct but interchangeable terms.
- Practice aimed at fostering learner development needs to address all three dimensions rather than regulation alone.
Where Pith is reading between the lines
- The same large-scale semantic method could be applied to other contested educational constructs to detect and measure jingle-jangle problems.
- Designers of AI learning environments could use the three-dimension map as a checklist when deciding which learner behaviors to scaffold.
Load-bearing premise
The semantic analysis pipeline recovers the true underlying conceptual dimensions without artifacts introduced by embedding models, clustering choices, or the particular publication corpus.
What would settle it
Re-running the identical semantic pipeline on an independent corpus of definitions and scale items and obtaining a different number or character of dimensions would falsify the three-dimension resolution.
Figures
read the original abstract
Learner agency and autonomy are foundational to personal development, yet a pervasive "jingle-jangle" fallacy (i.e. identical terms denoting different constructs, distinct terms denoting identical ones) has substantially hindered cumulative knowledge. Treating meaning as a phenomenon constituted through use in linguistic practice, we extracted 8,954 definitions and 2,700 scale items from over 14,000 publications, to investigate how researchers actually used learner agency and autonomy with a semantic analysis pipeline. The definitional landscape of two constructs resolves into three dimensions: regulation and control of learning (task), intrinsic motivation and internal decision-making (person), and social-relational action (sociocultural), thereby empirically quantifying the jingle-jangle fallacy. Existing scales, however, systematically underrepresent the sociocultural dimension. Critically, current generative AI research in education concentrates on learning regulation and control, narrowing the behavioral repertoire that AI-mediated learning environments are designed to cultivate. Beyond conceptual clarification, this work carries direct implications for conceptualization, measurement, and practice towards supporting the multidimensional learner agency and autonomy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript extracts 8,954 definitions and 2,700 scale items from over 14,000 publications on learner agency and autonomy. It applies a semantic analysis pipeline to these texts and reports that the definitional landscape resolves into three dimensions—regulation and control of learning (task), intrinsic motivation and internal decision-making (person), and social-relational action (sociocultural)—thereby quantifying the jingle-jangle fallacy. Existing scales are said to underrepresent the sociocultural dimension, while generative AI research in education concentrates on the task dimension, with implications for conceptualization, measurement, and AI-mediated learning environments.
Significance. If the semantic pipeline is shown to be robust, the work supplies a large-scale empirical basis for a multidimensional view of two central constructs in educational research. The scale of the corpus and the downstream claims about measurement gaps and AI design priorities would constitute a substantive contribution to clarifying construct validity and guiding future instrument development and technology applications.
major comments (1)
- [Methods (semantic analysis pipeline)] The central claim that definitions and scale items resolve into three stable dimensions (and thereby quantify the jingle-jangle fallacy) rests entirely on the semantic analysis pipeline. The manuscript supplies no information on the embedding model, distance metric, clustering algorithm or hyperparameters, corpus filtering criteria, or any validation (human agreement on held-out data, cluster stability via permutation tests, or sensitivity to alternative embeddings). Without these details, it cannot be determined whether the reported three-way split reflects conceptual structure or pipeline artifacts, directly affecting the claims about scale underrepresentation and GenAI focus.
Simulated Author's Rebuttal
We thank the referee for the careful reading and for highlighting the need for greater transparency in our semantic analysis pipeline. This is a fair and substantive point. We address it directly below and will incorporate the requested details into a revised manuscript.
read point-by-point responses
-
Referee: The central claim that definitions and scale items resolve into three stable dimensions (and thereby quantify the jingle-jangle fallacy) rests entirely on the semantic analysis pipeline. The manuscript supplies no information on the embedding model, distance metric, clustering algorithm or hyperparameters, corpus filtering criteria, or any validation (human agreement on held-out data, cluster stability via permutation tests, or sensitivity to alternative embeddings). Without these details, it cannot be determined whether the reported three-way split reflects conceptual structure or pipeline artifacts, directly affecting the claims about scale underrepresentation and GenAI focus.
Authors: We agree that the current manuscript does not supply adequate methodological detail on the semantic pipeline. In the revised version we will expand the Methods section to report: (1) the exact embedding model and version, (2) the distance metric, (3) the clustering algorithm together with all hyperparameters and the procedure used to select them, (4) explicit corpus filtering criteria, and (5) validation results including human agreement on a held-out sample, cluster stability checks, and sensitivity analyses across alternative embeddings. These additions will allow readers to evaluate whether the three-dimensional structure is robust or artifactual and will directly support the downstream claims about measurement gaps and generative-AI focus. revision: yes
Circularity Check
No circularity: dimensions derived from external corpus analysis
full rationale
The paper extracts 8,954 definitions and 2,700 scale items from an external corpus of over 14,000 publications and applies a semantic analysis pipeline to identify three dimensions. This is an empirical process on independent data with no equations, fitted parameters, or self-citations that reduce the reported dimensions to quantities defined by the authors' own prior work. No self-definitional steps, uniqueness theorems, or ansatzes smuggled via citation appear in the derivation chain. The result is self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Meaning of scientific terms is constituted through their use in linguistic practice
- domain assumption Semantic embeddings and clustering recover stable, interpretable dimensions from definitional text
Reference graph
Works this paper leans on
-
[1]
Abdalgader, K., Matroud, A. A., & Hossin , K. (2024). Experimental study on short - text clustering using transformer - based semantic similarity measure. PeerJ Computer Science , 10 , e2078. https://doi.org/10.7717/peerj - cs.2078 Achuthan, K. (2025). Artificial intelligence and learner autonomy: A meta - analysis of self - regulated and self - directed ...
-
[2]
G., Muldowney, S., Eichstaedt, J
https://doi.org/10.1146/annurev.anthro.30.1.109 Bai, H., V oelkel, J. G., Muldowney, S., Eichstaedt, J. C., & Willer, R. (2025). LLM - generated messages can persuade humans on policy issues. Nature Communications , 16 (1),
-
[5]
M., Gebru, T., McMillan - Major, A., & Shmitchell, S
https://doi.org/10.1146/annurev.psych.52.1.1 Bender, E. M., Gebru, T., McMillan - Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM C onference on F airness, A ccountability, and T ransparency (pp. 610 – 623). Association for Computing Machinery. https://doi.org/10.114...
-
[6]
https://doi.org/10.1017/S0261444806003958 Biesta, G., & Tedder, M. (2007). Agency and learning in the lifecourse: Towards an ecological perspective. Studies in the Education of Adults , 39 (2), 132 –
-
[7]
https://doi.org/10.1080/02660830.2007.11661545 Boleda, G. (2020). Distributional semantics and linguistic theory. Annual Review of Linguistics , 6 (1), 213 –
-
[8]
https://doi.org/10.1146/annurev - linguistics - 011619 - 030303 Brewer, M. B. , & Crano, W. D. (20 24 ). Research design and issues of validity. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology ( pp. 115 – 135 ). Cambridge University Press. https://doi.org/10.1017/9781009170123.007 Code, J. (2020). Agenc...
-
[9]
https://doi.org/10.3389/feduc.2020.00019 Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin , 52 (4), 281 –
-
[10]
https://doi.org/10.1037/h0040957 Chirkov, V ., Ryan, R. M., Kim, Y ., & Kaplan, U. (2003). Differentiating autonomy from individualism and independence: a self - determination theory perspective on internalization of cultural orientations and well - being. Journal of P ersonality and S ocial P sychology , 84 (1) ,
-
[12]
https://doi.org/10.1038/s41467 - 024 - 45563 - x Deci, E. L., & Ryan, R. M. (1987). The support of autonomy and the control of behavior. Journal of Personality and Social Psychology , 53 (6), 1024 –
-
[13]
https://doi.org/10.1037/0022 - 3514.53.6.1024 Deci, E. L., & Ryan, R. M. (2000). The "what" and "why" of goal pursuits: Human needs and the self - determination of behavior. Psychological Inquiry , 11 (4), 227 –
-
[14]
https://doi.org/10.1207/S15327965PLI1104_01 Dorison, C. A., & Charlesworth, T. E . (2025). What Is Rationality, Whom Is It Ascribed To, and Why Does It Matter? Evidence From Internet Text for 66 Social Groups and 101 Occupations. Psychological Science , 36 (9), 713 –
-
[15]
https://doi.org/10.1177/09567976251362120 Dowell, N. M., Nixon, T. M., & Graesser, A. C. (2019). Group communication analysis: A computational linguistics approach for detecting sociocognitive roles in multiparty interactions. Behavior R esearch M ethods , 51 (3), 1007 –
-
[17]
https://doi.org/10.1038/s41586 - 024 - 07522 - w Fenwick, T., Edwards, R., & Sawchuk, P . ( 201 5 ). Emerging approaches to educational research: Tracing the sociomaterial . Routledge. https://doi.org/10.4324/9780203817582 Gerring, J. (1999). What makes a concept good? A criterial framework for understanding concept formation in the social sciences. Polit...
-
[18]
https://doi.org/10.2307/3235246 Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text - annotation tasks. Proceedings of the National Academy of Sciences , 120 (30), e2305016120. https://doi.org/10.1073/pnas.2305016120 Goertz, G. (2006). Social science concepts: A user's guide . Princeton University Press. https://doi.o...
-
[19]
https://doi.org/10.2307/2095141 Gonzalez, O., MacKinnon, D. P., & Muniz, F. B. (2021). Extrinsic convergent validity evidence to prevent jingle and jangle fallacies. Multivariate Behavioral Research , 56 (1), 3 –
-
[20]
https://doi.org/10.1080/00273171.2019.1707061 Greeno, J. G. (1998). The situativity of knowing, learning, and research. American Psychologist , 53 (1), 5 –
-
[21]
https://doi.org/10.1037/0003 - 066X.53.1.5 Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class - based TF - IDF procedure. arXiv . https://doi.org/10.48550/arXiv.2203.05794 Hang, Y ., Peng, Y ., & Guo, J. (2025). An interactive journey with multiple coding of self: exploring graduate students’ self - formation during academic socializati...
work page internal anchor Pith review Pith/arXiv arXiv doi:10.1037/0003 2022
-
[22]
https://doi.org/10.1007/s10734 - 025 - 01595 - w Harris, Z. S. (1954). Distributional Structure. W ord , 10 (2 – 3), 146 –
-
[23]
22 https://doi.org/10.1080/00437956.1954.11659520 Holec, H. (1981). Autonomy and foreign language learning . Oxford: Pergamon Press . Huang, Z., Long, Y ., Peng, K., & Tong, S. (2025). An embedding - based semantic analysis approach: A preliminary study on redundancy detection in psychological concepts operationalized by scales. Journal of Intelligence , 13 (1),
-
[24]
https://doi.org/10.3390/jintelligence13010011 Hussain, Z., Binz, M., Mata, R., & Wulff, D. U. (2024). A tutorial on open - source large language models for behavioral science. Behavior Research Methods , 56 (8), 8214 –
-
[25]
https://doi.org/10.3758/s13428 - 024 - 02455 - 8 Jacobs, J. A. (2014). In defense of disciplines: Interdisciplinarity and specialization in the research university . University of C hicago Press. Krakowski, S. (2025). Human - AI agency in the age of generative AI. Information and Organization , 35 (1), 100560. https://doi.org/10.1016/j.infoandorg.2025.100...
-
[26]
Y ., & Reigeluth, C
Lin, C. Y ., & Reigeluth, C. M. (2019). Scaffolding learner autonomy in a wiki‐supported knowledge building community and its implications for mindset change. British Journal of Educational Technology , 50 (5), 2667 –
2019
-
[27]
https://doi.org/10.1111/bjet.12713 Lincoln, Y . S., & Guba, E. G. (1985). Naturalistic inquiry . Sage Publications. Little, D. (1991). Learner autonomy 1: Definitions, issues and problems . Authentik. Little, D. (1995). Learning as dialogue: The dependence of learner autonomy on teacher autonomy. System , 23(2), 175 –
-
[28]
https://doi.org/10.1016/0346 - 251X(95)00006 - 6 Littlewood, W. (1996). Autonomy: An anatomy and a framework. System , 24 (4), 427 –
-
[29]
https://doi.org/10.1016/S0346 - 251X(96)00039 - 5 Manyukhina, Y ., & Wyse, D. (2019). Learner agency and the curriculum: a critical realist perspective . The Curriculum Journal , 30 (3), 223 – 243 . https://doi.org/10.1080/09585176.2019.1599973 Marsh, H. W. (1994). Sport motivation orientations: Beware of jingle – jangle fallacies. Journal of Sport & Exer...
-
[30]
https://doi.org/10.1123/jsep.16.4.365 Matz, S. C., Teeny, J. D., V aid, S. S., Peters, H., Harari, G. M., & Cerf, M. (2024). The potential of generative AI for personalized persuasion at scale. Scientific Reports , 14 (1),
-
[31]
https://doi.org/10.1038/s41598 - 024 - 53755 - 0 Mercer, S. (2011). Understanding learner agency as a complex dynamic system. System , 39 (4), 427 –
-
[32]
https://doi.org/10.1016/j.system.2011.08.001 Moore, M. G. (1972). Learner autonomy: The second dimension of independent learning. Convergence , 5 (2), 76 –
-
[33]
Murray, G. (2014). Exploring the social dimensions of autonomy in language learning. In Social D imensions of A utonomy in L anguage L earning (pp. 3 – 11). London: Palgrave Macmillan UK. https://doi.org/10.1057/9781137290243_1 Muthusami, R., Mani Kandan, N., Saritha, K., Narenthiran, B., Nagaprasad, N., & Ramaswamy, K. (2024). Investigating topic modelin...
-
[34]
https://www.oecd.org/en/about/projects/future - of - education - and - skills - 2030.html OECD
A series of concept notes. https://www.oecd.org/en/about/projects/future - of - education - and - skills - 2030.html OECD . ( 2025 ) . Education for Human Flourishing . A Conceptual Framework . https://www.oecd.org/en/publications/education - for - human - flourishing_73d7cb96 - en.html Rathje, S., Mirea, D. M., Sucholutsky, I., Marjieh, R., Robertson, C....
-
[35]
https://doi.org/10.1016/j.cedpsych.2011.05.002 Rousseeuw, P. J. (1987). Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of C omputational and A pplied M athematics , 20 , 53 –
-
[36]
https://doi.org/10.1016/0377 - 0427(87)90125 - 7 Salvi, F., Horta Ribeiro, M., Gallotti, R., & West, R. (2025). On the conversational persuasiveness of GPT -
-
[38]
https://doi.org/10.1146/annurev.clinpsy.3.022806.091415 Searle, J. R. (1969). Speech Acts: An Essay in the Philosophy of Language . Cambridge University Press. Teng, M. F. (2019). Autonomy, agency, and identity in teaching and learning English as a foreign language . Springer. https://doi.org/10.1007/978 - 981 - 13 - 0728 - 7 UNESCO. (2021). AI and educat...
-
[39]
https://doi.org/10.1007/s10964 - 012 - 9847 - 7 V oppel, A., Ciampelli, S., Kircher, T., Liddle, P. F., Massuda, R., Stein, F., ... & Palaniyappan, L. (2025). Analysis of conceptual overlap among formal thought disorder rating scales in psychosis: a systematic semantic synthesis. Schizophrenia , 12 , 1 – 9 . https://doi.org/10.1038/s41537 - 025 - 00712 - ...
-
[40]
https://doi.org/10.1177/09637214251382083 Wulff, D. U., & Mata, R. (2025b). Semantic embeddings reveal and address taxonomic 24 incommensurability in psychological measurement. Nature Human Behaviour , 9 (5), 944 –
-
[41]
https://doi.org/10.1038/s41562 - 024 - 02089 - y Yang, L., Lee, S., & Oldac, Y . I. (2023). Agency and student development in higher education: a cross - cultural and cross - disciplinary exploration. In Student Agency and Self - Formation in Higher Education (pp. 67 – 87). Cham: Springer Nature Switzerland. Zai, F., & Zhou, X. (2026). The impact of AI - ...
-
[42]
https://doi.org/10.3390/bs16030379 25 Supplementary Information for Large - scale semantic mapping of learner agency and autonomy reveals what measurement and generative AI research overlook Literature C orpus C onstruction To assemble the literature corpus, we conducted systematic searches in Web of Science and Scopus using database - specific b oolean q...
-
[43]
is define d as,
APIs , with Zotero ’ s full - text retrieval function used to recover PDFs not obtained through these APIs (Corporation for Digital Scholarship, 2024). For non - open - access records, we first checked whether the DOI was associated with text and data mining authorization. When such authorization was confirmed, we retrieved the content through publisher -...
2024
-
[44]
a trainee’s ability to complete a procedure independently, with 31 minimal attending supervision and participation
and was therefore treated as an outlier cluster, leaving three substantive clusters in the retained solution. Inspection of the definitions in this small excluded cluster confirmed that they reflected outl ier content rather than the main semantic structure of autonomy - and agency - related definitions. This configuration achieved the highest silhouette ...
1981
-
[45]
autonom*
was cited by 84 articles in our corpus . We then resolved DOIs for the extracted citations via the Crossref and OpenAlex APIs and deduplicated the resulting records to obtain 939 unique candidate scale - source articles. To improve coverage of recently developed instruments that had not yet been ci ted within our literature corpus, we conducted supplement...
2025
-
[46]
generative AI ,
4 %), scale - based quantitative operationalization ( n = 276, 12.9%), experimental or intervention - based operationalization ( n = 179, 8.4%), and behavioral quantitative operationalization ( n = 161, 7.5%). N ote that some articles addres sed both constructs, so the two subsets overlap and their counts exceed the total corpus size . Furthermore, the th...
2022
-
[47]
L., Van der Kaap - Deeder, J.,
Chen, B., Vansteenkiste, M., Beyers, W., Boone, L., Deci, E. L., Van der Kaap - Deeder, J., ... & Verstuyf, J. (2015). Basic psychological need satisfaction, need frustration, and need strength across four cultures. Motivation and E motion , 39 (2), 216 –
2015
-
[48]
Corporation for Digital Scholarship. (2024). Zotero (Version
2024
-
[49]
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
[Software]. https://www.zotero.org Crossref. (2024). Crossref REST API [API service]. https://www.crossref.org/documentation/retrieve - metadata/rest - api/ Elsevier. (2024). Elsevier text and data mining (TDM) APIs [API service]. https://www.elsevier.com/about/open - science/research - data/text - and - data - mining Glm , T., Zeng, A., Xu, B., Wang, B.,...
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.12793 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.