pith. sign in

arxiv: 2604.22959 · v1 · submitted 2026-04-24 · 💻 cs.HC

Understanding teens' self-beliefs when learning to construct and deconstruct AI/ML systems: Developing a survey instrument

Luis Morales-Navarro , Deborah Fields , Michael T. Giang , Daniel J. Noh , Yasmin B. Kafai , Dana\'e Metaxa This is my paper

Pith reviewed 2026-05-08 10:30 UTC · model grok-4.3

classification 💻 cs.HC
keywords AI literacysurvey validationself-beliefsteenagersconstructiondeconstructiondesign justiceconfirmatory factor analysis
0
0 comments X

The pith

A new survey instrument measures teenagers' self-beliefs about constructing and deconstructing AI systems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops and tests a survey to capture how teens view their abilities in building AI systems creatively and solving problems, as well as auditing those systems for issues. It also includes beliefs about design justice and the worth of learning AI. Researchers gave the survey to 124 teens and used confirmatory factor analysis to verify that the questions group into six distinct factors. They discovered that beliefs in design justice are closely tied to self-beliefs in problem-solving, auditing, and creative expression. This tool can help educators understand what shapes teens' engagement with AI literacy activities that involve both making and critiquing technology.

Core claim

The central discovery is the development of a validated survey instrument consisting of six constructs: creative expression self-beliefs and problem-solving self-beliefs for construction activities, auditing self-efficacy and fascination with auditing for deconstruction activities, plus design justice beliefs and the value of learning about AI/ML. Administered to 124 teenagers, confirmatory factor analysis confirmed the six-factor structure, and analysis showed strong correlations between design justice beliefs and the constructs of problem-solving, auditing self-efficacy, and creative expression.

What carries the argument

The six-factor survey instrument, validated using confirmatory factor analysis, which measures specific self-beliefs related to AI construction, deconstruction, design justice, and learning value.

If this is right

  • The instrument enables assessment of teens' self-beliefs in educational AI programs.
  • Strong correlations indicate that design justice beliefs may influence or relate to confidence in creative and auditing tasks.
  • Educators can use the survey to identify areas for supporting teens' engagement in building and auditing AI.
  • The validated structure supports further research on how these beliefs affect participation in computational empowerment activities.
  • Findings highlight the interconnectedness of justice-oriented beliefs with technical self-efficacy in AI learning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the instrument proves reliable over time, it could track changes in self-beliefs as teens gain more experience with AI deconstruction tasks.
  • Emphasizing design justice in AI education might boost related self-beliefs, leading to more equitable learning outcomes.
  • Similar surveys could be developed for other age groups or specific AI topics to broaden understanding of youth empowerment.
  • The correlations suggest potential causal links worth testing in intervention studies.

Load-bearing premise

The survey questions accurately reflect the intended self-belief constructs and the sample of 124 teens is sufficient for confirmatory factor analysis to reliably validate the six-factor structure.

What would settle it

Administering the survey to another group of teenagers and finding that the confirmatory factor analysis does not support the six-factor model or that the correlations with design justice beliefs do not hold would challenge the instrument's validity.

read the original abstract

Despite growing calls to foster AI literacy, there are few available survey instruments designed for children and youth that study computational empowerment alongside construction and deconstruction activities. In such activities, learners' beliefs about their abilities and attributes can impact their engagement. In this paper, we introduce and validate a survey instrument with constructs related to construction (creative expression and problem-solving self-beliefs) and deconstruction (auditing self-efficacy and fascination with auditing), along with more general self-beliefs related to design justice and the value of learning about AI/ML. We administered the instrument to 124 teenagers and assessed the six-factor structure of the instrument using confirmatory factor analysis. In addition to confirming the structure, we found that design justice beliefs strongly correlated with problem-solving, auditing self-efficacy, and creative expression.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces and validates a new survey instrument measuring six self-belief constructs among teenagers in AI/ML education: creative expression and problem-solving self-beliefs (construction), auditing self-efficacy and fascination with auditing (deconstruction), plus design justice and value of learning AI/ML. The instrument was administered to 124 teens; confirmatory factor analysis (CFA) is reported to confirm the six-factor structure, with additional findings of strong correlations between design justice beliefs and problem-solving, auditing self-efficacy, and creative expression.

Significance. If the validation holds, the instrument addresses a clear gap in youth-focused measures of computational empowerment and AI literacy self-beliefs, enabling future studies on how these beliefs influence engagement with construction and auditing activities. The reported correlations between design justice and other constructs could inform the design of equitable AI education interventions.

major comments (3)
  1. [Methods/Results] Methods and Results sections: No model fit indices (CFI, RMSEA, SRMR, chi-square), factor loadings, or reliability coefficients (e.g., Cronbach's alpha, omega) are reported for the CFA, making it impossible to evaluate whether the six-factor structure is adequately confirmed or whether items load cleanly on intended factors.
  2. [Methods] Sample and validation procedure: With n=124 and six factors (likely 20+ observed variables), the CFA is performed on the development sample without a separate hold-out or independent validation sample, and without reported power analysis or handling of sample limitations; this risks unstable parameter estimates and capitalizing on chance, undermining the claim that the scales are ready for educational use.
  3. [Methods] Item development: Details on how the initial item pool was generated, refined, or mapped to the six constructs (e.g., via expert review, pilot testing, or EFA) are absent, which is load-bearing for construct validity claims.
minor comments (2)
  1. [Abstract] Abstract and introduction: The claim that the structure was 'confirmed' should be qualified with the specific fit criteria used, as standard CFA reporting requires explicit thresholds.
  2. [Discussion] Discussion: The interpretation of correlations between design justice and other factors would benefit from effect-size benchmarks or comparison to prior self-belief scales in related domains.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback on our manuscript. The comments highlight important aspects of reporting and methodological transparency that we will address to strengthen the paper. We provide point-by-point responses below.

read point-by-point responses

  1. Referee: [Methods/Results] Methods and Results sections: No model fit indices (CFI, RMSEA, SRMR, chi-square), factor loadings, or reliability coefficients (e.g., Cronbach's alpha, omega) are reported for the CFA, making it impossible to evaluate whether the six-factor structure is adequately confirmed or whether items load cleanly on intended factors.

    Authors: We agree that these statistics are essential for readers to evaluate the CFA. In the revised manuscript, we will report the full set of model fit indices (including CFI, RMSEA, SRMR, and chi-square with degrees of freedom), standardized factor loadings for all items, and reliability coefficients (Cronbach's alpha and McDonald's omega) for each of the six constructs. This addition will directly address the concern and allow assessment of whether the six-factor structure is adequately supported. revision: yes

  2. Referee: [Methods] Sample and validation procedure: With n=124 and six factors (likely 20+ observed variables), the CFA is performed on the development sample without a separate hold-out or independent validation sample, and without reported power analysis or handling of sample limitations; this risks unstable parameter estimates and capitalizing on chance, undermining the claim that the scales are ready for educational use.

    Authors: We acknowledge the sample size limitation for CFA. With 124 participants, the ratio of observations to parameters is modest, and performing CFA on the same sample used for item refinement carries risks of overfitting. We will revise the Methods and Discussion sections to explicitly note this limitation, cite relevant guidelines on minimum sample sizes for CFA, and describe the findings as preliminary validation rather than definitive. We will also add that future studies should include independent validation samples and cross-validation. A formal a priori power analysis was not conducted, but we will discuss post-hoc considerations of statistical power. These changes will temper the claims appropriately while preserving the instrument's value as an initial contribution. revision: partial

  3. Referee: [Methods] Item development: Details on how the initial item pool was generated, refined, or mapped to the six constructs (e.g., via expert review, pilot testing, or EFA) are absent, which is load-bearing for construct validity claims.

    Authors: We regret the omission of these details. The initial item pool was generated by adapting validated scales from related domains (self-beliefs in STEM, computational thinking, and AI literacy) and creating new items grounded in the theoretical constructs of construction, deconstruction, design justice, and learning value. Items were reviewed by three experts in AI education and adolescent development for content validity and age-appropriateness, then piloted with 12 teens for clarity and comprehension, leading to minor wording revisions. No exploratory factor analysis was performed because we had strong a priori theoretical expectations for the six-factor structure. We will expand the Methods section with a dedicated subsection detailing this process, including the number of initial items, expert review criteria, pilot feedback summary, and the mapping of items to constructs. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical validation rests on new data and standard CFA

full rationale

The paper introduces novel constructs for teens' self-beliefs in AI/ML construction and deconstruction, creates survey items, administers them to an independent sample of 124 teenagers, and applies confirmatory factor analysis to assess the six-factor structure. This chain relies on fresh empirical observations and conventional statistical methods without any reduction of results to prior fitted parameters, self-definitions, or self-citation chains. No equations or claims equate outputs to inputs by construction; the reported correlations emerge from the data rather than being presupposed. The derivation is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work applies standard psychometric validation techniques without introducing new free parameters, ad-hoc axioms, or invented entities beyond established survey constructs.

axioms (1)
  • standard math Confirmatory factor analysis assumptions hold, including appropriate sample size and data distribution for the six-factor model.
    The paper uses CFA to confirm the instrument structure on 124 participants.

pith-pipeline@v0.9.0 · 5457 in / 1209 out tokens · 33635 ms · 2026-05-08T10:30:58.325604+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages

  1. [1]

    1997.Self-efficacy: The exercise of control

    Albert Bandura. 1997.Self-efficacy: The exercise of control. Vol. 11. Freeman

    work page 1997

  2. [2]

    Karl-Emil Kjær Bilstrup. 2023. Embodied computational empowerment: designing educational tools for critical reflection on computational technologies. (2023)

    work page 2023

  3. [3]

    Lorena Casal-Otero, Alejandro Catala, Carmen Fernández-Morante, Maria Taboada, Beatriz Cebreiro, and Senén Barro. 2023. AI literacy in K-12: a systematic literature review.International Journal of STEM Education10, 1 (2023), 29

    work page 2023

  4. [4]

    Cannady, Christian Schunn, and Rena Dorph

    Ying-Fang Chen, Matthew A. Cannady, Christian Schunn, and Rena Dorph. 2017.Measures Technical Brief: Competency Beliefs in STEM. Technical Report. The Lawrence Hall of Science, University of Califor- nia, Berkeley. http://www.activationlab.org/wp-content/uploads/2017/06/ CompetencyBeliefs_STEM-Report_20170403.pdf Retrieved from Activation Lab

    work page 2017

  5. [5]

    Chung, M

    J. Chung, M. A. Cannady, C. Schunn, R. Dorph, and M. Bathgate. 2016.Measures Technical Brief: Valuing Science. Technical Report. The Learning Activation Lab. http://www.activationlab.org/wp-content/uploads/2016/02/Values-Report- 3.2-20160331.pdf Retrieved from Activation Lab website

    work page 2016

  6. [6]

    Sasha Costanza-Chock. 2018. Design justice, AI, and escape from the matrix of domination.Journal of Design and Science3, 5 (2018), 1–14

    work page 2018

  7. [7]

    2020.Design justice: Community-led practices to build the worlds we need

    Sasha Costanza-Chock. 2020.Design justice: Community-led practices to build the worlds we need. MIT press

    work page 2020

  8. [8]

    Adrienne Decker and Monica M McGill. 2019. A topical review of evaluation instruments for computing education. InProceedings of the 50th ACM Technical Symposium on Computer Science Education. 558–564

    work page 2019

  9. [9]

    Christian Dindler, Rachel Smith, and Ole Sejer Iversen. 2020. Computational empowerment: participatory design in education.CoDesign16, 1 (2020), 66–80

    work page 2020

  10. [10]

    Rena Dorph, Meghan E Bathgate, Christian D Schunn, and Matthew A Cannady

    work page

  11. [11]

    When I grow up: The relationship of science learning activation to STEM career preferences.International Journal of Science Education40, 9 (2018), 1034– 1057

    work page 2018

  12. [12]

    Stefania Druga and Amy J Ko. 2021. How do children’s perceptions of machine intelligence change when training and coding smart programs?. InProceedings of the 20th annual ACM interaction design and children conference. 49–61

    work page 2021

  13. [13]

    Joanna Goode, Gail Chapman, and Jane Margolis. 2012. Beyond curriculum: the exploring computer science program.ACM Inroads3, 2 (June 2012), 47–53. doi:10.1145/2189835.2189851

    work page doi:10.1145/2189835.2189851 2012

  14. [14]

    Idit Harel and Seymour Papert. 1990. Software design as a learning environment. Interactive learning environments1, 1 (1990), 1–32

    work page 1990

  15. [15]

    1999.The construction of the self: A developmental perspective

    Susan Harter. 1999.The construction of the self: A developmental perspective. Guilford press

    work page 1999

  16. [16]

    Ole Sejer Iversen, Rachel Charlotte Smith, and Christian Dindler. 2018. From computational thinking to computational empowerment: a 21st century PD agenda. InProceedings of the 15th Participatory Design Conference: Full Papers - Volume 1(Hasselt and Genk, Belgium)(PDC ’18). Association for Computing Machinery, New York, NY, USA, Article 7, 11 pages. doi:1...

    work page doi:10.1145/3210586.3210592 2018

  17. [17]

    Juho Kahila, Henriikka Vartiainen, Nicolas Pope, and Matti Tedre. 2026. Engaging young minds in AI through co-designing and making their own AI-Driven Apps in the classroom. InSocial robots and artificial intelligence in education: Integrating AI in K-12 and higher education. Springer, 187–210

    work page 2026

  18. [18]

    Tomáš Lintner. 2024. A systematic review of AI literacy scales.npj Science of Learning9, 1 (2024), 50

    work page 2024

  19. [19]

    Alex Lishinski and Aman Yadav. 2019. 28 motivation, attitudes, and dispositions

    work page 2019

  20. [20]

    Pekka Mertala, Janne Fagerlund, and Oscar Calderon. 2022. Finnish 5th and 6th grade students’ pre-instructional conceptions of artificial intelligence (AI) and their implications for AI literacy education.Computers and Education: Artificial Intelligence3 (2022), 100095

    work page 2022

  21. [21]

    Luis Morales-Navarro, Michelle A Gan, Evelyn Yu, Lauren Vogelstein, Yasmin Kafai, and Danaé Metaxa. 2025. Learning AI Auditing: A Case Study of Teenagers Auditing a Generative AI Model.Proceedings of the ACM on Human-Computer Interaction9, 7 (2025), 1–29

    work page 2025

  22. [22]

    Luis Morales-Navarro, Michael T Giang, Deborah A Fields, and Yasmin B Kafai

    work page

  23. [23]

    Computer Science Education34, 3 (2024), 387–413

    Connecting beliefs, mindsets, anxiety and self-efficacy in computer science learning: An instrument for capturing secondary school students’ self-beliefs. Computer Science Education34, 3 (2024), 387–413

    work page 2024

  24. [24]

    Luis Morales-Navarro, Daniel J Noh, and Yasmin B Kafai. 2025. High school students building babyGPTs: Engaging in data practices and addressing ethical issues through the construction of generative language models.International Journal of Child-Computer Interaction(2025), 100769

    work page 2025

  25. [25]

    Davy Tsz Kit Ng, Wenjie Wu, Jac Ka Lok Leung, Thomas Kin Fung Chiu, and Samuel Kai Wah Chu. 2024. Design and validation of the AI literacy questionnaire: The affective, behavioural, cognitive and ethical approach.British Journal of Educational Technology55, 3 (2024), 1082–1104

    work page 2024

  26. [26]

    You Can Actually Do Something

    Daniel J. Noh, Deborah A. Fields, Yasmin B. Kafai, and Danaé Metaxa. 2026. "You Can Actually Do Something”: Shifts in High School Computer Science Teachers’ Conceptions of AI/ML Systems and Algorithmic Justice. arXiv:2602.16123 [cs.HC] https://arxiv.org/abs/2602.16123

    work page arXiv 2026

  27. [27]

    Noh, Deborah A

    Daniel J. Noh, Deborah A. Fields, Luis Morales-Navarro, Alexis M. Cabrera-Sutch, Yasmin B. Kafai, and Danaé Metaxa. 2025. Youth as Advisors in Participatory Design: Situating Teens’ Expertise in Everyday Algorithm Auditing with Teachers and Researchers. InProceedings of the 24th Interaction Design and Children (IDC ’25). Association for Computing Machiner...

    work page arXiv 2025

  28. [28]

    2025.Empowering Learners for the Age of AI: An AI Literacy Frame- work for Primary and Secondary Education

    OECD. 2025.Empowering Learners for the Age of AI: An AI Literacy Frame- work for Primary and Secondary Education. Technical Report. Organisation for Economic Co-operation and Development (OECD), Paris, France. https: //ailiteracyframework.org Accessed: 2026-01-11

    work page 2025

  29. [29]

    Alannah Oleson, Brett Wortzman, and Amy J Ko. 2020. On the role of design in K-12 computing education.ACM Transactions on Computing Education (TOCE) 21, 1 (2020), 1–34

    work page 2020

  30. [30]

    Aadarsh Padiyath, Casey Fiesler, Mark Guzdial, and Barbara Ericson. 2025. Vali- dation of the Critical Reflection and Agency in Computing Index: Do Computing Ethics Courses Make a Difference?. InProceedings of the 2025 ACM Conference on International Computing Education Research V.1 (ICER ’25). Association for Computing Machinery, New York, NY, USA, 66–81...

    work page doi:10.1145/3702652.3744208 2025

  31. [31]

    Yves Rosseel. 2012. lavaan: An R package for structural equation modeling. Journal of statistical software48 (2012), 1–36

    work page 2012

  32. [32]

    Jean Salac, Alannah Oleson, Lena Armstrong, Audrey Le Meur, and Amy J Ko

    work page

  33. [33]

    InProceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1

    Funds of knowledge used by adolescents of color in scaffolded sensemak- ing around algorithmic fairness. InProceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1. 191–205

    work page 2023

  34. [34]

    Michael James Scott and Gheorghita Ghinea. 2014. Measuring enrichment: the assembly and validation of an instrument to assess student self-beliefs in CS1. In Proceedings of the tenth annual conference on International computing education research. 123–130

    work page 2014

  35. [35]

    Rachel Charlotte Smith, Marie-Monique Schaper, Mariana Aki Tamashiro, Maarten Van Mechelen, Marianne Graves Petersen, and Ole Sejer Iversen. 2023. A research agenda for computational empowerment for emerging technology education.International Journal of Child-Computer Interaction38 (2023), 100616

    work page 2023

  36. [36]

    Jaemarie Solyst, Cindy Peng, Wesley Hanwen Deng, Praneetha Pratapa, Amy Ogan, Jessica Hammer, Jason Hong, and Motahhare Eslami. 2025. Investigat- ing Youth AI Auditing. InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. 2098–2111

    work page 2025

  37. [37]

    Matti Tedre, Peter Denning, and Tapani Toivonen. 2021. CT 2.0. InProceedings of the 21st Koli Calling International Conference on Computing Education Research. 1–8

    work page 2021

  38. [38]

    Jeffrey C Valentine, David L DuBois, and Harris Cooper. 2004. The relation between self-beliefs and academic achievement: A meta-analytic review.Educa- tional psychologist39, 2 (2004), 111–133

    work page 2004

  39. [39]

    1974.The self-concept: Theory and research on selected topics

    Ruth C Wylie. 1974.The self-concept: Theory and research on selected topics. Vol. 2. U of Nebraska Press

    work page 1974

  40. [40]

    Helen Zhang, Anthony Perry, and Irene Lee. 2025. Developing and validating the artificial intelligence literacy concept inventory: An instrument to assess artificial intelligence literacy among middle school students.International Journal of Artificial Intelligence in Education35, 1 (2025), 398–438

    work page 2025