Pith. sign in

REVIEW 3 major objections 215 references

Strong code-solving models still fail at lowering cognitive demand for learners.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 13:48 UTC pith:UO2SFACE

load-bearing objection Clean measurement paper: matched Qwen pair shows models raise Bloom demand easily and lower it poorly; judge-on-mutations is the real soft spot, not the design. the 3 major comments →

arxiv 2607.08009 v1 pith:UO2SFACE submitted 2026-07-09 cs.CL cs.CY

From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs

Yi Zhang , Julia Rayz This is my paper
classification cs.CL cs.CY
keywords educational controlBloom's Taxonomylarge language modelsprogramming educationtask mutationcognitive demandrepresentation probingdifficulty control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the skill of solving programming tasks is not the same as the skill of adapting those tasks for education. Educational control means keeping a task's teaching purpose intact while deliberately shifting how much cognitive work it demands. The authors measure that control with revised Bloom's Taxonomy as a six-level scale of cognitive demand, using two kinds of rewrite instructions: make the task harder or easier, and aim for higher or lower Bloom levels. Across 2,520 programming tasks and a matched general versus code-specialized model pair, both models reliably push tasks upward in cognitive demand but often fail to push them downward. Lexical mutation patterns and layer-wise representation probes help explain how the models rewrite tasks and where the harder-versus-easier and higher-versus-lower contrasts become separable inside the networks. The practical claim is that execution strength alone is not enough for intelligent tutoring: systems must also be able to simplify tasks for novices, not only escalate them.

Core claim

Across 2,520 programming tasks from three benchmarks, a matched general and code-specialized model pair both raise Bloom-level cognitive demand under Harder and Higher prompts, yet struggle to lower it under Easier and Lower prompts. Target-zone accuracy for Higher levels is high, while accuracy for Lower levels stays below about 30 percent overall, and general Easier requests still produce positive average cognitive shifts. Within this pair, the general model shows clearer middle-layer linear separability for both instruction contrasts, while the coder model is weaker on difficulty contrasts and peaks deeper for Bloom contrasts. Strong execution performance therefore does not automatically

What carries the argument

A Bloom-aligned educational-control framework that rewrites each task under matched Harder/Easier and Higher/Lower prompts, then scores outcomes with Observed Cognitive Shift (average movement on the 1-6 Bloom scale) and Target Zone Accuracy (whether the rewrite lands in the intended Bloom zone), with supporting semantic-delta clustering of surface mutations and layer-wise Fisher's Discriminant Ratio probes of residual-stream separability.

Load-bearing premise

The main results rest on an automated judge's Bloom labels for rewritten tasks, validated mainly by strong agreement with humans on original tasks rather than a full human audit of the mutated tasks themselves.

What would settle it

If independent human raters labeled a large sample of the same Harder/Easier and Higher/Lower rewrites and found that the models do in fact land in Remember/Understand zones about as often as Higher zones, or that Easier rewrites systematically lower Bloom level rather than raise it, the directional-asymmetry claim would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes a Bloom-aligned framework for measuring educational control in LLMs: the ability to preserve a task’s instructional intent while shifting cognitive demand along revised Bloom’s Taxonomy. Using a matched Qwen3-Next general/coder pair on 2,520 programming tasks from BigCodeBench, LiveCodeBench, and SWE-Bench-Verified, it compares general difficulty interventions (Harder/Easier) with Bloom-targeted interventions (Higher/Lower). Outcomes are quantified by Observed Cognitive Shift (OCS; Eq. 1) and Target Zone Accuracy (TZA; Eq. 2) via an external Claude-3.5-Haiku judge, then characterized with semantic-delta keyword clustering and layer-wise Fisher’s Discriminant Ratio (FDR) probing of residual-stream activations. The central empirical claim is a robust upward asymmetry—both models raise cognitive demand more readily than they lower it—and that strong code-execution specialization does not automatically yield Bloom-aligned educational control.

Significance. If the directional asymmetry holds under stronger validation, the paper supplies a concrete, reusable evaluation protocol for a gap that execution benchmarks systematically miss: whether models can adapt tasks for learners rather than only solve them. Strengths include a carefully matched general/coder comparison (same architecture and tokenizer), fixed prompts and decoding, multi-benchmark coverage, explicit metrics (OCS/TZA), and complementary external (semantic-delta) and internal (FDR/projections) diagnostics. The work is timely for CS education and LLM tutoring, and the representation results offer a natural hook for future steering experiments. The contribution is primarily empirical and methodological rather than theoretical, but that is appropriate for the claim.

major comments (3)
  1. §3.2 and Limitations: The load-bearing OCS/TZA results (Tables 1–2) rest on Claude-3.5-Haiku labels of mutated tasks. Judge validity is grounded mainly in Gwet’s AC2 0.95 on a 150-question original-task subset, while the paper itself notes that mutations may leave that distribution. Relative-use is a reasonable design choice, but without a human audit (or second-judge agreement) on a stratified sample of mutated tasks—especially Easier/Lower failures and Appendix C-style surface rewrites—the upward-asymmetry claim remains only partially secured. A modest mutated-set audit with reported agreement would substantially strengthen the central result.
  2. §4.1–4.2 / Tables 1–2: “Preserve instructional intent” is part of the definition of educational control in the abstract and introduction, yet OCS and TZA measure only Bloom-level movement, not intent preservation (topic, learning objective, or solution-space fidelity). Without an explicit intent-preservation check—even a coarse human or secondary-judge rating on a sample—it is hard to distinguish successful educational adaptation from topic drift or problem replacement. Adding such a check, or narrowing the claim language to cognitive-level shift alone, is needed for the framework’s stated definition.
  3. §5.2 / Algorithm 2 and Figures 1–3: FDR and mean-difference projections are presented as representation diagnostics of instruction contrasts, which is appropriate, but the discussion sometimes invites a stronger causal reading (e.g., encoding loci, future steering). The paper correctly flags the need for causal interventions in Limitations; the main text should keep the same boundary more consistently so that weaker coder-model separability is not over-interpreted as the mechanistic cause of poorer downward control.

Circularity Check

0 steps flagged

No circular derivation: OCS/TZA and FDR are independent of the generators; the asymmetry is an empirical observation, not a fit or self-definition.

full rationale

The paper's central claim is an empirical comparison of two matched models under fixed intervention prompts, scored by an external judge and by parameter-free representation diagnostics. Observed Cognitive Shift and Target Zone Accuracy (Eqs. 1–2) are defined from judge-assigned Bloom labels of original vs. mutated tasks; they are not fitted parameters and do not redefine success in terms of the generators' own outputs. The judge is Claude-3.5-Haiku, justified by prior human agreement (Gwet's AC2 0.95) on a 150-question original-task subset, not by self-scoring from the Qwen models under test. Semantic-delta clustering and layer-wise FDR (Eq. 4) characterize observed contrasts rather than force the behavioral result. Self-citations (e.g., Zhang & Rayz 2026 on the judge; related Bloom/SOLO work) supply background and judge validation; they do not import a uniqueness theorem or ansatz that makes the upward-asymmetry claim true by construction. Distribution-shift risk for the judge is a validity concern, not circularity. The derivation chain is observational and self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 3 invented entities

The central claim rests on treating revised Bloom's Taxonomy as an ordinal cognitive-demand scale for programming tasks, on LLM-judge labels for mutated tasks, and on a matched open-weight model pair as a controlled general-vs-coder comparison. No physical constants or fitted scientific laws are involved; free choices are methodological (decoding, clustering, probe token).

free parameters (3)
  • generation decoding settings = T=0.1, top-p=0.95, freq_pen=0.1, max_tokens=2048, seed=42
    Temperature 0.1, top-p 0.95, frequency penalty 0.1, max tokens 2048, seed 42 are fixed by hand and can affect mutation style.
  • semantic-delta clustering hyperparameters = UMAP dims=5; min_cluster_size=15
    UMAP n_components=5 and HDBSCAN min_cluster_size=15 shape the keyword clusters used to interpret strategies.
  • FDR probe token choice = LAST_INST residual stream
    Activations are captured at the last instruction token; this design choice affects where separability is measured.
axioms (5)
  • domain assumption Revised Bloom's Taxonomy levels form a usable ordinal scale of cognitive demand for programming tasks.
    Used throughout as the operational educational-control axis (Introduction; Eqs. 1-2).
  • domain assumption Claude-3.5-Haiku Bloom labels are reliable enough for relative movement comparisons on mutated tasks.
    Justified by prior AC2=0.95 on original tasks, then applied to mutations (Section 3.2).
  • domain assumption Matched architecture and tokenizer make the Qwen3-Next general/coder pair a controlled comparison of specialization effects.
    Stated as the reason for model selection (Section 3.1).
  • standard math Fisher's Discriminant Ratio on residual-stream activations measures linear separability of instruction contrasts across depth.
    Standard linear-separability diagnostic used in Algorithm 2 and Equation 4.
  • domain assumption Semantic delta vectors E_mut - E_orig isolate mutation intent under the linear representation hypothesis.
    Used to justify BERTopic-style clustering of mutations (Section 3.3, Equation 3).
invented entities (3)
  • Observed Cognitive Shift (OCS) no independent evidence
    purpose: Average signed movement on the Bloom ordinal scale after mutation.
    Paper-defined metric (Equation 1); useful but not an external physical entity.
  • Target Zone Accuracy (TZA) no independent evidence
    purpose: Fraction of mutations landing in the requested Bloom zone.
    Paper-defined precision metric (Equation 2).
  • Educational control (Bloom-aligned) no independent evidence
    purpose: Name the ability to preserve instructional intent while shifting cognitive demand.
    Conceptual construct operationalized by the framework; no independent measurement outside this setup.

pith-pipeline@v1.1.0-grok45 · 21956 in / 3106 out tokens · 25828 ms · 2026-07-10T13:48:23.804341+00:00 · methodology

0 comments
read the original abstract

We introduce a Bloom-aligned framework for measuring educational control in Large Language Models (LLMs): the ability to preserve a task's instructional intent while shifting its cognitive demand toward specified learning objectives. We apply this framework to programming tasks in computer science education to study the gap between solving tasks and adapting them for learners. Using revised Bloom's Taxonomy as an operational scale of cognitive demand, we evaluate two intervention settings: general difficulty control, where models are asked to make tasks harder or easier, and Bloom's control, where models are asked to target higher or lower Bloom's levels. We evaluate a matched Qwen3-Next model pair, comparing Qwen3-Next-80B-A3B-Instruct with Qwen3-Coder-Next across 2,520 tasks from three benchmarks. The framework reveals a robust directional asymmetry: both models reliably increase cognitive demand, but struggle to lower it. We further characterize these outcomes with semantic-delta clustering and layer-wise Fisher's Discriminant Ratio probing. Within this controlled comparison, the general model shows clearer middle-layer separability for both general difficulty and Bloom-control contrasts, whereas the coder model shows weaker separability for general difficulty and a deeper peak for Bloom-control contrasts. These results show that strong execution performance does not automatically entail Bloom-aligned educational control.

Figures

Figures reproduced from arXiv: 2607.08009 by Julia Rayz, Yi Zhang.

Figure 1
Figure 1. Figure 1: Fisher’s Discriminant Ratio (FDR) curves showing where difficulty and Bloom [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Projection of internal activations onto the mean difference direction for difficulty [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Projection of internal activations onto the mean difference direction for targeted [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Transition heatmaps for Qwen3-Next-80B-A3B (General) across different interven [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Transition heatmaps for Qwen3-Coder-Next across different intervention settings. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: PCA projection of layer embeddings for the general model under general difficulty [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: PCA projection of layer embeddings for the coder model under general difficulty [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: PCA projection of layer embeddings for the general model under Bloom’s control [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: PCA projection of layer embeddings for the coder model under Bloom’s control [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

215 extracted references · 215 canonical work pages · 60 internal anchors

  1. [1]

    Abdin, Marah and Aneja, Jyoti and Awadalla, Hany and Awadallah, Ahmed and Awan, Ammar Ahmad and Bach, Nguyen and Bahree, Amit and Bakhtiari, Arash and Bao, Jianmin and Behl, Harkirat and Benhaim, Alon and Bilenko, Misha and Bjorck, Johan and Bubeck, S. Phi-3. doi:10.48550/arXiv.2404.14219 , archiveprefix =. 2404.14219 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2404.14219

  2. [2]

    Categorical Data Analysis , author =

    work page

  3. [3]

    Understanding intermediate layers using linear classifier probes

    Understanding Intermediate Layers Using Linear Classifier Probes , author =. doi:10.48550/arXiv.1610.01644 , archiveprefix =. 1610.01644 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1610.01644

  4. [4]

    Allamanis, Miltiadis and Sutton, Charles , year = 2013, pages =. Mining. 2013 10th

    work page 2013

  5. [5]

    and Inkpen, Kori and Teevan, Jaime and

    Amershi, Saleema and Weld, Dan and Vorvoreanu, Mihaela and Fourney, Adam and Nushi, Besmira and Collisson, Penny and Suh, Jina and Iqbal, Shamsi and Bennett, Paul N. and Inkpen, Kori and Teevan, Jaime and. Guidelines for. Proceedings of the 2019. doi:10.1145/3290605.3300233 , isbn =

    work page doi:10.1145/3290605.3300233 2019

  6. [6]

    and Conrad, Frederick G

    Anderson, John R. and Conrad, Frederick G. and Corbett, Albert T. , year = 1989, month = oct, journal =. Skill. doi:10.1207/s15516709cog1304_1 , langid =

    work page doi:10.1207/s15516709cog1304_1 1989

  7. [7]

    A Taxonomy for Learning, Teaching, and Assessing:

    Anderson, Lorin W and Krathwohl, David R , year = 2001, publisher =. A Taxonomy for Learning, Teaching, and Assessing:

    work page 2001

  8. [8]

    and Lilly, J

    Andrews, Robert W. and Lilly, J. Mason and Srivastava, Divya and Feigh, Karen M. , year = 2023, month = mar, journal =. The Role of Shared Mental Models in Human-. doi:10.1080/1463922X.2022.2061080 , langid =

    work page doi:10.1080/1463922x.2022.2061080 2023

  9. [9]

    Claude 3.5 Haiku , author =

    work page

  10. [10]

    Claude 3.7 Sonnet and Claude Code , author =

    work page

  11. [11]

    Claude 4.5 Haiku , author =

    work page

  12. [12]

    and Offerdahl, Erika G

    Arneson, Jessie B. and Offerdahl, Erika G. , editor =. Visual. CBE---Life Sciences Education , volume =. doi:10.1187/cbe.17-08-0178 , langid =

    work page doi:10.1187/cbe.17-08-0178

  13. [13]

    Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and Sutton, Charles , year = 2021, month = aug, number =. Program. doi:10.48550/arXiv.2108.07732 , archiveprefix =. 2108.07732 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2108.07732 2021

  14. [14]

    Transactions on Machine Learning Research , issn =

    Beyond the. Transactions on Machine Learning Research , issn =

    work page

  15. [15]

    Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

    Becker, Joel and Rush, Nate and Barnes, Elizabeth and Rein, David , year = 2025, month = jul, number =. Measuring the. doi:10.48550/arXiv.2507.09089 , archiveprefix =. 2507.09089 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.09089 2025

  16. [16]

    Belinkov, Yonatan , year = 2022, month = apr, journal =. Probing. doi:10.1162/coli_a_00422 , langid =

    work page internal anchor Pith review doi:10.1162/coli_a_00422 2022

  17. [17]

    Graph of Thoughts: Solving Elaborate Problems with Large Language Models

    Besta, Maciej and Blach, Nils and Kubicek, Ales and Gerstenberger, Robert and Gianinazzi, Lukas and Gajda, Joanna and Lehmann, Tomasz and Podstawski, Michal and Niewiadomski, Hubert and Nyczyk, Piotr and Hoefler, Torsten , year = 2023, month = nov, number =. Graph of. arXiv , keywords =:2308.09687 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  18. [18]

    Evaluating the Quality of Learning: The

    Biggs, John Burville and Biggs, John Burville and Collis, Kevin Francis , year = 1982, series =. Evaluating the Quality of Learning: The

    work page 1982

  19. [19]

    and Engelhart, Max D

    Bloom, Benjamin S. and Engelhart, Max D. , year = 1956, publisher =. Taxonomy of Educational Objectives: The Classification of Educational Goals

    work page 1956

  20. [20]

    Imitation and

    Boisseau,. Imitation and. Minds and Machines , volume =. doi:10.1007/s11023-024-09698-6 , langid =

    work page doi:10.1007/s11023-024-09698-6

  21. [21]

    Nature Reviews Genetics , volume =

    Constraint-Based Models Predict Metabolic and Associated Cellular Functions , author =. Nature Reviews Genetics , volume =. doi:10.1038/nrg3643 , copyright =

    work page doi:10.1038/nrg3643

  22. [22]

    Towards Monosemanticity:

    Bricken, Trenton and Templeton, Adly and Batson, Joshua and Chen, Brian and Jermyn, Adam and Conerly, Tom and Turner, Nicholas and Anil, Cem and Denison, Carson and Askell, Amanda and Lasenby, Robert and Wu, Yuhuai and Kravec, Shauna and Schiefer, Nicholas and Maxwell, Todd and Joseph, Nicholas and. Towards Monosemanticity:

    work page

  23. [23]

    Bu. To. Proceedings of the ACM on Human-Computer Interaction , volume =. doi:10.1145/3449287 , langid =

    work page internal anchor Pith review doi:10.1145/3449287

  24. [24]

    Campbell, Murray and Hoane, A.Joseph and Hsu, Feng-hsiung , year = 2002, month = jan, journal =. Deep. doi:10.1016/S0004-3702(01)00129-1 , copyright =

    work page doi:10.1016/s0004-3702(01)00129-1 2002

  25. [25]

    Campello, Ricardo J. G. B. and Moulavi, Davoud and Sander, Joerg , editor =. Density-. Advances in. doi:10.1007/978-3-642-37456-2_14 , isbn =

    work page doi:10.1007/978-3-642-37456-2_14

  26. [26]

    Cao, Ruisheng and Chen, Mouxiang and Chen, Jiawei and Cui, Zeyu and Feng, Yunlong and Hui, Binyuan and Jing, Yuheng and Li, Kaixin and Li, Mingze and Lin, Junyang and Ma, Zeyao and Shum, Kashun and Wang, Xuwu and Wei, Jinxi and Yang, Jiaxi and Zhang, Jiajun and Zhang, Lei and Zhang, Zongmeng and Zhao, Wenting and Zhou, Fan , year = 2026, month = feb, numb...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2603.00729 2026

  27. [27]

    Chan, Lawrence and. Causal

    work page

  28. [28]

    BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

    Chen, Guoxin and Meng, Fanzhe and Zhao, Jiale and Li, Minghao and Cheng, Daixuan and Song, Huatong and Chen, Jie and Lin, Yuzhi and Chen, Hui and Zhao, Xin and Song, Ruihua and Liu, Chang and Chen, Cheng and Jia, Kai and Wen, Ji-Rong , year = 2026, month = mar, number =. doi:10.48550/arXiv.2603.03194 , archiveprefix =. 2603.03194 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2603.03194 2026

  29. [29]

    Evaluating Large Language Models Trained on Code

    Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and Pinto, Henrique Ponde de Oliveira and Kaplan, Jared and Edwards, Harri and Burda, Yuri and Joseph, Nicholas and Brockman, Greg and Ray, Alex and Puri, Raul and Krueger, Gretchen and Petrov, Michael and Khlaaf, Heidy and Sastry, Girish and Mishkin, Pamela and Chan, Brooke and Gray, Scott and...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2107.03374

  30. [30]

    Chen, Xuelu and Hu, Ziniu and Sun, Yizhou , year = 2022, month = jun, journal =. Fuzzy

    work page 2022

  31. [31]

    Structure-

    Chen, Jiaao and Yang, Diyi , year = 2021, pages =. Structure-. Proceedings of the 2021. doi:10.18653/v1/2021.naacl-main.109 , langid =

    work page doi:10.18653/v1/2021.naacl-main.109 2021

  32. [32]

    Chen, Liguo and Guo, Qi and Jia, Hongrui and Zeng, Zhengran and Wang, Xin and Xu, Yijiang and Wu, Jian and Wang, Yidong and Gao, Qing and Wang, Jindong and Ye, Wei and Zhang, Shikun , year = 2025, month = mar, number =. A. doi:10.48550/arXiv.2408.16498 , archiveprefix =. 2408.16498 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.16498 2025

  33. [33]

    Church, Alonzo , year = 1936, month = apr, journal =. An. 2371045 , eprinttype =

    work page 1936

  34. [34]

    Training Verifiers to Solve Math Word Problems

    Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John , year = 2021, month = nov, number =. Training. doi:10.48550/arXiv.2110.14168 , archiveprefix =. 2110.14168 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2110.14168 2021

  35. [35]

    Cohen, Jacob , year = 1960, month = apr, journal =. A. doi:10.1177/001316446002000104 , copyright =

    work page doi:10.1177/001316446002000104 1960

  36. [36]

    Statistical

    Cohen, Jacob , year = 2013, month = may, edition =. Statistical. doi:10.4324/9780203771587 , isbn =

    work page doi:10.4324/9780203771587 2013

  37. [37]

    Weighted Kappa:

    Cohen, Jacob , year = 1968, journal =. Weighted Kappa:. doi:10.1037/h0026256 , langid =

    work page doi:10.1037/h0026256 1968

  38. [38]

    Variables and Distributions in

    CRAM. Variables and Distributions in. Mathematical Methods of Statistics (. j.ctt1bpm9r4.10 , eprinttype =

    work page

  39. [39]

    De Bruijn, N. G. , editor =. Automation of. doi:10.1007/978-3-642-81955-1_11 , isbn =

    work page doi:10.1007/978-3-642-81955-1_11

  40. [40]

    Understanding

    Desolda, Giuseppe and Esposito, Andrea and Greco, Francesco and Tucci, Cesare and Buono, Paolo and Piccinno, Antonio , year = 2025, month = aug, number =. Understanding. doi:10.48550/arXiv.2502.02194 , archiveprefix =. 2502.02194 , primaryclass =

    work page doi:10.48550/arxiv.2502.02194 2025

  41. [41]

    Insufficient

    Dewayani, Gracia Putri and Ngadiso, Ngadiso and Sarosa, Teguh , year = 2020, month = oct, journal =. Insufficient

    work page 2020

  42. [42]

    Communications of the ACM , volume =

    The Humble Programmer , author =. Communications of the ACM , volume =. doi:10.1145/355604.361591 , langid =

    work page doi:10.1145/355604.361591

  43. [43]

    Evaluating

    Du, Xueying and Liu, Mingwei and Wang, Kaixin and Wang, Hanlin and Liu, Junwei and Chen, Yixuan and Feng, Jiayi and Sha, Chaofeng and Peng, Xin and Lou, Yiling , year = 2024, month = apr, pages =. Evaluating. Proceedings of the. doi:10.1145/3597503.3639219 , isbn =

    work page doi:10.1145/3597503.3639219 2024

  44. [44]

    Thirty-Seventh

    Faith and. Thirty-Seventh

    work page

  45. [45]

    Elazar, Yanai and Ravfogel, Shauli and Jacovi, Alon and Goldberg, Yoav , year = 2021, month = mar, journal =. Amnesic. doi:10.1162/tacl_a_00359 , langid =

    work page doi:10.1162/tacl_a_00359 2021

  46. [46]

    A Mathematical Framework for Transformer Circuits , author =

    work page

  47. [47]

    Elhage, Nelson and Hume, Tristan and Olsson, Catherine and Schiefer, Nicholas and Henighan, Tom and Kravec, Shauna and. Toy. doi:10.48550/arXiv.2209.10652 , archiveprefix =. 2209.10652 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2209.10652

  48. [48]

    Elkins, Sabina and Kochmar, Ekaterina and Cheung, Jackie C. K. and Serban, Iulian , year = 2024, month = mar, journal =. How

    work page 2024

  49. [49]

    The Rating of Chessplayers, Past and Present , author =

    work page

  50. [50]

    The Computer Boys Take over: Computers, Programmers, and the Politics of Technical Expertise , shorttitle =

    Ensmenger, Nathan , year = 2010, series =. The Computer Boys Take over: Computers, Programmers, and the Politics of Technical Expertise , shorttitle =

    work page 2010

  51. [51]

    and Cicchetti, Domenic V

    Feinstein, Alvan R. and Cicchetti, Domenic V. , year = 1990, month = jan, journal =. High Agreement but Low. doi:10.1016/0895-4356(90)90158-L , copyright =

    work page doi:10.1016/0895-4356(90)90158-l 1990

  52. [52]

    CodeBERT: A Pre-Trained Model for Programming and Natural Languages

    Feng, Zhangyin and Guo, Daya and Tang, Duyu and Duan, Nan and Feng, Xiaocheng and Gong, Ming and Shou, Linjun and Qin, Bing and Liu, Ting and Jiang, Daxin and Zhou, Ming , year = 2020, month = sep, number =. doi:10.48550/arXiv.2002.08155 , archiveprefix =. 2002.08155 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2002.08155 2020

  53. [53]

    Fisher, R. A. , year = 1922, month = jan, journal =. On the. 2340521 , eprinttype =

    work page 1922

  54. [54]

    Building a Bridge between Skills and Thresholds:

    Folk, Amanda and Blocksidge, Katie and Hammons, Jane and Primeau, Hanna , year = 2024, month = jun, journal =. Building a Bridge between Skills and Thresholds:. doi:10.11645/18.1.58 , copyright =

    work page doi:10.11645/18.1.58 2024

  55. [55]

    ACM SIGCSE Bulletin , volume =

    Developing a Computer Science-Specific Learning Taxonomy , author =. ACM SIGCSE Bulletin , volume =. doi:10.1145/1345375.1345438 , langid =

    work page doi:10.1145/1345375.1345438

  56. [56]

    LLM-based NLG Evaluation: Current Status and Challenges

    Gao, Mingqi and Hu, Xinyu and Ruan, Jie and Pu, Xiao and Wan, Xiaojun , year = 2024, month = feb, number =. doi:10.48550/arXiv.2402.01383 , archiveprefix =. 2402.01383 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2402.01383 2024

  57. [57]

    Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor , year = 2020, month = dec, number =. The. doi:10.48550/arXiv.2101.00027 , archiveprefix =. 2101.00027 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2101.00027 2020

  58. [58]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Gao, Yunfan and Xiong, Yun and Gao, Xinyu and Jia, Kangxiang and Pan, Jinliu and Bi, Yuxi and Dai, Yi and Sun, Jiawei and Wang, Meng and Wang, Haofen , year = 2024, month = mar, number =. Retrieval-. doi:10.48550/arXiv.2312.10997 , archiveprefix =. 2312.10997 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.10997 2024

  59. [59]

    Scaling and Evaluating Sparse Autoencoders , booktitle =

    Gao, Leo and. Scaling and Evaluating Sparse Autoencoders , booktitle =

    work page

  60. [60]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and. The. doi:10.48550/arXiv.2407.21783 , archiveprefix =. 2407.21783 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783

  61. [61]

    BERTopic: Neural topic modeling with a class-based TF-IDF procedure

    Grootendorst, Maarten , year = 2022, month = mar, number =. doi:10.48550/arXiv.2203.05794 , archiveprefix =. 2203.05794 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2203.05794 2022

  62. [62]

    Gupta, Akshat and Yeung, Jay and Anumanchipalli, Gopala and Ivanova, Anna , year = 2026, month = mar, number =. How. doi:10.48550/arXiv.2510.18871 , archiveprefix =. 2510.18871 , primaryclass =

    work page doi:10.48550/arxiv.2510.18871 2026

  63. [63]

    Language Models Represent Space and Time , booktitle =

    Gurnee, Wes and Tegmark, Max , editor =. Language Models Represent Space and Time , booktitle =

    work page

  64. [64]

    Learner-

    Guzdial, Mark , year = 2016, series =. Learner-. doi:10.1007/978-3-031-02216-6 , copyright =

    work page doi:10.1007/978-3-031-02216-6 2016

  65. [65]

    British Journal of Mathematical and Statistical Psychology , volume =

    Computing Inter-rater Reliability and Its Variance in the Presence of High Agreement , author =. British Journal of Mathematical and Statistical Psychology , volume =. doi:10.1348/000711006X126600 , copyright =

    work page doi:10.1348/000711006x126600

  66. [66]

    Artificial

    Habiballa, Hashim and Kotyrba, Martin and Volna, Eva and Bradac, Vladimir and Dusek, Martin , year = 2025, month = jan, journal =. Artificial. doi:10.3390/app15020581 , copyright =

    work page doi:10.3390/app15020581 2025

  67. [67]

    The Thirteenth International Conference on Learning Representations , author =

    work page

  68. [68]

    Thirty-Fifth

    Measuring. Thirty-Fifth

    work page

  69. [69]

    International

    Measuring. International

    work page

  70. [70]

    Hochreiter, Sepp and Schmidhuber, J. Long. Neural Computation , volume =. doi:10.1162/neco.1997.9.8.1735 , langid =

    work page doi:10.1162/neco.1997.9.8.1735 1997

  71. [71]

    Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions

    Hong, Pengfei and Majumder, Navonil and Ghosal, Deepanway and Aditya, Somak and Mihalcea, Rada and Poria, Soujanya , year = 2024, month = nov, number =. Evaluating. doi:10.48550/arXiv.2401.09395 , archiveprefix =. 2401.09395 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2401.09395 2024

  72. [72]

    Hou, Irene and Mettille, Sophia and Man, Owen and Li, Zhuo and Zastudil, Cynthia and MacNeil, Stephen , year = 2024, month = jan, pages =. The. Proceedings of the 26th. doi:10.1145/3636243.3636248 , isbn =

    work page doi:10.1145/3636243.3636248 2024

  73. [73]

    Proceedings of the 31st

    Huber, Thomas and Niklaus, Christina , editor =. Proceedings of the 31st

    work page

  74. [74]

    DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation

    Hu, Wenhao and Duan, Jinhao and Wei, Chunchen and Zhang, Li and Zhang, Yue and Xu, Kaidi , year = 2025, month = mar, number =. doi:10.48550/arXiv.2503.10452 , archiveprefix =. 2503.10452 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.10452 2025

  75. [75]

    CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

    Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc , year = 2020, month = jun, number =. doi:10.48550/arXiv.1909.09436 , archiveprefix =. 1909.09436 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1909.09436 2020

  76. [76]

    and Hollan, James D

    Hutchins, Edwin L. and Hollan, James D. and Norman, Donald A. , year = 1985, month = dec, journal =. Direct. doi:10.1207/s15327051hci0104_2 , langid =

    work page doi:10.1207/s15327051hci0104_2 1985

  77. [77]

    Iyer, Srinivasan and Konstas, Ioannis and Cheung, Alvin and Zettlemoyer, Luke , year = 2018, month = aug, number =. Mapping. doi:10.48550/arXiv.1808.09588 , archiveprefix =. 1808.09588 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1808.09588 2018

  78. [78]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Jain, Naman and Han, King and Gu, Alex and Li, Wen-Ding and Yan, Fanjia and Zhang, Tianjun and Wang, Sida and. doi:10.48550/arXiv.2403.07974 , archiveprefix =. 2403.07974 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2403.07974

  79. [79]

    Designing and

    Jangra, Anubhav and Muresan, Smaranda , year = 2026, month = feb, number =. Designing and. doi:10.48550/arXiv.2510.21087 , archiveprefix =. 2510.21087 , primaryclass =

    work page doi:10.48550/arxiv.2510.21087 2026

  80. [80]

    Design Studies , volume =

    Design Fixation , author =. Design Studies , volume =. doi:10.1016/0142-694X(91)90003-F , copyright =

    work page doi:10.1016/0142-694x(91)90003-f

Showing first 80 references.