Pith. sign in

REVIEW 3 major objections 6 minor 8 references

HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Claude-3.5-Sonnet-1022 is the most consistent coder in a 65-task multi-file benchmark.

desk verdict Useful new multi-file project benchmark with a real consistency protocol, but the headline consistency claim rests on shaky statistics. read the letter →

arxiv 2502.00226 v1 pith:DD6MWZHP submitted 2025-01-31 cs.LG cs.SE

Jun Xing , Mayur Bhatia , Sahil Phulwani , Darshan Suresh , Rafik Matta This is my paper · ORCID
classification cs.LGcs.SE
keywords LLMevaluationcodegenerationbenchmarkmulti-fileprojectsmodelconsistencyfront-enddevelopmentpass@1medianstandarddeviationsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces HackerRank-ASTRA, a benchmark of 65 project-based coding problems that require reading an average of 12 source and configuration files and editing about 2.3 files each. Its purpose is to measure not just whether large language models can solve realistic multi-file tasks, but how consistently they solve them across repeated attempts. Across 32 fresh runs per problem, o1, o1-preview, and Claude-3.5-Sonnet-1022 earn nearly identical mean scores around 75 percent, with no statistically significant difference between them. The paper's central claim is that Claude-3.5-Sonnet-1022 is the most consistent of the evaluated models, with a median per-problem standard deviation of 0.0497, and that this consistency gap is statistically significant. If true, that places reliability as a measurable axis of model quality that is separate from mean correctness.

What carries the argument

The load-bearing mechanism is the ASTRA evaluation protocol: a structured prompt that pairs the problem statement with all relevant project files, a format-enforcing step that asks for XML or JSON output, post-processing that repairs escaping issues, integration of the candidate solution into a Dockerized project, and execution against hidden test cases. Correctness is aggregated as the mean fraction of tests passed across k=32 fresh conversations per problem, and consistency is summarized as the median of the 65 per-problem standard deviations. The paired t-test applied to those per-problem standard deviations is what carries the statistical claim that Claude-3.5-Sonnet-1022 is more consistent than the other models.

What would settle it

Recompute the consistency comparison with a paired permutation test on the 65 per-problem standard deviations (or with a Wilcoxon signed-rank test) and apply a multiple-comparison correction across the model pairs; if the corrected p-values fail to stay below 0.05 for Claude-3.5-Sonnet-1022 versus the other models, the paper's headline consistency claim would be falsified.

Watch

Extended reading notes

Core claim

The v1 ASTRA benchmark consists of 65 multi-file, project-based problems drawn from front-end development, covering 10 primary skill domains and 34 subskills across frameworks such as Node.js, React.js, Angular.js, Django, Java Spring Boot, Ruby on Rails, and .NET. Each model receives the problem statement plus the project files, returns code in XML or JSON, and is scored by the fraction of test cases passed, with 32 independent runs per problem. On mean score, o1 (75.80%), o1-preview (75.55%), and Claude-3.5-Sonnet-1022 (75.07%) are statistically tied, while GPT-4o-0513 trails at 69.52%. On consistency, defined as the median across problems of each model's per-problem standard deviation, Claude-3.5-Sonnet-1022 scores 0.0497, compared with 0.11 for o1, 0.13 for Gemini-1.5-pro, 0.17 for o1-preview, and 0.20 for GPT-4o-0513, and the authors report that this difference is statistically significant via a paired t-test.

Load-bearing premise

The consistency result depends on a paired t-test applied to per-problem standard deviations, even though the paper itself notes that those standard deviations often deviate from a normal distribution and applies no multiple-comparison correction across the model pairs.

Editorial extensions

If this is right

  • Model choice for production coding should weigh consistency alongside mean correctness, since two models with the same average score can have very different run-to-run reliability.
  • The benchmark's sub-skill results imply that no single model dominates: Claude-3.5-Sonnet-1022 wins on API integration and database interaction, while o1 wins on form handling and pagination, so task-specific selection is warranted.
  • The consistent XML-over-JSON advantage across models indicates that output format is a practical lever in evaluating or deploying code LLMs on multi-file tasks.
  • The moderate negative correlation between output length and score (-0.560) suggests that longer generated solutions tend to be less correct in these project-based settings, pointing to over-generation as a failure mode.
  • The near-tie among top models at about 75% implies that further progress on multi-file tasks will require benchmarks that can separate models on reliability or specialized sub-skills rather than aggregate accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to test whether the consistency ranking is stable under alternative dispersion measures, such as the interquartile range or the variance of scores, since the paper's own distributional caveat suggests median SD may be sensitive to the test choice.
  • The k=32 repeated-run design makes the benchmark expensive; if consistency differences replicate on a small subset of problems, a cheaper screening protocol could use fewer runs for models already known to be consistent.
  • Because the 65 problems come from a proprietary item bank, an independent replication using open-sourced problems would clarify whether Claude's low median SD is a general property or an artifact of this particular task distribution.
  • The format effect (XML over JSON) could itself be a useful diagnostic: models with smaller XML-vs-JSON gaps might be more robust to prompt formatting, a property worth checking when a model is deployed through varied interfaces.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces HackerRank-ASTRA, a benchmark of 65 multi-file, project-based coding problems (mostly front-end) with evaluation over k=32 independent runs per problem. It defines three metrics: mean score, mean pass@1, and median standard deviation of scores across problems as a consistency measure. On an initial evaluation of five frontier models, the paper reports that o1, o1-preview, and Claude-3.5-Sonnet-1022 achieve comparable mean scores around 75%, and that Claude-3.5-Sonnet-1022 shows the highest consistency (median SD=0.0497), a difference claimed to be statistically significant. The benchmark and dataset are open-sourced, and the paper includes taxonomy-level and error-analysis results.

Significance. If the results are substantiated, HackerRank-ASTRA is a useful benchmark contribution: it addresses a real gap by evaluating multi-file project-level tasks rather than isolated functions, and it introduces a consistency metric based on repeated runs, which is rare in code-generation benchmarks. Strengths include the release of the dataset, the relatively large k=32 per problem, the structured XML/JSON output pipeline, and the taxonomy-level breakdown. The central statistical claims, however, are currently underreported, and the headline consistency advantage is not adequately supported by the evidence presented.

major comments (3)
  1. [Section 4.1, Table 3] The claim that Claude-3.5-Sonnet-1022's lower median SD is 'statistically significant based on the paired t-test' is not substantiated. Section 2.5 states that per-problem standard deviations 'often deviate from a normal distribution,' which is the stated motivation for reporting the median; applying a paired t-test to those same non-normal per-problem SDs is inconsistent without reporting diagnostics for the 65 paired differences. The manuscript reports no p-values, effect sizes, or confidence intervals, does not specify whether the comparison is Claude versus each other model or Claude versus the pooled rest, and gives no multiple-comparison correction. Please replace or supplement the t-test with a nonparametric paired test (e.g., Wilcoxon signed-rank) or a bootstrap/permutation procedure on the paired differences, report exact p-values and effect sizes, and justify the pairing and the correction for multiple tests.
  2. [Section 4.1] The sentence 'a paired t-test reveals that, with the exception of GPT-4o-0513, the differences between model performances are not statistically significant' is ambiguous and does not report the actual test results. The abstract's claim of 'no statistically significant differences' among the top three models is not the same as the text's statement involving an exception for GPT-4o-0513. The authors should provide the full matrix of pairwise comparisons with test statistics, p-values, and a stated multiple-comparison correction; without this, neither claim can be verified, and a non-significant result with 65 problems and high variance is not evidence of equivalence.
  3. [Section 7 and HuggingFace link] The manuscript says the 65 project questions are open-sourced, but it does not state whether the hidden test cases, the Docker-based evaluation harness, and the aggregation scripts are also released. Since the benchmark's value depends on independent verification of the reported scores, please explicitly enumerate what is included in the release; if the test cases and harness are not public, the leaderboard cannot be reproduced by third parties, which would undercut the benchmark's utility.
minor comments (6)
  1. [Section 2.1] There is a typo: 'preapred' should be 'prepared'.
  2. [Section 7] The word 'argentic' should be 'agentic'.
  3. [Abstract and Table 3] The abstract reports Claude-3.5-Sonnet-1022's median SD as 0.0497, while Table 3 reports 0.05; please reconcile the two values or explicitly state that the table value is rounded.
  4. [Section 4.4 and Tables 5-6] The claim that 'XML demonstrated statistically significant superiority over JSON' also lacks the test used and the p-values; please report the statistical details or soften the claim.
  5. [Section 4.4, Tables 5 and 6] The model o1 is missing from the XML and JSON format comparison tables, although it appears in the main leaderboard; please clarify why o1 is excluded.
  6. [Section 2.5, Eq. (3)] Equation (3) uses a population standard deviation with division by k rather than k-1; since k=32, the difference is minor, but the choice should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark evaluation is external, self-contained, and does not derive its conclusions from fitted parameters or self-citations.

full rationale

This paper does not contain a derivation chain that reduces to its inputs. The HackerRank-ASTRA benchmark evaluates models against pre-existing HackerRank test cases: model outputs are executed in a Docker container and scored by the proportion of passed test cases, so the correctness and consistency metrics are computed from externally defined ground truth rather than from a fitted model or an assumed ansatz. No parameter is fitted to a subset of the data and then reported as a prediction of the same data; the mean score, mean pass@1, and median standard deviation are summary statistics of the measured run outcomes. The paper cites no self-authored prior work as load-bearing evidence, and the benchmark's claims do not depend on any uniqueness theorem or imported result from the authors. The consistency significance claim rests on a paired t-test over per-problem standard deviations, and Section 2.5 notes that such standard deviations often deviate from a normal distribution; however, this is a question of statistical validity and evidentiary strength, not circularity. Similarly, concerns about multiple comparisons, unreported p-values, and low power for the 'no statistically significant differences' claim are methodological caveats that belong under correctness risk, not under circularity. The dataset is curated by the authors' organization, which raises independence and external-validity considerations, but that is not a case of a conclusion being equivalent to its inputs by construction. Overall, the paper is self-contained as an external benchmark evaluation, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted and no mathematical derivation is presented. The central claims rest on the validity of the HackerRank problem set, the execution harness, and the statistical test used for consistency comparisons. No new physical or conceptual entities are postulated.

assumptions (4)
  • domain assumption HackerRank's 65 problems and their test cases are valid, unbiased ground truth for real-world software development skills.
    Section 2.2 states the problems come from HackerRank's proprietary library and were curated by its Content Creation team; the benchmark's validity depends on this curation being representative and correct.
  • domain assumption The Docker-based integration and test execution pipeline correctly measures solution correctness after post-processing.
    Section 3.2 describes the pipeline; if post-processing or integration introduces errors, the scores are biased.
  • domain assumption The paired t-test is valid for comparing per-problem standard deviations across models.
    Section 4.1 reports statistical significance of consistency differences, while Section 2.5 notes SDs deviate from a normal distribution, making this assumption load-bearing and questionable.
  • domain assumption k=32 runs and median standard deviation meaningfully capture model consistency.
    Section 2.5 defines consistency this way; no external validation shows this metric predicts real-world reliability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems." pith.science (2026). https://pith.science/paper/DD6MWZHP

@misc{pith2026250200226,
  author       = {Pith},
  title        = {Pith review of: HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DD6MWZHP}},
  note         = {Machine review of arXiv:2502.00226}
}
read the original abstract

Evaluating the real-world applicability of large language models (LLMs) provides valuable insights for their development and use in software development tasks. Existing benchmarks often focus on standalone coding problems or specific libraries, overlooking multi-file, project-based scenarios and lacking a rigorous evaluation of consistency. The HackerRank-ASTRA Benchmark introduces project-based coding problems that mirror real-world scenarios. It evaluates model consistency through 32 runs (k = 32) and median standard deviation while incorporating taxonomy-level analysis to assess sub-skill capabilities. Initial evaluations on 65 problems show that the top three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved comparable average scores of 75%, with no statistically significant differences in performance. Notably, Claude-3.5-Sonnet-1022 demonstrated the highest consistency across problems, with low variability (SD = 0.0497), which was statistically significant compared to other models, highlighting its reliability for real-world software development tasks.

Figures

Figures reproduced from arXiv: 2502.00226 by the authors.

Figure 1
Figure 1. Distribution of v1 HackerRank-ASTRA benchmark main skill frequency. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of v1 HackerRank-ASTRA benchmark sub-skill frequency. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Project structure of a sample RESTful API problem. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: Diagram of v1 HackerRank-ASTRA benchmark evaluation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: XML prompt of v1 HackerRank-ASTRA benchmark. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: JSON prompt of v1 HackerRank-ASTRA benchmark. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Model performance comparison by main skill categories. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Model performance comparison by sub-skill categories (1/3). [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Model performance comparison by sub-skill categories (2/3). [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Model performance comparison by sub-skill categories (3/3). [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Scatter Plot of Average Output vs Average Score vs Input Length. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: The routes defined by GPT-4o-0513 for handling products. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: The app.js provided by GPT-4o-0513. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Fixes required in app.js. This example underscores the importance of validating LLM outputs in complex pro￾gramming tasks. While models can generate syntactically correct code for modular com￾ponents, they often miss essential integrations, leading to functionality ga…
Figure 15
Figure 15. Figure 15: A form is expected to behave dynamically by providing real-time feedback to [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: A useEffect hook should be used to re-evaluate the form’s validity whenever [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Render error elements only when there is an actual error message. [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: The handleChange function should update the error state dynamically based [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Length Converter functionality. This problem reflects real-world use cases for unit conversion, commonly seen in in￾dustries like e-commerce, architecture, and engineering, where accurate and dynamic conversions are critical. For example, online platforms often provid…
Figure 20
Figure 20. Figure 20: This binds the input1 field to the component’s input1 property. Any changes [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: The conversion logic should ensure that the corresponding input field is up [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: The ngOnInit() lifecycle hook initializes the component’s state. Initially, the [PITH_FULL_IMAGE:figures/full_fig_p022_22.png]
Figure 23
Figure 23. Figure 23: Fix: Adding the current user method to the AuthenticateUser concern to ensure that the method is available for use in the controller actions [PITH_FULL_IMAGE:figures/full_fig_p023_23.png]
Figure 24
Figure 24. Figure 24: Fix: Adding validations ensured that only valid job entries were created, [PITH_FULL_IMAGE:figures/full_fig_p023_24.png]
Figure 25
Figure 25. Figure 25: Fix: The response was updated to include details about the associated job [PITH_FULL_IMAGE:figures/full_fig_p024_25.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 8 linked inside Pith

  1. [1]

    Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation

    Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. arXiv preprint arXiv:2412.21199, 2024. 15

    arXiv 2024

  2. [2]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    arXiv 2023

  3. [3]

    Jimenez, Alexander L

    John Yang, Carlos E. Jimenez, Alexander L. Zhang, Kilian Lieret, Jiani Yang, Xinyun Wu, Ofir Press, Nils Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024

    arXiv

  4. [4]

    Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories

    Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories. arXiv preprint arXiv:2405.19856, 2024

    arXiv 2024

  5. [5]

    Enhancing large language models in coding through multi-perspective self-consistency

    Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. Enhancing large language models in coding through multi-perspective self-consistency. arXiv preprint arXiv:2309.17272, 2024

    arXiv 2024

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and Xiaokang Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2024

    arXiv 2024

  7. [7]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023

    arXiv 2023

  8. [8]

    Mlagentbench: Eval- uating language agents on machine learning experimentation

    Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Eval- uating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2024. A Appendix A.1 Errors Observed in a Restful project problem This example highlights a RESTful API development task from the ASTRA Benchmark Dataset, illustrating both the design requi...

    arXiv 2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.