Pith. sign in

REVIEW 5 major objections 4 minor 11 references

Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams?

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language models ace Brazilian university entrance exams but stumble on math.

desk verdict A useful new Portuguese-language exam benchmark; the main result is plausible, but the unvalidated visual-filter step needs to be addressed before the math/engineering decline claims are taken at face value. read the letter →

arxiv 2508.15835 v1 pith:I4BEE7V6 submitted 2025-08-19 cs.CL cs.AI

Henrique Godoy This is my paper · ORCID
classification cs.CLcs.AI
keywords BrazilianuniversityentranceexamslanguagemodelevaluationPortuguesebenchmarkmathematicalreasoningconfidencecalibrationcost-efficiencyBloom'staxonomyENEM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Alvorada-Bench asks whether language models can genuinely handle Brazilian university entrance exams, which test Portuguese cultural knowledge alongside academic reasoning. The paper assembles 4,515 text-only multiple-choice questions from ENEM, FUVEST, UNICAMP, IME, and ITA, and evaluates 20 models under zero-shot, role-playing, and chain-of-thought prompts. Its central finding is that the best models exceed 94% average accuracy and are near-perfect on humanities, yet performance drops sharply on mathematics and on the computation-heavy engineering exams IME and ITA. The paper also reports that models' self-reported confidence tracks their actual accuracy closely, and that near-92% accuracy can be reached for under $2 per 1K tokens. Together, these claims suggest that language models have crossed a threshold of Portuguese-curriculum competence, with multi-step quantitative reasoning as the remaining bottleneck.

What carries the argument

Alvorada-Bench, a curated corpus of 4,515 text-only multiple-choice questions with official answer keys, built from five Brazilian exams through PDF extraction, pattern matching, an LLM-based filtering stage that removes image-dependent items, and text normalization. The evaluation protocol wraps each question in a structured JSON output requirement—selected alternative, confidence score (0–10), perceived difficulty (0–10), and a Bloom's-taxonomy label—which is what allows the paper to measure accuracy, calibration, and cognitive-complexity profiles on the same responses.

What would settle it

A concrete test would be to have human annotators judge a random sample of both filtered-out and retained questions for whether an image is truly required, then run the same 20 models on freshly written, unpublished questions in each subject; if humanities accuracy falls sharply or the mathematics gap changes, the reported ceiling and subject-level gap reflect filtering or contamination rather than genuine capability.

Watch

Extended reading notes

Core claim

The paper's central claim is that, on a new benchmark built from real Brazilian entrance examinations, current language models perform at or above the level of the average Brazilian student in most subjects while retaining a persistent weakness in mathematics and engineering-style problem solving. Top models reach 94.6% overall accuracy; human sciences reach 93.9%; mathematics falls to 62.7% for baseline models, although reasoning-enhanced models recover to roughly 94% on mathematics. The evaluation also collects structured confidence and difficulty self-reports, and the paper shows that low-confidence responses still exceed 90% accuracy while accuracy degrades monotonically as confidence fa

Load-bearing premise

The key assumption is that the automatic language-model filter that removes questions requiring figures or diagrams is accurate; if it silently keeps image-dependent items or silently excludes text-solvable ones, the reported accuracies describe only the filtered subset, not the full exams.

Editorial extensions

If this is right

  • If the reported results hold, educational deployment is no longer blocked by cost: DeepSeek Reasoner and O3 Mini deliver roughly 92% accuracy at under $2 per 1K tokens.
  • Confidence self-reports can serve as a practical routing signal, because low-confidence responses reliably mark likely errors and uncertainty correlates with perceived difficulty.
  • Prompt engineering has little effect on reasoning-optimized models, with O3 varying by only about 0.1 percentage point across prompting strategies, so evaluation protocols can be standardized without much performance cost.
  • Application-level Bloom tasks are the main failure tier for conventional models, meaning progress in knowledge recall has outpaced progress in computation and applied problem solving.
  • The 24.8-percentage-point drop from ENEM to IME/ITA indicates that engineering entrance exams, not general curriculum exams, are the sharper test of symbolic manipulation and multi-step reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own contamination caveat implies a falsifiable prediction: on a freshly written, never-published exam, accuracy on culturally specific humanities content would drop if public-exam training data explains the near-perfect scores.
  • The LLM-based filtering stage could be inverted as a diagnostic: comparing performance on retained text questions versus excluded image-dependent questions would quantify how much visual reasoning modern models still lack, an extension the paper does not pursue.
  • A natural next experiment is to run the same 759 mathematics questions with tool use, such as a calculator or symbolic solver, and measure whether the residual IME/ITA gap closes; the paper explicitly leaves tool use out of scope.
  • Because only final answers are scored, human partial-credit grading of the same 270,900 responses would test whether correct answers are reached through sound reasoning or spurious correlations, especially on mathematics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. Alvorada-Bench is a 4,515-question, text-only multiple-choice benchmark drawn from five Brazilian university entrance exams (ENEM, FUVEST, UNICAMP, IME, ITA), spanning 1981–2025. The paper evaluates 20 language models from OpenAI, Anthropic, and DeepSeek under zero-shot, role-playing, and chain-of-thought prompting, yielding 270,900 responses with structured self-reports of confidence, difficulty, and Bloom level. The main empirical claims are that top models exceed 94% overall accuracy; accuracy is high on humanities and languages but declines sharply on Mathematics (62.7%) and on the engineering-oriented IME (61.4%) and ITA (68.1%) exams; confidence is well calibrated; and cost-accuracy frontiers make high accuracy available under $2 per 1K tokens. The paper also compares model performance with human baselines on ENEM 2024, reporting that all models surpass humans in most domains.

Significance. If the findings hold, Alvorada-Bench is a valuable, publicly released resource for Portuguese-language evaluation, addressing an important gap in multilingual and culturally specific benchmarking. The scale (4,515 questions, 20 models, 270,900 responses) and the availability of data/code are concrete strengths that support reproducibility. The observed pattern—strong cultural/humanities performance with weaker quantitative reasoning—is a useful, potentially actionable result for the community. However, the central quantitative claims are not yet fully established: the visual-question filtering step is unvalidated, no statistical uncertainty is reported, and some internal numerical inconsistencies appear. These issues are fixable and do not invalidate the dataset's potential, but they must be addressed before the paper's conclusions can be taken as reliable.

major comments (5)
  1. [§2.2 / §4.1] The filtering stage uses an unvalidated language model to detect and remove questions requiring visual interpretation. No precision, recall, or manual audit of the filter is reported. Because IME/ITA and Mathematics questions frequently contain figures, diagrams, and geometric drawings, any image-dependent item left in the text-only benchmark would depress accuracy artificially, misattributing a missing-modality artifact to 'multi-step reasoning.' Conversely, over-filtering would skew the surviving mathematics subset. The Limitations section acknowledges the exclusion of multimodal questions but provides no check that the exclusion succeeded. I request a manual audit of a random sample (ideally all) of included and excluded items, with report of filter accuracy and agreement, and a re-estimation or sensitivity analysis of the subject/exam-level results.
  2. [§3.1, Table 2, §3.5, §3.6] All accuracy numbers are point estimates without confidence intervals or variance. The gap between O3 Pro (94.63%) and O3 (94.55%) is 0.08 percentage points, and the difference between ENEM (86.2%) and UNICAMP (86.1%) is 0.1 points; these are almost certainly within sampling error. The paper's ranking and 'decline' claims need bootstrap confidence intervals or Bayesian credible intervals, especially for per-subject and per-exam breakdowns where item counts are smaller. Without these, comparisons such as 'O1 vs DeepSeek Reasoner' or 'IME vs ITA' are over-precise. Please report the number of questions per subject/exam after filtering and provide interval estimates for the headline numbers.
  3. [§3.2] The human comparison on ENEM 2024 is not sufficiently documented. The paper states that all 20 models surpass human baselines in Humanities, Natural Sciences, and Languages, and that GPT-4.1 Nano only underperforms humans in Mathematics, but it does not report the human accuracy values, the number of items per domain, or whether the human baseline was computed on the same text-only filtered subset used for the models. Without this information, the 'decisive shift' conclusion is not verifiable. Please provide the baseline numbers from reference [6], clarify whether the same questions and scoring were used, and note any differences in administration conditions.
  4. [§3.3] The cost-accuracy frontier contains an internal inconsistency. Table 2 gives O3 Mini an accuracy of 88.15% and O4 Mini an accuracy of 91.50%, but §3.3 says 'DeepSeek Reasoner (92.71%, $1.82) and O3 Mini (91.50%, $1.95) dominate the cost–accuracy frontier.' The 91.50% value belongs to O4 Mini, not O3 Mini. This changes the cost-accuracy claim and must be corrected. Please also specify the date and basis of API pricing, since provider prices change over time and the cost comparison may not generalize.
  5. [§3.4] Calibration claims are supported only by qualitative descriptions and figures. The abstract and §3.4 state that confidence is 'well calibrated' and 'correlates with perceived difficulty,' but no quantitative metrics are reported. I request standard calibration measures (expected calibration error, Brier score, correlation coefficient between confidence and accuracy, and between uncertainty and difficulty), with confidence intervals. This is necessary to support the paper's contribution on uncertainty quantification.
minor comments (4)
  1. [Abstract / §1] Minor language issues: 'this paper introduce' should be 'this paper introduces'; the abstract sentence beginning 'Evaluating twenty models...' is a fragment. A proofread pass is recommended.
  2. [Figures 5, 7, 9] The figures would benefit from error bars or shaded confidence regions, explicit sample sizes, and clearer axis labels. In particular, Figure 5's calibration panels should include a reference diagonal and quantitative summaries.
  3. [§3.8] Bloom classifications are model-generated and used as measurements without validation against expert labels. Please state this limitation explicitly—it is currently only implicit—and note that the 'application-level bottleneck' claim depends on the reliability of these self-reported labels.
  4. [References] Reference [6] is cited for the human baseline in §3.2, but the citation context is not fully described. Please specify which tables/figures in [6] support the human values used.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Alvorada-Bench is an empirical evaluation benchmark, not a derivation with fitted parameters or self-citation load-bearing claims.

full rationale

This paper constructs a benchmark and reports measured model accuracies, confidence ratings, and cost figures. There are no equations derived from assumptions, no fitted parameters renamed as predictions, and no load-bearing self-citations: the author is Henrique Godoy, and the cited related works ([1] BLUEX and [6] Brazilian exam LLM behavior) are external and do not overlap with the author. The central claims (top models exceed 94% overall; accuracy declines on Mathematics and IME/ITA) are direct aggregates of prompted model responses against official answer keys. The self-reported confidence, difficulty, and Bloom classifications are analyzed as model outputs, not used to define accuracy, so no self-definitional circularity arises. The main validity concern is the unvalidated language-model filtering stage in §2.2, which the paper itself flags in §4.1 as excluding multimodal questions; if the filter is imperfect, the math/engineering results could be partially an artifact of missing figures. However, that is a measurement-validity or data-quality threat, not a circularity step, because the filtering decision is not derived from the accuracy results and the reported numbers are not forced by construction. Data contamination (§4.1) is likewise an acknowledged external-validity caveat rather than circularity. The evaluation is self-contained against external benchmarks and human baselines, so the appropriate finding is no significant circularity (score 0).

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several unverified assumptions. The most fragile is the LLM-based filter that removes visual questions; its failure mode is unmeasured. The official answer keys are assumed correct, and the self-report protocol is treated as trustworthy. No free parameters are fitted, because this is an evaluation rather than a derivation.

assumptions (4)
  • domain assumption Official answer keys are correct for all 4,515 questions
    The dataset relies on these keys as ground truth, as stated in §2.2.
  • domain assumption Regex pattern matching correctly segments questions and aligns them with answer keys
    Pipeline step described in §2.2; no manual verification is reported.
  • ad hoc to paper The language model used for filtering correctly identifies all visual questions
    The filter model is unnamed, its accuracy is unreported, and its errors directly affect benchmark composition.
  • domain assumption Self-reported confidence and Bloom levels are meaningful model outputs
    These outputs are analyzed in §3.4 and §3.8 as reliable measurements, but they are unverified model responses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams?." pith.science (2026). https://pith.science/paper/I4BEE7V6

@misc{pith2026250815835,
  author       = {Pith},
  title        = {Pith review of: Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I4BEE7V6}},
  note         = {Machine review of arXiv:2508.15835}
}
abstract

Language models are increasingly used in Brazil, but most evaluation remains English-centric. This paper presents Alvorada-Bench, a 4,515-question, text-only benchmark drawn from five Brazilian university entrance examinations. Evaluating twenty models under zero-shot, role-playing, and chain-of-thought prompting, producing 270,900 responses with structured self-reports of confidence, perceived difficulty, and Bloom level. The top models exceed 94% accuracy overall, but accuracy declines on Mathematics and on the engineering oriented IME and ITA exams, indicating persistent weaknesses in multi-step reasoning. Confidence is well calibrated and correlates with perceived difficulty, revealing that models can accurately assess their own certainty capabilities. A cost accuracy analysis shows that high accuracy is achievable at under $2 per 1K tokens. On ENEM 2024 the top model (O3) achieved perfect scores in Languages subject questions while even the weakest system (GPT-4.1 Nano) only underperforms humans in Mathematics. Through exams that distill decades of Brazilian educational priorities and assess millions of students yearly, Alvorada-Bench establishes whether language models can navigate the intersection of language, culture, and reasoning that defines academic readiness in Brazil.

Figures

Figures reproduced from arXiv: 2508.15835 by the authors.

Figure 1
Figure 1. Representative Question Examples from Alvorada-Bench [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Dataset Construction Pipeline The construction pipeline, illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of LLM models performance with Brazilian student baseline on ENEM 2024 across different [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: (a) Cost-efficiency frontier: price per 1K tokens vs accuracy. (b) Temporal evolution of model performance [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: (a) Model calibration: predicted vs actual accuracy. (b) Accuracy vs self-reported uncertainty. (c) Uncertainty [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Subject-level performance analysis across academic domains. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Model performance across different Brazilian entrance exams. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Analysis of prompt engineering effects across models and subjects. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Model accuracy stratified by Bloom’s taxonomy. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 10 canonical work pages

  1. [6]

    Examining the Behavior of LLM Architectures Within the Framework of Standardized National Exams in Brazil

    Marcelo Sartori Locatelli, Matheus Prado Miranda, Igor Joaquim da Silva Costa, Matheus Torres Prates, Victor Thomé, Mateus Zaparoli Monteiro, Tomas Lacerda, Adriana Pagano, Eduardo Rios Neto, Wagner Meira Jr., and Virgilio Almeida. Examining the Behavior of LLM Architectures Within the Framework of Standardized National Exams in Brazil. arXiv preprint arX...

    work page Pith review arXiv 2024

  2. [1]

    BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams

    Thales Sales Almeida, Thiago Laitz, Giovana K. Bonás, and Rodrigo Nogueira. BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams. arXiv preprint arXiv:2307.05410 , 2023. URL https: //arxiv.org/abs/2307.05410

    work page Pith review arXiv 2023

  3. [2]

    GPT-4 Technical Report

    OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2024. URL https://arxiv.org/abs/ 2303.08774

    arXiv 2024

  4. [3]

    Multilingual Performance Biases of Large Language Models in Education

    Vansh Gupta, Sankalan Pal Chowdhury, Vilém Zouhar, Donya Rooein, and Mrinmaya Sachan. Multilingual Performance Biases of Large Language Models in Education. arXiv preprint arXiv:2504.17720 , 2025. URL https://arxiv.org/abs/2504.17720

    work page Pith review arXiv 2025

  5. [4]

    Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs

    Yanzhu Guo, Simone Conia, Zelin Zhou, Min Li, Saloni Potdar, and Henry Xiao. Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs. arXiv preprint arXiv:2410.15956, 2025. URL https://arxiv.org/abs/2410.15956

    work page Pith review arXiv 2025

  6. [5]

    Comparing Large Language Models and Human Programmers for Generating Programming Code

    Wenpin Hou and Zhicheng Ji. Comparing Large Language Models and Human Programmers for Generating Programming Code. Advanced Science, 12(8), 2024. DOI: 10.1002/advs.202412279

    work page doi:10.1002/advs.202412279 2024

  7. [7]

    Primeiro , i d e n t i f i q u e o que a questao esta pedindo

    work page

  8. [8]

    D e c o m p o n h a os conceitos - chave ou as i n f o r m a c o e s f o r n e c i d a s

    work page

Show all 11 references
  1. [9]

    Analise cada a l t e r n a t i v a em relacao aos r e q u i s i t o s da questao

  2. [10]

    Elimine a l t e r n a t i v a s i n c o r r e t a s com j u s t i f i c a t i v a

  3. [11]

    resposta

    S el ec ion e a melhor resposta e avalie suas c a r a c t e r i s t i c a s Formato JSON exigido : {" resposta ": " letra " , " d i f i c u l d a d e ": 0 -10 , " co nf ia nc a ": 0 -10 , " bloom ": " Remember / U n d e r s t a n d / Apply / Analyze / Evaluate / Create "} ** I...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.