{"id":"390e14e0-9fd6-4f75-b5d2-7a7ea889fb34","arxiv_id":"2502.08655","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper presents a hybrid expert-system and LLM adaptive LMS prototype and a benchmark of ten LLMs on standardized tests, showing self-hosted models are competitive with proprietary ones in reading, writing, and coding but weak in mathematics.","lead":"This paper describes a prototype adaptive learning management system that combines an expert system with large language models and benchmarks ten LLMs on Canadian standardized test questions in math, reading, writing, reasoning, and coding. It finds that self-hosted models perform nearly as well as proprietary models in most categories, which matters for educators weighing cost, privacy, and performance in AI tutoring tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hardware-utilization half of the central claim is unsupported: Section V-G measures local client CPU/memory for API calls, not the remote inference hardware, so the Section VII comparison of self-hosted vs. proprietary resource use compares different physical systems.","rationale":"I read the strongest claim as two-part: self-hosted models are comparable to API-accessed proprietary models in problem-solving competency and in hardware utilization. The reader's concern about unmatched human norms is legitimate and would weaken the 'weak/strong' summary in Section VI-A6, but that comparison is not necessary for the first part of the strongest claim. The more direct threat to the stated conclusion is the hardware-utilization comparison. Table IV metrics are local process/system metrics; for API calls they exclude the inference hardware. Section VI-B's 'surprising' finding that proprietary calls use equal or slightly more local resources than self-hosted inference is an artifact of measuring the client rather than the model. This does not refute the problem-solving results, so the reader's CONDITIONAL verdict remains appropriate; a revised version should remove or reframe the hardware-utilization claim and clarify that the resource metrics are a local-footprint comparison. I partially agree with the reader: the human-norm issue is real and should be fixed, but the single most load-bearing part of the stated conclusion is the hardware measurement.","tokens_in":18664,"tokens_out":9160,"duration_ms":94473,"concrete_test":"Re-run or re-analyze the resource-utilization experiment with instrumentation that separates client-side overhead from inference-side usage. For the three OpenAI models, log local CPU, memory, and execution time for a pure no-op network request to the same endpoint, and also record server-side inference time or cost (e.g., from API response timing or provider billing) for each question; then compare self-hosted local inference resources against this full API-side resource picture. If the local-only API metrics still match self-hosted inference metrics, the Section VII hardware-utilization claim remains; if the remote inference resources are substantial and unmeasured, the claim must be removed or reframed as a client-footprint comparison only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section VII's strongest claim is that self-hosted LLMs are 'very capable, both in terms of problem-solving competency and hardware utilization, relative to proprietary models accessed via API.' The evidence in Sections V-G and VI-B cannot support the hardware-utilization part. Table IV defines Avg CPU%, Avg Mem%, system time, user time, and execution time as metrics of the local test application and its subprocess tree. For API models (GPT-3.5, GPT-3.5 Instruct, GPT-4), the actual LLM inference occurs on OpenAI's servers and never appears in these counters; the measured CPU, memory, and time belong to the HTTP client, TLS/JSON handling, and local polling. For self-hosted models, the same counters include local Ollama inference. Pooling these two conditions under the label 'hardware utilization' does not compare the models' hardware demands. Section VI-B's 'surprising' finding that proprietary calls used equal or slightly more local resources than self-hosted inference is an artifact of measuring the client rather than the model. This directly affects the conclusion that an ALMS can choose self-hosted models without a hardware penalty: the data show only that API calls are not free of local overhead, not that remote inference hardware is comparable to local inference hardware. The problem-solving comparison may survive, but the hardware-utilization claim as stated does not follow from the measurements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the design and development of an adaptive learning management system (ALMS) called Komodo, built by integrating an expert-system-style test-bank search with multiple large language models (LLMs). The system was developed in three phases: a Django/React expert system with OCR and question matching, an LLM question-answering interface with retrieval-augmented generation, and a benchmark study comparing ten LLMs (including proprietary GPT-3.5/GPT-4 and self-hosted 7B models plus Phi-2) on mathematics, reading, writing, reasoning, and coding questions drawn from standardized tests. The authors conclude that self-hosted LLMs are comparable to proprietary API models in both problem-solving competency and hardware utilization, and that LLMs are relatively strong in reading, writing, and coding, moderate in reasoning, and weak in mathematics.","tokens_in":18870,"tokens_out":7020,"duration_ms":76726,"significance":"The paper's practical claim that small self-hosted models can substitute for expensive proprietary APIs in an adaptive LMS is potentially valuable for privacy-conscious, low-cost deployments. The work contributes a concrete system architecture and a benchmark that uses real standardized-test items, and it has several strengths: the benchmark is empirical with no fitted parameters, the essay grading was blinded with model names omitted, and the authors explicitly discuss confounding variables such as responses without identifiers. However, the significance of the empirical conclusions is currently limited by methodological gaps, as detailed in the major comments: the reported dataset size is inconsistent with the stated test design, the human-performance comparisons rely on published norms rather than a matched control group, the hardware-utilization claims conflate local client overhead with model inference, and the mathematics difficulty paradox is explained with an internally contradictory guess-probability argument.","major_comments":[{"comment":"The reported total of 3,780 data points is inconsistent with the question counts in Table II and the statement in Section V-A that each model was tested on the entire battery three times over. Summing the question-round products for all categories (Math: 90, Reading: 90, Writing: 61, Reasoning: 90, Coding: 15, all multiplied by 10 LLMs) gives 3,460 data points, not 3,780. The dataset size should be corrected in all three locations and the void-response rate in Section VI-C recomputed.","section":"Sections V-A, V-H, VI-C"},{"comment":"The comparison to human performance is based on published norm statistics from EQAO, ACT, OSSLT, and LSAT rather than on a human control group that sat the identical test battery under the same conditions (text-only input, no diagrams, single-letter response format, and volunteer-graded essays). Because the administration protocol differs substantially, the qualitative labels 'weak in mathematics,' 'strong in reading,' and similar claims relative to humans are not established. The authors should either add a matched human control condition or explicitly limit these conclusions to 'performance relative to published norms collected under different administration conditions.'","section":"Sections VI-A6 and VII"},{"comment":"The hardware utilization metrics (CPU%, memory%, system/user time, execution time) are measured on the local client and its process tree. For API-based models (GPT-3.5, GPT-3.5 Instruct, GPT-4), the actual model inference runs on remote servers and is not included in these counters; the measurements capture the HTTP client, JSON handling, and local polling. Consequently, the conclusion in Section VII that self-hosted LLMs are 'very capable ... in terms of hardware utilization' relative to proprietary API models is not supported. The authors should distinguish 'local client resource usage' from 'model inference resource usage' and restrict their hardware claims accordingly.","section":"Sections V-G, VI-B, VII"},{"comment":"The explanation of the mathematics difficulty paradox is internally contradictory. The authors state that hard questions have five multiple-choice options whereas easy and medium questions have four, and they claim this leads to a 'higher number of lucky guesses' for the hard set. In fact, a larger number of options reduces the probability of a correct random guess, so this mechanism cannot explain the observed inverse relationship between difficulty and score. The passage should be corrected or the paradox left explicitly unresolved.","section":"Section VI-A1"},{"comment":"Model rankings and comparative statements are presented without any measure of variability or statistical testing. For example, Section VI-A5 states that Falcon 7B 'scored the highest, but only by a single point' on the basis of five questions per difficulty, and Section VI-A3 reports that GPT-4 scored 33 versus Phi-2's 30 on a single essay per model. Because the number of items per cell is small (often five or ten, and one for the hard writing test), these differences may not be reliable. The authors should report standard errors, confidence intervals, or significance tests, or temper the comparative conclusions.","section":"Sections VI-A and VI-B"}],"minor_comments":[{"comment":"The phrase 'Informal experimentation was performed on the self-hosted and proprietary models' capabilities in system prompt modification' is awkward; consider 'with' instead of 'in' and rephrasing to make the object of experimentation clear.","section":"Section IV-B"},{"comment":"The entries 'GPT3 5' and 'GPT3 5 I' are unclear; they likely denote GPT-3.5 and GPT-3.5 Instruct but should be written out. The CodeLlama 7B row is missing an entry in the Parameters column, and the 'HDD Space' column for proprietary models uses '-' without explanation.","section":"Table III"},{"comment":"The sentence 'Each model was tested on the entire battery three times over' conflicts with Table II, which shows one round for the hard writing question and the coding questions; clarify that the 'three times' applies only to the 10-question sets.","section":"Section V-A"},{"comment":"The phrase 'anti-perfect scores' is nonstandard; it presumably means that every question was answered incorrectly, but the terminology should be defined or replaced.","section":"Section VI-A3"},{"comment":"The sentence 'the range between the highest and lowest-performing models was so small that this was not a significant concern' uses 'significant' informally; since no statistical test is performed, avoid the statistical term or provide a quantitative basis for the claim.","section":"Section VI-B"},{"comment":"The header 'Total V oid Responses' contains a stray space, and the column label 'Explanation Only' is unclear; it appears to mean 'provided an explanation but no identifier,' which should be stated explicitly.","section":"Section VI-C, Table V"},{"comment":"The phrase 'hardware intensive' should be hyphenated as 'hardware-intensive' for consistency with standard usage.","section":"Section VII"},{"comment":"In Section VI-A3, the claim 'All models had perfect scores on the easy test' is ambiguous because the easy category was not a single test; clarify that it refers to the easy writing questions.","section":"Section V-D"},{"comment":"The description of cgroup usage says that getrusage() 'accesses cgroup usage data,' but getrusage() returns process resource usage statistics and does not directly access cgroups; the explanation should be clarified.","section":"Section V-G"},{"comment":"Table I is titled 'Requirements by Parameter Count' but only lists memory demand; since the text discusses hardware requirements more broadly, either rename the table or add other resources such as GPU/CPU considerations.","section":"Section II-C, Table I"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an engineering report with a genuinely new dataset—ten LLMs, including several 7B self-hosted models, run on Canadian standardized test questions (EQAO, OSSLT, ACT, LSAT). The authors deserve credit for transparent grading: they describe the volunteer essay graders, the pass/fail unit tests for coding, and the 35 void responses in Table V. The qualitative findings (weak math, strong reading/writing, sensitivity to wording) match prior literature, which is a healthy sanity check. The paper ships no code or data, but it clearly documents how the numbers were produced.\n\nThe soft spots are real and load-bearing in places. Most important: Section VII's claim that self-hosted models are \"very capable... in terms of hardware utilization\" does not follow from the measurements. Section V-G records CPU, memory, and execution time of the local test application and its subprocess tree. For API models, that captures the HTTP client, TLS, JSON handling, and polling—not the actual inference on OpenAI's servers. For self-hosted models, the same counters include local Ollama inference. Pooling these as \"hardware utilization\" compares different physical systems. The problem-solving comparison survives, but the hardware conclusion should be dropped or re-measured.\n\nSecond, the paper states 3,780 data points, but Table II's counts (with ten models) sum to 3,460. That's a 320-row discrepancy, appearing in Sections V-A and V-H. It needs correction. Third, the human comparison relies on published norms, not a matched control group taking the same test under the same protocol (text-only, no diagrams, single-letter output constraints). The \"strong/weak relative to humans\" language in Section VI-A6 is therefore approximate at best; the paper should frame those labels as indirect comparisons.\n\nMinor: no error bars or significance tests, though with 10 questions per cell that's a known limitation rather than a fatal one. The math difficulty paradox explanation is muddled—the different option counts don't obviously explain an inverse score-difficulty relationship—but it's a side remark.\n\nOverall: the empirical benchmark is worth a serious look, but the paper overclaims in its conclusion. A revision that fixes the count, re-scopes or removes the hardware claim, and releases the data/scripts would be a solid contribution to the educational-LLM community. I'd send it to review, conditional on major revision, not desk-reject it.","headline":"Useful benchmark with a clearly unsupported hardware-utilization claim and an arithmetic slip in the data count; the problem-solving half is plausible and worth a careful revision.","tokens_in":19436,"tokens_out":2723,"would_cite":false,"duration_ms":26346,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark of ten language models finds that small self-hosted models perform about as well as large proprietary API models across most educational tasks, making low-cost, private adaptive tutoring feasible.","keywords":["adaptive learning management system","large language models","self-hosted LLMs","retrieval augmented generation","personalized learning","standardized test benchmarking","LLM resource utilization","hallucination mitigation"],"falsifier":"Administer the same 150-question battery to a matched group of human students under the identical text-only, no-diagram, forced-choice conditions and compare their scores against the published norms the paper relies on; if the human sample deviates substantially from those norms, the paper's relative rankings of LLM strengths and weaknesses lose their anchor. A second check is to rerun the ten models on a fresh set of never-published standardized questions and see whether the self-hosted/proprietary parity and the sensitivity-to-wording effects persist.","tokens_in":18413,"feed_emoji":"🎓","tokens_out":5770,"duration_ms":51281,"temperature":0.7,"pith_summary":"This paper argues that an adaptive learning management system (ALMS) can deliver personalized education by combining a small set of large language models rather than relying on one general-purpose chatbot. To support that, it benchmarks ten LLMs--several small self-hosted models and larger proprietary API models--on 3,780 responses across mathematics, reading, writing, reasoning, and coding, using questions drawn from standardized tests. The paper's central conclusion is that self-hosted 7-billion-parameter models perform about as well as proprietary API models on most tasks while using comparable hardware resources, so a system can delegate each task to the cheapest suitable model without sacrificing quality. If right, this makes privacy-preserving, low-cost adaptive tutoring feasible for schools and institutions with limited instructor availability.","feed_headline":"Self-hosted LLMs match paid APIs in tutoring benchmark","feed_subtitle":"Ten-model standardized-test sweep finds small local AI near parity with big API models; math remains the weak spot.","key_machinery":"The load-bearing mechanism is task-specialized model delegation inside an expert-system-style pipeline: instead of one general LLM, the ALMS routes each question or learning task to the LLM best suited to it. The paper's evidence for this mechanism is a stratified benchmark battery of ten models across five categories and three difficulty levels, with grading automated for multiple-choice items, rubric-based for essays, and unit-test-based for code. Supporting machinery includes retrieval-augmented generation with vector embedding, which lets the system answer from a curated knowledge base rather than from the model's training memory, and resource-utilization tracking that compares CPU, memory, and execution time between self-hosted and API models. The benchmark is what carries the claim that delegation plus RAG can match proprietary API quality at lower cost and higher privacy.","core_discovery":"According to the paper, when ten LLMs were tested over three rounds on questions from EQAO, ACT, LSAT, OSSLT, and custom coding problems, the models collectively came out weak in mathematics, strong in reading and writing, strong in CS1-level coding, and moderate in reasoning relative to published human norms. The key observation was not just the ranking but the parity: small self-hosted models such as a 7-billion-parameter Mistral and the 2.3-billion-parameter Phi-2 scored close to, and sometimes above, proprietary models with far larger parameter counts, and resource-utilization measurements showed no systematic hardware penalty for self-hosting. The paper therefore concludes that an ALMS can rely on self-hosted models for most subtasks, reserving APIs only where needed, while using retrieval-augmented generation to curb hallucination and keep answers grounded in curated sources.","pith_inferences":["Beyond the paper's benchmark, the observed self-hosted/proprietary parity suggests the per-student cost of AI tutoring could drop sharply, since self-hosted models avoid per-token API fees.","The near-identical wording across different models on the easy writing prompts points to shared fine-tuning data or model distillation; a similarity analysis over response corpora could test this directly.","A stronger test of the paper's design would deploy the ALMS in a real classroom and measure learning gains against a standard LMS, since the present study measures model aptitude rather than student outcomes."],"forward_implications":["An adaptive LMS can be built around a small set of self-hosted models, with proprietary APIs reserved for specific tasks, preserving data privacy and cutting operating cost.","Mathematics is the category where LLM tutoring needs the most scaffolding, since all tested models, including GPT-4, performed poorly relative to human norms.","Retrieval-augmented generation from a curated knowledge base is a practical guard against hallucination, since RAG tests produced 'not found' answers instead of confident fabrications.","Wording sensitivity is a systematic failure mode: models answered consistently but were strongly influenced by phrasing, so prompt engineering should be part of any ALMS.","Because self-hosted 7B models matched API models on hardware metrics, institutions with mid-range hardware can realistically host their own tutoring stack."],"supporting_citations":[{"why":"Supplies the prior evidence that LLMs are weak at mathematics, which the paper's own math results confirm.","marker":"[21]"},{"why":"Establishes retrieval-augmented generation as a method for reducing hallucination, a core component of the proposed ALMS.","marker":"[23]"},{"why":"Shows how RAG can be used in an AI tutor to keep answers grounded in course content, the architecture the paper adapts.","marker":"[24]"},{"why":"Provides the basis for expecting LLMs to be strong at reading and general-knowledge tasks.","marker":"[17]"},{"why":"Motivates the shift to self-hosted models by documenting hallucination and privacy risks of general LLMs.","marker":"[8]"},{"why":"Supports the educational premise that tutoring systems can approach human-tutor effectiveness, justifying the ALMS goal.","marker":"[10]"},{"why":"Argues hallucinations are inherent to LLM training, motivating the paper's reliance on RAG and curated sources.","marker":"[37]"},{"why":"Supplies the comparison point that LLMs can produce CS1-level code, which the coding results extend.","marker":"[20]"}],"fun_headline_variants":["Tiny AI models rival cloud APIs in tutoring trials","Small self-hosted models keep pace with big LLMs in exams","Self-hosted LLMs near parity with paid APIs; math weak spot","Local LLMs match big APIs on tests, but math stumps all","7B model matches GPT-class API in tutoring benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison of LLM performance to human performance rests on the assumption that published norm statistics for tests like EQAO, ACT, and LSAT are directly comparable to LLM scores obtained under a different protocol, with text-only input, no diagrams, single-letter output constraints, and volunteer rubric grading.","fun_headline_variants_meta":{"raw":{"variants":["Tiny AI models rival cloud APIs in tutoring trials","Small self-hosted models keep pace with big LLMs in exams","Self-hosted LLMs near parity with paid APIs; math weak spot","Local LLMs match big APIs on tests, but math stumps all","7B model matches GPT-class API in tutoring benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001517,"raw_usage":{"total_tokens":6048,"prompt_tokens":886,"completion_tokens":5162,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":5085}},"tokens_in":502,"tokens_out":5162,"duration_ms":36746,"temperature":1.0,"reasoning_tokens":5085,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:45:09.759582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Administer the same 150-question battery to a matched group of human students under the identical text-only, no-diagram, forced-choice conditions and compare their scores against the published norms the paper relies on; if the human sample deviates substantially from those norms, the paper's relative rankings of LLM strengths and weaknesses lose their anchor. A second check is to rerun the ten models on a fresh set of never-published standardized questions and see whether the self-hosted/proprietary parity and the sensitivity-to-wording effects persist.","supporting_citations":[{"cited_title":"Active retrieval augmented generation,","cited_arxiv_id":null,"evidence_quote":"Establishes retrieval-augmented generation as a method for reducing hallucination, a core component of the proposed ALMS."},{"cited_title":"LLM lies: Hallucinations are not bugs, but features as adversarial examples,","cited_arxiv_id":null,"evidence_quote":"Motivates the shift to self-hosted models by documenting hallucination and privacy risks of general LLMs."},{"cited_title":"The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems,","cited_arxiv_id":null,"evidence_quote":"Supports the educational premise that tutoring systems can approach human-tutor effectiveness, justifying the ALMS goal."},{"cited_title":"Llm lies: Hallucinations are not bugs, but features as adversarial examples,","cited_arxiv_id":null,"evidence_quote":"Argues hallucinations are inherent to LLM training, motivating the paper's reliance on RAG and curated sources."},{"cited_title":"Examining student use of ai in cs1 and cs2,","cited_arxiv_id":null,"evidence_quote":"Supplies the comparison point that LLMs can produce CS1-level code, which the coding results extend."}],"review_version":1}