{"id":"ca85d2b8-2d9d-4817-b268-af3ba9ede910","arxiv_id":"2505.16590","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A fine-tuned 14B small open-source model with LoRA and RAG outperforms larger proprietary LLMs on automated Java logging statement generation in AL-Bench point estimates.","lead":"This paper tests small open-source language models for writing software logging statements, and finds a 14-billion-parameter model fine-tuned with LoRA and RAG beats larger proprietary models on this benchmark. The result matters because it suggests companies can build accurate logging tools locally, avoiding privacy risks and high API costs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ3 confounds model size with fine-tuning: all SOLMs are LoRA-tuned on 26,713 in-distribution instances while every LLM baseline is prompt-only, so the 'larger is not always better' title and Finding 8 remain unsupported until a larger model is tuned with the same recipe.","rationale":"The reader's weakest_assumption identifies the same confound as the most load-bearing issue: RQ3 compares fine-tuned SOLMs against prompt-only LLM baselines, so the observed advantage cannot be attributed to model size. I agree with that assessment and do not think a stronger objection is needed; the paper retains real value as an empirical study of practical, locally deployable logging generation, and the replication package plus multi-repo robustness checks are genuine supporting evidence. The central 'larger is not always better' claim, however, is not proven by the table as designed. A single missing ablation—fine-tuning a larger model with the same LoRA+RAG recipe—would settle whether the effect is scale or adaptation. The paper's own internal evidence (Table 4's large base-to-LoRA gains and the RQ4.2 long-code degradation) reinforces this concern rather than resolving it. I also note the acknowledged validity threats in Sections 6.1.2 and 6.3.2; they are secondary to the RQ3 confound. No statistical significance testing is reported, but adding it would not fix the confound. Verdict remains CONDITIONAL: the paper should either add the large-model fine-tuning comparison or rephrase the scope and title to claim that adapted small models are competitive, efficient alternatives, not that larger models are generally worse.","tokens_in":40745,"tokens_out":3723,"duration_ms":34289,"concrete_test":"Fine-tune a larger open model, e.g., Qwen2.5-coder-32B, with the exact LoRA configuration, RAG prompt, one-epoch training, learning rate 1e-4, and the same 26,713-instance training split used for Qwen2.5-coder-14B, then evaluate on the same 3,003-instance test set. If the 32B model's PA, LA, F1, BLEU-4, or ROUGE-L exceed Qwen2.5-coder-14B's 66.20, 69.92, 59.93, 24.05, and 46.51, the headline 'larger is not always better' is not established and the paper should be re-scoped to claim that fine-tuned SOLMs are competitive with, not superior to, prompt-only large LLMs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The title claim and Finding 8 both assert a scale effect: fine-tuned sub-14B SOLMs, especially Qwen2.5-coder-14B, outperform larger LLMs and existing tools on AL-Bench. The decisive evidence is Table 7 in RQ3, but that table varies two things at once. Every SOLM row is a LoRA-fine-tuned model using an RAG prompt, while every proprietary/LLM baseline row is used zero-shot or with prompting only; none of the large models receives the same 26,713-instance in-distribution fine-tuning. Thus the comparison shows that a specialized task-adapted small model beats generic prompted models; it does not show that smaller scale is the cause of the advantage. The paper's own RQ2.1 (Table 4) shows that fine-tuning is the dominant intervention: Qwen2.5-coder PA jumps from 3.40 to 62.40 after LoRA. Unless a large open model fine-tuned with the identical LoRA+RAG protocol fails to surpass Qwen2.5-coder-14B, the observed result is fully explained by fine-tuning rather than by 'larger is not always better.' The RQ4.2 long-code result (Table 11) further weakens the scale narrative: the fine-tuned SOLM degrades more than the LLM baseline in position accuracy on long code, which is the opposite of a clean scale advantage. The practical contribution—that a locally deployable fine-tuned SOLM is competitive and useful—remains credible, but the headline claim requires an ablation that the paper does not provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of small open-source language models (SOLMs, defined as models under 14B parameters) for automated logging statement generation on the AL-Bench benchmark. It evaluates four SOLMs with four prompting strategies and four parameter-efficient fine-tuning techniques, and reports that RAG prompting and LoRA fine-tuning are the most effective choices. The main claim is that fine-tuned SOLMs, especially Qwen2.5-coder-14B with LoRA and RAG, outperform existing logging tools and larger proprietary LLM baselines on location accuracy and statement quality, based on metrics such as PA, LA, PMR, F1, BLEU-4, ROUGE-L, and an LLM-as-a-judge scoring scheme. The paper also studies generalization across repositories and the effect of code length.","tokens_in":41072,"tokens_out":4367,"duration_ms":36260,"significance":"If the headline claim were established, the result would be practically valuable: a locally deployable, sub-14B model that beats GPT-4o and Claude3.7-sonnet on a well-known logging benchmark would give enterprises a privacy-preserving and cost-effective alternative. The paper has genuine strengths: it uses an external benchmark (AL-Bench) with a file-level split, explicitly addresses data leakage, releases a replication package, and conducts a fairly systematic comparison of prompting and PEFT strategies. However, the central scale claim is not currently supported by the experimental design, because the SOLM/LLM comparison in RQ3 varies model scale and fine-tuning simultaneously. The practical message about efficient, task-adapted SOLMs is credible; the theoretical message 'larger is not always better' is not yet established.","major_comments":[{"comment":"The headline comparison in Table 7 conflates model size with fine-tuning. Every SOLM row is LoRA-fine-tuned on 26,713 in-distribution AL-Bench instances and evaluated with a RAG prompt, while every proprietary and open LLM baseline is prompt-only. Table 4 shows that LoRA alone increases Qwen2.5-coder's position accuracy from 3.40 to 62.40, so fine-tuning is a dominant intervention. To support the title claim that 'larger is not always better' and Finding 8, the paper needs an ablation in which a large open model (e.g., Qwen2.5-coder-32B, LLaMA-3.1-70B, or an equivalent model) is fine-tuned with the identical LoRA+RAG protocol and shown not to match or exceed Qwen2.5-coder-14B. Without such an ablation, the data only support the narrower claim that a specialized, in-domain fine-tuned small model can beat generic prompted models.","section":"Section 4.3, Table 7 and Finding 8"},{"comment":"The headline comparisons are single-run point estimates without confidence intervals or significance tests. Several decisive margins are very small: position accuracy is 66.20 for Qwen2.5-coder-14B versus 65.90 for Claude3.7-sonnet-RAG, and the average LLM-judge score is 1.506 versus 1.489. These gaps may be within sampling noise, especially given test-set and repository variability. The paper should report confidence intervals (e.g., bootstrap by test instance or by repository) or repeated running of the stochastic components, and it should use a statistical test before claiming that a method 'outperforms' all baselines across all metrics.","section":"Section 4.3, Tables 7 and Figure 6"},{"comment":"The long-code robustness result is in tension with Finding 8 and the title. Table 11 shows that the fine-tuned Qwen2.5-coder-14B loses 13.43 PA points on long code, while the Deepseek-coder-v3-RAG baseline loses only 9.97; the paper itself attributes this to 'smaller model scale may face greater challenges' in Section 5.4.3. This is evidence that the larger baseline is more robust in one important dimension, which undercuts the general claim that smaller scale is preferable. The authors should either reconcile this with the scale narrative (for example, by testing large fine-tuned models on long code) or explicitly restrict the scope of the central claim to short- to medium-length code.","section":"Section 4.4, Table 11, Finding 13, and Section 5.4.3"}],"minor_comments":[{"comment":"The column header prints 'LOD' but Section 3.5.1 defines and uses AOD; rename the column to AOD for consistency.","section":"Table 7"},{"comment":"The approach paragraph enumerates only LANCE, LEONID, Unilog, and Fastlog as existing methods, but Table 7 and the subsequent results include SCLogger; add SCLogger to the list.","section":"Section 4.3 approach text"},{"comment":"In the base+RAG Short row, the ROUGE-L value is reported as 15.10, which conflicts with Table 3's ROUGE-L of 37.57 for the same configuration; this appears to be a copy-paste from the BLEU-4 column.","section":"Table 10"},{"comment":"The column header 'LOLS' is not defined; define it as 'lines of logging statements' or rename it for clarity.","section":"Table 2"},{"comment":"Several typos should be fixed in copy editing: 'Tradictional evaluating metrics' (Section 3.5.1), 'Empircial' (Section 4 heading), 'investegates' (Section 3), and 'automoated' (Findings 2 and 3).","section":"Throughout"},{"comment":"Reference [70] for AL-Bench is missing venue and date information ('[n. d.]'); update it to the full arXiv or publication entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's practical contribution is solid and the replication package is a plus, but the title and abstract currently overstate what the evidence shows. The editor may wish to require either the large-model fine-tuning ablation or a reframed title and claims. The authors should also be asked to avoid treating point estimates separated by fractions of a percentage point as definitive wins."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a solid empirical study of small open-source language models (SOLMs) for logging statement generation. What's genuinely new: it systematically evaluates four sub-14B models across prompting strategies, PEFT techniques, model sizes, and instruct variants on the external AL-Bench benchmark, with a file-level split and a promised replication package. The RAG plus LoRA synergy finding (LoRA+RAG beats LoRA with other prompts) is credible and useful. The scaling analysis within the Qwen family (0.5B to 14B) shows that within the fine-tuned SOLM class, larger models generally do better, which is a nuanced and believable result.\n\nThe soft spot is the title claim. \"Larger is not always better\" and Finding 8 compare fine-tuned SOLMs against prompt-only proprietary LLMs in Table 7. That comparison varies two things at once: fine-tuning (26,713 in-distribution instances, LoRA) and model scale. Given that Table 4 shows LoRA lifting Qwen's position accuracy from 3.40 to 62.40, the advantage of the SOLM over Claude or Deepseek is almost certainly driven by task-specific fine-tuning, not by smaller size. The paper does not run the reverse ablation—fine-tuning a large open or proprietary model with the same LoRA+RAG protocol—so the scale narrative is unsupported. The paper's own RQ4.2 shows the fine-tuned SOLM drops more than the LLM baseline on long-code position accuracy, which cuts against a clean scale advantage.\n\nOther concerns are real but minor. No confidence intervals or significance tests; the margin over Deepseek-coder-v3-RAG on PA is 1.0 point. The LLM-as-judge is used without validating its scores against human judges, so treat that as suggestive. The study is Java-only, which is acceptable but should be caveated more strongly.\n\nThe evaluation is not circular: AL-Bench is external, the split is file-level, and RAG retrieval from the validation set does not touch the test set. The threats-to-validity section is honest.\n\nThis paper deserves a serious referee. The practical message—that a locally deployable, fine-tuned sub-14B model can be very competitive for this specialized task—is valuable and well-supported. The fix is to either add a same-protocol fine-tuned large-model comparison or reframe the claims to \"a specialized fine-tuned SOLM can beat generic prompted LLMs\" and drop the size narrative. I'd accept it for review with that expected.","headline":"Useful empirical study of fine-tuned small models for logging generation; the 'larger is not always better' headline is not established because model size is confounded with fine-tuning.","tokens_in":41680,"tokens_out":2705,"would_cite":true,"duration_ms":23013,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned small open-source language models under 14 billion parameters, especially Qwen2.5-coder-14B with LoRA and RAG, beat existing tools and larger proprietary LLMs at predicting where to log and what to write.","keywords":["automated logging statement generation","small open-source language models","parameter-efficient fine-tuning","Low-Rank Adaptation (LoRA)","retrieval-augmented generation","empirical study","AL-Bench","software maintenance"],"falsifier":"Fine-tune a large (e.g., 70B-parameter) open model with the same LoRA+RAG protocol on the same 26,713 AL-Bench training instances and evaluate it on the same test set; if it matches or exceeds Qwen2.5-coder-14B's 66.20% position accuracy, the claim that smaller models are the better choice collapses. A supporting check is to repeat the evaluation on logging statements collected after these models' training cutoffs to rule out memorization of AL-Bench.","tokens_in":40524,"feed_emoji":"🪵","tokens_out":12117,"duration_ms":83956,"temperature":0.7,"pith_summary":"This paper reports the first large-scale empirical study of small open-source language models (SOLMs, under 14 billion parameters) for automated logging statement generation. The authors test four SOLMs on the AL-Bench dataset, varying prompting strategies and parameter-efficient fine-tuning, and find that a specific recipe—an instruction-tuned model, LoRA fine-tuning, and retrieval-augmented generation with a similar-example prompt—lets Qwen2.5-coder-14B exceed both specialized logging tools and much larger proprietary LLMs on location accuracy and statement quality. If the finding holds, enterprises could deploy locally hosted logging assistants that preserve code privacy, lower cost, and still reach top results. The claim rests on the assumption that in-distribution fine-tuning, not model scale, is what the comparison actually isolates.","feed_headline":"Larger is not always better: a 14B model wins at logging","feed_subtitle":"LoRA tuning plus retrieval prompts lets a sub-14B open model beat proprietary LLMs on the AL-Bench logging task.","key_machinery":"The load-bearing mechanism is the LoRA+RAG synergy: LoRA injects low-rank adapter matrices that specialize a frozen base model to the logging task using only a small fraction of trainable parameters, while RAG retrieves the single most similar code snippet (via BM25) from a validation pool and prepends it as a one-shot example during inference. The paper shows this combination beats either technique alone and remains the best configuration across code lengths, and the AL-Bench benchmark (10 Java repositories, 33,224 instances split at file level to avoid leakage) supplies the measuring instrument, with position accuracy, level accuracy, variable precision and F1, BLEU-4, ROUGE-L, and an LLM-based judge as the metrics.","core_discovery":"The central discovery is that a well-optimized sub-14B open-source model can be the top performer in automated logging. On AL-Bench, fine-tuned SOLMs with LoRA and RAG prompts outperform all five existing tools and four larger LLM baselines across position accuracy, level accuracy, variable matching, and text-generation metrics, and the LLM-as-a-judge evaluation agrees. The best configuration is Qwen2.5-coder-14B with LoRA and RAG, reaching 66.20% position accuracy, 69.92% level accuracy, 59.93% variable F1, and the highest judge scores; the paper reads this as evidence that for specialized logging tasks targeted adaptation outweighs raw model scale.","pith_inferences":["Beyond the paper: the headline comparison may reflect in-distribution fine-tuning advantage rather than model scale, since the large LLM baselines were prompted without being fine-tuned on the same 26,713 training instances.","Beyond the paper: the same LoRA+RAG recipe could be tested on other code-generation tasks, such as comment generation or test generation, to see whether the synergy transfers beyond logging.","Beyond the paper: because RAG-based retrieval quality depends on the validation pool, using a per-project knowledge base may matter more than model size in real deployments, and the observed failures to invoke out-of-context methods suggest that adding static-analysis call-graph context could improve the approach further."],"forward_implications":["A sub-14B model fine-tuned for under six hours on a single A100 GPU can match or beat large proprietary LLMs at automated logging, enabling local, privacy-preserving deployment.","The LoRA+RAG combination is synergistic: task-level specialization from fine-tuning and instance-level context from retrieval together define the best-performing configuration.","Logging performance scales with size only from 3B parameters upward; models below 3B scale inconsistently, and a single fine-tuning epoch is enough to avoid overfitting.","SOLMs generalize to unseen repositories, and training on projects with consistent logging conventions, such as Apache-style practices, noticeably improves that generalization.","Longer code hurts logging-position accuracy but improves level, variable, and text quality, posing a design trade-off for real-world tools."],"supporting_citations":[{"why":"Supplies the AL-Bench dataset of 10 Java repositories used to train, validate, and test every model and tool in the study.","marker":"[70]"},{"why":"Defines Low-Rank Adaptation (LoRA), the parameter-efficient fine-tuning technique that carries the performance gains.","marker":"[22]"},{"why":"Defines Retrieval-Augmented Generation (RAG), the prompting strategy that provides the instance-specific example at inference.","marker":"[36]"},{"why":"SCLogger, the strongest specialized logging-tool baseline that the fine-tuned SOLMs are compared against.","marker":"[43]"},{"why":"UniLog, the LLM in-context-learning logging baseline that this study extends and competes with.","marker":"[77]"},{"why":"FastLog, a state-of-the-art end-to-end logging tool baseline used in the RQ3 comparison.","marker":"[76]"},{"why":"LANCE, the pioneering deep-learning logging tool whose task formulation and results anchor the comparison.","marker":"[55]"},{"why":"The prior empirical study of LLMs for automated logging that defines the task framing and evaluation metrics used here.","marker":"[42]"},{"why":"BM25, the retrieval algorithm used to select the most similar code snippet for the RAG prompt.","marker":"[63]"},{"why":"The LLM-as-a-judge methodology the paper uses to corroborate the traditional metric scores.","marker":"[72]"}],"fun_headline_variants":["Sub-14B open model beats larger LLMs at logging","Small open model wins logging with LoRA and RAG","14B model outperforms bigger LLMs in logging","Smaller models, better logs: Qwen2.5-14B wins","Fine-tuned 14B beats GPT-sized models on logging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that fine-tuning on 26,713 in-distribution AL-Bench examples is not the true cause of the small models' wins; if a large model fine-tuned the same way also surpassed them, the 'larger is not always better' conclusion would fall.","fun_headline_variants_meta":{"raw":{"variants":["Sub-14B open model beats larger LLMs at logging","Small open model wins logging with LoRA and RAG","14B model outperforms bigger LLMs in logging","Smaller models, better logs: Qwen2.5-14B wins","Fine-tuned 14B beats GPT-sized models on logging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000838,"raw_usage":{"total_tokens":3624,"prompt_tokens":883,"completion_tokens":2741,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2654}},"tokens_in":499,"tokens_out":2741,"duration_ms":14425,"temperature":1.0,"reasoning_tokens":2654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:58:19.484753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a large (e.g., 70B-parameter) open model with the same LoRA+RAG protocol on the same 26,713 AL-Bench training instances and evaluate it on the same test set; if it matches or exceeds Qwen2.5-coder-14B's 66.20% position accuracy, the claim that smaller models are the better choice collapses. A supporting check is to repeat the evaluation on logging statements collected after these models' training cutoffs to rule out memorization of AL-Bench.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Retrieval-Augmented Generation (RAG), the prompting strategy that provides the instance-specific example at inference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SCLogger, the strongest specialized logging-tool baseline that the fine-tuned SOLMs are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UniLog, the LLM in-context-learning logging baseline that this study extends and competes with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FastLog, a state-of-the-art end-to-end logging tool baseline used in the RQ3 comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LANCE, the pioneering deep-learning logging tool whose task formulation and results anchor the comparison."},{"cited_title":"Briand, and Michael R","cited_arxiv_id":null,"evidence_quote":"The prior empirical study of LLMs for automated logging that defines the task framing and evaluation metrics used here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BM25, the retrieval algorithm used to select the most similar code snippet for the RAG prompt."}],"review_version":1}