Pith. sign in

REVIEW 3 major objections 6 minor 45 references

COSMosFL: Ensemble of Small Language Models for Fault Localisation

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A voting ensemble of small open-source language models can match or beat the best single model's fault-localisation accuracy at the same or lower cost, offering a middle option between expensive closed LLMs and weaker individual small…

desk verdict A useful empirical study of SLM ensembles for fault localization whose headline Pareto-optimality claim is not supported by the current cost-accuracy comparisons. read the letter →

arxiv 2502.02908 v1 pith:RHDVEP6D submitted 2025-02-05 cs.SE cs.LG

Hyunjoon Cho , Sungmin Kang , Gabin An , Shin Yoo This is my paper · ORCID
classification cs.SEcs.LG
keywords faultlocalisationsmalllanguagemodelsensemblemethodsvotingself-consistencydifferentialevolutioncost-benefittrade-offDefects4J
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a task-level voting ensemble of small open-source language models can match or beat the fault-localisation accuracy of the best single model at the same or lower cost, giving users a middle option between expensive closed LLMs and weaker individual small models. The authors build COSMosFL by replacing the repeated self-consistency runs of one model in AutoFL with runs from four complementary SLMs, then aggregating their ranked suspicious-method scores by voting. On 353 Defects4J bugs they report that ensembles outperform every individual model at higher rank cutoffs, and that the ensemble sits on the Pareto frontier when accuracy is traded against energy, inference time, and token count. The work matters because it suggests that locally deployable open-source models can be combined to narrow the accuracy gap with large proprietary models without surrendering privacy or blowing up cost.

What carries the argument

The load-bearing object is the task-level voting ensemble: COSMosFL runs each member SLM several times under the AutoFL agent protocol, converts each run's predicted buggy set into per-method confidence scores (each predicted method gets a score of 1 divided by the set size), and sums these scores across runs and models to produce a final ranking of suspicious methods. The ensemble's power comes from model orthogonality: the four chosen models (Llama3 8B, Llama3.1 8B, Mistral NeMo 12B, and Qwen2.5-Coder 7B) rank largely disjoint bug sets at the top, so their votes cover each other's blind spots. Weight optimisation via Differential Evolution, with 10-fold cross-validation to limit overfitting, is the second mechanism, though the reported results show that equal weights perform about as well as optimised weights.

What would settle it

Run COSMosFL's model-selection protocol on a different fault-localisation benchmark, such as a fresh set of real-world bugs not used during selection, and check whether the chosen four-model ensemble still matches or beats the best individual model at the same token budget; if it does not, the Pareto-optimality claim is specific to the original dataset.

Watch

Extended reading notes

Core claim

The central discovery claimed is that heterogeneous small language models make mistakes on largely disjoint sets of bugs, and that a simple vote over their ranked fault hypotheses exploits this orthogonality: the ensemble's acc@k rises above every member model at higher k, while its cost lies between the cheapest and most expensive member. When voting weights are optimised with Differential Evolution, accuracy does not improve over equal weighting and the learned weights stay near uniform, suggesting the complementary behaviour already carries the benefit. The paper also reports a cost-accuracy analysis showing that under constraints on energy, inference time, or token count, the ensemble can be Pareto-optimal: for a given budget there is an ensemble composition whose accuracy is at least as high as any single model at that budget. The authors adapt the AutoFL agent for small models by dropping the class-level coverage tool, which the small models tended to call redundantly.

Load-bearing premise

The ensemble members were chosen using preliminary runs on the exact same 353 bugs used for the reported evaluation, so the claimed Pareto gains may not survive on a different dataset.

Editorial extensions

If this is right

  • A user with a fixed energy or token budget gets better fault-localisation accuracy from the ensemble than from any single small model at that budget.
  • Because all members are open-source and 4-bit quantised, the full setup can run on a single consumer GPU without sending code to a closed API.
  • Equal-weight voting captures most of the ensemble benefit, so practitioners do not need to tune voting weights to see gains.
  • The ensemble's accuracy advantage grows as the developer inspects a short ranked list rather than only the top hit, since acc@k gains are larger for k = 3 to 5.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The model-selection step, which picked four of seven models using five-run preliminary accuracy on the same 353 bugs used in the reported evaluation, is a potential overfit point; a held-out dataset would confirm whether the orthogonality benefit transfers to new projects.
  • The same task-level voting recipe could be lifted to other self-consistency-boosted software engineering tasks, such as program repair or test generation, wherever candidate outputs can be scored and ranked.
  • If model strengths are bug-type specific, a per-bug router that delegates to the most suitable member could beat the static ensemble, a direction the paper itself flags.
  • The energy and time measurements are distorted by an Ollama infinite-generation bug that produces extreme outliers; re-running with a fixed serving stack could shift the Pareto frontier for those two cost axes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces COSMos, a task-level voting ensemble of small open-source language models, and instantiates it for fault localization as COSMosFL. The approach replaces repeated sampling of a single LLM (self-consistency in AutoFL) with repeated runs of four selected SLMs, aggregates their per-method confidence scores using either equal weights or weights optimized by differential evolution, and evaluates the resulting rankings on 353 Defects4J bugs. The authors report accuracy (acc@k, with emphasis on acc@1) and three cost metrics (GPU energy, execution time, token count) as the number of runs varies. The headline claim is that ensembles can achieve Pareto-optimality in the accuracy-cost plane relative to individual models.

Significance. The paper addresses a timely and practically relevant problem: reducing dependence on large closed LLMs in software engineering while preserving acceptable accuracy through ensembles of locally deployable SLMs. Its strengths include a reproducible public implementation, a substantial experimental effort (30 runs per model and 20 re-sampled estimates), direct measurement of energy and time rather than only token counts, and consideration of both equal and optimized voting. The paper is also, to my knowledge, the first task-level voting ensemble for LLM-based fault localization, which is a useful novelty. If the Pareto claim were properly supported, the contribution would be valuable. As it stands, however, the evidence demonstrates only that ensemble configurations occupy intermediate cost-accuracy positions, and the model-selection step on the same benchmark weakens external validity. The paper is a useful empirical data point and a reasonable starting point, but the central claim needs additional analysis or a more modest reframing.

major comments (3)
  1. [Section IV-B, Figures 8a-8c] The Pareto-optimality claim in the abstract is not established by the reported cost-accuracy comparisons. The single-model points represent R runs of one model with R from 4 to 24, while the ensemble points represent R=4*Rm runs with Rm=1..6 distributed across four models. Since the four SLMs have different per-run energy, time, and token costs, an ensemble point and a single-model point with the same R have different total costs. The observation in the text that ensemble cost and performance 'tend to lie between these two extremes' is a description of location in cost-accuracy space, not a demonstration of Pareto dominance. To support the claim, the authors need to compare, for each ensemble configuration and each cost budget, against the best single-model configuration (including any number of repetitions) that fits within the same budget. Such cost-matched baselines are absent; without them, one cannot rule out that spending the ensemble's cost on additional runs of Qwen2.5-Coder alone would yield both higher acc@1 and lower cost.
  2. [Section III-A, RQ1] The ensemble membership is not chosen externally. The four member models were selected from seven candidates based on five-run preliminary evaluations on the same 353 Defects4J bugs that are later used for all reported accuracy and cost results. The 10-fold cross-validation described in Section III-D applies only to the DE weight optimization, not to the model-selection step, so the reported ensemble accuracy is partially fit to the evaluation benchmark. The paper should either perform the member selection inside a nested cross-validation or a leave-one-project-out scheme, or report results on a separate set of bugs; at minimum, the current limitation must be stated explicitly and its implications for the Pareto claim discussed.
  3. [Section IV-A, Figure 5] The paper itself reports that the equal-weight ensemble does not surpass the best single model when the number of runs is held equal, and that DE-optimized weights underperform equal weights (Figures 5 and 6). This means the contribution cannot be framed as accuracy improvement at equal run counts; the entire case rests on the cost-accuracy trade-off. Given that framing, the missing cost-matched analysis in the first major comment is decisive: the current figures do not show that the ensemble is on the Pareto frontier, only that it is between the extremes. The discussion should be revised either to add the required cost-matched baselines or to state explicitly that the current evidence supports only an intermediate cost-accuracy position.
minor comments (6)
  1. [Section II.B and Figure 1] The text states that 'we remove the class-level coverage tool for COSMosFL,' but Figure 1(b) appears to list get_class_covered among the available tools for COSMosFL; please reconcile this inconsistency.
  2. [Section IV.B] The text says 'The best performing model, Qwen2.5-Coder consumes the most amount of energy and time,' but Figures 9a and 9b show that Qwen2.5-Coder's median per-run consumption is not the highest; the sentence should specify that it refers to mean consumption and explain the outlier effect.
  3. [Section IV.A, paragraph after Figure 7] The statement that ensembles perform well 'despite generally outperforming individual models' is hard to reconcile with the Figure 5 caption ('The ensemble does not surpass the best single model's performance'); please clarify which individual-model baseline is meant.
  4. [Section III.D] The DE hyperparameters (population size 40, 30 generations, differential weight 1.5, crossover 0.8) are stated but no sensitivity analysis or rationale is given; a sentence on why these values were chosen would help reproduce the optimization.
  5. [Figure 3] The comparison with GPT-3.5 is not controlled because COSMosFL removes the class-level coverage tool and switches from chat-completion to instruction-following; the text acknowledges the R difference but should also state these task differences.
  6. [Section III.B] There is a typo in the model name: 'grantie3 8B' should read 'Granite 3.0 8B'.

Circularity Check

1 steps flagged · score 5.0 of 10

Ensemble membership is selected on the same Defects4J benchmark used for the reported accuracy, making the central effectiveness claim partially circular.

  1. fitted input called prediction [Section III-A, RQ1 (Experimental Setup), ensemble member selection paragraph]
    "To address this, we conducted initial runs with seven open-source SLMs to assess model orthogonality, running each model five times. Based on these preliminary results, we selected four models that demonstrated the most complementary fault localisation performance in combination. For the final evaluation, we ran AutoFL on each selected model 30 times and sampled a varying number of runs to compare two ensemble weighting strategies: equal and DE-optimised weightings."

    The four ensemble members are selected from seven candidates using preliminary fault-localisation performance on the same 353 Defects4J bugs that later constitute the reported evaluation dataset (Section III-C). The paper then reports that the resulting ensemble 'can build effective ensembles' on that same benchmark. This is a fitted input called a prediction: the discrete model-selection parameter is optimised against the evaluation set, and the evaluation set is then used as evidence that the selection is effective. No held-out validation of the model-selection step is reported, so the reported ensemble accuracy is a post-selection measurement rather than an independent prediction.

full rationale

The main circular step is the ensemble-member selection: the paper tunes which four of seven SLMs to include based on five-run preliminary accuracy on the same 353 Defects4J bugs used for the reported evaluation, and then presents the resulting ensemble accuracy as evidence that COSMos works. This does not make the measured accuracy true by definition, but it makes the central 'effective ensemble' claim partially fitted to the benchmark. The DE weight optimisation is at least described as using 10-fold cross-validation, which weakens but does not eliminate the same concern; the paper does not report a fully external validation of the whole pipeline. The Pareto-optimality claim in the abstract is also not backed by cost-matched comparisons, since ensemble points and single-model points are compared at equal numbers of runs rather than equal cost; however, that is an evidentiary gap rather than a circular reduction and therefore does not by itself raise the circularity score. Self-citations to AutoFL [17] are present and the authors overlap, but the new ensemble contribution is evaluated independently of the cited prior result, so the self-citation is not load-bearing circularity. Overall, the paper's effectiveness evaluation is partially circular because the model-selection input is fitted to the evaluation data, yielding a score of 5.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on one fitted set of voting weights and on several domain assumptions inherited from AutoFL and the Defects4J setup. No new physical or conceptual entities are introduced.

free parameters (3)
  • DE-optimized voting weights = 0.15 to 0.37 (mean across folds and samples, Fig. 6)
    Weights applied to each model's votes are optimized by differential evolution on the Defects4J subset with 10-fold cross-validation; they are fitted to the same benchmark on which final accuracy is reported.
  • DE hyperparameters = pop=40, gen=30, differential weight=1.5, crossover prob=0.8
    Chosen manually following Storn et al. [30]; they control the weight optimization but are not fitted to data.
  • Number of runs R and sampling repetitions = R=4..24, 20 samples
    Experimental design choices that define the cost-accuracy operating points; the ensemble splits R runs across M=4 models.
assumptions (4)
  • domain assumption Self-consistency voting improves LLM fault localization accuracy
    Taken from AutoFL [17] and Wang et al. [4]; COSMosFL assumes the benefit transfers from repeated sampling of one LLM to voting across heterogeneous SLMs (Section II-A).
  • domain assumption The AutoFL tool set, minus class-level coverage, is sufficient for SLMs
    The authors removed the class-level coverage tool after observing redundant calls by small models (Section II-B); the remaining tools are assumed to give SLMs enough context.
  • domain assumption The 353-bug Defects4J subset is a representative benchmark
    Standard dataset, but the subset comes from AutoFL [17] and is used for both model selection and final evaluation (Section III-C).
  • domain assumption GPU energy measured by nvidia-smi approximates inference cost
    Only GPU energy is tracked; CPU, RAM, and idle power are ignored, which may bias the cost comparison across models (Section III-D).

how reviews work

0 comments
Cite this review

Pith. "Pith review of COSMosFL: Ensemble of Small Language Models for Fault Localisation." pith.science (2026). https://pith.science/paper/RHDVEP6D

@misc{pith2026250202908,
  author       = {Pith},
  title        = {Pith review of: COSMosFL: Ensemble of Small Language Models for Fault Localisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RHDVEP6D}},
  note         = {Machine review of arXiv:2502.02908}
}
read the original abstract

LLMs are rapidly being adopted to build powerful tools and agents for software engineering, but most of them rely heavily on extremely large closed-source models. This, in turn, can hinder wider adoption due to security issues as well as financial cost and environmental impact. Recently, a number of open source Small Language Models (SLMs) are being released and gaining traction. While SLMs are smaller, more energy-efficient, and therefore easier to locally deploy, they tend to show worse performance when compared to larger closed LLMs. We present COSMos, a task-level LLM ensemble technique that uses voting mechanism, to provide a broader range of choice between SLMs and LLMs. We instantiate COSMos with an LLM-based Fault Localisation technique, AutoFL, and report the cost-benefit trade-off between LLM accuracy and various costs such as energy consumption, inference time, and the number of tokens used. An empirical evaluation using Defects4J shows that COSMos can build effective ensembles that can achieve Pareto-optimality in terms of FL accuracy and inference cost, when compared to individual models.

Figures

Figures reproduced from arXiv: 2502.02908 by the authors.

Figure 1
Figure 1. Overview of our approach against AutoFL [17] with differences colored in red. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overlap of bugs ranked at first by Llama3, Llama3.1, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. acc@k for R=20 for each model and ensemble ap [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Mean of optimised weights for each model over cross [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 5
Figure 5. Figure 5: Mean of acc@1 across runs for four single models and two ensemble approaches. Note that the ensemble techniques are only available at multiples of four runs. High accuracy at the first rank is crucial for FL tasks, so we focus on acc@1 in [PITH_FULL_IMAGE:figures/full…
Figure 8
Figure 8. Figure 8: Cost-Benefit Tradeoff: average energy consumption, inference time, and # of input and output tokens, across 20 samples [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Boxplots of cost measures per model. Note that [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Landscapes of Pairwise acc@1 explored by grid search fault localisation results into prompts, though its scope was limited to student programming assignments. Alongside these advancements, efforts to utilize open-source language models in fault localisation have gaine…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 38 canonical work pages

  1. [1]

    Large language models for software engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in Proceedings of the 45th IEEE/ACM International Conference on Software Engineering: Future of Software Engineering, ICSE-FoSE, pp. 31–53, May 2023

    work page 2023

  2. [2]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, Curran Associates, Inc., 2017

    2017

  3. [3]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS ’22, (Red Hook, NY , USA), Curran Associates Inc., 2024

    work page 2024

  4. [4]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdh- ery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171 , 2022

    arXiv 2022

  5. [5]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in Pro- ceedings of the International Conference on Learning Representation , ICLR 2023, 2023

    work page 2023

  6. [6]

    Towards autonomous testing agents via conversational large language models,

    R. Feldt, S. Kang, J. Yoon, and S. Yoo, “Towards autonomous testing agents via conversational large language models,” in Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), ASE 2023, pp. 1688–1693, 2023

    work page 2023

  7. [7]

    Repairagent: An autonomous, llm-based agent for program repair,

    I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” 2024

    work page 2024

  8. [8]

    Intent-driven mobile gui testing with autonomous large language model agents,

    J. Yoon, R. Feldt, and S. Yoo, “Intent-driven mobile gui testing with autonomous large language model agents,” in Proceedings of the 16th IEEE International Conference on Software Testing, Verification and Validation, ICST 2024, pp. 129–139, 2024

    work page 2024

Show all 45 references
  1. [9]

    Energy and policy consid- erations for deep learning in NLP,

    E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy consid- erations for deep learning in NLP,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (A. Korhonen, D. Traum, and L. M `arquez, eds.), (Florence, Italy), pp. 3645–3650,...

    2019

  2. [10]

    Risks and benefits of large language models for the environment,

    M. C. Rillig, M. ˚Agerstrand, M. Bi, K. A. Gould, and U. Sauerland, “Risks and benefits of large language models for the environment,” Environmental Science & Technology , vol. 57, no. 9, pp. 3464–3466, 2023

    2023

  3. [11]

    The Llama 3 Herd of Models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, and et al., “The Llama 3 Herd of Models,” Aug. 2024

    2024

  4. [12]

    Gemma 2: Improving Open Language Models at a Practical Size,

    G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, and et al., “Gemma 2: Improving Open Language Models at a Practical Size,” Oct. 2024

    2024

  5. [13]

    Qwen2.5- Coder Technical Report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, and et al., “Qwen2.5- Coder Technical Report,” Nov. 2024

    2024

  6. [14]

    GPT-4 Technical Report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “GPT-4 Technical Report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  7. [15]

    Evaluating diverse large language models for automatic and general bug reproduction,

    S. Kang, J. Yoon, N. Askarbekkyzy, and S. Yoo, “Evaluating diverse large language models for automatic and general bug reproduction,” IEEE Transactions on Software Engineering , vol. 50, no. 10, pp. 2677– 2694, 2024

    2024

  8. [16]

    Better patching using llm prompting, via self-consistency,

    T. Ahmed and P. Devanbu, “Better patching using llm prompting, via self-consistency,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pp. 1742–1746, 2023

    2023

  9. [17]

    A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization,

    S. Kang, G. An, and S. Yoo, “A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization,” Proc. ACM Softw. Eng., vol. 1, pp. 64:1424–64:1446, July 2024

    2024

  10. [18]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Pro- ceedings of the 2014 international symposium on software testing and analysis, pp. 437–440, 2014

    2014

  11. [19]

    Differential evolution-a simple and efficient adaptive scheme for global optimization over continuous spaces,

    R. Storn and K. Price, “Differential evolution-a simple and efficient adaptive scheme for global optimization over continuous spaces,” Inter- national computer science institute , 1995

    1995

  12. [20]

    Differential evolution: A survey of the state-of-the-art,

    S. Das and P. N. Suganthan, “Differential evolution: A survey of the state-of-the-art,” IEEE transactions on evolutionary computation , vol. 15, no. 1, pp. 4–31, 2010

    2010

  13. [21]

    Arachne: Search-based repair of deep neural networks,

    J. Sohn, S. Kang, and S. Yoo, “Arachne: Search-based repair of deep neural networks,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 4, pp. 1–26, 2023

    2023

  14. [22]

    Measuring and Improving the Energy Efficiency of Large Language Models Inference,

    M. F. Argerich and M. Pati ˜no-Mart´ınez, “Measuring and Improving the Energy Efficiency of Large Language Models Inference,” IEEE Access, vol. 12, pp. 80194–80207, 2024

    2024

  15. [23]

    “Ollama.” https://github.com/ollama/ollama, Nov. 2024

    2024

  16. [24]

    Code Llama: Open Foundation Models for Code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, and et al., “Code Llama: Open Foundation Models for Code,” Jan. 2024

    2024

  17. [25]

    Granite 3.0 Language Mod- els

    Granite Team, IBM, “Granite 3.0 Language Mod- els.” https://github.com/ibm-granite/granite-3.0-language- models/blob/main/paper.pdf, 2024

    2024

  18. [26]

    Mistral NeMo

    Mistral AI, “Mistral NeMo.” https://mistral.ai/news/mistral-nemo/, July 2024

    2024

  19. [27]

    A practical guide for using statistical tests to assess randomized algorithms in software engineering,

    A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” in Proceedings of the 33rd International Conference on Software Engineering , ICSE ’11, pp. 1–10, ACM, 2011

    2011

  20. [28]

    cloc: v1.92,

    A. Danial, “cloc: v1.92,” Dec. 2021

    2021

  21. [29]

    DEAP: Evolutionary algorithms made easy,

    F.-A. Fortin, F.-M. De Rainville, M.-A. Gardner, M. Parizeau, and C. Gagn ´e, “DEAP: Evolutionary algorithms made easy,” Journal of Machine Learning Research , vol. 13, pp. 2171–2175, jul 2012

    2012

  22. [30]

    On the usage of differential evolution for function optimiza- tion,

    R. Storn, “On the usage of differential evolution for function optimiza- tion,” in Proceedings of North American Fuzzy Information Processing , pp. 519–523, June 1996

    1996

  23. [31]

    Nvidia system management interface,

    N. Developer, “Nvidia system management interface,” NVIDIA System Management Interface, 2021

    2021

  24. [32]

    Ollama gets stuck in an infinite loop sometimes and has to be restarted · Issue #2805 · ollama/ollama

    “Ollama gets stuck in an infinite loop sometimes and has to be restarted · Issue #2805 · ollama/ollama.” https://github.com/ollama/ollama/issues/2805

  25. [33]

    Mixture-of-experts with expert choice routing,

    Y . Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Zhao, A. M. Dai, Q. V . Le, J. Laudon, et al. , “Mixture-of-experts with expert choice routing,” Advances in Neural Information Processing Systems , vol. 35, pp. 7103– 7114, 2022

    2022

  26. [34]

    Routellm: Learning to route llms with preference data,

    I. Ong, A. Almahairi, V . Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica, “Routellm: Learning to route llms with preference data,” arXiv preprint arXiv:2406.18665 , 2024

    2024 arXiv

  27. [35]

    Learning to decode collaboratively with multiple language models,

    S. Z. Shen, H. Lang, B. Wang, Y . Kim, and D. Sontag, “Learning to decode collaboratively with multiple language models,” arXiv preprint arXiv:2403.03870, 2024

    2024 arXiv

  28. [36]

    Large Language Models in Fault Localisation,

    Y . Wu, Z. Li, J. M. Zhang, M. Papadakis, M. Harman, and Y . Liu, “Large Language Models in Fault Localisation,” Oct. 2023

    2023

  29. [37]

    Demystifying Faulty Code: Step-by-Step Reasoning for Explainable Fault Localization,

    R. Widyasari, J. W. Ang, T. G. Nguyen, N. Sharma, and D. Lo, “Demystifying Faulty Code: Step-by-Step Reasoning for Explainable Fault Localization,” in IEEE International Conference on Software Analysis, Evolution and Reengineering , pp. 568–579, Mar. 2024

    2024

  30. [38]

    Large Language Models for Test-Free Fault Localization,

    A. Z. H. Yang, C. Le Goues, R. Martins, and V . Hellendoorn, “Large Language Models for Test-Free Fault Localization,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ICSE ’24, (New York, NY , USA), pp. 1–12, Association for Computing Mac...

    2024

  31. [39]

    Empirical Evaluation of Large Language Models for Novice Program Fault Localization,

    Y . Liu, H. Liu, Z. Yang, Z. Li, and Y . Liu, “Empirical Evaluation of Large Language Models for Novice Program Fault Localization,” in 2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS) , pp. 180–191, July 2024

    2024

  32. [40]

    Search-based fault localization,

    S. Wang, D. Lo, L. Jiang, Lucia, and H. C. Lau, “Search-based fault localization,” in 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011) , pp. 556–559, Nov. 2011

    2011

  33. [41]

    Learning to Combine Multiple Ranking Metrics for Fault Localization,

    J. Xuan and M. Monperrus, “Learning to Combine Multiple Ranking Metrics for Fault Localization,” in 2014 IEEE International Conference on Software Maintenance and Evolution , pp. 191–200, Sept. 2014

    2014

  34. [42]

    Why train-and-select when you can use them all? ensemble model for fault localisation,

    J. Sohn and S. Yoo, “Why train-and-select when you can use them all? ensemble model for fault localisation,” in Proceedings of the Genetic and Evolutionary Computation Conference , GECCO ’19, (New York, NY , USA), pp. 1408–1416, Association for Computing Machinery, July 2019

    2019

  35. [43]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,

    D. Jiang, X. Ren, and B. Y . Lin, “Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,” arXiv preprint arXiv:2306.02561, 2023

    2023 arXiv

  36. [44]

    InfeRE: Step-by-Step Regex Generation via Chain of Inference,

    S. Zhang, X. Gu, Y . Chen, and B. Shen, “InfeRE: Step-by-Step Regex Generation via Chain of Inference,” in 2023 38th IEEE/ACM Interna- tional Conference on Automated Software Engineering (ASE), pp. 1505– 1515, Sept. 2023

    2023

  37. [45]

    An Ensemble Method for Bug Triaging using Large Language Models,

    A. Kumar Dipongkor, “An Ensemble Method for Bug Triaging using Large Language Models,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE-Companion ’24, (New York, NY , USA), pp. 438– 440, Association for Comp...

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.