REVIEW 6 cited by
Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Natural Language Processing (NLP) is witnessing a remarkable breakthrough driven by the success of Large Language Models (LLMs). LLMs have gained significant attention across academia and industry for their versatile applications in text generation, question answering, and text summarization. As the landscape of NLP evolves with an increasing number of domain-specific LLMs employing diverse techniques and trained on various corpus, evaluating performance of these models becomes paramount. To quantify the performance, it's crucial to have a comprehensive grasp of existing metrics. Among the evaluation, metrics which quantifying the performance of LLMs play a pivotal role. This paper offers a comprehensive exploration of LLM evaluation from a metrics perspective, providing insights into the selection and interpretation of metrics currently in use. Our main goal is to elucidate their mathematical formulations and statistical interpretations. We shed light on the application of these metrics using recent Biomedical LLMs. Additionally, we offer a succinct comparison of these metrics, aiding researchers in selecting appropriate metrics for diverse tasks. The overarching goal is to furnish researchers with a pragmatic guide for effective LLM evaluation and metric selection, thereby advancing the understanding and application of these large language models.
Forward citations
Cited by 6 Pith papers
-
Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs
Multi-modal RAG (text plus UI screenshots) with reward-based polishing generates acceptance criteria from user stories that three industry experts rated near 4/5 on relevance, correctness, and understandability.
-
A Conceptual Framework for AI Capability Evaluations
A descriptive conceptual framework with seven elements (target, task, subject, inputs, instance, measurement, result analysis) for systematizing analysis of AI capability evaluations.
-
Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
LLM explanations split into local and mechanistic tracks; the paper argues they are trustworthy only if they pass causal and contrastive stress tests, adapt to the explainee, and satisfy eight trust principles.
-
WALL: A Web Application for Automated Quality Assurance using Large Language Models
WALL combines SonarQube with GPT-3.5 Turbo and GPT-4o to automatically fix detected code issues, reporting 71.6% and 85.5% revision rates plus cost reductions from a hybrid workflow.
-
Large Language Models in the Travel Domain: An Industrial Experience
Mixtral 8x7B generated more complete and less hallucinated hotel descriptions than a fine-tuned Mistral 7B, at about ten times the hourly compute cost.
-
Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements
Generating fixes with GPT-3.5 Turbo and GPT-4o, prompted with SonarQube findings and web-retrieved examples, removed most flagged bugs, vulnerabilities, and code smells from one codebase, with success judged solely by...
Discussion (0). Sign in to comment.