REVIEW 4 cited by
SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Instruction-tuned Large Language Models (LLMs) have recently showcased remarkable advancements in their ability to generate fitting responses to natural language instructions. However, many current works rely on manual evaluation to judge the quality of generated responses. Since such manual evaluation is time-consuming, it does not easily scale to the evaluation of multiple models and model variants. In this short paper, we propose a straightforward but remarkably effective evaluation metric called SemScore, in which we directly compare model outputs to gold target responses using semantic textual similarity (STS). We conduct a comparative evaluation of the model outputs of 12 prominent instruction-tuned LLMs using 8 widely-used evaluation metrics for text generation. We find that our proposed SemScore metric outperforms all other, in many cases more complex, evaluation metrics in terms of correlation to human evaluation. These findings indicate the utility of our proposed metric for the evaluation of instruction-tuned LLMs.
Forward citations
Cited by 4 Pith papers
-
SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generation
SafeDrive228K is a 228K-example multimodal QA benchmark for traffic safety, and a graph-based RAG method improves VLM accuracy on it by 4.7 to 14.6 points across five models.
-
Generation of Synthetic Clinical Text: A Systematic Review
A systematic review of synthetic clinical free-text generation maps the field into six purposes, four technique families, and four evaluation aspects, with GPT-style transformers and utility-based evaluation dominating.
-
Pre-Training Curriculum for Multi-Token Prediction in Language Models
A forward curriculum that gradually increases the number of predicted tokens helps small models retain MTP's decoding speed benefits and often improves generation quality, but does not consistently improve next-token ...
-
Probabilistic Concept-Aware Steering for Trustworthy LLM Inference
PCS improves steering direction accuracy by adaptively sampling the intervention coefficient from a cosine-similarity-conditioned Gaussian, but its evaluation is partly circular because the optimal coefficient is chos...
Discussion (0). Continue with ORCID to comment.