A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation

Anna Grigoryan; Bhaskarjit Sarmah; Dhagash Mehta; Kriti Dutta; Sachin Tiwari; Stefano Pasquali

REVIEW 5 major objections 8 minor 3 cited by

A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation

T0 review · 5 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automated prompt optimizers can tune an LLM's hallucination detector to agree with human labels, with MIPROv2 and Bootstrap Few Shot Random Search leading on the HaluBench test set.

desk verdict Legitimate but statistically underpowered comparison of DSPy teleprompters; the ranking isn't established, but the study is a fair empirical data point worth a careful look. read the letter →

arxiv 2412.15298 v1 pith:OHG4ZQDS submitted 2024-12-19 cs.CL cs.AIcs.LGq-fin.STstat.ME

Bhaskarjit Sarmah , Kriti Dutta , Anna Grigoryan , Sachin Tiwari , Stefano Pasquali , Dhagash Mehta This is my paper · ORCID

classification cs.CLcs.AIcs.LGq-fin.STstat.ME

keywords DSPyteleprompterspromptoptimizationhallucinationdetectionLLMasjudgeHaluBenchfaithfulnessevaluationGPT-4o

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that automated prompt optimizers, called teleprompters, can tune an LLM's hallucination-detection prompt so that the LLM's judgments reproduce human labels more faithfully than a standard prompt does. Using the HaluBench benchmark as a testbed, it compares five teleprompter algorithms and finds that optimized prompts beat the baseline GPT-4o prompt and two off-the-shelf faithfulness checkers (RAGAS and DeepEval). The best optimizers, MIPROv2 and Bootstrap Few Shot Random Search, reach the highest weighted F1 (0.8248) and macro F1 (0.8115) respectively on the test set. If the result holds, it means LLM-as-judge evaluation can be aligned with human judgment cheaply by optimizing the prompt, without fine-tuning model weights.

What carries the argument

Teleprompters are prompt optimizers inside the DSPy programming model: they generate candidate prompt instructions and few-shot demonstrations, evaluate them against an objective metric on a labeled training set, and return the best-performing program. In this paper the objective used for optimization is exact-match accuracy against human Pass/Fail labels on the training subset, and the final evaluation is micro, macro, and weighted F1 on a held-out test set. The two optimizers that carry the result are MIPROv2, which jointly optimizes instructions and demonstrations in stages, and Bootstrap Few Shot Random Search, which samples diverse prompt programs.

What would settle it

Re-running the five teleprompters on several independently drawn stratified 1,500-sample splits of the cleaned HaluBench data and computing 95% confidence intervals for each metric would settle whether the 85.87% vs 80.91% accuracy gap and the F1 differences are systematic. If the intervals overlap between MIPROv2, Bootstrap Few Shot Optuna, and the baseline, the claim that optimized prompts align better with human labels is unsupported.

Watch

Extended reading notes

Core claim

MIPROv2, a multi-stage instruction-and-demonstration optimizer, produces a prompt that achieves the highest weighted F1 score of 0.8248 on the HaluBench test set, while Bootstrap Few Shot Random Search achieves the highest macro F1 of 0.8115. The baseline GPT-4o prompt achieves 0.8125 weighted F1 and 0.8019 macro F1, so the optimizers improve class balance and overall alignment with human annotations. Optimized prompts also outperform the RAGAS and DeepEval faithfulness implementations, which score around 0.61–0.67 F1. The paper concludes that teleprompters can effectively align LLM-based hallucination evaluation with human labels on this benchmark, though the gains are concentrated in structured sub-datasets and unstructured domains like FinanceBench remain challenging.

Load-bearing premise

The reported rankings rest on the assumption that the 375-example stratified test set faithfully represents the full HaluBench distribution and that the observed differences between optimizers are real rather than sampling noise, since the paper reports no error bars or significance tests.

Editorial extensions

If this is right

LLM-as-judge evaluation can be aligned to human labels by prompt optimization alone, without fine-tuning, which is cheaper and faster for deployment.
Automated prompt selection can replace manual prompt engineering for hallucination detection, removing a trial-and-error step.
The best optimizer choice depends on the objective: MIPROv2 for overall weighted performance, Bootstrap Few Shot Random Search for minority-class balance.
The approach is not uniformly beneficial: unstructured sub-datasets like FinanceBench and DROP show smaller or negative gains, so domain-specific adaptation is still needed.
The pipeline generalizes to other evaluation tasks such as reasoning, toxicity, and bias assessment, as the paper proposes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the test set is a stratified 375-example sample of HaluBench, the reported metric gaps (e.g., 85.87% vs 80.91% accuracy) are not shown to be statistically significant; without error bars or significance tests, the ranking of teleprompters could be within sampling noise.
The observed advantage of optimization on structured datasets (CovidQA, PubMedQA) may partly reflect model pretraining exposure to those public benchmarks, as the authors themselves note; a controlled test on proprietary or deliberately unseen data would separate prompt-optimization gains from memorization.
The proposed fix for data-source bias, stratified sampling during teleprompter optimization, is a concrete testable extension: future work could compare aggregate F1 and per-domain F1 with and without source-stratified demonstration selection.
For practitioners, the paper's design implies that optimizing the prompt against human labels can produce sharper score distributions and crisper decision thresholds, which connects to the companion work on threshold selection for LLM evaluation metrics.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a legitimate, clearly written empirical comparison of five DSPy teleprompters for hallucination detection with GPT-4o as judge on HaluBench. What's new is the specific comparison and the per-dataset breakdown; all components are off-the-shelf. It deserves a referee, but the ranking claim is not yet established because the test set is a single 375-sample split with no error bars, and the tables don't agree with each other.

What it does well: it runs the real teleprompters from the DSPy library with sensible hyperparameters, evaluates against human labels on a public benchmark, and reports sub-dataset results. The discussion honestly notes that teleprompters can overfit to high-performing sources and suggests stratified sampling. The footnote about default thresholds for RAGAS/DeepEval is appropriately scoped.

Where it's soft: First, the headline differences are small. MIPROv2's weighted F1 beats the random-search variant by 0.0051 and the baseline by 0.0123; at n=375 that's within plausible noise. No confidence intervals, no repeated runs. Second, Table 3 reports MIPROv2 accuracy 85.87% while Table 4 reports micro-F1 0.8187 for the same system. In binary classification micro-F1 equals accuracy. Either the prediction sets differ between the tables or there's an error. That undermines the numerical basis for the ranking. Third, MIPROv2's macro-F1 of 0.4082 sits oddly with its weighted F1 of 0.8248; the paper notes class imbalance but doesn't explain why the 'best' optimizer is so poor on the minority class. These issues are fixable but load-bearing.

The coarser finding—that optimized prompts beat the fixed baseline and the default RAGAS/DeepEval thresholds—is more robust, though that comparison is explicitly at default thresholds.

This paper is for practitioners using DSPy for LLM-as-judge, not for theorists. I'd send it out for review with a request for major revision: fix the F1/accuracy inconsistency, add repeated runs or confidence intervals, justify the sample size, and ideally evaluate on the full cleaned set (9,616 examples) to check the sample isn't driving the conclusions.

Referee Report

5 major / 8 minor

Summary. This manuscript reports a comparative study of five DSPy teleprompter algorithms (COPRO, MIPROv2, BootstrapFewShot, BootstrapFewShot with Optuna, and KNN Few Shot) for aligning an LLM-based hallucination judge (GPT-4o) with human labels on the HaluBench benchmark. The authors clean and subsample HaluBench to 1,500 examples (750 train, 375 validation, 375 test), optimize prompts with each teleprompter, and report accuracy and micro/macro/weighted F1 on the held-out test set, together with comparisons against RAGAS and DeepEval. They report that MIPROv2 achieves the highest weighted F1 (0.8248), Bootstrap Few Shot Random Search achieves the highest macro F1 (0.8115), and all optimized prompts outperform the RAGAS/DeepEval baselines. The paper's central claim is that DSPy teleprompters can align hallucination evaluation with human annotations and that some teleprompters outperform others on this benchmark.

Significance. If the results were statistically supported, this would be a useful practical comparison for practitioners choosing among DSPy optimizers for LLM-as-judge tasks. The paper's strengths include its use of a public benchmark (HaluBench), explicit hyperparameter tables, a held-out test split, and comparisons against two established faithfulness toolkits. However, the quantitative evidence for the headline ranking is not yet secure: the reported differences are small, the evaluation is unreplicated, and several numbers are internally inconsistent across tables. The paper does not provide code, seeds, or confidence intervals, so the reproducibility of the exact numeric claims is limited.

major comments (5)

[Section 5, Tables 3 and 4] The paper reports a single evaluation on a 375-example test set and gives no confidence intervals, bootstrap replicates, or significance tests. The headline differences are small: MIPROv2's weighted F1 (0.8248) exceeds Bootstrap Few Shot Random Search (0.8197) by 0.0051 and baseline GPT-4o (0.8125) by 0.0123. At n=375, differences of this size are plausibly within sampling noise, so the claim in the abstract and Section 5.1 that 'certain teleprompters outperform the others' is not yet supported. The authors should add repeated runs or resampling-based uncertainty estimates, and ideally report paired tests across the same test items.
[Section 5, Tables 3-4] For a binary classification task, micro-F1 is mathematically identical to accuracy, yet the reported accuracy and micro-F1 disagree for most rows (e.g., MIPROv2: 85.87% vs 0.8187; Bootstrap Few Shot Random Search: 84.00% vs 0.8160; KNN Few Shot: 83.47% vs 0.7754). This suggests the two tables were computed on different prediction sets, with different label parsing, or with a different definition of micro-F1. The manuscript must state exactly how each metric was computed; as written, the numerical basis for the ranking is unclear and the inconsistency undermines the comparative claim.
[Section 5.1 and Table 4] The text says MIPROv2 'balanced class-specific sensitivity and global accuracy' and 'balanced both class-specific sensitivity and global accuracy,' but Table 4 reports MIPROv2's macro F1 as 0.4082, far below Bootstrap Few Shot Random Search's 0.8115 and even below baseline GPT-4o's 0.8019. Macro F1 is the metric that directly reflects class-level balance, so this result contradicts the qualitative claim. The authors should either revise the interpretation or explain why weighted F1, rather than macro F1, is the appropriate measure of balance for this comparison.
[Section 4.1.1] The cleaned data pipeline removes all answers with three or fewer tokens and discards the QUESTION component from the input, then further subsamples 9,616 examples to 1,500 with a single stratified split (750 train, 375 validation, 375 test). The representativeness of this 375-sample test set for the full HaluBench benchmark is assumed rather than demonstrated. The paper should report the per-subdataset and per-class composition of the final test set, compare it with the original HaluBench distribution, or evaluate on a larger or repeated split; otherwise the external validity of the accuracy/F1 numbers is unclear.
[Section 4.3.2 and footnote 5] RAGAS and DeepEval are evaluated with their default faithfulness thresholds of 0.5, and footnote 5 acknowledges that threshold selection is beyond the paper's scope. Because the paper's comparative claim includes outperforming these benchmark methods, comparing an optimized prompt against uncalibrated default thresholds is not an apples-to-apples comparison. At minimum, the authors should report the sensitivity of RAGAS/DeepEval metrics to threshold choices over a grid, or use thresholds derived from the validation split, and note the resulting caveat in the conclusion.

minor comments (8)

[Abstract and Section 1] The manuscript uses both 'teleprompters' and 'telemprompters'; please standardize the spelling throughout.
[Section 2] There are several grammatical errors, including 'A signatures is' and 'One can also device modules'; the section should be proofread.
[Appendix A, Tables A1-A6] The appendix tables omit the 'Bootstrap Few Shot (8 Examples)' row that appears in Tables 3-4; either add it or state why it is excluded.
[Footnote 5] Footnote 5 is difficult to parse ('using default values of the implementations for specific dataset with ground truth label') and should be rewritten as a complete sentence.
[Section 4.3.2 and Tables 3-4] The terminology alternates between 'weighted-macro F1' and 'weighted F1'; please pick one term and define it explicitly.
[References] Reference [11] lacks a publication venue and arXiv identifier; please complete the citation.
[Table 1] In the MIPROv2 row, 'num_trials¯15' appears to contain a formatting artifact; please render it as num_trials=15.
[Section 6] The discussion of 'Potential Model Bias due to Public Data Exposure' is speculative and not tied to any measurement; consider moving it to a limitations paragraph or removing it.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the DSPy teleprompter comparison uses a standard train/validation/test split, and the sole self-citation is explicitly non-load-bearing.

full rationale

The paper's derivation chain is a conventional supervised prompt-optimization pipeline: teleprompters fit prompts and few-shot demonstrations against exact-match agreement with human ground-truth labels on a 750-sample training split, are tuned on a 375-sample validation split, and are then evaluated on a disjoint 375-sample test split (Sections 4.1.1 and 4.3.2, Tables 3 and 4). No equation or reported metric reduces to a fitted parameter renamed as a prediction; the test-set numbers are computed on prompts that were not fitted on the test split. The comparison against RAGAS and DeepEval uses their default thresholds, which is methodologically debatable, but that is a fairness/correctness concern rather than a circularity concern, because those thresholds are fixed external defaults rather than quantities derived from this paper's own fitted values. The only self-citation is footnote 5 and reference [5], which point to the authors' companion threshold-selection work; the paper explicitly states that it uses default thresholds 'as a general purpose benchmark only,' so the central claim does not rest on that self-citation. Numerical inconsistencies such as micro-F1 disagreeing with accuracy in Tables 3 and 4 are potential correctness or reporting issues, not circular reductions. Overall, the core comparison is self-contained against a held-out test set, and no load-bearing circular step is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the correctness of the HaluBench labels, the assumed parseable GPT-4o output, and the representativeness of the 375-sample test set after aggressive cleaning and sampling. The only hand-chosen parameters are the optimizer hyperparameters and the baseline threshold for RAGAS/DeepEval.

free parameters (4)

K (number of nearest neighbors) = 8
Chosen by hand for the KNN Few Shot teleprompter (Table 1).
num_candidate_programs = 3
Used for Bootstrap Few Shot Random Search and Optuna variants (Table 1).
num_trials = 15
MIPROv2's search trials, set by the authors (Table 1).
threshold = 0.5
Default threshold for RAGAS and DeepEval faithfulness scores, acknowledged as potentially inappropriate (footnote 5).

assumptions (4)

domain assumption HaluBench human labels are correct ground truth
The entire evaluation treats the dataset's Pass/Fail labels as the target for alignment (Section 3).
domain assumption GPT-4o produces parseable JSON outputs matching the specified format
The pipeline relies on REASONING and SCORE keys for all evaluations (Section 4.3).
domain assumption Stratified sampling preserves the original class and sub-dataset distribution
The paper claims proportional representation after subsampling to 1,500 examples (Section 4.1.1).
domain assumption DSPy teleprompters optimize the stated objective (exact match accuracy)
The optimizers select prompts based on the metric, which the paper discusses in Section 6.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation." pith.science (2026). https://pith.science/paper/OHG4ZQDS

@misc{pith2026241215298,
  author       = {Pith},
  title        = {Pith review of: A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OHG4ZQDS}},
  note         = {Machine review of arXiv:2412.15298}
}

read the original abstract

We argue that the Declarative Self-improving Python (DSPy) optimizers are a way to align the large language model (LLM) prompts and their evaluations to the human annotations. We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO), Multi-Stage Instruction Prompt Optimization (MIPRO), BootstrapFewShot, BootstrapFewShot with Optuna, and K-Nearest Neighbor Few Shot, within the DSPy framework with respect to their ability to align with human evaluations. As a concrete example, we focus on optimizing the prompt to align hallucination detection (using LLM as a judge) to human annotated ground truth labels for a publicly available benchmark dataset. Our experiments demonstrate that optimized prompts can outperform various benchmark methods to detect hallucination, and certain telemprompters outperform the others in at least these experiments.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

From Errors to Rules: Iterative Prompt Optimization for Text Classification
cs.AI 2026-06 conditional novelty 6.0 of 10

Error-driven prompt optimization (ERGO) beats demonstration and search methods on boundary-learnable tasks (TREC 90.0, CLINC150 94.4), but no paradigm dominates overall.
Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation
cs.CL 2025-07 conditional novelty 4.0 of 10

A two-stage pipeline combining human-curated Singlish examples with embedding-based LLM ranking selects GPT-4o mini for toxicity-preserving translation, reaching gold-level human scores for Chinese and Malay but not Tamil.
Data Diversification Methods In Alignment Enhance Math Performance In LLMs
cs.AI 2025-07 reject novelty 4.0 of 10

DTS, which generates diverse solution strategies before writing solutions, improves GSM8K by 7.1 points and MATH by 4.2 points over an untuned base model at 1.03x baseline compute.

Reference graph

Works this paper leans on

15 extracted references · 6 canonical work pages · cited by 3 Pith papers

[1]

A compre- hensive overview of large language models

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, S aeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A compre- hensive overview of large language models. arXiv preprint arXiv:2307.06435 , 2023

arXiv 2023
[2]

A systematic survey of prompt engineering i n large lan- guage models: Techniques and applications

Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija J ain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering i n large lan- guage models: Techniques and applications. arXiv preprint arXiv:2402.07927 , 2024

arXiv 2024
[3]

Can an unsupervised clustering algorithm re produce a catego- rization system? In Proceedings of the 5th ACM International Conference on AI in Finance, pages 213–221, 2024

Nathalia M Castellanos, Sebastian Frank, Dhruv Desai, S tefano Pasquali, and Dhagash Mehta. Can an unsupervised clustering algorithm re produce a catego- rization system? In Proceedings of the 5th ACM International Conference on AI in Finance, pages 213–221, 2024

work page 2024
[4]

Human-calibrate d automated testing and validation of generative language models: An overview

Agus Sudjianto and Srinivas Neppalli. Human-calibrate d automated testing and validation of generative language models: An overview. A vailable at SSRN, 2024

work page 2024
[5]

How to choose a t hreshold for an evaluation metric for large language models, 2024

Bhaskarjit Sarmah, Mingshu Li, Jingrao Lyu, Sebastian F rank, Nathalia Castel- lanos, Stefano Pasquali, and Dhagash Mehta. How to choose a t hreshold for an evaluation metric for large language models, 2024

work page 2024
[6]

Dspy: Compiling declarative language m odel calls into self-improving pipelines

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyu an Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, T homas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language m odel calls into self-improving pipelines. arXiv preprint arXiv:2310.03714 , 2023

arXiv
[7]

Parameter- eﬃcient ﬁne-tuning for large models: A comprehensive surve y

Zeyu Han, Chao Gao, Jinyang Liu, Jeﬀ Zhang, and Sai Qian Zh ang. Parameter- eﬃcient ﬁne-tuning for large models: A comprehensive surve y. arXiv preprint arXiv:2403.14608, 2024

arXiv 2024
[8]

Domain specialization as the key to make large language models disr uptive: A compre- hensive survey

Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et a l. Domain specialization as the key to make large language models disr uptive: A compre- hensive survey. arXiv preprint arXiv:2305.18703 , 2023

arXiv 2023

Show all 15 references

[9]

Lynx: An open source hallucination evaluatio n model

Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannapp an, Douwe Kiela, and Rebecca Qian. Lynx: An open source hallucination evaluatio n model. arXiv preprint arXiv:2407.08488, 2024

2024 arXiv
[10]

Teach better or show smarter? on instructions and exemplars in automatic prompt optimization

Xingchen Wan, Ruoxi Sun, Hootan Nakhost, and Sercan O Ar ik. Teach better or show smarter? on instructions and exemplars in automatic prompt optimization. arXiv preprint arXiv:2406.15708 , 2024

2024 arXiv
[11]

Dspy guardrails: Building safe ll m applications via self- reﬁning language model pipelines

Boxi Yu and Pinjia He. Dspy guardrails: Building safe ll m applications via self- reﬁning language model pipelines
[12]

Optimizing ins tructions and demonstrations for multi-stage language model programs

Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christo- pher Potts, Matei Zaharia, and Omar Khattab. Optimizing ins tructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695, 2024

2024 arXiv
[13]

Optuna: A next-generation hyperparameter optimiz ation framework

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru O hta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimiz ation framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2623–2631, 2019

2019
[14]

Ra- gas: Automated evaluation of retrieval augmented generati on

Shahul Es, Jithin James, Luis Espinosa-Anke, and Steve n Schockaert. Ra- gas: Automated evaluation of retrieval augmented generati on. arXiv preprint arXiv:2309.15217, 2023

2023 arXiv
[15]

Fin e-tuning and prompt op- timization: Two great steps that work better together

Dilara Soylu, Christopher Potts, and Omar Khattab. Fin e-tuning and prompt op- timization: Two great steps that work better together. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process ing, pages 10696– 10710. Association for Computational L...

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

A compre- hensive overview of large language models

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, S aeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A compre- hensive overview of large language models. arXiv preprint arXiv:2307.06435 , 2023

arXiv 2023

[2] [2]

A systematic survey of prompt engineering i n large lan- guage models: Techniques and applications

Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija J ain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering i n large lan- guage models: Techniques and applications. arXiv preprint arXiv:2402.07927 , 2024

arXiv 2024

[3] [3]

Can an unsupervised clustering algorithm re produce a catego- rization system? In Proceedings of the 5th ACM International Conference on AI in Finance, pages 213–221, 2024

Nathalia M Castellanos, Sebastian Frank, Dhruv Desai, S tefano Pasquali, and Dhagash Mehta. Can an unsupervised clustering algorithm re produce a catego- rization system? In Proceedings of the 5th ACM International Conference on AI in Finance, pages 213–221, 2024

work page 2024

[4] [4]

Human-calibrate d automated testing and validation of generative language models: An overview

Agus Sudjianto and Srinivas Neppalli. Human-calibrate d automated testing and validation of generative language models: An overview. A vailable at SSRN, 2024

work page 2024

[5] [5]

How to choose a t hreshold for an evaluation metric for large language models, 2024

Bhaskarjit Sarmah, Mingshu Li, Jingrao Lyu, Sebastian F rank, Nathalia Castel- lanos, Stefano Pasquali, and Dhagash Mehta. How to choose a t hreshold for an evaluation metric for large language models, 2024

work page 2024

[6] [6]

Dspy: Compiling declarative language m odel calls into self-improving pipelines

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyu an Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, T homas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language m odel calls into self-improving pipelines. arXiv preprint arXiv:2310.03714 , 2023

arXiv

[7] [7]

Parameter- eﬃcient ﬁne-tuning for large models: A comprehensive surve y

Zeyu Han, Chao Gao, Jinyang Liu, Jeﬀ Zhang, and Sai Qian Zh ang. Parameter- eﬃcient ﬁne-tuning for large models: A comprehensive surve y. arXiv preprint arXiv:2403.14608, 2024

arXiv 2024

[8] [8]

Domain specialization as the key to make large language models disr uptive: A compre- hensive survey

Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et a l. Domain specialization as the key to make large language models disr uptive: A compre- hensive survey. arXiv preprint arXiv:2305.18703 , 2023

arXiv 2023