Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Talking to Data: Designing Smart Assistants for Humanities Databases

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that a RAG chatbot combining hybrid search, text-to-SQL, and semantic filtering lets researchers query the Prozhito diary archive in natural language, with the best hybrid retrieval configuration outperforming…

desk verdict Useful applied RAG system with a real Prozhito evaluation, but the hybrid-search gain is within noise and alpha was tuned on the test set. read the letter →

arxiv 2506.00986 v1 pith:QF3VXROZ submitted 2025-06-01 cs.CL

Alexander Sergeev , Valeriya Goloviznina , Mikhail Melnichenko , Evgeny Kotelnikov This is my paper · ORCID
classification cs.CL
keywords Retrieval-AugmentedGenerationhybridsearchtext-to-SQLsemanticfilteringhumanitiesdatabasesdiaryarchivechatbotLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a retrieval-augmented chatbot, built from hybrid search, text-to-SQL, and semantic filtering, can give researchers and non-specialists natural-language access to a humanities database. The assistant is tested on the Prozhito diary archive, and the experiments show that combining lexical and semantic retrieval improves precision for the best encoder, reaching Precision@5 of 0.572. The paper also shows that open chat models can produce accurate answers while all tested models are vulnerable to past-tense jailbreaking, a concrete risk for public archival assistants. If the design holds, it lowers the technical barrier to searching large textual archives with relational metadata.

What carries the argument

The load-bearing mechanism is a generalized retrieval scoring formula that fuses three retrieval routes before the LLM writes an answer: $S = \gamma(\alpha S_{\mathrm{sem}} + (1-\alpha)S_{\mathrm{ft}}) + (1-\gamma)\frac{1}{|C|}\sum_{c\in C} S_c$, where $S_{\mathrm{sem}}$ is cosine similarity from dense embeddings, $S_{\mathrm{ft}}$ is a normalized lexical (tf-idf/BM25-style) score, and each $S_c$ is the cosine similarity between the query and a short metadata field such as an author biography. The workflow also regenerates the search query from conversation history and uses text-to-SQL to turn date and numeric constraints into SQL filters. This machinery is what lets one interface handle free diary prose and structured author metadata at once.

What would settle it

Ask working historians to write a fresh set of questions against the full 60,240-entry Prozhito corpus, then measure whether hybrid search (te-3-large plus tf-idf) still beats semantic-only retrieval and whether bge-m3 plus tf-idf still underperforms bge-m3 alone; if hybrid does not consistently win on these real queries, the paper's central claim about hybrid search would be falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a RAG architecture extended with query generation, text-to-SQL filtering, and semantic filtering lets users converse with a relational humanities database, and that hybrid retrieval is an effective way to find precise diary fragments. In the search experiments, te-3-large embeddings combined with tf-idf give the highest Precision@5 (0.572), beating te-3-large alone (0.548); bge-m3 alone reaches 0.568, but adding tf-idf lowers it to 0.556, which the paper attributes to bge-m3 already encoding full-text similarity through self-knowledge distillation. In generation, DeepSeek-V3 achieves the best accuracy score (4.54 out of 5) and o3-mini the best ethics score (4.46 out of 5), while every model showed some tendency to add facts absent from the source fragments and to answer dangerous historical questions despite refusals.

Load-bearing premise

The evaluation's conclusions rest on the assumption that the 50 expert-written questions and 125 diary entries, each with exactly five relevant passages, reflect how real researchers actually query the archive; if real queries look different, the measured precision and quality scores may not transfer.

Editorial extensions

If this is right

  • Researchers can query a large diary corpus in plain language and trace each answer to a hyperlinked source entry, removing the need to learn query syntax.
  • Hybrid search should be tuned per encoder: the paper's results show fusing tf-idf with bge-m3 can reduce precision, so the fusion weight and even the fusion itself are design choices.
  • The text-to-SQL and semantic-filtering stages extend the assistant beyond full-text search to dates, biographies, and numeric fields, making the same architecture portable to other humanities databases.
  • The past-tense jailbreak results imply that a public-facing assistant needs a dedicated safety layer, because prompt-level instructions alone did not stop detailed unsafe answers.
  • All experiments are reproducible from the public repository, including the prompts, dataset topics, and evaluation criteria.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same architecture could be applied to other long-text collections with relational metadata, such as oral histories, letters, or court records; the paper notes the design is domain-agnostic but does not demonstrate this.
  • Editorial inference: because the test set was expanded by paraphrasing with GPT-4o and guarantees exactly five relevant passages per question, measured precision may be higher than it would be on real, messier researcher queries; a field study with independently posed questions would test this.
  • Editorial inference: the finding that past-tense framing defeats safety training suggests a concrete safety intervention not tested in the paper—refusing or redacting procedural details in historical questions, or verifying user intent—which could be evaluated with the same provocative-question subset.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents an LLM-based RAG assistant for natural-language interaction with humanities databases, applied to the Prozhito diary archive. The system combines hybrid search (semantic plus TF-IDF), automatic query generation, text-to-SQL filtering, and semantic filtering of metadata fields. The authors evaluate retrieval with four encoders under semantic-only and hybrid settings, and answer generation with five LLMs using expert annotation of accuracy and ethics. The central claim is that the proposed architecture enables accurate and ethical question answering, and that hybrid search improves retrieval precision over semantic-only search.

Significance. The problem is worthwhile: making digital humanities archives accessible through natural language is an increasingly important application of LLMs. The system description is a useful integration of existing components, and the empirical evaluation on a real Russian-language diary corpus is a concrete contribution. Strengths include the use of a real archive, human expert evaluation with reported inter-annotator agreement, and a discussion of the ethical risks of temporal-framing jailbreaks. However, the main evidence for the hybrid-search improvement is weak—the largest reported advantage over the best semantic-only baseline is 0.004 in Precision@5, alpha was tuned on the same test set, and no uncertainty quantification is provided. The answer-generation results also lack error bars or significance tests. The paper is a reasonable system demonstration but does not currently substantiate its headline claims.

major comments (3)
  1. [Section 5.1, Table 2, Section 7] The conclusion that 'the retrieval experiment results demonstrate the effectiveness of hybrid search' is not supported by the reported numbers. The best hybrid configuration (te-3-large + tf-idf) achieves P@5=0.572, only 0.004 above the best semantic-only model (bge-m3, 0.568), and the same section states that alpha was set to 0.9 'based on the experiments,' i.e., tuned on the same 50 questions used for reporting. With exactly five gold passages per question, these differences correspond to a handful of passages, and no confidence intervals or significance tests are given. The authors should either select alpha on a held-out set or report a sensitivity analysis across alpha values, and should provide bootstrap or permutation tests before claiming a hybrid advantage.
  2. [Section 4] The evaluation dataset is small and artificially constructed: 125 entries and 50 questions, with exactly five relevant passages per question, including GPT-4o paraphrases of original entries. The paper does not explain how the 90-entry subset was selected, whether the paraphrase process was verified for factual preservation, or how the gold relevance judgments were obtained (e.g., single expert or multiple annotators with agreement). Because every quantitative conclusion in the paper rests on this dataset, the lack of detail on its construction and representativeness is a load-bearing limitation that should be addressed explicitly.
  3. [Section 5.2, Table 4] The answer-generation scores have no variance measures, confidence intervals, or significance tests. The differences among models (e.g., DeepSeek-V3 4.54 vs DeepSeek-R1 4.51 for Accuracy) are likely within annotation noise, especially given the moderate inter-annotator agreement (Krippendorff's alpha=0.722) for Ethics and that the Ethics scores for provocative questions are based on only eight questions. The paper should report standard deviations or per-model score distributions and, at minimum, acknowledge which differences are not statistically distinguishable.
minor comments (5)
  1. [Section 4] The text states that experts selected a 90-entry subset grouped into 25 topics, and that after paraphrasing each topic contained 5 entries, for a final dataset of 125 entries. These numbers are inconsistent: 25 topics x 5 entries = 125, so the role of the initial 90 entries is unclear. Please clarify the data construction process.
  2. [Section 5.1] For reproducibility, the paper should specify the TF-IDF preprocessing (lemmatization/stemming, stop-word removal) and the exact normalization procedure applied to S_sem and S_ft before computing the hybrid score in Eq. (1).
  3. [Appendices A-F] All appendices refer only to the GitHub repository rather than containing any content in the paper. At least the prompt templates, scoring criteria, and the list of dataset topics should be included in the paper itself so that the evaluation can be inspected without external resources.
  4. [Section 2] The sentence 'In [12] was developed Al assistant specializing in research instrument validation' contains a typo ('Al' should be 'AI') and is ungrammatical; please revise.
  5. [Section 3.2] The statement that full-text search 'demonstrates sensitivity to phrasing variability, synonyms, and morphology' is ambiguous: it should say that full-text search fails to capture semantic similarity, not that it is sensitive to those variations.

Circularity Check

1 steps flagged · score 4.0 of 10

Hybrid-search gain rests on α tuned to the same 50-question test set; reported P@5 differences are in-sample and not significance-tested.

  1. fitted input called prediction [Section 5.1 (Eq. 1, Table 2) and Section 7]
    "For hybrid search evaluation, a linear combination of similarity scores from each semantic model and tf-idf scores was computed using the formula (1). Based on the experiments, the weight parameter α was set to 0.9. ... The retrieval experiment results demonstrate the effectiveness of hybrid search for precise text fragment extraction."

    The conclusion that hybrid search is effective is supported by P@5 numbers in Table 2, but the hybrid score in Eq. (1) contains a free weight α that Section 5.1 says was 'set to 0.9' 'based on the experiments' — the same 50-question experiments reported in Table 2. Thus the reported hybrid P@5 values are in-sample results after tuning α on the evaluation data, not independent predictions. The best hybrid (te-3-large + tf-idf, 0.572) beats its semantic-only baseline by 0.024, while adding tf-idf to bge-m3 lowers P@5 (0.568→0.556); with 50 questions and no significance test, the claimed advantage is consistent with α-selection and noise. The Section 7 claim therefore partially reduces to a fitted parameter rather than a robust out-of-sample effect.

full rationale

No load-bearing self-citation, uniqueness-from-authors, ansatz-smuggling, or renaming patterns are present. The system pipeline is described independently, and the answer-generation evaluation is a separate expert annotation with reported inter-annotator agreement. The only circularity-like step is the retrieval comparison: α in Eq. (1) was tuned on the same test set used to demonstrate hybrid-search superiority, making the headline retrieval conclusion an in-sample, parameter-fitted comparison rather than a derived prediction. Because the central claim still depends on real retrieval scores and not solely on the fitted value, this is a moderate partial circularity/confound rather than a full reduction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. Its main free parameter is the hybrid search weight alpha, tuned on the evaluation set, and its claims rely on domain assumptions about the dataset and evaluation methodology.

free parameters (1)
  • alpha = 0.9
    Weight in the hybrid search score (Eq. 1), chosen based on experiments on the same evaluation set, as stated in Section 5.1.
assumptions (4)
  • domain assumption The 125-entry dataset with 50 questions and five relevant entries per question is representative of real humanities queries.
    Section 4 describes the expert-selected and GPT-4o-paraphrased dataset, but there is no evidence that these entries and questions match how real researchers query Prozhito.
  • domain assumption The expert scoring criteria for Accuracy and Ethics are reliable and appropriate.
    Section 5.2 reports inter-annotator agreement, but assumes these criteria capture the quality dimensions that matter for users.
  • ad hoc to paper The linear combination of retrieval scores in Eq. 1 with a single alpha is a valid way to merge semantic and lexical relevance.
    No justification is given for the linear form beyond convenience, and alpha is tuned on test data.
  • domain assumption Precision@5 with exactly five relevant fragments per question is a sufficient retrieval metric.
    Section 5.1 equates Recall@5 with Precision@5 because each question has exactly five relevant fragments, but this assumes a single correct fragment set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Talking to Data: Designing Smart Assistants for Humanities Databases." pith.science (2026). https://pith.science/paper/QF3VXROZ

@misc{pith2026250600986,
  author       = {Pith},
  title        = {Pith review of: Talking to Data: Designing Smart Assistants for Humanities Databases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QF3VXROZ}},
  note         = {Machine review of arXiv:2506.00986}
}
read the original abstract

Access to humanities research databases is often hindered by the limitations of traditional interaction formats, particularly in the methods of searching and response generation. This study introduces an LLM-based smart assistant designed to facilitate natural language communication with digital humanities data. The assistant, developed in a chatbot format, leverages the RAG approach and integrates state-of-the-art technologies such as hybrid search, automatic query generation, text-to-SQL filtering, semantic database search, and hyperlink insertion. To evaluate the effectiveness of the system, experiments were conducted to assess the response quality of various language models. The testing was based on the Prozhito digital archive, which contains diary entries from predominantly Russian-speaking individuals who lived in the 20th century. The chatbot is tailored to support anthropology and history researchers, as well as non-specialist users with an interest in the field, without requiring prior technical training. By enabling researchers to query complex databases with natural language, this tool aims to enhance accessibility and efficiency in humanities research. The study highlights the potential of Large Language Models to transform the way researchers and the public interact with digital archives, making them more intuitive and inclusive. Additional materials are presented in GitHub repository: https://github.com/alekosus/talking-to-data-intersys2025.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 22 canonical work pages

  1. [1]

    S., Xu, M., Patros, P., Wu, H., Kaur, R., Kaur, K., Fuller, S., Singh, M., Arora, P., Kumar Parlikad, A., Stankovski, V., Abraham, A., Ghosh, S

    Gill, S. S., Xu, M., Patros, P., Wu, H., Kaur, R., Kaur, K., Fuller, S., Singh, M., Arora, P., Kumar Parlikad, A., Stankovski, V., Abraham, A., Ghosh, S. K., Lutfiyya, H., Kanhere, S. S., Bahsoon, R., Rana, O., Dustdar, S., Sakellariou, R., Uhlig, S., Buyya, R.: Transformative effects of ChatGPT on modern education: Emerging Era of AI Chatbots. Internet o...

    work page 2024

  2. [2]

    arXiv:2404.01644 (2024)

    Weng, L., Wang, X., Lu, J., Feng, Y., Liu, Y., Feng, H., Huang, D., Chen, W.: InsightLens: Augmenting LLM-Powered Data Analysis with Interactive Insight Management and Navi- gation. arXiv:2404.01644 (2024)

    arXiv 2024

  3. [3]

    Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 1 -13 (2024)

    Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., Myers, B.: Using an LLM to Help With Code Understanding. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 1 -13 (2024)

    work page 2024

  4. [4]

    arXiv:2005.11401 (2020)

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval -Augmented Generation for Knowledge -Intensive NLP Tasks. arXiv:2005.11401 (2020)

    arXiv 2020

  5. [5]

    N., Abdul Rahman, M

    Chee, K. N., Abdul Rahman, M. H., Yahaya, N., Ibrahim, N., Razak, R., Sugino, C.: Ex- ploring the Trend and Potential Distribution of Chatbot in Education: A Systematic Review. International Journal of Information and Education Technology, 13, 516 -525 (2023)

    work page 2023

  6. [6]

    Ait Baha, T., El Hajji, M., Es-Saady, Y. et al. The impact of educational chatbot on student learning experience. Educ Inf Technol 29, 10153 –10176 (2024)

    work page 2024

  7. [7]

    V., Yajaman, V

    Yoganand, B. V., Yajaman, V. G., Madugula, A., Hajari, M., Fathima, N., Chandramauli, A., Sharma, G.: Humanistic artificial intelligence context for automated customer assistance in online banking. AIP Conf. Proc. 21 April 2025; 3157 (1): 020003

    work page 2025

  8. [8]

    Inf Syst Front 26, 899 –919 (2024)

    Behera, R.K., Bala, P.K., Ray, A.: Cognitive Chatbot for Personalised Contextual Customer Service: Behind the Scene and beyond the Hype. Inf Syst Front 26, 899 –919 (2024)

    work page 2024

Show all 34 references
  1. [9]

    Educ Inf Technol 29, 6357 –6382 (2024)

    Bilquise, G., Ibrahim, S., Salhieh, S.M.: Investigating student acceptance of an academic advising chatbot in higher education institutions. Educ Inf Technol 29, 6357 –6382 (2024)

    2024

  2. [10]

    Figueroa-Torres, M.: The Three Social Dimensions of Chatbot Technology. Philos. Technol. 38, 1 (2025). 14

    2025

  3. [11]

    Mannheimer S., Bond N., Young S. W. H., Kettler H. S.: Responsible AI Practice in Librar- ies and Archives: A Review of the Literature. Information Technology and Libraries. 43(3) (2024)

    2024

  4. [12]

    T., Villarino, M

    Villarino, R. T., Villarino, M. L.: Advancing Instrument Validation in Social Sciences: An AI-Powered Chatbot and Interactive Website based on Research Instrument Validation Framework (RIVF) (2024)

    2024

  5. [13]

    Educ Inf Technol 29, 17035 –17069 (2024)

    Yeti̇şensoy, O., Karaduman, H.: The effect of AI -powered chatbots in social studies educa- tion. Educ Inf Technol 29, 17035 –17069 (2024)

    2024

  6. [14]

    Journal of Web Librarianship 16(2), 120 –142 (2022)

    Ehrenpreis M., DeLooper J.: Implementing a Chatbot on a Library Website. Journal of Web Librarianship 16(2), 120 –142 (2022)

    2022

  7. [15]

    Reference Services Review

    Rodriguez S., Mune C.: Uncoding library chatbots: deploying a new virtual reference tool at the San Jose State University library. Reference Services Review. 50(3/4), 392 –405 (2022)

    2022

  8. [16]

    Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization (UMAP '23 Adjunct), 388 –396 (2023)

    Tsitseklis K., Stavropoulou G., Zafeiropoulos A., Thanou A., Papavassiliou S.: RECBOT: Virtual Museum navigation through a Chatbot assistant and personalized Recommenda- tions. Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization (UMA...

    2023

  9. [17]

    S., Ramesh G., Meghana Reddy T

    Potluri J., Gummadi H., Bhogi M., Katta Y. S., Ramesh G., Meghana Reddy T. S.: Unveiling Covert Conversational Agents: Enhancing Insight, Archives, and Dialog Acts with ChatGPT. 7th International Conference on I -SMAC (IoT in Social, Mobile, Analytics and Cloud), 766 –772 (2023)

    2023

  10. [18]

    A., Lagana, A.: A RAG Chatbot for Precision Medicine of Multiple Myeloma

    Quidwai, M. A., Lagana, A.: A RAG Chatbot for Precision Medicine of Multiple Myeloma. medRxiv 2024.03.14.24304293 (2024)

    2024

  11. [19]

    A., Amirlatifi, A., Mittal, S., Rahimi, S.: From Questions to Insightful Answers: Building an Informed Chatbot for University Resources

    Subash, N., Hossain, E., Keith, J., Tripathi, H., Ghiasi, F., Golilarz, N. A., Amirlatifi, A., Mittal, S., Rahimi, S.: From Questions to Insightful Answers: Building an Informed Chatbot for University Resources. arXiv:2405.08120 (2024)

    2024 arXiv

  12. [20]

    Journal of documentation, 28(1), 11 –21 (1972)

    Sparck Jones, K.: A statistical interpretation of term specificity and its application in r e- trieval. Journal of documentation, 28(1), 11 –21 (1972)

    1972

  13. [21]

    In Proceedings of the 17th International Workshop on Treebanks and Linguistic Theories (TLT 2018), December 13–14, 2018, Oslo University, Norway, 155, 52 –65 (2018)

    Droganova, K., Lyashevskaya, O., Zeman, D.: Data Conversion and Consistency of Mono- lingual Corpora: Russian UD Treebanks. In Proceedings of the 17th International Workshop on Treebanks and Linguistic Theories (TLT 2018), December 13–14, 2018, Oslo University, Norway, 155, 52...

    2018

  14. [22]

    arXiv:1908.10084 (2019)

    Reimers, N., Gurevych, I.: Sentence -BERT: Sentence Embeddings using Siamese BERT - Networks. arXiv:1908.10084 (2019)

    2019 arXiv

  15. [23]

    arXiv:2212.03533 (2022)

    Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., Wei, F.: Text Embeddings by Weakly -Supervised Contrastive Pre -training. arXiv:2212.03533 (2022)

    2022 arXiv

  16. [24]

    arXiv:2402.03216 (2024)

    Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., Liu, Z.: BGE M3 -Embedding: Multi-Lin- gual, Multi -Functionality, Multi -Granularity Text Embeddings Through Self -Knowledge Distillation. arXiv:2402.03216 (2024)

    2024 arXiv

  17. [25]

    arXiv: 2406.08426 (2024)

    Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., Huang, X.: Next -Generation Database Interfaces: A Survey of LLM -based Text -to-SQL. arXiv: 2406.08426 (2024)

    2024

  18. [26]

    OpenAI blog, 1(8), 9 (2019)

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9 (2019)

    2019

  19. [27]

    S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large Language Models are Zero- Shot Reasoners

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large Language Models are Zero- Shot Reasoners. arXiv:2205.11916 (2022)

    2022 arXiv

  20. [28]

    Y., Chen, Z., Zhang, T., Deng, X., Sun, H.: Exploring Chain-of-Thought Style Prompting for Text-to-SQL

    Tai, C. Y., Chen, Z., Zhang, T., Deng, X., Sun, H.: Exploring Chain-of-Thought Style Prompting for Text-to-SQL. arXiv:2305.14215 (2023)

    2023 arXiv

  21. [29]

    arXiv:2305.20050 (2023)

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., Cobbe, K.: Let's Verify Step by Step. arXiv:2305.20050 (2023). 15

    2023 arXiv

  22. [30]

    T., Gupta, S., Berrospi, C., Mishra, L., Dolfi, M., Staar, P., Vagenas, P.: Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems

    de Lima, R. T., Gupta, S., Berrospi, C., Mishra, L., Dolfi, M., Staar, P., Vagenas, P.: Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems. arXiv: 2411.19710 (2024)

    2024 arXiv

  23. [31]

    Findings of the Association for Computational Linguistics: ACL 2024, 11065 –11082 (2024)

    Long, L., Wang, R., Xiao, R., Zhao, J,, Ding, X., Chen, G., Wang, H.: On LLMs -Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. Findings of the Association for Computational Linguistics: ACL 2024, 11065 –11082 (2024)

    2024

  24. [32]

    S., Parnin, C., Sarkar, A.: Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation In- structions

    Murugadoss, B., Poelitz, C., Drosos, I., Le, V., McKenna, N., Negreanu, C. S., Parnin, C., Sarkar, A.: Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation In- structions. arXiv:2408.08781 (2024)

    2024

  25. [33]

    SAGE Publica- tions (2019)

    Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. SAGE Publica- tions (2019)

    2019

  26. [34]

    Appendix A

    Andriushchenko, M., Flammarion, N.: Does Refusal Training in LLMs Generalize to the Past Tense? arXiv:2407.11969 (2024). Appendix A. Chatbot interface Chatbot interface is available at the repository, Appendix A . Appendix B. Prompt with instruction for solving the Text -to- S...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.