pith. sign in

arxiv: 2606.20588 · v1 · pith:YLFLRKWAnew · submitted 2026-05-15 · 💻 cs.HC · cs.AI

AInterviewer: A Platform for Designing and Conducting AI-led Qualitative Interviews

Tobias Priesholm Gardhus , Nikolas Vitsakis , Fie Lejre Frederiksen , Anna Rogers , Hjalmar Bang Carlsen This is my paper

Pith reviewed 2026-06-30 19:35 UTC · model grok-4.3

classification 💻 cs.HC cs.AI
keywords qualitative interviewslarge language modelsopen source platformmulti-agent systemssurvey softwaredata collectionsocial science research
0
0 comments X

The pith

AInterviewer is an open-source multi-agent platform that blends survey-style question control with LLM flexibility for qualitative interviews.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents AInterviewer to solve problems in existing AI interview systems that depend on proprietary LLMs and lack control over question order and wording. It proposes a multi-agent pipeline to merge the standardization of survey software with the adaptability of LLMs, while allowing local model use for better security and reproducibility. The platform includes tools for every stage of qualitative data collection in social science research. A sympathetic reader would care because this setup could make automated interviews more reliable and accessible without sacrificing research standards. The approach aims to follow established best practices in interviewing while keeping conversation natural.

Core claim

AInterviewer is an opensource solution based on a multi-agent pipeline that combines controlled question administration of survey software with the flexibility of LLMs and can run with locally hosted models to ensure security, transparency, and reproducibility.

What carries the argument

The multi-agent pipeline, which integrates controlled question administration from survey software with LLM flexibility for handling interview tasks.

If this is right

  • The platform supports the full interview process through a web-based GUI, including guide design, pilot testing, distribution, and monitoring.
  • Local model hosting allows data collection without sending sensitive information to external providers.
  • It applies social science standards for qualitative interviewing to an automated system.
  • Question wording and order remain standardized even as LLMs handle conversational elements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The system could be adapted for hybrid data collection that mixes structured survey items with open qualitative probes.
  • Real-world deployment in social science projects would test whether it maintains interview quality at scale.
  • Similar multi-agent designs might apply to other research tasks like automated coding of responses.

Load-bearing premise

That a multi-agent pipeline can simultaneously enforce standardization and best practices from qualitative interviewing while retaining LLM flexibility without introducing new biases or losing natural conversation flow.

What would settle it

A controlled study measuring response consistency, adherence to the interview guide, and participant engagement in interviews run by AInterviewer versus human interviewers or single-LLM systems.

Figures

Figures reproduced from arXiv: 2606.20588 by Anna Rogers, Fie Lejre Frederiksen, Hjalmar Bang Carlsen, Nikolas Vitsakis, Tobias Priesholm Gardhus.

Figure 1
Figure 1. Figure 1: System design of the AInterviewer platform. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The implementation of the multi-agent AI interview system used in our demonstration. Before generating [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The chat interface that interviewees interact with the AInterviewer. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The full workflow for Interview Designers. The setup section (Consent, Interview guide) contains [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
read the original abstract

There are now multiple proposals for systems based on Large Language Models (LLMs) to conduct automated qualitative interviews, but most of the current solutions rely on proprietary LLMs, which compromises reproducibility and data security. They also rely on LLMs for all interview tasks, which limits standardisation of question wording as well as control over question order. To address these issues, we introduce the AInterviewer platform, an opensource solution based on a multi-agent pipeline that combines controlled question administration of survey software with the flexibility of LLMs. AInterviewer is an interdisciplinary effort designed to implement best practices of qualitative interviewing in social science, and it can run with locally hosted models to ensure security, transparency, and reproducibility. Our platform provides a web-based GUI supporting each phase of data collection: from interview guide design and pilot testing to interview distribution and data collection monitoring.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces AInterviewer, an open-source web-based platform for AI-led qualitative interviews. It describes a multi-agent pipeline that integrates controlled question administration from survey software with LLM adaptability, supports local model hosting for security/transparency/reproducibility, implements social-science best practices for interviewing, and provides a GUI covering interview guide design, pilot testing, distribution, and monitoring.

Significance. If the platform's claims hold, it would offer a reproducible, secure alternative to proprietary LLM interview systems and could support more standardized qualitative data collection in HCI and social sciences. The emphasis on open-source code, local models, and interdisciplinary best practices is a strength. However, the complete absence of any empirical validation, pilot data, or comparisons means the practical significance remains potential rather than demonstrated.

major comments (2)
  1. [Abstract] Abstract: The central claim that the multi-agent pipeline 'combines controlled question administration of survey software with the flexibility of LLMs' while implementing best practices is load-bearing for the contribution, yet the manuscript supplies only architectural descriptions and GUI screenshots with no pilot data, inter-rater reliability metrics, ablation studies on agent roles, or comparisons to human interviewers or single-agent baselines.
  2. [Platform architecture description] Platform architecture description: The assertion that the system enforces standardization of question wording and order while retaining natural conversation flow without introducing new biases is presented without any implementation details on agent coordination, error handling, or flow-control mechanisms, leaving the technical feasibility of the claimed balance unexamined.
minor comments (2)
  1. [GUI screenshots] Figure captions for GUI screenshots could include annotations or callouts to highlight specific interface elements referenced in the text.
  2. [Conclusion] The manuscript would benefit from a dedicated limitations or future-work subsection that explicitly addresses the current lack of empirical evaluation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review and for highlighting both the platform's potential strengths and the need for greater clarity on its claims. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that the multi-agent pipeline 'combines controlled question administration of survey software with the flexibility of LLMs' while implementing best practices is load-bearing for the contribution, yet the manuscript supplies only architectural descriptions and GUI screenshots with no pilot data, inter-rater reliability metrics, ablation studies on agent roles, or comparisons to human interviewers or single-agent baselines.

    Authors: We agree that the manuscript contains no empirical validation, pilot data, or comparative metrics. The work is framed as a systems paper whose primary contribution is the open-source platform design, architecture, and GUI that integrates survey-style control with LLM flexibility while supporting local models. We will revise the abstract, introduction, and add an explicit limitations and future-work section to state the scope more precisely and note that empirical evaluations (including reliability metrics and baselines) are planned separately. revision: partial

  2. Referee: [Platform architecture description] Platform architecture description: The assertion that the system enforces standardization of question wording and order while retaining natural conversation flow without introducing new biases is presented without any implementation details on agent coordination, error handling, or flow-control mechanisms, leaving the technical feasibility of the claimed balance unexamined.

    Authors: We will expand the platform architecture section with concrete implementation details on the multi-agent pipeline, including agent coordination logic, error-handling routines, and flow-control mechanisms that maintain question order and wording while allowing natural follow-up. These additions will make the technical feasibility of the claimed balance explicit. revision: yes

Circularity Check

0 steps flagged

No circularity: platform description with no derivations or fitted claims

full rationale

The paper is a system description of an open-source interview platform. It contains no equations, no fitted parameters, no predictions of quantities, and no derivation chain. Claims about the multi-agent pipeline combining survey control with LLM flexibility are presented as design choices and implementation goals, not as results derived from prior fitted data or self-citations that reduce to the inputs. No load-bearing self-citation, uniqueness theorem, or ansatz smuggling is present. This is the expected non-finding for a software platform paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical content, fitted parameters, or new entities; this is a software platform description.

pith-pipeline@v0.9.1-grok · 5692 in / 1016 out tokens · 22438 ms · 2026-06-30T19:35:07.088669+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

121 extracted references · 57 canonical work pages · 2 internal anchors

  1. [1]

    , month = nov, year =

    Salganik, Matthew J. , month = nov, year =. Bit by

  2. [2]

    Thinking through methods:

    Martin, John Levi , year =. Thinking through methods:

  3. [3]

    2011 , pages =

    The Public Opinion Quarterly , author =. 2011 , pages =

    2011

  4. [4]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  5. [5]

    PloS one , author =

    Wiki surveys:. PloS one , author =. 2015 , note =

    2015

  6. [6]

    Method and meaning in polls and surveys , url =

    Schuman, Howard , year =. Method and meaning in polls and surveys , url =

  7. [7]

    Social Science Computer Review , author =

    A. Social Science Computer Review , author =. 2021 , note =. doi:10.1177/0894439319885317 , abstract =

    work page doi:10.1177/0894439319885317 2021

  8. [8]

    Proceedings of the 17th international web for all conference , pages=

    Accessible conversational user interfaces: considerations for design , author=. Proceedings of the 17th international web for all conference , pages=

  9. [9]

    Competitive Social Science Research Journal , author=

    Development of Qualitative Semi-Structured Interview Guide for Case Study Research , volume=. Competitive Social Science Research Journal , author=. 2022 , month=

    2022

  10. [10]

    The SAGE handbook of qualitative methods in health research , volume=

    Qualitative interviewing techniques and styles , author=. The SAGE handbook of qualitative methods in health research , volume=. 2010 , publisher=

    2010

  11. [11]

    Journal of advanced nursing , volume=

    Systematic methodological review: developing a framework for a qualitative semi-structured interview guide , author=. Journal of advanced nursing , volume=. 2016 , publisher=

    2016

  12. [12]

    arXiv preprint arXiv:2503.16789 , year=

    Conversational user-ai intervention: A study on prompt rewriting for improved llm response generation , author=. arXiv preprint arXiv:2503.16789 , year=

    work page arXiv

  13. [13]

    Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems , pages=

    Privacy Meets Explainability: Managing Confidential Data and Transparency Policies in LLM-Empowered Science , author=. Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems , pages=

  14. [14]

    Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence , articleno =

    Herlihy, Christine and Neville, Jennifer and Schnabel, Tobias and Swaminathan, Adith , title =. Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence , articleno =. 2024 , publisher =

    2024

  15. [15]

    Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region , pages =

    Staudinger, Moritz and Kusa, Wojciech and Piroi, Florina and Lipani, Aldo and Hanbury, Allan , title =. Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region , pages =. 2024 , isbn =. doi:10.1145/3673791.3698432 , abstract =

    work page doi:10.1145/3673791.3698432 2024

  16. [16]

    Whispers in the Machine: Confidentiality in Agentic Systems

    Whispers in the machine: Confidentiality in llm-integrated systems , author=. arXiv preprint arXiv:2402.06922 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2601.05918 , year=

    Agentic LLMs as Powerful Deanonymizers: Re-identification of Participants in the Anthropic Interviewer Dataset , author=. arXiv preprint arXiv:2601.05918 , year=

    work page arXiv

  18. [18]

    arXiv preprint arXiv:2406.12934 , year=

    Current state of LLM Risks and AI Guardrails , author=. arXiv preprint arXiv:2406.12934 , year=

    work page arXiv

  19. [19]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    The only way is ethics: A guide to ethical research with large language models , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  20. [20]

    Minimalist style of UI interface design in the age of self-media , author=. Proc. Of International Conference on Information and Social Science , pages=

  21. [21]

    ti 5.0: A comparative review of two popular qualitative data-analysis programs , author=

    NVivo 2.0 and ATLAS. ti 5.0: A comparative review of two popular qualitative data-analysis programs , author=. Field methods , volume=. 2004 , publisher=

    2004

  22. [22]

    Journal of the Medical Library Association: JMLA , volume=

    NVivo , author=. Journal of the Medical Library Association: JMLA , volume=

  23. [23]

    Conducting Research in Conservation , pages=

    Qualitative interviews and focus groups , author=. Conducting Research in Conservation , pages=. 2010 , publisher=

    2010

  24. [24]

    Journal of official statistics , author =

    The status of computer-assisted telephone interviewing:. Journal of official statistics , author =. 1986 , note =

    1986

  25. [25]

    Social Science Computer Review , author =

    New. Social Science Computer Review , author =. 1992 , note =. doi:10.1177/089443939201000202 , abstract =

    work page doi:10.1177/089443939201000202 1992

  26. [26]

    Sociological Methods & Research , author =

    Computational. Sociological Methods & Research , author =. 2017 , pages =. doi:10.1177/0049124117729703 , abstract =

    work page doi:10.1177/0049124117729703 2017

  27. [27]

    Big Data & Society , author =

    Computational grounded theory revisited:. Big Data & Society , author =. 2022 , note =

    2022

  28. [28]

    Quality & Quantity , author =

    Quantitative narrative analysis software options compared:. Quality & Quantity , author =. 2013 , note =

    2013

  29. [29]

    The international encyclopedia of communication research methods , author =

    Qualitative analysis software (. The international encyclopedia of communication research methods , author =. 2017 , note =

    2017

  30. [30]

    Qualitative sociology , author =

    The. Qualitative sociology , author =. 1991 , note =

    1991

  31. [31]

    Sociological Research Online , author =

    Choosing. Sociological Research Online , author =. 1998 , pages =. doi:10.5153/sro.178 , abstract =

    work page doi:10.5153/sro.178 1998

  32. [32]

    Qualitative and

    Salmona, Michelle and Lieber, Eli and Kaczynski, Dan , month = aug, year =. Qualitative and

  33. [33]

    Political Analysis , author =

    Synthetic. Political Analysis , author =. 2024 , keywords =. doi:10.1017/pan.2024.5 , abstract =

    work page doi:10.1017/pan.2024.5 2024

  34. [34]

    Sociological Methods & Research , author =

    Generative. Sociological Methods & Research , author =. 2025 , note =. doi:10.1177/00491241251327130 , abstract =

    work page doi:10.1177/00491241251327130 2025

  35. [35]

    Social Science Computer Review , author =

    The. Social Science Computer Review , author =. 2025 , note =. doi:10.1177/08944393251334977 , abstract =

    work page doi:10.1177/08944393251334977 2025

  36. [36]

    Social Science Computer Review , author =

    Navigating the. Social Science Computer Review , author =. 2025 , note =. doi:10.1177/08944393251366243 , abstract =

    work page doi:10.1177/08944393251366243 2025

  37. [37]

    Social Science Computer Review , author =

    Prompting the. Social Science Computer Review , author =. 2025 , note =. doi:10.1177/08944393251344865 , abstract =

    work page doi:10.1177/08944393251344865 2025

  38. [38]

    Social Science Computer Review , author =

    Dialogues. Social Science Computer Review , author =. 2025 , note =. doi:10.1177/08944393251370354 , abstract =

    work page doi:10.1177/08944393251370354 2025

  39. [39]

    Proceedings of the National Academy of Sciences , author =

    Can. Proceedings of the National Academy of Sciences , author =. 2024 , pages =. doi:10.1073/pnas.2314021121 , abstract =

    work page doi:10.1073/pnas.2314021121 2024

  40. [40]

    International Journal of Social Research Methodology , author =

    Clarifying question meaning in standardized interviews can improve data quality even though wording may change: a review of the evidence , volume =. International Journal of Social Research Methodology , author =. 2021 , note =. doi:10.1080/13645579.2020.1824627 , abstract =

    work page doi:10.1080/13645579.2020.1824627 2021

  41. [41]

    PLOS ONE , author =

    Precision and. PLOS ONE , author =. 2015 , note =. doi:10.1371/journal.pone.0128337 , abstract =

    work page doi:10.1371/journal.pone.0128337 2015

  42. [42]

    Topics in Cognitive Science , author =

    When. Topics in Cognitive Science , author =. 2018 , note =. doi:10.1111/tops.12330 , abstract =

    work page doi:10.1111/tops.12330 2018

  43. [43]

    Public opinion quarterly , author =

    Does conversational interviewing reduce survey measurement error? , url =. Public opinion quarterly , author =. 1997 , note =

    1997

  44. [44]

    Applied Cognitive Psychology , author =

    Bringing features of human dialogue to web surveys , volume =. Applied Cognitive Psychology , author =. 2007 , note =. doi:10.1002/acp.1335 , abstract =

    work page doi:10.1002/acp.1335 2007

  45. [45]

    Journal of Official Statistics , author =

    Promoting uniform question understanding in today's and tomorrow's surveys , volume =. Journal of Official Statistics , author =. 2005 , note =

    2005

  46. [46]

    Journal of official statistics , author =

    New strategies for pretesting survey questions , volume =. Journal of official statistics , author =. 1991 , note =

    1991

  47. [47]

    Public Opinion Quarterly , author =

    Interviewer. Public Opinion Quarterly , author =. 2010 , pages =. doi:10.1093/poq/nfp089 , abstract =

    work page doi:10.1093/poq/nfp089 2010

  48. [48]

    Public Opinion Quarterly , author =

    Questions for. Public Opinion Quarterly , author =. 2011 , pages =. doi:10.1093/poq/nfr048 , abstract =

    work page doi:10.1093/poq/nfr048 2011

  49. [49]

    Public Opinion Quarterly , author =

    Clarifying. Public Opinion Quarterly , author =. 2000 , pages =. doi:10.1086/316757 , abstract =

    work page doi:10.1086/316757 2000

  50. [50]

    Narayanan, Arvind and Kapoor, Sayash , month = dec, year =

  51. [51]

    , month = jul, year =

    Briggs, Charles L. , month = jul, year =. Learning

  52. [52]

    Strangers in their own land:

    Hochschild, Arlie Russell , year =. Strangers in their own land:

  53. [53]

    Interviews:

    Kvale, Steinar and Brinkmann, Svend , year =. Interviews:

  54. [54]

    Sociological Focus , author =

    A. Sociological Focus , author =. 2019 , pages =. doi:10.1080/00380237.2018.1544514 , language =

    work page doi:10.1080/00380237.2018.1544514 2019

  55. [55]

    Qualitative Research , author =

    ‘. Qualitative Research , author =. 2013 , note =. doi:10.1177/1468794112446106 , abstract =

    work page doi:10.1177/1468794112446106 2013

  56. [56]

    Comparing

    Guven, Semra Yuksel and G. Comparing. 2025 , note =

    2025

  57. [57]

    Soudani, Heydar and Petcu, Roxana and Kanoulas, Evangelos and Hasibi, Faegheh , month = may, year =. A. doi:10.48550/arXiv.2405.13003 , abstract =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2405.13003

  58. [58]

    Balloccu, Simone and Schmidtová, Patrícia and Lango, Mateusz and Dušek, Ondřej , year =. Leak,

  59. [59]

    Conversations at

    Geiecke, Friedrich and Jaravel, Xavier , year =. Conversations at

  60. [60]

    Conducting

    Chopra, Felix and Haaland, Ingar , year =. Conducting

  61. [61]

    and Würschinger, Quirin and Kreuter, Frauke , year =

    Wuttke, Alexander and Aßenmacher, Matthias and Klamm, Christopher and Lang, Max M. and Würschinger, Quirin and Kreuter, Frauke , year =

  62. [62]

    , year =

    Weiss, Robert S. , year =. Learning from

  63. [63]

    Sociological Methods & Research , author =

    Sources of. Sociological Methods & Research , author =. 2019 , pages =. doi:10.1177/0049124117729698 , number =

    work page doi:10.1177/0049124117729698 2019

  64. [64]

    Sociological Methods & Research , author =

    Deceptively. Sociological Methods & Research , author =. 2023 , pages =. doi:10.1177/00491241221140431 , number =

    work page doi:10.1177/00491241221140431 2023

  65. [65]

    Nature , author =

    Why. Nature , author =. 2023 , pages =. doi:10.1038/d41586-023-01295-4 , number =

    work page doi:10.1038/d41586-023-01295-4 2023

  66. [66]

    Nature Computational Science , author =

    Using. Nature Computational Science , author =. 2024 , pages =. doi:10.1038/s43588-023-00585-1 , number =

    work page doi:10.1038/s43588-023-00585-1 2024

  67. [67]

    Ethnography , author =

    Interpretive. Ethnography , author =. 2017 , pages =. doi:10.1177/1466138115592418 , number =

    work page doi:10.1177/1466138115592418 2017

  68. [68]

    AI-interview , url=

    Carlsen, Hjalmar B , year=. AI-interview , url=

  69. [69]

    Krippendorff, Klaus , year =. Content

  70. [70]

    ACM Transactions on Interactive Intelligent Systems (TiiS) , volume=

    Bridging the gap between ethics and practice: guidelines for reliable, safe, and trustworthy human-centered AI systems , author=. ACM Transactions on Interactive Intelligent Systems (TiiS) , volume=. 2020 , publisher=

    2020

  71. [71]

    and Wang, Haohan , booktitle =

    Jin, Haibo and Zhou, Andy and Menke, Joe D. and Wang, Haohan , booktitle =. Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters , url =. doi:10.52202/079017-1896 , editor =

    work page doi:10.52202/079017-1896

  72. [72]

    An Audit on the Perspectives and Challenges of Hallucinations in NLP

    Narayanan Venkit, Pranav and Chakravorti, Tatiana and Gupta, Vipul and Biggs, Heidi and Srinath, Mukund and Goswami, Koustava and Rajtmajer, Sarah and Wilson, Shomir. An Audit on the Perspectives and Challenges of Hallucinations in NLP. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp...

    work page doi:10.18653/v1/2024.emnlp-main.375 2024

  73. [73]

    ACM Transactions on Information Systems , volume=

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions , author=. ACM Transactions on Information Systems , volume=. 2025 , publisher=

    2025

  74. [74]

    Qualitative Health Research , author =

    Protecting. Qualitative Health Research , author =. 2009 , pages =. doi:10.1177/1049732309350879 , number =

    work page doi:10.1177/1049732309350879 2009

  75. [75]

    Comparing Text and Voice Answers to Open Questions on Sensitive Topics in Smartphone Surveys , author =

    Typing or. Comparing Text and Voice Answers to Open Questions on Sensitive Topics in Smartphone Surveys , author =

  76. [76]

    Ethnography , author =

    ‘. Ethnography , author =. 2009 , pages =. doi:10.1177/1466138108099586 , number =

    work page doi:10.1177/1466138108099586 2009

  77. [77]

    Sociological Methods & Research , author =

    Gender. Sociological Methods & Research , author =. 1985 , pages =. doi:10.1177/0049124185014001002 , number =

    work page doi:10.1177/0049124185014001002 1985

  78. [78]

    , year =

    Goldthorpe, John H. , year =. On

  79. [79]

    Gerson, Kathleen and Damaske, Sarah , year =. The

  80. [80]

    Field Methods , author =

    Survey. Field Methods , author =. 2005 , pages =. doi:10.1177/1525822X04269550 , number =

    work page doi:10.1177/1525822x04269550 2005

Showing first 80 references.