Pith. sign in

REVIEW 4 major objections 5 minor 27 references

Facilitating Trustworthy Human-Agent Collaboration in LLM-based Multi-Agent System oriented Software Engineering

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A RACI-based framework can keep humans accountable when LLM agents do software work

desk verdict A modest but useful work-in-progress: the RACI framework is a sensible, internally consistent role-assignment tool for human-LLM collaboration, but the trust and accountability claims outrun the evidence. read the letter →

arxiv 2505.04251 v1 pith:TOTVGTBZ submitted 2025-05-07 cs.SE cs.AIcs.MA

Krishna Ronanki This is my paper · ORCID
classification cs.SEcs.AIcs.MA
keywords LargeLanguageModelsLLM-BasedMulti-AgentSystemsSoftwareEngineeringTrustworthyAIHuman-AgentCollaborationDevOpsRACImatrixAccountability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This work-in-progress paper proposes a RACI-based framework (Responsible, Accountable, Consulted, Informed) for assigning roles between humans and LLM-based multi-agent systems in software engineering. The framework provides a nine-step implementation guideline and a worked example for a DevOps planning phase. The paper argues that this structured role assignment can facilitate efficient collaboration, ensure accountability, and mitigate risks from LLM-driven automation, aligning with the EU's Trustworthy AI guidelines. A sympathetic reading takes the proposal as a concrete answer to the open problem of how to allocate tasks between humans and LMA systems.

What carries the argument

The central object is the RACI responsibility-assignment matrix, a standard project-management tool here adapted to LMA-oriented software engineering. Responsibilities are defined as: Responsible does the work, Accountable delegates and validates the outcome, Consulted provides input, and Informed is kept in the loop. The machinery is the constraint set that forces a human Accountable for every LLM-Responsible task, plus the nine-step implementation guideline that connects tasks, actors, regulatory constraints, and workflow design. The matrix enforces human oversight at the point of output validation.

What would settle it

A multi-case study with expert walkthroughs that applies the nine steps to SE tasks whose outputs cannot be automatically verified would show whether Step-1's artefact-based precondition holds. If teams cannot assign a human Accountable actor to such tasks without a verifiable output, the framework's central accountability guarantee would be limited to a subset of software work.

Watch

Extended reading notes

Core claim

The paper's central claim is that applying the RACI matrix to human-agent collaboration solves the task-allocation problem in LLM-based multi-agent software engineering. In this adapted RACI, humans hold the Accountable role whenever an LLM-agent is Responsible for producing an artefact, so every automated output passes through a human validation gate before the task is complete. The framework's three constraints enforce this: each task needs at least one Responsible and one Accountable actor, and no LLM-agent can be Responsible without at least one human being Accountable. The example DevOps planning matrix demonstrates how this works for tasks such as creating user stories, building the product backlog, and sprint planning.

Load-bearing premise

The framework assumes every task it governs is artefact-based, so its outputs can be compared against ground truth automatically; for tasks like stakeholder negotiation or product vision alignment, the paper offers no mechanism to keep humans accountable for LLM-agent output.

Editorial extensions

If this is right

  • Teams can use the nine-step method to produce a RACI matrix for any artefact-based task, phase, or the whole SDLC, giving a concrete division of labour.
  • Human validation becomes a mandatory step before any LLM-agent-produced artefact is accepted, which operationalises accountability in practice.
  • The framework offers a systematic way to demonstrate compliance with EU AI Act deployer obligations, since every LLM output has a named human accountable actor.
  • The example DevOps planning matrix can be reused as a template for other SDLC phases such as development, testing, and deployment.
  • The distinction between Responsible and Accountable clarifies when LLM agents may act autonomously versus when humans must stay in the loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the framework proves out, it suggests a general principle for human-AI collaboration: accountability should be structurally assigned to humans at the validation point, not at the point of task execution, and this could extend to other artefact-producing AI domains such as data analysis or document drafting.
  • The artefact-based precondition means the framework may not cover early-SDLC activities like stakeholder negotiation or product vision alignment; extending it to such tasks would require specifying new validation mechanisms for non-verifiable outputs.
  • A testable extension would be to compare teams using RACI assignments against teams without them on metrics like the share of LLM-generated artefacts merged without human sign-off, or the number of incidents traced to unvalidated outputs.
  • The paper's planned groupware walkthrough could be sharpened to specifically probe Step-1, asking experts which SDLC tasks resist ground-truth verification and whether the framework still guarantees accountability for those.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This work-in-progress paper proposes a RACI-based framework (Responsible, Accountable, Consulted, Informed) for assigning roles between humans and LLM-based agents in software engineering (SE) workflows. The framework is presented as nine implementation steps plus a set of constraints, and a worked example builds a RACI matrix for the planning phase of a DevOps SDLC, involving three human actors and three LLM agents. The central claim is that this structured role assignment 'can facilitate efficient collaboration, ensure accountability, and mitigate potential risks associated with LLM-driven automation while aligning with the Trustworthy AI guidelines.' The paper explicitly notes that no empirical validation has been performed and outlines a planned groupware-walkthrough multi-case study as future work.

Significance. If the framework's central claim survives scrutiny, it would give practitioners a concrete, low-cost tool for injecting human oversight and accountability into LLM-driven SE automation, a pressing problem given the EU AI Act's oversight requirements. Strengths of the paper include the clearly enumerated nine-step process, the explicit set of framework constraints in §3.1.1, and a worked example whose RACI matrix is internally consistent with the accompanying narrative in most rows. The authors also deserve credit for openly stating that the framework has not been empirically validated and for identifying a concrete evaluation method. However, the paper's core accountability mechanism rests on the 'artefact-based' precondition, which is not precisely defined and appears violated by the very tasks in the example implementation; this undermines the strength of the central claim as currently stated.

major comments (4)
  1. [§3.1 Step-1 and §3.2 Example Implementation (Table 1)] The framework's Step-1 requires that all tasks be 'artefact-based,' and §1 argues that artefact-based evaluation against ground truth can be automated. Yet the example includes 'sprint planning and task allocation' and 'create product roadmap' as tasks where LLM-agents are Responsible or Consulted (Table 1, rows 5-6 and row 2). A sprint plan, task allocation, or product roadmap is a collaborative commitment with no objective ground-truth oracle, so the Step-8 human validation step has no verifiable target. This creates a dilemma: if 'artefact-based' means 'produces a document,' the Step-1 filter is vacuous and the accountability guarantee does not follow; if it means 'automatically verifiable against ground truth,' the example violates its own Step-1. The paper needs to define 'artefact-based' precisely and either restrict the framework to tasks with automatic verifiability or demonstrate how human validation alone anchors accountability without a ground-truth oracle.
  2. [§3.1.1 Constraint (2) and §3.2 Step-6 / Table 1, 'Create product roadmap' row] The framework's second constraint states that there must be at least one 'Accountable' assignment for each task, with an exception only when LLM-agents are assigned 'Informed' or no assignment. In Table 1, the 'Create product roadmap' row assigns LLM-agent A and LLM-agent C as 'Consulted,' yet assigns no 'Accountable' actor at all. Step-6 (point 2) adds a further inconsistency: it says 'There is no accountable assignment in this scenario as per the second framework constraint. The responsible actor is the accountable actor as well.' If the responsible actor is also the accountable actor, an A assignment should appear; if no A appears, the constraint is violated because the exception does not apply when LLM-agents are Consulted. This internal inconsistency directly affects the framework's internal validity and must be resolved in the example and in the statement of the exception.
  3. [§3.2 Step-7 and §4 Conclusion] The paper claims that the AIA's obligations are satisfied whenever a human actor holds the Accountable role for a task in which an LLM-agent is Responsible (§3.2 Step-7), and this is used to support the overall claim of alignment with Trustworthy AI guidelines. The EU AI Act, however, imposes specific requirements (risk classification, transparency, human oversight, etc.) that are not mapped to framework elements anywhere in the paper. The assertion that a single human 'Accountable' assignment resolves all compliance obligations is presented without supporting analysis or reference to specific AIA provisions. This is a load-bearing step for the 'trustworthy' part of the central claim; the authors should either provide a concrete mapping between framework constructs and AIA obligations or temper the claim to indicate that the framework provides a structure that could support compliance rather than ensure it.
  4. [§4 Conclusion] The paper states as a matter of fact that the framework 'ensures accountability' and 'mitigates potential risks associated with LLM-driven automation,' while also admitting in the same section that it 'has not yet undergone empirical validation.' For a work-in-progress paper, the abstract and conclusion should use modal or provisional language (e.g., 'is intended to facilitate,' 'may help ensure') so that the claims are not stronger than the evidentiary basis. This is not a fatal flaw, but it is important for readers to accurately calibrate the state of the research.
minor comments (5)
  1. [Abstract] The abstract contains a grammatical error: 'challenges that spans' should be 'challenges that span.' Please proofread throughout.
  2. [§3.1 Step-7 and Step-9] In Step-7, the phrase 'are in do not conflict' should be 'do not conflict.' In Step-9, 'If yes, return to Step-8' uses 'yes' where 'so' or 'conflicts exist' would be clearer. Similar typos appear in Step-7's reference to 'stepStep-5.'
  3. [§3.2 Step-4 (5)] The sentence 'Sprint planning and task allocation, which is are typically performed...' should be 'which is typically performed' or 'which are typically performed.'
  4. [Table 1] The row 'Task allocation' is identical to 'Sprint planning' in the matrix but has no separate explanation in Step-6. Please clarify whether these are intended as distinct tasks or as a single combined task; if combined, the table should have one row.
  5. [References] Reference [10] lists 'Tse-Husn' as an author; this appears to be a garbled name. The authors should verify the correct spelling, likely 'Tse-Hsun Chen.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed RACI framework is a design proposal built from an external project-management construct; its accountability guarantee is definitional rather than a fitted or self-cited derivation.

full rationale

The paper makes no empirical prediction that is fitted to data; it proposes a RACI-based responsibility-assignment method for human-LLM-agent collaboration. The central claim ('can facilitate efficient collaboration, ensure accountability, and mitigate potential risks') is presented as a design goal, and the framework's mechanism - constraint (3) requiring a human Accountable whenever an LLM-agent is Responsible - implements that goal by construction. This is a design invariant, not a derivation of an observed outcome from inputs that already contain the conclusion. The RACI construct itself is imported from external project-management literature (PMBOK [7]), not invented by the authors. The only in-house citation, [19] (Ronanki et al. 2023), is used to support the qualitative claim that the EU AI Act is 'the most comprehensive in terms of coverage of applicable AI principles'; this claim is not load-bearing because the same Step-3 also directly cites the AIA texts [13,15], and the framework's compliance checks do not depend on the self-citation. Similarly, the agent profiles are based on external prior work ([5,6,25]). The 'ensure all tasks are artefact-based' precondition (Step-1) is an untested assumption and may be violated by tasks such as sprint planning in the worked example; however, that is a validity/coverage risk, not a circularity, because the framework does not use the outcomes of those tasks to define its own correctness. No equation or fitted parameter is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked to force the framework's choice.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The paper is a process framework, not a quantitative model. It introduces no free parameters or fitted constants. It relies on domain assumptions about artefact-based tasks, AI Act compliance mapping, and the transferability of RACI semantics to human-agent teams. The only invented entity is the hypothetical LLM-agent C, explicitly flagged as an assumption.

assumptions (4)
  • domain assumption SE tasks to which the framework applies must be artefact-based (Step-1, §3.1).
    The framework's workflows rely on verifiable artefacts for human oversight; non-artefact-based tasks (e.g., stakeholder negotiation) fall outside the method.
  • ad hoc to paper EU AI Act obligations can be satisfied by human Accountable assignments on tasks where LLM agents are Responsible (Step-7, §3.2).
    The paper equates role assignment with AIA compliance without demonstrating that this mapping satisfies the legal requirements.
  • domain assumption RACI role definitions transfer from human-only project management to human-agent teams without adaptation (§2.3, §3).
    The framework imports RACI unchanged and assumes accountability semantics remain meaningful when the Responsible actor is an LLM agent.
  • domain assumption LLM agents can be made technically proficient for assigned tasks via modelling on existing agents or fine-tuning (Step-5, §3.1).
    The example relies on agent A, B, and C proficiency; for agent C this is explicitly hypothetical (footnote 1).
invented entities (1)
  • LLM-agent C (scrum implementation assistant)
    purpose: To populate the RACI matrix in the DevOps planning example (Responsible for sprint planning and task allocation).
    Footnote 1 states the agent does not currently exist and is assumed by the authors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Facilitating Trustworthy Human-Agent Collaboration in LLM-based Multi-Agent System oriented Software Engineering." pith.science (2026). https://pith.science/paper/TOTVGTBZ

@misc{pith2026250504251,
  author       = {Pith},
  title        = {Pith review of: Facilitating Trustworthy Human-Agent Collaboration in LLM-based Multi-Agent System oriented Software Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TOTVGTBZ}},
  note         = {Machine review of arXiv:2505.04251}
}
read the original abstract

Multi-agent autonomous systems (MAS) are better at addressing challenges that spans across multiple domains than singular autonomous agents. This holds true within the field of software engineering (SE) as well. The state-of-the-art research on MAS within SE focuses on integrating LLMs at the core of autonomous agents to create LLM-based multi-agent autonomous (LMA) systems. However, the introduction of LMA systems into SE brings a plethora of challenges. One of the major challenges is the strategic allocation of tasks between humans and the LMA system in a trustworthy manner. To address this challenge, a RACI-based framework is proposed in this work in progress article, along with implementation guidelines and an example implementation of the framework. The proposed framework can facilitate efficient collaboration, ensure accountability, and mitigate potential risks associated with LLM-driven automation while aligning with the Trustworthy AI guidelines. The future steps for this work delineating the planned empirical validation method are also presented.

Figures

Figures reproduced from arXiv: 2505.04251 by the authors.

Figure 1
Figure 1. Responsibility Assignment Framework • Step-4: Identify the tasks that can be automated using LLM￾agents. • Step-5: Ensure the LLM-agent(s) selected are technically proficient in automating the identified task(s). • Step-6: Assign each actor across the row an R, A, C, or I to indicate the role they’ll play within the workflow to mark them as complete. • Step-7: Ensure the assignments from Step-6 are in do not conflic… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 12 canonical work pages

  1. [1]

    Malak Abdullah, Alia Madain, and Yaser Jararweh. 2022. ChatGPT: Funda- mentals, Applications and Social Impacts. In2022 Ninth International Con- ference on Social Networks Analysis, Management and Security (SNAMS). 1–8. doi:10.1109/SNAMS58071.2022.10062688

    arXiv 2022

  2. [2]

    Maria Christakis and Christian Bird. 2016. What Developers Want and Need From Program Analysis: An Empirical Study. InProceedings of the 31st IEEE/ACM Inter- national Conference on Automated Software Engineering(Singapore, Singapore) (ASE ’16). 332–343. doi:10.1145/2970276.2970347

    arXiv 2016

  3. [3]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engi- neering: Survey and Open Problems. In2023 IEEE/ACM International Confer- ence on Software Engineering: Future of Software Engineering (ICSE-FoSE). 31–53. doi:10.1109/ICSE-FoSE59343.2023.00008

    arXiv 2023

  4. [4]

    João Faustino, Daniel Adriano, Ricardo Amaro, Rubén Pereira, and Miguel Mira da Silva. 2022. DevOps Benefits: A Systematic Literature Review.Soft- ware: Practice and Experience52, 9 (2022), 1905–1926. doi:10.1002/spe.3096 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/spe.3096

    work page doi:10.1002/spe.3096 2022

  5. [5]

    Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead.ACM Trans. Softw. Eng. Methodol.(2025). doi:10.1145/3712003

    doi:10.1145/3712003 2025

  6. [6]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber

  7. [7]

    Project Management Institute. 2000. A Guide to the Project Management Body of Knowledge (PMBOK Guide). Project Management Institute

    work page 2000

  8. [8]

    Jennings

    Nicholas R. Jennings. 1999. Agent-Oriented Software Engineering. InMulti-Agent System Engineering, Francisco J. Garijo and Magnus Boman (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1–7

    work page 1999

Show all 27 references
  1. [9]

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. InAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, an...

    2023

  2. [10]

    Feng Lin, Dong Jae Kim, Tse-Husn, and Chen. 2024. When LLM-based Code Generation Meets the Software Development Process. arXiv:2403.15852 [cs.SE] https://arxiv.org/abs/2403.15852

    2024 arXiv

  3. [11]

    Emerson Murphy-Hill, Chris Parnin, and Andrew P. Black. 2012. How We Refactor, and How We Know It.IEEE Transactions on Software Engineering38, 1 (2012), 5–18. doi:10.1109/TSE.2011.41

    2012 doi

  4. [12]

    Niedober, Nhut T

    David J. Niedober, Nhut T. Ho, Gina Masequesmay, Kolina Koltai, Mark Skoog, Artemio Cacanindin, Walter Johnson, and Joseph B. Lyons. 2014. Influence of Cultural, Organizational and Automation Factors on Human-Automation Trust: A Case Study of Auto-GCAS Engineers and Developmen...

    2014

  5. [13]

    Future of Life Institute. 2024. High-level Summary of the AI Act. https:// artificialintelligenceact.eu/high-level-summary/

    2024

  6. [14]

    2019.Ethics Guidelines for Trustworthy Artificial Intelligence (AI)

    High-Level Expert Group on Artificial Intelligence. 2019.Ethics Guidelines for Trustworthy Artificial Intelligence (AI). www.aepd.es/sites/default/files/2019- 12/ai-ethics-guidelines.pdf

    2019

  7. [15]

    European Parliament. 2024. Artificial Intelligence Act. https://www.europarl. europa.eu/doceo/document/TA-9-2024-0138_EN.pdf

    2024

  8. [16]

    David Pinelle and Carl Gutwin. 2002. Groupware Walkthrough: Adding Context to Groupware Usability Evaluation. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Minneapolis, Minnesota, USA)(CHI ’02). 455–462. doi:10.1145/503376.503458

    2002

  9. [17]

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. InProceedings of the 62nd Annual Meeting of the As...

    2024 doi

  10. [18]

    Gema Rodríguez-Pérez, Reza Nadri, and Meiyappan Nagappan. 2021. Perceived Diversity in Software Engineering: A Systematic Literature Review.Empirical Software Engineering26 (2021), 1–38

    2021

  11. [19]

    Krishna Ronanki, Beatriz Cabrero-Daniel, Jennifer Horkoff, and Christian Berger

  12. [20]

    Simin Wang, Liguo Huang, Amiao Gao, Jidong Ge, Tengfei Zhang, Haitao Feng, Ishna Satyarth, Ming Li, He Zhang, and Vincent Ng. 2023. Machine/Deep Learning for Software Engineering: A Systematic Literature Review.IEEE Transactions on Software Engineering49, 3 (2023), 1188–1231. ...

    2023

  13. [21]

    InProceedings of the First International Symposium on Trustworthy Autonomous Systems(Edinburgh, United Kingdom)(TAS ’23)

    RE-centric Recommendations for the Development of Trustworthy(er) Autonomous Systems. InProceedings of the First International Symposium on Trustworthy Autonomous Systems(Edinburgh, United Kingdom)(TAS ’23). Article 1, 8 pages. doi:10.1145/3597512.3599697

  14. [22]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agar- w...

    2022

  15. [23]

    Adam Waytz, Joy Heafner, and Nicholas Epley. 2014. The Mind in the Machine: Anthropomorphism Increases Trust in an Autonomous Vehicle.Journal of Exper- imental Social Psychology52 (2014), 113–117. doi:10.1016/j.jesp.2014.01.005

    2014 doi

  16. [24]

    Herbsleb, Alexandra Holloway, and Scott Davidoff

    David Gray Widder, Laura Dabbish, James D. Herbsleb, Alexandra Holloway, and Scott Davidoff. 2021. Trust in Collaborative Automation in High Stakes Software Engineering Work: A Case Study at NASA. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems(Y...

    2021

  17. [25]

    Jim Whitehead. 2007. Collaboration in Software Engineering: A Roadmap. In Future of Software Engineering (FOSE ’07). 214–225. doi:10.1109/FOSE.2007.4

    2007 doi

  18. [27]

    Zheying Zhang, Maruf Rayhan, Tomas Herda, Manuel Goisauf, and Pekka Abra- hamsson. 2024. LLM-Based Agents for Automating the Enhancement of User Story Quality: An Early Report. InAgile Processes in Software Engineering and Extreme Programming. Springer Nature Switzerland, Cham...

    2024

  19. [2023]

    arXiv:2308.00352 [cs.AI] https://arxiv.org/abs/2308.00352

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. arXiv:2308.00352 [cs.AI] https://arxiv.org/abs/2308.00352

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.