REVIEW 5 major objections 5 minor 32 references
Leveraging Large Language Models for Tacit Knowledge Discovery in Organizational Contexts
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read An LLM-driven agent interviewing simulated employees reconstructs full table knowledge in 94.9% of 864 runs, without ever reaching the sole domain expert.
desk verdict The paper is a sensible LLM-agent simulation study, but its headline 94.9% recall metric is far weaker than it looks because the agent already knows the column names at the start. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a prompt-chained LLM agent whose knowledge state is a running table description. After each employee answer, the agent scores the description, writes a critique, lists gaps, and revises before choosing the next question or next employee from a stack ordered by hierarchy and by names mentioned in the conversation. On the simulation side, knowledge dissemination follows a Susceptible-Infectious model with waning infectivity, $\beta(t) = \beta_0 e^{-\gamma t}$, treating each table column as a separate fact that spreads from a patient zero through formal hierarchy and informal relationship networks. The self-critical loop is the part that does the work: it turns vague replies into concrete next questions and keeps the search focused, which is how the agent recovers facts without ever reaching the sole expert.
What would settle it
Run the same agent against simulated employees whose backgrounds do not tell them what their colleagues know, and who answer only with facts they personally hold, then compare full-knowledge recall; if it falls well below 94.9%, the original figure was carried by leaked metadata in the simulation rather than by conversational reconstruction.
Extended reading notes
Core claim
The central claim is that autonomous conversational agents can serve as a knowledge-discovery method inside organizations, reconstructing a complete dataset description from scattered partial knowledge. In the paper's simulation, each column of a table is a fact that spreads through the company like an infection, and the agent enters after the knowledge has already fragmented. Across 864 runs the agent achieved 94.9% full-knowledge recall, and in the 44 failures the final description still retained about 77% of the columns. The low correlation (-0.06) between faithfulness and patient-zero contact is the key evidence: high-quality reconstructions occur even where the original domain specialist was never reached, so the method is not just find-the-expert. The authors present this as a step toward capturing tacit knowledge that formal documentation misses.
Load-bearing premise
The load-bearing premise is that the LLM-simulated employees behave about as helpfully and as knowledgeably as real employees would, and that the generated ground-truth descriptions are representative of real organizational knowledge; if those two hold, the 94.9% recall is a genuine property of the conversational method rather than an artifact of the simulation.
Editorial extensions
If this is right
- If the 94.9% figure holds outside simulation, organizations can turn documentation into a set of short conversational interviews instead of a search for the right expert.
- Because self-critical scores track external reference metrics, a deployed agent could monitor its own documentation quality without ground truth.
- Parameter effects give practical guidance: more informal connections shorten the conversation path and reduce reliance on hubs, so network structure, not just prompt quality, determines retrieval cost.
- The near-zero dependence on patient zero implies that knowledge can be reconstructed from partial holders, which matters when the original expert has left the company or is overloaded.
Reading between the lines
- Beyond the paper: the simulation's employee backgrounds include awareness of what their colleagues know, which real employees may lack; a deeper test would disable that information and measure how much recall drops.
- Beyond the paper: the same self-critical loop could be pointed at other fragmented artifacts, such as codebases, design decisions, and incident postmortems, where the table descriptions are replaced by whatever document needs reconstruction.
- Beyond the paper: the framework implies that organizational network structure is a first-class variable in LLM-agent evaluation, so future benchmarks should report conversation graphs, not just final scores.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an LLM-based agent system that reconstructs documentation of a data table by conversing with simulated employees in synthetic organizational networks. Knowledge dissemination is modeled as an SI epidemic with waning infectivity over combined formal and informal networks, and the agent iteratively asks questions, updates a description, and self-critiques. The authors run 864 simulations with varying organizational parameters and report a 94.9% full-knowledge recall rate, correlations between the agent's self-critical scores and external reference-based metrics, and evidence that high-quality descriptions can be obtained without contacting the single domain specialist (patient zero). The paper also releases code, data, and prompts.
Significance. If the claims held, the paper would offer a scalable simulation-based methodology for studying tacit-knowledge elicitation in organizations and a useful benchmark for LLM-agent interaction. The work has concrete strengths: a large simulation corpus (over 300,000 interactions), a clearly specified epidemic-inspired knowledge-dissemination model, a reproducible experimental design with a public code/data deposit, and a multi-metric evaluation that includes both a non-LLM metric (METEOR) and LLM-as-judge scores. The analysis of how organizational parameters (hierarchy depth, informal connections, decay, alpha) affect retrieval is a genuinely useful contribution. However, the central quantitative claims are weakened by a near-trivial recall criterion, missing baselines, and a modest correlation between self-critique scores and external metrics, so the paper's main conclusions need substantial revision before they can be considered established.
major comments (5)
- [Section III.A and Section V (Full-knowledge recall)] The headline result of 94.9% full-knowledge recall does not test knowledge reconstruction. Section III.A states that the agent begins with knowledge k0 that includes the table's name and columns, and Section V.a defines full-knowledge recall as a final description that included a mention of every column originally disseminated. Since k0 already contains every column name, this metric only measures whether the agent preserves a list that was given to it in the initial prompt; it does not require recovery of column semantics, data types, example values, variable interactions, or any other content that the epidemic model actually disseminates. The claim in Section V that the agent can recover information without contacting patient zero is therefore not supported by the 94.9% number. The authors should either redefine full-knowledge recall to require recovery of non-trivial content (types, meanings, examples, interactions), or explicitly evaluate the column-list baseline and report the recall of substantive facts separately.
- [Section V (Table II) and Abstract] The abstract claims that self-critical feedback scores strongly correlate with external literature critic scores, but Table II shows Spearman correlations of only 0.26 (cMETEOR vs. SCS) and 0.35 (cGEvalCoh vs. SCS), with the strongest correlation being 0.50 (cGEvalFaith vs. SCS). These are weak-to-moderate correlations, not strong ones. Only the self-critical score with context (SCS vs. SCS+C, 0.73) is strong, and that comparison is between two variants of the same LLM-based critic rather than an external evaluation. The wording should be corrected, and the interpretation of Table II should be revised to acknowledge that the agent's self-critique has limited ability to rank final descriptions according to external quality.
- [Section IV and Section V (Evaluation design)] The evaluation lacks baselines and ablations that would establish the contribution of the proposed agent architecture. For example, there is no comparison against: (a) an agent that simply writes a report from its k0 column list without any conversation, (b) an agent that randomly selects employees instead of using self-critique-guided questions, or (c) a non-agentic prompting baseline that reads the same employee responses without the prompt-chaining/self-critique loop. Without such controls, the 94.9% recall and the correlation findings cannot be attributed to conversational knowledge retrieval or to the SI-based network traversal. The authors should add at least one baseline and one ablation, or clearly state what the incremental contribution of each component is.
- [Section IV (Simulation) and Section V (G-Eval)] The validation loop is largely internal to the same LLM family: the ground-truth table descriptions are generated by GPT-4o mini, the simulated employees are GPT-4o mini, the agent is GPT-4o mini, and the G-Eval judge is an LLM (presumably also GPT-4o or similar). Only METEOR provides a non-LLM anchor. This creates a risk that the reported success partly reflects self-consistency of LLM-generated content rather than recovery of an independent ground truth. The paper should add a limitations paragraph discussing this circularity, and ideally include a small human-evaluated subsample or at least an analysis of which claims survive under the purely lexical METEOR metric (e.g., report the METEOR scores by semantic content type rather than an average of 0.17).
- [Section V (Patient-zero analysis and Figure 4)] The conclusion that the agent's effectiveness does not depend on reaching patient zero is based on the correlation between cGEvalFaith and % p0 (-0.06) and on the UMAP visualization. However, cGEvalFaith is an LLM-judge score on semantic faithfulness, and the correlation is computed across configurations with only three repetitions each. More importantly, since the recall metric is trivially satisfied, the evidence that the agent reconstructs knowledge without patient zero is weak. The authors should report the distribution of METEOR or cGEvalFaith conditional on whether patient zero was reached, with confidence intervals, and should avoid claiming independence from patient zero when the strongest metric (94.9% recall) does not measure substantive knowledge recovery.
minor comments (5)
- [Section III.B, Eq. (2)] Equation (2) is garbled: 'bd ≈ hierarchy depth p number of employees' should presumably be the depth-th root of the number of employees, i.e., bd ≈ (number of employees)^(1/hierarchy depth). Please fix the notation.
- [Section IV] The sentence 'For each repetition, a table subject was randomly selected from a predefined list, which was then used to generate the simulated complete knowledge s∗ through an LLM' introduces the symbol s∗, but the text elsewhere calls this knowledge k∗. Please use a single consistent notation.
- [Section V (Table I)] In the caption and text, the paper refers to 'average and mean self-critical score' for patient zero, which is redundant; please clarify which column corresponds to which statistic.
- [Section I and Conclusion] The introduction says the method 'facilitates the crowdsourcing of this process using LLM agents', but the conclusion describes it as a method for 'tacit knowledge retrieval'. Please reconcile the terminology, since the simulation only measures reproduction of given facts, not the acquisition of genuinely tacit (unarticulated) knowledge.
- [Throughout] There are some typographical issues (e.g., 'elected a range of possible values' in Section IV, 'company structures' in the conclusion). A careful proofreading pass would be helpful.
Circularity Check
Headline 94.9% full-knowledge recall reduces to the initial column list, and the semantic validation loop stays inside GPT-4o mini.
-
self definitional
[Section III.A (initial knowledge state) and Section V (metric a, Full-knowledge recall)]
"The agent begins with a basic understanding k0 of the table or organizational knowledge, knowing only the table's name and columns. [...] Full-knowledge recall: We first checked if the agent was able to generate a table description that included a mention of every column originally disseminated in the company's network. An agent is considered to have achieved full-knowledge recall in an experiment run if it succeeded in at least reproducing all table columns in its final report."
The success criterion for the headline 94.9% result is only that the final description mentions every column of the table, and the agent is initialized with exactly the table's columns in k0. The metric therefore measures whether the LLM preserves the list it was given at time zero, not whether it recovered the disseminated knowledge (column meanings, data types, example values, variable interactions, tacit knowledge). Because the column list is an input to the agent, 'full-knowledge recall' is satisfied by construction in any run where the model does not drop a name. This also explains the near-zero correlation with reaching patient zero: a column list supplied at the start does not require contacting anyone.
-
other
[Section IV (knowledge generation and model choice) and Section V (G-Eval judge)]
"For each repetition, a table subject was randomly selected from a predefined list, which was then used to generate the simulated complete knowledge s* through an LLM. [...] All experiments were conducted on OpenAI's GPT-4o mini model. [...] We then used an LLM-as-judge approach to better capture the semantic equivalency between the original and generated descriptions."
The reference knowledge k*, the simulated employees' knowledge and backgrounds, the questioning agent, and the G-Eval 'faithfulness' judge are all generated or run by the same GPT-4o mini model family. The object to be recovered and the recovery system therefore share the same priors and the same sampler, so agreement between the final description and the reference does not demonstrate recovery of real external tacit knowledge. The only non-LLM metric, METEOR, averages only 0.17, which weakens the claim of high semantic correspondence. The network topology and SI dissemination structure are externally specified, so the circularity is confined to the semantic-content loop rather than the whole simulation framework.
full rationale
The paper is not a self-citation chain, and the epidemic/network simulation is an independently defined setup; METEOR provides a non-LLM anchor, so the circularity is not total. However, the central claim is undermined by two specific reductions from the paper's own definitions. First, the primary success metric, full-knowledge recall, is defined as mentioning the table's columns, and the agent is initialized with exactly those columns; the 94.9% figure therefore tracks list persistence rather than knowledge reconstruction, and the 'without patient zero' claim rests on a metric that is satisfied without contacting anyone. Second, the semantic ground truth, the simulated employees, the agent, and the main judge are all GPT-4o mini, so the faithfulness evaluation compares same-model outputs rather than recovered external knowledge. Additionally, each simulated employee is given a description of 'what partial knowledge they are aware their connections have,' so the agent's routing success is partly pre-wired. These are specific reductions of the paper's own measurement choices, not disagreements with external consensus. The conversational loop, parameter analyses, and METEOR comparisons retain independent content, so the paper is only partially circular rather than fully derivable from its inputs.
Assumptions & free parameters
free parameters (7)
- Alpha (knowledge sharing probability) =
0.1, 0.5
- Decay (waning infectivity rate, gamma) =
0.5, 0.8
- Number of informal connections (NIC) =
0, 2.5, 5
- Self-critic score threshold =
8
- Maximum hierarchical depth =
2, 5, 10, 20
- Number of employees =
20, 75, 200
- Number of table columns =
5, 20
assumptions (5)
- domain assumption SI model with waning infectivity (beta(t)=beta0 e^{-gamma t}) approximates real knowledge dissemination.
- domain assumption LLM-generated employee backgrounds produce realistic responses.
- domain assumption LLM-generated ground truth k* is a valid gold standard for table descriptions.
- domain assumption Balanced tree-like company hierarchy for all simulated organizations.
- domain assumption G-Eval and METEOR are valid measures of description quality.
Cite this review
Pith. "Pith review of Leveraging Large Language Models for Tacit Knowledge Discovery in Organizational Contexts." pith.science (2026). https://pith.science/paper/RDF5CIOG
@misc{pith2026250703811,
author = {Pith},
title = {Pith review of: Leveraging Large Language Models for Tacit Knowledge Discovery in Organizational Contexts},
year = {2026},
howpublished = {\url{https://pith.science/paper/RDF5CIOG}},
note = {Machine review of arXiv:2507.03811}
}
read the original abstract
Documenting tacit knowledge in organizations can be a challenging task due to incomplete initial information, difficulty in identifying knowledgeable individuals, the interplay of formal hierarchies and informal networks, and the need to ask the right questions. To address this, we propose an agent-based framework leveraging large language models (LLMs) to iteratively reconstruct dataset descriptions through interactions with employees. Modeling knowledge dissemination as a Susceptible-Infectious (SI) process with waning infectivity, we conduct 864 simulations across various synthetic company structures and different dissemination parameters. Our results show that the agent achieves 94.9% full-knowledge recall, with self-critical feedback scores strongly correlating with external literature critic scores. We analyze how each simulation parameter affects the knowledge retrieval process for the agent. In particular, we find that our approach is able to recover information without needing to access directly the only domain specialist. These findings highlight the agent's ability to navigate organizational complexity and capture fragmented knowledge that would otherwise remain inaccessible.
Figures
Reference graph
Works this paper leans on
-
[1]
Weber, The theory of social and economic organization
M. Weber, The theory of social and economic organization . Simon and Schuster, 1947
work page 1947
-
[2]
F. W. Taylor, Scientific management. Routledge, 2004
work page 2004
-
[3]
Can epidemic models describe the diffusion of topics across disciplines?
I. Z. Kiss, M. Broom, P. G. Craze, and I. Rafols, “Can epidemic models describe the diffusion of topics across disciplines?” Journal of Informetrics, vol. 4, no. 1, pp. 74–82, 2010
work page 2010
-
[4]
H.-M. Zhu, S.-T. Zhang, Y .-Y . Zhang, F. Wang et al., “Tacit knowledge spreading based on knowledge spreading model on networks with consideration of intention mechanism,” Journal of Digital Information Management, vol. 13, no. 4, pp. 293–300, 2015
work page 2015
-
[5]
M. Polanyi, “The tacit dimension,” in Knowledge in organisations . Routledge, 2009, pp. 135–146
work page 2009
-
[6]
The knowledge-creating company,
I. Nonaka and H. Takeuchi, “The knowledge-creating company,” Har- vard business review, vol. 85, no. 7/8, p. 162, 2007
work page 2007
-
[7]
M. U. Hadi, Q. Al Tashi, A. Shah, R. Qureshi, A. Muneer, M. Irfan, A. Zafar, M. B. Shaikh, N. Akhtar, J. Wuet al., “Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects,” Authorea Preprints, 2024
work page 2024
-
[8]
Detecting llm-generated text in computing education: Comparative study for chatgpt cases,
M. S. Orenstrakh, O. Karnalim, C. A. Suarez, and M. Liut, “Detecting llm-generated text in computing education: Comparative study for chatgpt cases,” in 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) . IEEE, 2024, pp. 121–126
work page 2024
Show all 32 references
-
[9]
Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts,
J. Zamfirescu-Pereira, R. Y . Wong, B. Hartmann, and Q. Yang, “Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , 2023, pp. 1–21
2023
-
[10]
Emergent analogical reasoning in large language models,
T. Webb, K. J. Holyoak, and H. Lu, “Emergent analogical reasoning in large language models,” Nature Human Behaviour , vol. 7, no. 9, pp. 1526–1541, 2023
2023
-
[11]
Selfcheck: Using llms to zero-shot check their own step-by-step reasoning,
N. Miao, Y . W. Teh, and T. Rainforth, “Selfcheck: Using llms to zero-shot check their own step-by-step reasoning,” arXiv preprint arXiv:2308.00436, 2023
2023 arXiv
-
[12]
Blau, Exchange and power in social life
P. Blau, Exchange and power in social life . Routledge, 2017
2017
-
[13]
Formal and informal hierarchy in different types of organization,
T. Diefenbach and J. A. Sillince, “Formal and informal hierarchy in different types of organization,” Organization studies, vol. 32, no. 11, pp. 1515–1537, 2011
2011
-
[14]
Assumptions of hierarchy and heterarchy, with applications to the management of the multinational corporation,
G. Hedlund, “Assumptions of hierarchy and heterarchy, with applications to the management of the multinational corporation,” in Organization theory and the multinational corporation. Springer, 1993, pp. 211–236
1993
-
[15]
Hierarchical structure and search in complex organizations,
J. Mihm, C. H. Loch, D. Wilkinson, and B. A. Huberman, “Hierarchical structure and search in complex organizations,” Management science , vol. 56, no. 5, pp. 831–848, 2010
2010
-
[16]
Generative agents: Interactive simulacra of human behavior,
J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th annual acm symposium on user interface software and technology , 2023, pp. 1–22
2023
-
[17]
Artificial leviathan: Exploring social evolution of llm agents through the lens of hobbesian social contract theory,
G. Dai, W. Zhang, J. Li, S. Yang, S. Rao, A. Caetano, M. Sra et al. , “Artificial leviathan: Exploring social evolution of llm agents through the lens of hobbesian social contract theory,” arXiv preprint arXiv:2406.14373, 2024
2024 arXiv
-
[18]
Chatdev: Communicative agents for software development,
C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Cong et al. , “Chatdev: Communicative agents for software development,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2024, pp...
2024
-
[19]
Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts,
T. Wu, M. Terry, and C. J. Cai, “Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts,” in Proceedings of the 2022 CHI conference on human factors in computing systems, 2022, pp. 1–22
2022
-
[20]
Least-to-most prompting enables complex reasoning in large language models,
D. Zhou, N. Sch ¨arli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schu- urmans, C. Cui, O. Bousquet, Q. Le et al. , “Least-to-most prompting enables complex reasoning in large language models,” arXiv preprint arXiv:2205.10625, 2022
2022 arXiv
-
[21]
Chain-of-thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, a...
2022
-
[22]
A contribution to the mathe- matical theory of epidemics,
W. O. Kermack and A. G. McKendrick, “A contribution to the mathe- matical theory of epidemics,” Proceedings of the royal society of london. Series A, Containing papers of a mathematical and physical character , vol. 115, no. 772, pp. 700–721, 1927
1927
-
[23]
Sir-based mathematical modeling of infectious diseases with vaccination and waning immunity,
M. Ehrhardt, J. Ga ˇsper, and S. Kilianov ´a, “Sir-based mathematical modeling of infectious diseases with vaccination and waning immunity,” Journal of Computational Science , vol. 37, p. 101027, 2019
2019
-
[24]
A susceptible– infectious (si) model with two infective stages and an endemic equilib- rium,
S. Ahmetolan, A. H. Bilge, A. Demirci, and A. P. Dobie, “A susceptible– infectious (si) model with two infective stages and an endemic equilib- rium,” Mathematics and Computers in Simulation , vol. 194, pp. 19–35, 2022
2022
-
[25]
Key economic sectors and services,
D. J. Arent, R. S. Tol, E. Faust, J. P. Hella, S. Kumar, K. M. Strzepek, F. L. T ´oth, D. Yan, A. Abdulla, H. Kheshgi et al. , “Key economic sectors and services,” in Climate change 2014 impacts, adaptation and vulnerability: Part a: Global and sectoral aspects . Cambridge Uni...
2014
-
[26]
Key sectors in some underdeveloped countries
P. S. Laumas, “Key sectors in some underdeveloped countries.” Kyklos, vol. 28, no. 1, 1975
1975
-
[27]
An essay on industrial classifi- cations,
B. Guibert, J. Laganier, and M. V olle, “An essay on industrial classifi- cations,” ´Economie et statistique , vol. 20, pp. 1–18, 1971
1971
-
[28]
T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to algorithms. MIT press, 2022
2022
-
[29]
Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments,
A. Lavie and A. Agarwal, “Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments,” in Proceedings of the Second Workshop on Statistical Machine Translation, ser. StatMT ’07. USA: Association for Computational Linguistics, 2007, p. 228–231
2007
-
[30]
G-eval: Nlg evaluation using gpt-4 with better human alignment,
Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: Nlg evaluation using gpt-4 with better human alignment,” in Conference on Empirical Methods in Natural Language Processing , 2023, p. 2511
2023
-
[31]
UMAP: Uniform manifold approximation and projection for dimension reduction,
L. McInnes, J. Healy, and J. Melville, “UMAP: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018
2018 arXiv
-
[32]
Code and data for Leveraging Large Language Models for Tacit Knowledge Discovery in Organizational Contexts,
G. Zuin, “Code and data for Leveraging Large Language Models for Tacit Knowledge Discovery in Organizational Contexts,” 2025. [Online]. Available: https://doi.org/10.6084/m9.figshare.28785524
2025 doi
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.