Pith. sign in

REVIEW 2 major objections 4 minor 23 references

LLM groups gain from shorter communication paths only when their first guesses are randomized.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:40 UTC pith:CJT2YC3W

load-bearing objection A genuinely new, transparent study of network topology effects in LLM collectives, with a real statistical caveat about clustering that should be fixed before publication. the 2 major comments →

arxiv 2607.14574 v1 pith:CJT2YC3W submitted 2026-07-16 cs.AI cs.SI

Collaborative Spatial Learning with Multi-LLM Agents in Networked Social Experiments

Hao He , Chris J. Kuhlman , Xinwei Deng This is my paper
classification cs.AI cs.SI
keywords collective searchmulti-agent LLMnetwork structureexploration-exploitationBayesian optimizationsocial learningspatial searchaverage path length
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the network-efficiency effect seen in human collective search—groups connected by shorter paths earn higher collective payoffs—also appears when the group members are large language model (LLM) agents. Sixteen LLM agents played the Wildcat Wells spatial search game on eight different network topologies for fifteen rounds. The results show the effect transfers only partially: LLM agents instructed to randomize their first-round guesses earned significantly more on shorter-path networks, while default LLM agents, which almost always start at the grid center, showed no topology effect. A single first-round randomization sentence improved cumulative payoff by 9.44 points, more than three times the entire payoff difference across topologies. The paper also shows that mechanistic Bayesian optimization agents outperform the LLM agents and that LLM copying behavior is largely positional rather than payoff-conditional.

Core claim

Using 2,400 games across five agent types, eight network graphs, three landscape complexities, and twenty replicates, the paper finds that the relationship between average path length and cumulative group payoff is significantly negative for LLM agents with randomized initialization (β = −1.37, p = 0.003) and for both Gaussian-process Bayesian agents (EI and UCB), but not for the default LLM (β = −0.41, p = 0.619). The default LLM places almost all first-round guesses at the grid center (7,677 of 7,680), and the randomization instruction that fixes this yields a gain of 9.44 points—more than three times the roughly 3-point range of the topology effect. LLM agents copy neighbors at high rates

What carries the argument

The Wildcat Wells collective search game: a group of sixteen agents on a fixed 3-regular graph samples a two-dimensional payoff landscape over fifteen rounds, with each agent observing its own and its neighbors' full histories. The network-efficiency effect is measured by regressing cumulative game-level mean payoff on average path length, with complexity fixed effects. The decisive intervention is a one-sentence prompt appended only to round 1, instructing the agent to choose its first guess uniformly at random; this changes initialization without altering any later-round prompt. Behavioral measures—copying rate and spatial diversity—explain why topology effects appear only in early rounds.

Load-bearing premise

The central empirical claim rests on a single LLM configuration—one open-weight model, one prompt template, one reasoning-effort setting, and one landscape bank—so the centered initialization and positional copying, and therefore the partial transfer of the network effect, may be specific to that configuration rather than general to LLM agents.

What would settle it

Run the same Wildcat Wells protocol with a different open-weight LLM family using the default prompt; if that model does not begin at the grid center and still shows a significant negative slope of payoff on average path length, the paper's claim that randomization is necessary for the network effect would fail. Alternatively, a broader set of landscapes or prompt templates that eliminates LLM-RI's significant slope would refute the transfer claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the finding holds, network topology should be treated as a second-order design factor in multi-LLM systems; first-round initialization can dominate structural choices.
  • LLM collectives can reproduce a human social-learning regularity, but only when a diversity-preserving initialization is imposed.
  • The weak copying–payoff correlation implies that LLM agents imitate positions rather than rewarding outcomes, so social information may propagate less effectively than in human groups.
  • Bayesian optimization agents provide a strong mechanistic reference; LLM search strategies still trail task-specific optimization on this spatial task.
  • Prompt-level interventions of a single sentence can shift collective outcomes more than network architecture, suggesting prompt design deserves priority in system evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the center-start reflects a general LLM prior toward grid midpoints, other prompt formulations or model families might show topology effects without an explicit randomization instruction; this is testable by varying the initial-position context.
  • The rapid spatial homogenization of LLM groups suggests that asynchronous updating or communication-restricted designs could preserve diversity and amplify network effects; the paper itself flags asynchronous protocols as future work.
  • The positional-copying pattern implies that LLM groups might benefit from explicit payoff-conditional social learning rules, for example by weighting neighbor suggestions by their observed payoffs rather than their coordinates.
  • Because topology effects vanish on final-round payoff, the effective horizon of network structure is the early search phase; extending the game beyond 15 rounds would likely reduce cumulative differences further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper asks whether the Mason–Watts network-efficiency result—human groups perform better on shorter-path networks in a spatial search task—transfers to groups of LLM agents. The authors run 2,400 games (5 agent types × 8 graphs × 3 complexities × 20 replicates; 576,000 node-rounds) on 60 generated landscapes. Agents are gpt-oss-120b with default and randomized-initialization prompts, two Gaussian-process Bayesian optimization agents (EI, UCB), and a random baseline, with the original human data included as a reference. They report that cumulative payoff declines with APL for LLM-RI (β=-1.37, p=0.003) and both BO agents, but not for the default LLM; that a one-sentence first-round randomization instruction increases payoff by 9.44 points, several times the full topology effect; that BO agents outperform both LLM variants; and that LLM copying is frequent but weakly correlated with payoff.

Significance. If the statistical inference holds, the paper provides the first systematic test of communication-network effects on LLM collectives and a useful mechanistic comparison against GP-BO baselines. Its strengths are the balanced experimental design, fixed landscapes across conditions, open data/code, transparent dual-level regression reporting, and explicit limitation statements. The main reservation is that the graph-level APL inference effectively has only eight independent units, so precision may be overstated. The single-model, single-prompt configuration also limits external generalization, as the authors acknowledge. If the robustness check I request below is satisfied, the result would be a meaningful benchmark for multi-agent system design.

major comments (2)
  1. [Section 4.2, Table 2] The central network-efficiency claim rests on the significance of the APL slopes, but the reported inference treats observations as independent at a finer level than APL varies. APL is constant within each of the eight graphs, so the three complexity rows per graph are not independent replicates of a graph-level effect. The graph-level regression with n=24 and the game-level regression with n=480 both reuse the same eight APL values; reported p-values (LLM-RI p=0.003; UCB/EI p<0.001) may therefore be anti-conservative. Please add cluster-robust standard errors by graph, a permutation test that permutes APL across graphs, or an analysis at the eight graph means, and report whether the conclusions survive. This is load-bearing because the headline 'significant network-efficiency effect' for LLM-RI depends on these p-values. Even if the signs remain, the corrected p-values should be reporte
  2. [Section 4.4] The conclusion that LLM copying is 'positional rather than payoff-conditional' is inferred from the low correlation between individual copying rate and individual payoff. This is an aggregate association, not a direct test of whether copying decisions respond to neighbor payoffs. A multilevel model of the decision to copy as a function of neighbor payoff (versus spatial proximity) would make the mechanism claim more direct. As written, the claim is suggestive rather than demonstrated.
minor comments (4)
  1. [Section 4.3] Please report the decoding parameters used for the LLM (temperature, top-p, seed) and the number of repeated samples per prompt, if any. The near-deterministic center-start (7,677 of 7,680 first-round choices at (50,50)) could be an artifact of greedy decoding or of the specific model, rather than a robust behavioral prior.
  2. [Figure 7] The right panel legend labels all five agent types but the points are colored by graph; the legend and point labeling are confusing. Please make the panel for each agent type explicit, or use a common legend keyed to graph labels.
  3. [Abstract / Section 5] The abstract states the effect without qualification; given the acknowledged single-model, single-template limitation, consider phrasing as 'for the evaluated configuration' in both the abstract and the conclusions.
  4. [Section 3.1] The landscape-generation step says the payoff distribution is 'calibrated to match the tail targets in Table 1' but does not report the achieved calibration. A short supplementary table or text check would increase confidence that the complexity manipulation worked as intended.

Circularity Check

0 steps flagged

No circularity: the paper reports empirical measurements of LLM and Bayesian agents on fixed landscapes and networks; no fitted quantity is relabeled as a prediction.

full rationale

The paper's central results are empirical: cumulative payoffs of LLM, LLM-RI, EI, and UCB agents are measured on fixed landscapes and fixed Mason–Watts networks, and then regressed on average path length. LLM-RI is an explicit prompt intervention (a single added sentence instructing uniform random first-round choice), not a parameter fitted to the outcome. The Bayesian agents are independent mechanistic baselines using standard EI and UCB acquisition functions; their GP hyperparameters are tuned online to each agent's own and neighbors' observations, not to the payoff-topology effect being claimed. The claim that the initialization effect is more than three times the topology effect is arithmetic on measured means, not a fit renamed as a result. The paper invokes no self-citations as load-bearing evidence, and the Mason–Watts reference is an external human benchmark, not an assertion of the authors' own conclusion. The skeptical concern about graph-level versus game-level inference is a statistical validity issue about pseudo-replication, not a circular-reasoning issue: it concerns whether the reported p-values are anti-conservative, not whether the derivation is equivalent to its inputs. The stated limitations (single model, single prompt template, unmatched human landscapes) are acknowledged scope restrictions and do not turn the empirical derivation into a circular one.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central claim rests on the LLM model/prompt configuration being representative enough for a claim about 'LLM agents', on the eight MW graphs serving as network-efficiency proxy, and on the constructed landscapes being a valid task substrate. No invented entities are introduced. The only hand-chosen numeric parameters are the copying threshold and landscape calibration targets; GP hyperparameters are tuned online for baselines only.

free parameters (3)
  • copying threshold ε = 3 cells (3% of axis length)
    Hand-chosen in Section 4.4 to define copying; directly determines copying-rate and copying-payoff correlation claims, and no robustness check is reported.
  • landscape tail calibration targets = Low: ~20% ≥80, ~30% ≤20; Moderate: ~10%/40%; High: ~5%/50%
    Design parameters in Table 1 used to create difficulty gradient; not fitted to the outcome but define the task substrate.
  • GP hyperparameters (ℓ_short, ℓ_long, σ²_ε) and UCB κ_t = Tuned online each round
    Standard Bayesian-optimization practice; affects only baseline agent behavior, not the LLM topology claim.
axioms (3)
  • domain assumption A single fixed open-weight LLM (gpt-oss-120b, medium reasoning) with one prompt template is sufficient to test the existence of network-efficiency effects in LLM collectives.
    The paper states this as a limitation (Section 5), but the central empirical claim is drawn entirely from this one configuration.
  • domain assumption The eight Mason-Watts 3-regular graphs and their average-path-length ordering capture network efficiency in the same way as the original human experiment.
    Section 3.1 claims APL ranges 2.20-3.87 and that alternative measures rank graphs similarly; human comparability depends on this.
  • domain assumption The constructed landscapes, with unique global maximum and calibrated tail distributions, are an adequate substrate for the search task.
    Section 3.1 describes the generative procedure; the intended difficulty gradient is confirmed by performance declines across complexity levels.

pith-pipeline@v1.3.0-alltime-deepseek · 11232 in / 9563 out tokens · 104056 ms · 2026-08-02T01:40:27.757018+00:00 · methodology

0 comments
read the original abstract

Collective problem solving often requires that group members consider the tradeoff between exploitation of known solutions and exploration for new ones, where information of known solutions can be disseminated among individual members through communication networks. The Mason--Watts experiment (PNAS 2012) showed that human groups in shorter-path networks outperform those in longer-path networks on a two-dimensional search task. In this work, we focus on the investigation of such network-efficiency effects in the setting of a group of large language model (LLM) agents. Specifically, we consider groups of sixteen LLM agents playing the Mason--Watts experiment on the eight Mason--Watts network topologies. Moreover, we develop mechanistic Bayesian optimization agents such that the performance of LLM agents can be compared with both the mechanistic agents and the human experimental data. Our computational experiments indicate that the LLM agents show a significant network-efficiency effect when instructed to randomize their first-round choices, but not under the default initialization. In this experiment, adding a one-sentence first-round randomization instruction improves collective payoff by more than three times the estimated payoff difference across the eight network topologies. Also, the Bayesian optimization agents obtain higher payoffs than the evaluated LLM agents on this spatial search task. We further compare the agents' exploration--exploitation behavior, copying, and spatial diversity.

Figures

Figures reproduced from arXiv: 2607.14574 by Chris J. Kuhlman, Hao He, Xinwei Deng.

Figure 1
Figure 1. Figure 1: An illustration of the Mason–Watts experiment [ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The eight 3-regular graphs on 16 nodes from the Mason–Watts design [11], ordered by increasing average path length (APL), which corresponds to decreasing graph efficiency from A to H. The color scale (red → blue) matches the graph-level colors used in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prompt template used for the LLM and LLM-RI agents. Text in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Group-level learning curves by agent type and landscape complexity. Each curve [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cumulative fraction of games in which the group’s best observed payoff reaches at [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Baseline-corrected, normalized deviation from the round-level grand mean, by [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Cumulative game-level mean payoff versus average path length. Left: human [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Copying rate (left) and spatial diversity (right) over rounds. Human data (dashed) [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 4 linked inside Pith

  1. [1]

    Proceedings of the National Academy of Sciences118(36), e2101062118 (2021)

    Almaatouq, A., Alsobay, M., Yin, M., Watts, D.J.: Task complexity moderates group synergy. Proceedings of the National Academy of Sciences118(36), e2101062118 (2021)

    2021

  2. [2]

    Nature Communications7(1), 13109 (2016)

    Barkoczi, D., Galesic, M.: Social learning strategies modify the effect of network structure on group performance. Nature Communications7(1), 13109 (2016)

    2016

  3. [3]

    Proceedings of the National Academy of Sciences115(35), 8734–8739 (2018)

    Bernstein, E., Shore, J., Lazer, D.: How intermittent breaks in interaction improve collective intelligence. Proceedings of the National Academy of Sciences115(35), 8734–8739 (2018)

    2018

  4. [4]

    In: Proceedings of the 41st International Conference on Machine Learning (2024)

    Du, Y ., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I.: Improving factuality and reasoning in language models through multiagent debate. In: Proceedings of the 41st International Conference on Machine Learning (2024)

    2024

  5. [5]

    Harris, K., Slivkins, A.: Should you use your large language model to explore or exploit? arXiv preprint arXiv:2502.00225 (2025)

    Pith/arXiv arXiv 2025

  6. [6]

    Journal of Global Optimization13(4), 455–492 (1998)

    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. Journal of Global Optimization13(4), 455–492 (1998)

    1998

  7. [7]

    Proceedings of the National Academy of Sciences121(45), e2405460121 (2024)

    Kosinski, M.: Evaluating large language models in theory of mind tasks. Proceedings of the National Academy of Sciences121(45), e2405460121 (2024)

    2024

  8. [8]

    Krishnamurthy, A., Harris, K., Foster, D.J., Zhang, C., Slivkins, A.: Can large language models explore in-context? In: Advances in Neural Information Processing Systems. vol. 37 (2024)

    2024

  9. [9]

    Physical Review Letters 87(19), 198701 (2001)

    Latora, V ., Marchiori, M.: Efficient behavior of small-world networks. Physical Review Letters 87(19), 198701 (2001)

    2001

  10. [10]

    Administrative Science Quarterly52(4), 667–694 (2007)

    Lazer, D., Friedman, A.: The network structure of exploration and exploitation. Administrative Science Quarterly52(4), 667–694 (2007)

    2007

  11. [11]

    Proceedings of the National Academy of Sciences109(3), 764–769 (2012)

    Mason, W., Watts, D.J.: Collaborative learning in networks. Proceedings of the National Academy of Sciences109(3), 764–769 (2012)

    2012

  12. [12]

    arXiv preprint arXiv:2508.10925 (2025)

    OpenAI: gpt-oss-120b and gpt-oss-20b model card. arXiv preprint arXiv:2508.10925 (2025)

    Pith/arXiv arXiv 2025

  13. [13]

    PNAS Nexus4(12), pgaf317 (2025)

    Papachristou, M., Yuan, Y .: Network formation and dynamics among multi-LLMs. PNAS Nexus4(12), pgaf317 (2025)

    2025

  14. [14]

    In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

    Park, J.S., O’Brien, J., Cai, C.J., Morris, M.R., Liang, P., Bernstein, M.S.: Generative agents: Interactive simulacra of human behavior. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. pp. 1–22. ACM (2023)

    2023

  15. [15]

    In: ACM SIGGRAPH Computer Graphics

    Perlin, K.: An image synthesizer. In: ACM SIGGRAPH Computer Graphics. vol. 19, pp. 287–296. ACM (1985)

    1985

  16. [16]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (2024)

    Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y ., Li, J., Yang, C., Chen, W., Su, Y ., Cong, X., Xu, J., Li, D., Liu, Z., Sun, M.: ChatDev: Communicative agents for software development. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (2024)

    2024

  17. [17]

    MIT Press, Cambridge, MA (2006)

    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA (2006)

    2006

  18. [18]

    Science Advances10(45), eadp1528 (2024)

    Schoenegger, P., Tuminauskaite, I., Park, P.S., Valdece Sousa Bastos, R., Tetlock, P.E.: Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Science Advances10(45), eadp1528 (2024)

    2024

  19. [19]

    In: Proceedings of the 27th International Conference on Machine Learning

    Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.: Gaussian process optimization in the bandit setting: No regret and experimental design. In: Proceedings of the 27th International Conference on Machine Learning. pp. 1015–1022 (2010)

    2010

  20. [20]

    Nature Human Behaviour8, 1285–1295 (2024)

    Strachan, J.W.A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., Graziano, M.S.A., Becchio, C.: Testing theory of mind in large language models and humans. Nature Human Behaviour8, 1285–1295 (2024)

    2024

  21. [21]

    arXiv preprint arXiv:2302.08399 (2023)

    Ullman, T.: Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399 (2023)

    Pith/arXiv arXiv 2023

  22. [22]

    Nature393(6684), 440–442 (1998)

    Watts, D.J., Strogatz, S.H.: Collective dynamics of ‘small-world’ networks. Nature393(6684), 440–442 (1998)

    1998

  23. [23]

    arXiv preprint arXiv:2505.09901 (2025)

    Zhang, Z., Wang, D., Chen, N., Mansur, R., Sarhangian, V .: Comparing exploration-exploitation strategies of LLMs and humans: Insights from standard multi-armed bandit experiments. arXiv preprint arXiv:2505.09901 (2025)

    Pith/arXiv arXiv 2025