Pith. sign in

REVIEW 3 major objections 4 minor 16 references

A code-evidence pipeline yields 2,297 verified MCP projects on GitHub, mostly Python and TypeScript servers and clients.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 14:07 UTC pith:AODS7BLO

load-bearing objection Solid MSR dataset paper: first repo-first, evidence-tagged MCP corpus with roles; useful infrastructure, usual heuristic limits. the 3 major comments →

arxiv 2607.10123 v1 pith:AODS7BLO submitted 2026-07-11 cs.SE cs.AI

A Large-Scale Dataset of MCP Implementations on GitHub

Benny Toeppe , Amine Barrak , Emna Ksontini This is my paper
classification cs.SE cs.AI
keywords Model Context ProtocolMCPdatasetGitHubrepository miningopen source softwareGraphQLgateways
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The Model Context Protocol is becoming a shared way for language-model agents to discover and call external tools, but until now most maps of the ecosystem came from marketplace listings rather than from the code itself. This paper builds the first large repository-first dataset by searching GitHub for MCP-related projects, then confirming each one with multi-stage evidence: dependency fingerprints, directory and entry-point patterns, and deployment configuration. After filtering inactive and documentation-only repos and removing a further set of tutorial templates found by manual review, the authors release 2,297 validated operational projects classified as clients, servers, or gateways. A dual-author check on a representative sample gives 83 percent precision. The resulting public JSONL resource, complete with evidence tags and commit history, is intended as a reproducible benchmark so researchers can study how MCP is actually implemented, connected, and maintained rather than how it is advertised.

Core claim

A hybrid GitHub REST/GraphQL discovery pipeline followed by multi-stage code-level evidence checks and a targeted exclusion of non-operational templates produces a final dataset of 2,297 validated MCP implementations (from 3,238 candidates), role-classified as 1,962 servers, 1,462 clients, 80 gateways and 36 unclassified, with 83 percent precision on a dual-author manual sample and with Python and TypeScript as the dominant languages.

What carries the argument

Multi-stage evidence verification plus rule-based role scoring: repositories must show MCP-related dependencies, directory/entry-point patterns, or integration configs; each then receives numeric client/server/gateway scores from strong (source/manifest) and weak (metadata) signals, with fixed thresholds and a client override for editor plugins determining the primary role.

Load-bearing premise

The chosen dependency fingerprints, directory patterns, and scoring thresholds, together with one 170-repository dual-author sample, are enough to separate real operational MCP code from false positives and to assign primary roles correctly for the whole set.

What would settle it

An independent re-label of a fresh stratified sample of several hundred repositories (or of the published 170) that finds substantially lower precision than 83 percent, or that systematically reassigns roles under the same evidence rules, would overturn the claimed reliability of the final 2,297-project set.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers can ground claims about MCP security, maintainability, and evolution in repository-verified code rather than marketplace listings alone.
  • Language and role statistics (Python/TypeScript dominance, hybrid client-server codebases, gateway subtypes) become measurable baselines for ecosystem studies.
  • The public JSONL schema with evidence tags and commit/PR history supports longitudinal tracking of how MCP implementations change.
  • Future work on integration, connectivity, and compatibility can start from a shared, reproducible ground-truth set of operational projects.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the pipeline is fully documented and the evidence is stored, community re-runs can detect drift as MCP libraries and naming conventions evolve.
  • The high share of dual client-server repositories suggests that many early MCP projects are self-contained demos rather than pure protocol endpoints, which may shape how interoperability tests should be designed.
  • Gateway rarity (only 80) implies that mediation layers are still scarce; security and routing research may therefore need synthetic or constructed examples until more real gateways appear.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper constructs and releases a repository-centric dataset of Model Context Protocol (MCP) implementations mined from GitHub. Starting from 3,238 candidates discovered via five REST/GraphQL search queries (Jan 2024–Oct 2025), the authors apply activity filters, multi-stage code-level evidence checks (dependency manifests, directory/entry-point patterns, CI/Docker/Claude configs), and dual-author manual review of a 170-repository sample (83% precision, Wilson 95% CI [79%, 87%]). After removing 90 additional non-operational items they obtain 2,297 verified projects, each tagged with operational role (1,962 Servers, 1,462 Clients, 80 Gateways, 36 Unclassified), language, evidence signals, and commit/PR metadata, exported in a reproducible JSONL schema. Python and TypeScript dominate; hybrid client–server repositories are noted as common.

Significance. If the residual false-positive rate and role labels are reliable, this is a useful foundational artifact for the emerging MCP ecosystem: it is the first large-scale, code-evidence-driven (rather than registry-driven) corpus, supplies transparent multi-layer verification tags, dual-author sample validation, and public data/code. Those strengths directly enable follow-on work on security, maintainability, evolution, and interoperability that prior marketplace listings cannot support. The contribution is therefore of clear interest to the MSR community even if the absolute counts require modest qualification.

major comments (3)
  1. [§2.4] §2.4 reports 83% precision on a 170-repo dual-author sample drawn from the 2,387 automatically verified set, then removes 90 further non-operational repositories to reach 2,297. No second independent sample is drawn after the rule refinement, so the residual false-positive rate on the final corpus is not re-estimated. Because the central claim is that the released set constitutes a ‘validated’ foundational benchmark, a post-refinement precision (or at least a quantified upper bound on residual contamination) is load-bearing and currently missing.
  2. [§3.2] §3.2 defines role assignment via author-chosen numeric thresholds (Sc≥3 & Ss≤1 → Client; Ss≥3 & Sc≤1 → Server; Sg≥4 & others <2 → Gateway) plus a client-override rule for editor signals. These thresholds and the dual-role counting policy are never validated against the manual sample (which assessed only presence of MCP, not role correctness). Consequently the headline tallies (1,962 Servers / 1,462 Clients) and the abstract claim that ‘hybrid architectures [are] the most common design pattern’ rest on unmeasured classification error; a modest role-accuracy audit on the same or a fresh sample is required.
  3. [Abstract / §3.2] The abstract and §3 assert that hybrid architectures are the most common pattern, yet the paper never reports the cardinality of the dual-role intersection. Because repositories may be counted in both Server and Client tallies, the raw numbers alone do not establish that hybrids outnumber pure clients or pure servers. An explicit contingency table (pure-client / pure-server / hybrid / gateway / unclassified) is needed to substantiate the claim.
minor comments (4)
  1. [Table 1] Table 1 column headers are slightly ambiguous (‘Total repositories (response)’ vs ‘(Kept)’); a short footnote clarifying that ‘Kept’ already excludes duplicates/archived items would help.
  2. [Figure 3] Figure 3 schema diagram is dense; labeling the foreign-key edges or providing a one-sentence legend would improve readability.
  3. [§2.2–2.3] The nine-month inactivity cutoff (§2.2) and the exact list of dependency fingerprints are free parameters; listing them in an appendix or config file would aid exact reproduction.
  4. [§1] A few minor wording issues: ‘market centered efforts’ → ‘market-centered efforts’; ‘point in time metadata’ → ‘point-in-time metadata’.

Circularity Check

0 steps flagged

No circularity: empirical GitHub mining dataset with independent multi-stage evidence checks and held-out manual precision sample, not a self-referential derivation or fitted prediction.

full rationale

This is a repository-mining dataset paper, not a theoretical derivation that claims first-principles predictions. Candidate discovery uses public GitHub REST/GraphQL queries; verification applies explicit, inspectable code-level signals (manifest dependencies such as mcp/sdk or fastmcp, directory patterns like /src/mcp/, entry points, CI/Docker/Claude configs); role labels use a transparent rule-based scoring system with stated thresholds; and precision (83%, Wilson CI [79%,87%]) is estimated from dual-author manual review of a 170-repo sample followed by post-hoc removal of 90 non-operational cases. None of these steps reduce by construction to their inputs: the search queries do not force the evidence matches, the scoring thresholds are applied after discovery rather than fitted to produce the reported role counts, and the manual sample is independent of the automated pipeline. Self-citations (e.g., Barrak 2025) appear only as background and are not load-bearing for uniqueness or the dataset claims. No self-definitional loops, fitted-input-as-prediction, uniqueness theorems imported from the authors, ansatz smuggling, or renaming of known results occur. The work is self-contained against its own stated evidence criteria and external GitHub artifacts.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

As a repository-mining dataset paper, load-bearing content is methodological choices rather than physical free parameters or new particles. The central claim rests on search-query coverage, activity/documentation filters, evidence fingerprints, numeric role thresholds, and the representativeness of a 170-repo manual sample. No invented physical entities; invented constructs are the operational role taxonomy and evidence schema.

free parameters (4)
  • Role score thresholds (Sc≥3/Ss≤1 Client; Ss≥3/Sc≤1 Server; Sg≥4 Gateway)
    Numeric cutoffs and strong/weak point weights (2 vs 1) are chosen by the authors to decide primary role; different thresholds would change counts.
  • Nine-month inactivity cutoff
    Repositories with no commits in the last nine months are dropped; the window is a design choice affecting the active set size.
  • Manual sample size / margin (±7–8% at 95% CI, n=170)
    Sample size and assumed error band determine the precision claim extrapolated to the full set.
  • Time window January 2024–October 2025
    Restricts discovery to the MCP emergence period; changes the candidate pool.
axioms (4)
  • domain assumption GitHub REST Search + GraphQL metadata plus the five listed keyword queries sufficiently cover public MCP implementations of interest.
    Section 2.1; unindexed, private, or differently named projects are out of scope by construction.
  • ad hoc to paper Presence of listed dependency fingerprints, /src/mcp/ style paths, mcp-server entry points, or Claude Desktop/Docker/CI config is adequate evidence of an operational MCP implementation.
    Sections 2.3.1–2.3.3 define the multi-stage evidence layers used as ground truth for verification.
  • domain assumption Dual independent author review of 170 repositories with consensus resolution yields an unbiased precision estimate for the automated pipeline.
    Section 2.4; standard sampling assumption for dataset validation.
  • ad hoc to paper MCP operational roles reduce to Client, Server, Gateway (plus Unclassified) with the stated signal lists and client-override rule.
    Section 3.1–3.2 introduce the role model and scoring decision logic.
invented entities (2)
  • TRUE_MCP_REPOS evidence-tagged role schema (Client/Server/Gateway + subtypes) independent evidence
    purpose: Organize verified repositories, evidence strength, and collaboration metadata for reuse.
    Figure 3 and Section 3.4 define the export schema; it is a data model, not an external physical entity.
  • Six gateway subtypes (orchestration, language bridge, transport relay, auth policy, sandbox, discovery/registry) no independent evidence
    purpose: Finer classification of the 80 gateway repositories.
    Section 3.2; taxonomy invented for this dataset analysis.

pith-pipeline@v1.1.0-grok45 · 11624 in / 3340 out tokens · 30790 ms · 2026-07-14T14:07:24.805801+00:00 · methodology

0 comments
read the original abstract

The rapid emergence of the Model Context Protocol (MCP) has introduced a new standard for connecting large language models to external tools and services. Despite its rapid adoption in open-source development, systematic understanding of how MCP is implemented, structured, and maintained remains limited. This study presents the first large-scale, evidence-based dataset of real-world MCP implementation collected directly from GitHub. Using a hybrid pipeline that integrates the GitHub REST and GraphQL APIs with custom Python verification scripts, 3,238 candidate repositories were discovered, filtered, and validated through multi-stage evidence checks. Each verified project was classified by operational role (e.g., client, server, gateway) and exported in a reproducible JSONL schema. A manual review of a representative subset confirmed an overall precision of 83% at a 95% confidence level, and additionally revealed a set of repositories functioning primarily as educational samples, tutorials, or demonstration templates. A targeted exclusion rule was then applied to remove these non-operational repositories, resulting in a final dataset of 2,297 validated MCP projects. The analysis shows that Python and TypeScript dominate MCP development, with hybrid architectures emerging as the most common design pattern. By emphasizing transparent verification strategies, structured evidence tagging, and reproducible data organization, this work establishes a foundational benchmark for studying real-world MCP ecosystems and supports future research on integration, connectivity, and compatibility across the broader developer community.

Figures

Figures reproduced from arXiv: 2607.10123 by Amine Barrak, Benny Toeppe, Emna Ksontini.

Figure 1
Figure 1. Figure 1: Flow of dataset construction from initial GitHub [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MCP architecture showing the host, clients, and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Dataset Schema and Entity Relationships for the MCP Repository Dataset [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 1 canonical work pages

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    Toolformer: Language models can teach themselves to use tools , author=. Advances in Neural Information Processing Systems , volume=

  2. [2]

    2012 9th IEEE Working Conference on Mining Software Repositories (MSR) , pages=

    GHTorrent: GitHub's data from a firehose , author=. 2012 9th IEEE Working Conference on Mining Software Repositories (MSR) , pages=. 2012 , organization=

    2012

  3. [3]

    2025 , publisher=

    Building AI Agents with LLMs, RAG, and Knowledge Graphs: A practical guide to autonomous and modern AI agents , author=. 2025 , publisher=

    2025

  4. [4]

    Proceedings of the

    Amine Barrak , title =. Proceedings of the. 2025 , publisher =

    2025

  5. [5]

    2025 , publisher =

    Verified MCP Implementations Dataset , author =. 2025 , publisher =. doi:10.5281/zenodo.17573071 , url =

    work page doi:10.5281/zenodo.17573071 2025

  6. [6]

    Agentic AI: A Conceptual taxonomy, applications and challenges , journal =

    AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges , journal =. 2026 , issn =. doi:https://doi.org/10.1016/j.inffus.2025.103599 , url =

    doi:10.1016/j.inffus.2025.103599 2026

  7. [7]

    arXiv preprint arXiv:2508.02979 , year=

    Unified Tool Integration for LLMs: A Protocol-Agnostic Approach to Function Calling , author=. arXiv preprint arXiv:2508.02979 , year=

    Pith/arXiv arXiv

  8. [8]

    Proceedings of the Seventh ACM Conference on Learning@ Scale , pages=

    Human languages in source code: Auto-translation for localized instruction , author=. Proceedings of the Seventh ACM Conference on Learning@ Scale , pages=

  9. [9]

    arXiv preprint arXiv:2507.10593 , year=

    ToolRegistry: A Protocol-Agnostic Tool Management Library for Function-Calling LLMs , author=. arXiv preprint arXiv:2507.10593 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2504.19678 , year=

    From llm reasoning to autonomous ai agents: A comprehensive review , author=. arXiv preprint arXiv:2504.19678 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2508.13220 , year=

    Mcpsecbench: A systematic security benchmark and playground for testing model context protocols , author=. arXiv preprint arXiv:2508.13220 , year=

    arXiv

  12. [12]

    arXiv preprint arXiv:2306.05301 , year=

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases , author=. arXiv preprint arXiv:2306.05301 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2507.16044 , year=

    Making rest apis agent-ready: From openapi to mcp servers for tool-augmented llms , author=. arXiv preprint arXiv:2507.16044 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2509.22814 , year=

    Model Context Protocol for Vision Systems: Audit, Security, and Protocol Extensions , author=. arXiv preprint arXiv:2509.22814 , year=

    arXiv

  15. [15]

    arXiv preprint arXiv:2506.13538 , year=

    Model context protocol (mcp) at first glance: Studying the security and maintainability of mcp servers , author=. arXiv preprint arXiv:2506.13538 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2506.23474 , year=

    A Large-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysis , author=. arXiv preprint arXiv:2506.23474 , year=

    Pith/arXiv arXiv