Pith Number

pith:KPTUWT5P

pith:2026:KPTUWT5PVEXSJPU4HX7TPJUE7A

not attested not anchored not stored refs pending

Jagged AI in Scientific Peer Review: Evidence from POMP Data Analysis

Edward L. Ionides, Jin Wook Lee, William Szegda, Zhisheng Song

AI reviewers catch technical errors in POMP analyses that humans miss but fall short on interpretive and narrative checks.

arxiv:2605.07855 v2 · 2026-05-08 · stat.AP

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{KPTUWT5PVEXSJPU4HX7TPJUE7A}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

AI reviewers exhibited a jagged capability profile, proficiently catching human-overlooked technical errors and invalid inference methodology, while failing to match human standards in checking interpretive errors, narrative coherence, and domain-informed model critique. The jaggedness was found to be similar for all agents, consistent with it being primarily a property of the underlying AI model rather than the specific instructions.

C2weakest assumption

That the 72 anonymized student POMP projects and their human peer reviews form a representative and unbiased testbed for general AI performance in scientific peer review of mechanistic dynamic models.

C3one line summary

AI reviewers of POMP data analyses detect technical and methodological errors effectively but underperform humans on interpretive, narrative, and domain-informed critique, showing consistent jaggedness.

Receipt and verification

First computed	2026-05-20T00:05:46.622215Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

53e74b4fafa92f24be9c3dff37a684f81482fb73c1de33a3815e0ad503399afe

Aliases

arxiv: 2605.07855 · arxiv_version: 2605.07855v2 · doi: 10.48550/arxiv.2605.07855 · pith_short_12: KPTUWT5PVEXS · pith_short_16: KPTUWT5PVEXSJPU4 · pith_short_8: KPTUWT5P

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/KPTUWT5PVEXSJPU4HX7TPJUE7A \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 53e74b4fafa92f24be9c3dff37a684f81482fb73c1de33a3815e0ad503399afe

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "7274d62c7fd667135b959d4e7bfe91cfddc244e803c740f6bd4979525232139a",
    "cross_cats_sorted": [],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "stat.AP",
    "submitted_at": "2026-05-08T15:17:29Z",
    "title_canon_sha256": "e3292a929adb128f339ad421fbb49ee86a7651eca0c1f608784c14a871aa9dc3"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.07855",
    "kind": "arxiv",
    "version": 2
  }
}