Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

BrainPilot: Automating Brain Discovery with Agentic Research

T0 review · 5 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An open-source multi-agent system, BrainPilot, claims to run brain-science research workflows with traceable, agent-verified steps and to match far more expensive commercial agent harnesses on benchmark tasks.

desk verdict Real, open engineering contribution with an unusually honest evaluation; the cost/performance headline is plausible but not established until the single runs and local rescoring get an independent pass. read the letter →

arxiv 2607.15079 v2 pith:6TL74EQF submitted 2026-07-16 cs.AI

Haoxuan Li , Tianci Gao , Jianhe Li , Yang Fan , Runze Shi , Weiran Wang , Tianxiang Zhao , Zezhao Wu
show 8 more authors
Xiaoyang Jiang Qihui Zhang Jia Li Xiao Xiao Kai Du Xiaoxuan Jia Chao Xie Lu Mi
This is my paper · ORCID
classification cs.AI
keywords multi-agentAIsystemsbrain-scienceresearchautomationdomainknowledgegroundingskilllibraryGraphofTracefabricationcheckingscientific-agentbenchmarksopen-sourceagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BrainPilot is a multi-agent system aimed at automating brain-science research, from literature survey and experimental design through code execution and report writing, while keeping a human expert in the loop. The paper's central claim is that a coordinator ('principal investigator') agent, five specialist agents, a curated 7,233-item knowledge base, and a 72-skill methodology library can carry out benchmark brain-science tasks about as well as frontier commercial agent harnesses — and at 5–56 percent of their cost when driven by an open-source model. A second, structural claim is that research done this way can be made trustworthy: the append-only Graph of Trace records every subgoal, tool call, and evidence link, and an Auditor agent checks every numerical, file, and citation claim against the session's own artifacts before delivery. A fair reader would care because the system's aim — cheap, transparent, expert-verifiable AI for a discipline where conclusions feed downstream science — responds directly to the failures (fabrication, drift, few intervention points) that block agent use in science.

What carries the argument

The load-bearing mechanism is the principal-investigator (PI) agent architecture: one agent decomposes the request, delegates subtasks to five specialists — Librarian, Experimentalist, Engineer, Writer, Auditor — and synthesizes the result. Agents are grounded in two offline-built assets: a vector knowledge base of 7,233 indexed textbooks and papers, and a skill library of 72 reusable methodology units fetched on demand, so the team is not limited to what the language model remembers. Every completed subtask is appended to the Graph of Trace, a monotonic graph linking subgoals, tools, evidence, and claims; the Auditor then classifies each claim as confirmed, unverified, or disputed against t

What would settle it

Re-run the three Agents' Last Exam brain-science tasks inside the benchmark's official hosted environment, five or more repetitions per configuration, using the authors' adapter and one without it; if BrainPilot's score parity on the deterministic tasks or its 5–56 percent cost advantage does not reproduce, the headline claim fails. A second test: re-run BrainPilotBench multiple times per cell to see whether the reported single-run scores — including the higher-of-two RSC score — are stable, and whether the knowledge-base ablated condition (8 improved, 14 reduced, 4 unchanged across 26 cells)

Watch

Extended reading notes

Core claim

Brain-science workflows — literature survey, experiment design, code execution, interpretation, writing — can be handed to specialist AI agents without losing expert oversight. The design claim: a principal-investigator agent coordinating five specialists, grounded in a 7,233-item knowledge base and a 72-skill library, with every step logged in an append-only Graph of Trace and drafts checked by an Auditor, matches frontier agent harnesses at far lower cost. On three Agents' Last Exam tasks, an open-source backbone matched the two deterministic scores at 5–56 percent of reference-harness cost; on the authors' four-task benchmark, 53 of 56 runs scored. It also reports domain knowledge improve

Load-bearing premise

The comparative claim rests on the benchmark numbers being representative: the three Agents' Last Exam tasks were re-scored outside that benchmark's official hosted environment using the authors' own adapter, every headline configuration was a single unrepeated run, and BrainPilot's best score on its own benchmark was the higher of two runs.

Editorial extensions

If this is right

  • If the cost figures hold, brain-science analyses that currently require expensive frontier-model agent runs could be executed with an open-source backbone at as little as a few cents per task, widening access to labs without large API budgets.
  • Traceability becomes an explicit property of the pipeline: the append-only Graph of Trace links every claim to its subgoals, tools, and evidence, so a researcher can inspect and correct the workflow before an error propagates to a conclusion.
  • Fabrication checking is built in rather than left to the user: the Auditor's confirmed/unverified/disputed classification gives a defined checkpoint at which unsupported numerical, file, or citation claims are caught before delivery.
  • The design is extensible by construction: meta-skills that convert papers, software, and completed workflows into new skill units mean the library can grow with community contributions, and the released benchmark harness provides an artifact-graded, isolation-checked way to compare future brain-science agents.
  • The reported ablation is itself a caution: domain knowledge improved scores in only 8 of 26 comparable cells, so retrieval-grounded agents must justify their grounding component task by task rather than assume it always helps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The architecture is portable: because the knowledge base and skill library are built by pipeline (OCR-to-text, chunking, embedding, plus paper-to-skill and repo-to-skill meta-skills), the same PI-specialist design could be rebuilt for other knowledge-intensive fields, and the cost-advantage claim could be directly re-tested there.
  • Read as a whole, the paper's own evidence suggests that task structure, not agent architecture, currently dominates outcomes: every harness — including the most expensive commercial one — struggled on the open-ended neuron-tracking task, and domain knowledge helped only 8 of 26 comparable scores. A fair reader may infer that near-term gains will come from constraining open-ended tasks rather than
  • The in-paper case studies suggest the practical payoff is error discovery, not headline scores: the visual-hierarchy analysis caught an implausible latency estimate, added a responsiveness criterion, and reran; the fMRI report flagged minor inconsistencies before delivery. If that pattern generalizes, the system's real value is making its own mistakes visible and revisable — which is precisely wha
  • A cheap replication experiment follows immediately: repeat each single-run configuration to measure variance. If the run-to-run spread is large relative to the reported score differences, the comparative claims need re-baselining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper presents BrainPilot, an open-source multi-agent system for brain-science research. A principal-investigator (PI) agent decomposes tasks and coordinates specialist agents (Librarian, Experimentalist, Engineer, Writer, Auditor). The agents are grounded in a 7,233-item knowledge base and a 72-skill library, and all major steps are recorded in an append-only Graph of Trace. The authors evaluate the system on three brain-science tasks from Agents' Last Exam (ALE), on a new four-task benchmark called BrainPilotBench-v0, and in five end-to-end case studies. The headline claim is that BrainPilot with an open-source backbone achieves performance comparable to a state-of-the-art agent framework at 5-56% of the cost and 8-63% of the time, with explicit disclosure that ALE results are single unrepeated runs and are grader-faithful re-scorings generated outside ALE's official hosted environment.

Significance. If the headline comparison held, the paper would be a useful contribution: a fully open-source, traceable multi-agent system for neuroscience; a new benchmark with artifact-based scoring; and unusually concrete cost accounting based on official list prices. The authors also deserve credit for openly disclosing single-run results, non-significant case-study effects, and the mixed domain-knowledge ablation. However, the empirical evidence is currently too weak to support the 'comparable performance' claim: the key ALE numbers are single unrepeated runs re-scored outside the official environment, the own-benchmark headline uses best-of-two selection for BrainPilot only, and domain knowledge reduces measured scores in more cells than it improves. The contribution is real and the system is plausibly useful, but the quantitative claims need substantial additional validation before they can be accepted.

major comments (5)
  1. [§4.1, Table 2] The central claim that BrainPilot offers 'comparable performance' at 5-56% of the cost and 8-63% of the time rests entirely on one unrepeated run per configuration. The table itself shows extreme run-to-run sensitivity: e.g., BrainPilot GPT-5.5 T1 drops from 1.00 (DK-) to 0.00 (DK+), and Claude Code DeepSeek-V4-Pro T1 drops from 1.00 to 0.00. With reasoning backbones, single point estimates cannot support the comparative ratios stated in the abstract and §4.1. Please add repeated runs with distributions for at least the headline configurations, or explicitly withdraw the comparative claim and present the ALE results as illustrative single-case observations.
  2. [§4.1, Evaluation details] The ALE scores are 'grader-faithful re-scorings generated outside ALE's official hosted run.' Because the paper's main external anchor is comparison with ALE-graded systems, the local adapter must be validated against official ALE execution for at least a subset of tasks. Please provide official-hosted re-runs or an independent grader check; otherwise the comparability of these scores to other published ALE results is not established, and the risk of altered task setup, reveal timing, or environment cannot be dismissed.
  3. [Table 3 note; Appendix B.2] For BrainPilot's RSC score, the table reports the higher-scoring of two runs, while all other cells report one formal run. This asymmetric best-of-two selection directly affects the headline statement that 'BrainPilot with claude-opus-4.8 tied the highest RSC score' and the corresponding panel in Figure 6. Report both runs, use the identical selection rule for all harnesses, or use only the first run for every system.
  4. [§4.2, BrainPilotBench results] Across the 26 valid cells, domain knowledge increased 8 scores, reduced 14, and left 4 unchanged. The current text says that greater tool use 'did not yield uniform score improvements,' which understates a result that runs against the system's central design claim that curated domain grounding improves or underwrites research quality. Please analyze the conditions under which domain knowledge helps versus hurts, or revise the architecture-related claims so they are not contradicted by the benchmark evidence.
  5. [Appendix B.1, RSC scoring] The RSC primary score uses three pass/fail thresholds (0.2863, 0.4863, 0.5657) with no derivation. Because BrainPilotBench-v0 is introduced in this paper, these thresholds function as free parameters in the headline score. They should be justified from the source study or shown to be robust via a sensitivity analysis; otherwise the RSC scores in Table 3 and Figure 6 are not a stable benchmark signal.
minor comments (4)
  1. [Throughout] Please fix typographical and encoding issues: 'eabled' in §4.2, 'scientifc' in §3.1, 'accelarating' in §7, 'Alvzheimer' in Figure 4, and the rendering '/da◎abase−' and '/da◎abase+' in Tables 2, 3, and 4-7, which should read DK− and DK+.
  2. [§4.1 and §2.2] The phrase 'state-of-the-art agent framework' is undefined. The comparison is made only against Codex and Claude Code, which are general coding-agent harnesses, not against brain-science-specific agents such as BrainAgent that are cited in the related work. Please define the comparator class precisely, or narrow the claim to 'general-purpose coding agents.'
  3. [Figure 8 caption] The caption notes that anatomical scores are sourced from Siegle et al. (2021) but the agent cited Harris et al. (2019). This is a useful example of the need for human verification, but it appears only in a caption; it should be discussed in the main text with an explicit assessment of how the Auditor treated the mis-citation.
  4. [§4.2] The paper calls BrainPilotBench-v0 'preliminary' in some places and 'a benchmark' in others. Please state clearly the intended status of the benchmark and its validation stage, especially given the small number of tasks and the absence of external benchmarking of the benchmark itself.

Circularity Check

1 steps flagged · score 3.0 of 10

One load-bearing self-citation for Graph of Trace evaluation; the performance comparison itself is anchored externally and is not circular.

  1. self citation load bearing [Section 3.3, Graph of Trace (last sentence)]
    "The design and evaluation of the Graph of Trace is demonstrated in (Gao et al., 2026)."

    The Graph of Trace is a central design contribution claimed to make workflows 'traceable, human-inspectable, and agent-verified.' The only cited evaluation of this component is a companion paper by overlapping authors (Gao et al., 2026, including Tianci Gao, Haoxuan Li, Jianhe Li, Tianxiang Zhao, Runze Shi, Weiran Wang, Zezhao Wu, and Lu Mi). The present manuscript provides no independent evaluation, machine-checked proof, or code-reproduced validation of the Graph of Trace, so the claim that its design and evaluation are 'demonstrated' rests on the authors' own prior work rather than on external evidence.

full rationale

The headline performance claim—BrainPilot with an open-source backbone is comparable to state-of-the-art agent frameworks at lower cost—is anchored primarily by the three Agents' Last Exam tasks, whose graders are pre-existing, held-out references independent of the authors. The authors' own BrainPilotBench-v0 uses frozen references and held-out labels, so scores on it are not derived from BrainPilot's inputs by construction. The self-citation for Graph of Trace evaluation is the one clear load-bearing self-referential step, but it supports a design/auditability claim rather than the central performance comparison. Concerns about single-run reporting, best-of-two selection for one BrainPilot cell, local re-scoring of ALE, and the overlap between BrainPilotBench-v0 tasks and the paper's own case studies are evaluation-validity limitations and not, on inspection, reductions of predictions to inputs. Accordingly, there is no significant derivation circularity beyond the noted self-citation.

Assumptions & free parameters 1 free parameters · 5 assumptions · 2 invented entities

BrainPilot is an empirical systems paper, not a derivation, so the ledger records the benchmarks, statistical-treatment choices, and internal validation components on which the empirical claims rest. The thresholds in BrainPilotBench-v0 are hand-chosen but fixed before execution; the main hidden assumptions are the validity of the ALE rescoring, adequacy of single runs, public-data ground truth, list-price cost accounting, and the reliability of the Auditor/trace components, none of which is independently benchmarked.

free parameters (1)
  • RSC benchmark score thresholds = place-cell ratio in [0.2863, 0.4863]; decoding improvement ≥ 0.5657
    Chosen by hand in the new BrainPilotBench-v0 scoring formula (Appendix B.2). They define the task's official score; they were fixed before runs and passed Oracle/empty-submission gates, so they are not fitted to BrainPilot's outputs, but they are arbitrary design constants.
assumptions (5)
  • domain assumption ALE graders and withheld references remain valid when applied to outputs produced outside ALE's official hosted environment by the authors' adapter.
    Invoked in Section 4.1 Evaluation details; if local rescoring differs from official grading, the ALE comparison is not benchmark-faithful.
  • domain assumption Single runs of stochastic reasoning models are treated as sufficient evidence for the comparative performance claim.
    Section 4.1: 'we report one unrepeated run per task and configuration'; the paper acknowledges run-to-run variation but uses single-run numbers in all headline tables.
  • domain assumption Public datasets used for BrainPilotBench and case studies are valid ground truth for their scientific tasks.
    Section 4.2 and Section 5 assume Mao et al. 2020, Lee et al. 2021, Tangermann et al. 2012, and Kemp et al. 2000 provide correct references/labels.
  • domain assumption Official provider list prices are a fair proxy for true cost of running each configuration.
    Cost comparisons in Tables 2/3 are based on list pricing verified on 16 July 2026; actual costs may differ with caching, discounts, and infrastructure.
  • domain assumption The Auditor agent's claim-verification is reliable enough to support 'agent-verified results'.
    Section 3.1 describes the Auditor's procedure, but no error-injection benchmark or independent evaluation of audit accuracy is provided.
invented entities (2)
  • Graph of Trace (GoT)
    purpose: Append-only DAG recording subgoals, tool use, evidence, and claims for auditability of agent workflows.
    Introduced here and validated only in the same-author companion paper (Gao et al., 2026); no external benchmark demonstrates its audit benefit.
  • Auditor agent
    purpose: Checks numerical, file, and citation claims in drafts against session evidence before delivery.
    No quantitative evaluation (e.g., planted-hallucination tests) is reported; audit reports are presented as evidence of the mechanism's operation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BrainPilot: Automating Brain Discovery with Agentic Research." pith.science (2026). https://pith.science/paper/6TL74EQF

@misc{pith2026260715079,
  author       = {Pith},
  title        = {Pith review of: BrainPilot: Automating Brain Discovery with Agentic Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TL74EQF}},
  note         = {Machine review of arXiv:2607.15079}
}
read the original abstract

Understanding the brain increasingly depends on integrating evidence across scales, modalities, and disciplines. Addressing a single research question therefore requires a coordinated sequence of operations, from surveying prior work to executing analyses and interpreting results in light of domain knowledge. AI agents promise to accelerate this process, but current agents lack domain expertise in brain science, may fabricate claims, drift during multi-step reasoning, and offer few defined points for expert intervention. These failures are especially costly in brain science, where conclusions feed into downstream scientific claims and depend on laboratory-specific expertise and careful human judgment. We present \textbf{BrainPilot} a \textbf{fully open-source} multi-agent system that accelerates brain science research with traceable logs and agent-verified results. A principal investigator (PI) agent coordinates specialist agents grounded in curated domain knowledge: a unified brain science knowledge base containing 7{,}233 indexed items and a skill library of 72 reusable methodology units across seven research domains. Every major step is recorded in the Graph of Trace, an auditable record that links subgoals, tool use, evidence, and claims and allows researchers to follow and inspect the workflow. An Auditor agent further integrates fabrication checking into the workflow. For evaluation, we run three brain science tasks from Agents' Last Exam, introduce our own benchmark, \textbf{BrainPilotBench-v0}, and present additional end-to-end case studies. Across these evaluations, BrainPilot with an open-source backbone model attains performance comparable to state-of-the-art agent framework with less costs.

Figures

Figures reproduced from arXiv: 2607.15079 by the authors.

Figure 1
Figure 1. Overview of BrainPilot. (a) A curated knowledge base and skill library, built from neuro￾science textbooks and papers, feed a harness in which a principal investigator (PI) agent coordinates specialist agents. (b) The targeted breadth of brain science research, from systems and circuits to cognition, disease, and foundational machine learning. (c) The researcher interacts with the PI agent, which delegates tasks to … view at source ↗
Figure 2
Figure 2. BrainPilot system architecture. The researcher interacts with the principal investigator (PI) agent, which delegates to specialist agents. The agents draw on a knowledge base, a literature library, and a skill library and are evaluated by BrainPilotBench. The Graph of Trace along the bottom records every major step, with its nodes displaying per-agent status [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Knowledge-base construction. Neuroscience textbooks and papers are converted to text using OCR, split into overlapping passages, and embedded in a unified vector index. At query time, an agent’s question is embedded, candidate passages are retrieved by vector similarity, and a cross￾encoder reranker reorders them before they are returned as evidence. The Auditor agent has a deliberately narrow role to check if the o… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Composition of BrainPilot’s domain-knowledge assets. The skill library comprises 72 skills across seven research domains. The knowledge base contains 7,233 items assigned to 11 disciplinary groups. Its vocabulary is summarized from 377,868 content tokens in paper title…
Figure 5
Figure 5. Figure 5: BrainPilot on Agents’ Last Exam: performance and cost. Eight harness × backbone configurations on the three brain science tasks (T1 skull-stripping QC, T2 ROI resampling, T3 C. elegans neuron tracking), with domain knowledge enabled. (a) ALE grader scores; “F” marks a …
Figure 6
Figure 6. Figure 6: BrainPilotBench overview: performance, cost, and domain-resource use. (a) Scores across the four tasks: T1 RSC place-cell analysis, T2 TOPS-fMRI, T3 BCI Competition IV 2a, and T4 Sleep-EDF. T1 reports the domain-knowledge-disabled score; T2–T4 report the domain￾knowled…
Figure 7
Figure 7. Figure 7: Case study: spatial coding in the retrosplenial cortex. (a) Representative single-cell activity heatmaps with spatial tuning curves. (b) Population firing-rate map of detected place cells, sorted on odd trials and rendered on held-out even trials. (c) Bayesian position…
Figure 8
Figure 8. Figure 8: Case study: functional hierarchy in the mouse visual system. BrainPilot tested whether functional measures from spiking activity align with the anatomical hierarchy of eight mouse vi￾sual areas. (a–c) Three-session smoke-test outputs for response latency, intrinsic tim…
Figure 9
Figure 9. Figure 9: Case study: an fMRI pain connectivity signature and its transfer. (a) Full FC weight matrix of the learned signature. (b) Pain-region network of the signature weights. (c) Held-out true-versus-predicted pain ratings in the experimental tonic-pain cohort. (d) External r…
Figure 10
Figure 10. Figure 10: Case study: EEG motor-imagery decoding on BCI Competition IV 2a. (a) Model comparison (MIAgentModel vs. EEGNet baseline). (b) Learned spatial filters as scalp topogra￾phies. (c) Class activation topomaps (subject 3). (d) Right-minus-left contrast (subject 3). (e) Clas…
Figure 11
Figure 11. Figure 11: Case study: Sleep-EDF downstream sleep-staging decoder with physiological vali￾dation. (a) 20-fold leave-one-subject-out curves with DeepSleepNet reference lines. (b) Class-wise power spectral density. (c) Relative band-power heatmap across the five stages. (d) Held-o…
Figure 12
Figure 12. Figure 12: BrainPilot user interface. The interface presents the agent conversation, session history, agent activity, and workspace files. B BRAINPILOTBENCH SCORING AND PER-TASK STATISTICS The main text reports the cross-task summary ( [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding

    cs.AI 2026-08 conditional novelty 7.0 of 10

    A unified benchmark of 172 EEG analysis tasks shows that LLMs handle well-specified analyses better than long multi-step workflows, and that structured agent execution usually beats autonomous code generation.

Reference graph

Works this paper leans on

40 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    (1) Build a position tuning curve

    Place-cell screening (statistical test, P<0.05; exclude frames in which the running speed is below 1 cm/s). (1) Build a position tuning curve. a. Divide the entire linear track into N position bins. The number of bins should be chosen jointly from the animal's mean running speed and the signal sampling rate: bins must be wide enough that each is sampled b...

    2026

  2. [2]

    (1) For each place cell, draw a trial-by-position activity heatmap: x-axis is position, y-axis is trial, color encodes activity

    Representative single-cell visualization and population sorted visualization. (1) For each place cell, draw a trial-by-position activity heatmap: x-axis is position, y-axis is trial, color encodes activity. (2) Use odd trials to determine each cell's peak position from its spatial tuning curve, and sort all cells by this peak. Keep this ordering fixed and...

  3. [3]

    (1) Decode using the deconvolved dF/F activity of all imaged neurons, not only the screened place cells, so that the selection procedure does not bias the result

    Position decoding (Bayesian decoder). (1) Decode using the deconvolved dF/F activity of all imaged neurons, not only the screened place cells, so that the selection procedure does not bias the result. Restrict the analysis to time periods in which the animal's running speed exceeds 1 cm/s. (2) Divide the behavioral time series into non-overlapping short t...

    2026

  4. [4]

    (1) Within a single paradigm, partition the trials in their original temporal order into non-overlapping consecutive trial bins, each containing M trials

    Trial-bin correlation analysis. (1) Within a single paradigm, partition the trials in their original temporal order into non-overlapping consecutive trial bins, each containing M trials. Average the activity within each trial bin across the trial dimension to obtain a population vector for that bin. Arrange the bins in order to form a trial-bin population...

  5. [5]

    (1) For each trial, average the activity of all place cells across the cell dimension to obtain a per-trial population-mean firing rate

    Firing-rate dynamics. (1) For each trial, average the activity of all place cells across the cell dimension to obtain a per-trial population-mean firing rate. Plot this value as a function of trial. Please complete the analysis for each sub-task. # Reference information - Data path: <RSC_DATA_DIR> - Other notes: Read the README for the experimental paradi...

    2026

  6. [6]

    final_model

    Do not adjust the model based on any Study4/5 feedback. At minimum, report the following Study3 internal results: - overall Pearson r: predictions versus true pain ratings across all held-out test bins. - within-subject mean r: for each held-out subject, compute the correlation between predictions and true ratings across CAPS+REST bins, then report the me...

    2026

  7. [7]

    upper_triangle_row_major_i_lt_j

    The`edge_order`field in`signature_manifest.json`must contain the ordering actually established by validation. Only after confirmation may it be set to: ```json "upper_triangle_row_major_i_lt_j" ```

  8. [8]

    Do not directly assume that the input order equals: ```python np.triu_indices(279, k=1) ```

Show all 40 references
  1. [9]

    Prioritize verification of edge ordering using the following information: - the correspondence between`example_participant/ROI/`and `example_participant/FC/`

  2. [10]

    - upper triangle, MATLAB column-major

    At minimum, examine the following candidate orderings: - upper triangle, row-major,`i < j`. - upper triangle, MATLAB column-major. - lower triangle, MATLAB column-major

  3. [11]

    - the consistency metric between each candidate ordering and the example FC

    Save the validation procedure and results to: ```text results/edge_order_validation.json 35 2026-07-16 ``` The file must record at least: - the candidate orderings examined. - the consistency metric between each candidate ordering and the example FC. - the final edge ordering ...

    2026

  4. [13]

    source_feature_order_preserved_but_roi_mapping_unverified

    If the specific ROI-pair ordering cannot be confirmed: - Training and inference may still proceed using the original feature order. - No anatomical or ROI-pair-level interpretation of individual weights is allowed. -`signature_manifest.json`should contain: ```json "source_feat...

    2026

  5. [14]

    Use the CAPS and REST conditions from Study3 to train a tonic pain signature

  6. [15]

    Prioritize`dfc_5bin_dat`and`pain_5bin_dat`

  7. [16]

    38 2026-07-16

    Conduct Study3 internal held-out testing using subject-level splits; different time bins from the same subject must not appear in both the training and test sets. 38 2026-07-16

    2026

  8. [17]

    All hyperparameters, preprocessing choices, and model-selection decisions must be based only on Study3 training/validation data or Study3 internal held-out results

  9. [18]

    Do not use any externally pretrained signature weights

  10. [20]

    The main motor-imagery frequency bands are: - Mu rhythm: approximately 8--13 Hz - Beta rhythm: approximately 13--30 Hz - The input has already been band-pass filtered at 4--38 Hz

    Temporal convolution Include temporal convolution to capture Mu- and Beta-related patterns. The main motor-imagery frequency bands are: - Mu rhythm: approximately 8--13 Hz - Beta rhythm: approximately 13--30 Hz - The input has already been band-pass filtered at 4--38 Hz. Tempo...

  11. [21]

    Use information from C3, Cz, C4, and surrounding motor-cortex channels

    Spatial filtering / channel mixing 41 2026-07-16 Include spatial filtering or mixing across EEG channels. Use information from C3, Cz, C4, and surrounding motor-cortex channels. You may draw on the ideas of EEGNet / ShallowFBCSPNet: - temporal convolution for frequency-related...

    2026

  12. [22]

    Small-sample generalization BCI IV 2a is a small-sample EEG dataset, so avoid an oversized model. Recommended components include: - BatchNorm - Dropout - Pooling - depthwise / separable convolution - AdaptiveAvgPool - a lightweight classifier - reasonable channel counts and pa...

  13. [23]

    Its design should remain stable under CrossSessionEvaluation from A0xT to A0xE

    Class balance and robustness The model should not favor only one class in the four-class task. Its design should remain stable under CrossSessionEvaluation from A0xT to A0xE. Do not define the loss, class weights, training loop, or sampling strategy in the model file. # Code r...

  14. [25]

    Define: class MIAgentModel(nn.Module):

  15. [30]

    Do not call complete braindecode models, including: - EEGNet 42 2026-07-16 - ShallowFBCSPNet - Deep4Net - EEGConformer - ATCNet - any other complete EEG decoding architecture wrapper

    2026

  16. [32]

    subject_ids

    Do not output Markdown. # Tensor validity requirement The following test must pass: import torch model = MIAgentModel() x = torch.randn(2, 22, 512) y = model(x) assert y.shape == torch.Size([2, 4]) # Recommended architecture You may use the following design directions, but pro...

    2026

  17. [33]

    Conv1d is preferred because the input is a single-channel time series: (Batch, 1, 3000)

    Temporal convolution Include Conv1d or Conv2d to process the 30-second EEG waveform. Conv1d is preferred because the input is a single-channel time series: (Batch, 1, 3000)

  18. [34]

    45 2026-07-16

    Multi-scale temporal kernels Sleep stages depend on different frequency bands and morphologies: - Delta / slow wave: 0.5--4 Hz, mainly associated with N3 - Theta: 4--8 Hz, common in N1 / REM - Alpha: 8--13 Hz, associated with Wake - Sigma / spindle: 11--16 Hz, associated with ...

    2026

  19. [35]

    Local temporal context Although the input contains only one 30-second epoch, the model should capture local temporal structure within the epoch. Possible components include: - dilated Conv1d - TCN-style residual blocks - temporal pooling - a lightweight GRU/LSTM If a recurrent...

  20. [36]

    Small-sample generalization The evaluation dataset is small, so avoid an oversized model. Recommended components include: - BatchNorm1d - Dropout - AdaptiveAvgPool1d - reasonable channel counts - residual connections - a parameter-efficient classifier Do not use a very large T...

  21. [37]

    The model should avoid complete dependence on the majority classes

    Robustness to class imbalance Wake and N2 may be more frequent in Sleep-EDF, and N1 is usually the most difficult class. The model should avoid complete dependence on the majority classes. Use robust feature extraction and regularization to improve generalization. Do not defin...

  22. [38]

    Depend only on standard PyTorch: import torch import torch.nn as nn You may also use: import torch.nn.functional as F

  23. [39]

    Define: class SleepAgentModel(nn.Module):

  24. [40]

    The model must be independently importable

  25. [41]

    The model file must not execute training, evaluation, data loading, data downloading, or result saving

  26. [42]

    Do not hard-code the batch size in forward

  27. [43]

    Do not change the batch size in forward

  28. [44]

    Do not call complete braindecode models, including: 46 2026-07-16 - SleepStagerChambon2018 - DeepSleepNet - USleep - SleepStagerBlanco2020 - any other complete sleep-staging architecture wrapper

    2026

  29. [45]

    Do not write pseudocode

  30. [46]

    Do not output Markdown. # Tensor validity requirement The following test must pass: import torch model = SleepAgentModel() x = torch.randn(2, 1, 3000) y = model(x) assert y.shape == torch.Size([2, 5]) # Recommended architecture Use the following as design directions, not as ps...

    2026

  31. [2018]

    Russell A

    doi: 10.1016/j.conb.2018.04.007. Russell A. Poldrack and Krzysztof J. Gorgolewski. Making big data open: Data sharing in neu- roimaging.Nature Neuroscience, 17(11):1510–1517, 2014. doi: 10.1038/nn.3818. Ben Prystawski, Kushin Mukherjee, Daniel Wurgaft, Linas Nasvytis, Michael ...

    2018 arXiv

  32. [2022]

    Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone

    doi: 10.1038/s41593-021-00980-9. Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone. When combinations of humans and AI are useful: A systematic review and meta-analysis.Nature Human Behaviour, 8, 2024. Cheng Wang, Zhibin He, Zhihao Peng, Shengyuan Liu, Yufan Hu, Carl Yan...

    2024 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.