Insights from the Algonauts 2025 Winners

Mihir Tripathy; Paul S. Scotti

REVIEW 2 major objections 3 minor 1 cited by

Insights from the Algonauts 2025 Winners

T0 review · 2 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By reviewing the Algonauts 2025 winners, this paper argues that predicting fMRI responses to long, naturalistic movies now depends on multimodal, context-aware models that pass out-of-distribution tests.

desk verdict A timely perspective on Algonauts 2025 that promises insight but whose value hinges entirely on whether the full text critiques the winners' self-reports rather than relaying them. read the letter →

arxiv 2508.10784 v1 pith:XIKFF63N submitted 2025-08-14 q-bio.NC cs.CV

Paul S. Scotti , Mihir Tripathy This is my paper · ORCID

classification q-bio.NCcs.CV

keywords AlgonautsChallengefMRIencodingnaturalisticmoviesbraindecodingout-of-distributiongeneralizationmultimodalmodelswhole-brainparcelscomputationalneuroscience

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper examines the Algonauts 2025 Challenge, where teams predicted fMRI activity across 1,000 whole-brain parcels while four participants watched nearly 80 hours of naturalistic movies. The authors, themselves competitors, analyze the winning approaches and draw lessons about what drives successful brain encoding. They claim that the move from still images to long, multimodal movies changes the modeling challenge: success now hinges on capturing narrative context and generalizing to completely unseen films, not just on recognizing familiar categories. The paper positions out-of-distribution movie prediction as the new benchmark for evaluating encoding models, and reflects on what this shift means for the field's next steps.

What carries the argument

The central instrument is the challenge's evaluation design: thousands of whole-brain fMRI parcels, long multimodal movie stimuli, and a held-out set of six films that requires out-of-distribution generalization. This design is what separates models that memorize training statistics from models that capture generalizable stimulus-response relationships, and it is the engine that produces the insights the paper discusses.

What would settle it

If an independent reanalysis of the top models on a fresh set of participants and movies showed that models without the highlighted components (e.g., no pretrained multimodal features or no temporal modeling) generalize just as well, the paper's account of what won would collapse.

Watch

Extended reading notes

Core claim

The Algonauts 2025 Challenge used 65 hours of training data—episodes of a long-running sitcom and four feature films—and tested models on six held-out movies. The winners were the teams whose models best predicted fMRI responses in out-of-distribution films. The paper's central claim is that these winners' shared approaches reveal the current best-practice recipe for brain encoding: pretrained multimodal representations, temporal or contextual modeling, and per-participant mapping to whole-brain parcels. The authors read the challenge outcomes as evidence that the field is shifting from static, category-driven experiments to naturalistic, dynamic, and narratively rich stimuli, and that out-o

Load-bearing premise

The paper's lessons depend on the winners' public reports and the challenge's out-of-distribution scores accurately reflecting what actually drives brain-encoding performance, rather than quirks of one dataset.

Editorial extensions

If this is right

Brain-encoding research should adopt naturalistic, long-form stimuli and out-of-distribution evaluation as standard practice, rather than static image benchmarks.
Pretrained multimodal models are likely to become the default feature extractors, with simple per-participant decoders mapping features to whole-brain parcels.
Out-of-distribution generalization becomes the primary way to compare encoding models, because it tests whether a model understands narrative and context rather than merely recalling training signals.
Improved movie-brain encoding could enable more accurate prediction of individual brain responses in clinical or applied settings, such as decoding mental states during natural viewing.
Future challenges may need to add more diverse movies or more participants to push beyond the current benchmarks and expose what existing models still miss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If out-of-distribution movie prediction becomes the norm, encoding models will increasingly be evaluated on their ability to reason about social and narrative structure, not just low-level visual features.
The reliance on a single challenge dataset means the identified winning factors might be tailored to that specific set of films; a meta-analysis across all submitted models, not just winners, would more reliably isolate which components matter.
The paper's conclusions could be tested by building a deliberately 'shallow' model that lacks the proposed key ingredients and checking whether it still generalizes to held-out movies, which would falsify the claimed recipe.
Future editions could adopt continuously updated movie sets to prevent the field from overfitting to a fixed pool of naturalistic stimuli, keeping the out-of-distribution test genuinely challenging.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Only the abstract was in front of me, so this is a provisional read. The paper is, by the authors' own admission, a retrospective: the 4th-place MedARC team reflecting on what Algonauts 2025 tells us about brain encoding. There is no new dataset, no new method, no derivation. That's not necessarily a strike against it; a well-done perspective on a landmark challenge can be useful. The abstract is transparent about the setup, the data scale, and the authors' position, and it promises to synthesize lessons from the winning teams' public reports. If the full text actually does that synthesis critically—checking whether the winners' reported choices really explain their performance, whether compute or ensembling confounds the story—then the paper has value for people planning the next round of encoding models.

The soft spots are exactly where you'd expect. The insights are only as reliable as the winning teams' self-reports, which are not peer-reviewed. The abstract gives no hint that the authors did any independent reanalysis or control experiments; it sounds like a reflection on the winners' write-ups. That means the central claim—these approaches 'work'—could inherit whatever biases the top teams had, from selective reporting to hidden compute budgets. The authors may address this in the full text, but the abstract doesn't. Also, because the paper is a commentary, there's no way to evaluate soundness beyond plausibility; the reader's 'unverdictable' rating is fair.

On citation: I probably wouldn't cite this in my own work unless I needed a pointer to the competition results themselves, and even then the original challenge reports would be the better source. But for a reading group that follows brain encoding and benchmark culture, it could spark a useful discussion about what challenge wins actually tell us. So: maybe for reading group, no for citation.

My recommendation on peer review: if the full text delivers on the abstract's promise and treats the winners' reports as data to be interrogated rather than gospel, it deserves referee time. The abstract alone is too thin to justify publication, but the topic is current and the authors are insiders. I'd say yes to sending it to review, with the expectation that the reviewers push for evidence that the insights are more than paraphrased self-reports.

Referee Report

2 major / 3 minor

Summary. This paper is a commentary by members of the MedARC team (the 4th-place team) reflecting on the results of the Algonauts 2025 Challenge, which tasked participants with predicting fMRI responses to long, naturalistic movies from the CNeuroMod dataset. The authors claim that the winning teams' approaches reveal which modeling strategies are effective for brain encoding, that these insights characterize the current state of brain-encoding research, and that they point toward future directions. The abstract provides no quantitative results, derivations, or reanalyses; it is a high-level statement of the paper's theme.

Significance. If the insights are accurate and appropriately qualified, this perspective could be a useful, timely synthesis of state-of-the-art practice in fMRI encoding with naturalistic video, a setting that is more realistic than the static-image or short-clip benchmarks used in earlier Algonauts editions. The authors' insider status gives them access to the winning teams' public reports and to the competitive context, which adds value beyond a purely external analysis. However, the significance hinges on the reliability of those self-reports and on careful generalization from the specific challenge settings to the broader field. The abstract alone does not demonstrate these conditions.

major comments (2)

[Abstract] The central claim that the winners' approaches 'reveal what works in brain encoding' is based on the assumption that the top teams' public reports are accurate and sufficiently complete. The abstract does not indicate any independent verification (e.g., reimplementation, ablations, or sensitivity analyses) of these self-reports. Without such checks, the insights could be shaped by self-presentation biases, omitted details about ensembling or compute budget, or post-hoc selection effects. The authors should make this reliance explicit and, if full-text evidence allows, provide critical scrutiny of the reports' completeness rather than treating them as ground truth.
[Abstract (final sentence)] The claim that the insights reveal 'the current state of brain encoding research' requires generalization beyond the specific challenge setup: four participants, approximately 80 hours of a limited film set, and a particular evaluation metric. The abstract does not mention any discussion of these limits. If the manuscript does not already contain such a discussion, it should be added to avoid overclaiming; if it does, the abstract should convey that qualification.

minor comments (3)

[Abstract] The abstract does not name the winning teams or describe their approaches, making it difficult for a reader to assess the claimed insights from the abstract alone. A brief identification of the winning methods would improve clarity.
[Abstract / Introduction] The paper is authored by a 4th-place team, which is declared, but the title says 'Algonauts 2025 Winners.' Consider clarifying that the paper is a commentary on the winning approaches, not an authorial report from the winners themselves.
[General] Since the paper relies on public challenge reports, the authors should cite or link to those reports explicitly to allow readers to verify the basis of the insights.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: commentary piece with no derivation chain.

full rationale

This is an abstract-only commentary in which the MedARC team reflects on the Algonauts 2025 winners' public reports. There is no mathematical derivation, no fitted model, no predictive claim that is then validated. The central claim is an interpretation of external competition outcomes. Any concern about the reliability of self-reported winner accounts is an evidentiary or correctness issue, not a circularity issue. No equation or definition reduces to itself, and no load-bearing self-citation is present. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper's central claim rests on the challenge data and the winners' reports, both of which are external to this preprint. No free parameters or invented entities are introduced.

assumptions (2)

domain assumption The Algonauts challenge evaluation score (correlation between predicted and held-out fMRI parcel responses) is a valid measure of brain encoding quality.
The reflection's usefulness depends on treating the challenge metric as a meaningful proxy for brain encoding performance. This is standard in the field but is an assumption the paper likely inherits from the challenge design.
domain assumption The winners' self-reported methods in their public reports are accurate and complete.
The paper draws insights from the winners' reports; if those reports are inaccurate or omit crucial details, the reflections would be built on faulty evidence. The abstract offers no validation of these reports.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Insights from the Algonauts 2025 Winners." pith.science (2026). https://pith.science/paper/XIKFF63N

@misc{pith2026250810784,
  author       = {Pith},
  title        = {Pith review of: Insights from the Algonauts 2025 Winners},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XIKFF63N}},
  note         = {Machine review of arXiv:2508.10784}
}

read the original abstract

The Algonauts 2025 Challenge just wrapped up a few weeks ago. It is a biennial challenge in computational neuroscience in which teams attempt to build models that predict human brain activity from carefully curated stimuli. Previous editions (2019, 2021, 2023) focused on still images and short videos; the 2025 edition, which concluded last month (late July), pushed the field further by using long, multimodal movies. Teams were tasked with predicting fMRI responses across 1,000 whole-brain parcels across four participants in the dataset who were scanned while watching nearly 80 hours of naturalistic movie stimuli. These recordings came from the CNeuroMod project and included 65 hours of training data, about 55 hours of Friends (seasons 1-6) plus four feature films (The Bourne Supremacy, Hidden Figures, Life, and The Wolf of Wall Street). The remaining data were used for validation: Season 7 of Friends for in-distribution tests, and the final winners for the Challenge were those who could best predict brain activity for six films in their held-out out-of-distribution (OOD) set. The winners were just announced and the top team reports are now publicly available. As members of the MedARC team which placed 4th in the competition, we reflect on the approaches that worked, what they reveal about the current state of brain encoding, and what might come next.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Predicted Cortex Is Not a Domain-General Prior: A Matched-Control Audit of Brain-Encoding Features for Video Memorability
cs.CV 2026-07 conditional novelty 6.0 of 10

Predicted cortical responses from a brain-encoding model beat their own visual backbone on VideoMem but lose on Memento10k, so they are a dataset-specific memorability representation, not a domain-general prior.

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.