{"id":"edbc9249-cd8b-4e7c-bb22-5ceb1a429264","arxiv_id":"2606.11660","paper_version":1,"verdict":"ACCEPT","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Bergson is an open-source library providing scalable data attribution tools and the first open implementations of MAGIC, SOURCE, and TrackStar for large language models.","lead":"Bergson is an open-source library that supplies scalable implementations of data attribution methods for large language models, including the first public versions of MAGIC, SOURCE, and TrackStar. It reduces engineering effort for researchers studying how training data influences model behavior.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Scalability and faithfulness of MAGIC/SOURCE/TrackStar implementations asserted without benchmarks or verification in the paper","rationale":"The reader's weakest_assumption directly identifies the same gap. Because the manuscript supplies no internal evidence that the implementations meet the scalability requirement, the ACCEPT verdict should be conditioned on external verification of the repository. No other technical inconsistency is visible from the supplied abstract and claim description.","tokens_in":1654,"tokens_out":337,"duration_ms":11374,"concrete_test":"Clone https://github.com/EleutherAI/bergson, run the provided MAGIC implementation on a 7B+ parameter model (e.g., Pythia-6.9B) using the on-disk gradient store on a single node with 8xA100; record peak memory and wall-clock time per 10k examples. Compare against the scaling behavior reported in the original MAGIC paper; if memory exceeds 80 GB or time does not improve linearly with the on-disk store, the scalability claim does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the library's implementations of MAGIC, SOURCE, and TrackStar are both faithful to the originals and achieve the stated scalability on very large LMs and pre-training datasets. The manuscript provides no timing results, memory measurements, scaling curves, or side-by-side comparisons against the original papers; it only states that the library 'natively supports on-disk gradient stores and multi-node distributed training.' Because the paper contains no empirical section or appendix demonstrating these properties, the claim rests entirely on the external repository rather than on evidence internal to the work.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces Bergson, an open-source library for data attribution in machine learning. It aims to reduce engineering effort for applying attribution methods at scale on large language models and pre-training datasets by providing native support for on-disk gradient stores and multi-node distributed training, along with quality-of-life tools. The paper claims to deliver the first open-source implementations of the MAGIC, SOURCE, and TrackStar methods.","tokens_in":1748,"tokens_out":288,"duration_ms":22624,"significance":"If the implementations are faithful to the originals and achieve the claimed scalability, the library would lower barriers for data attribution research, enabling wider use of these techniques for model debugging and dataset curation. The open-source release itself is a concrete contribution that could accelerate progress in interpretability.","major_comments":[{"comment":"Abstract: The central claims that Bergson 'provides a host of techniques that scale to very large language models and pre-training datasets' and supplies 'the first open-source implementations of three leading data attribution methods: MAGIC, SOURCE, and TrackStar' are asserted without any benchmarks, timing results, memory measurements, scaling curves, or side-by-side verification of faithfulness to the source papers. The manuscript provides no empirical section demonstrating these properties and instead points to the external repository.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their thoughtful review and constructive feedback on our manuscript describing the Bergson library. We address the major comment below.","responses":[{"response":"We acknowledge the referee's observation that the abstract makes claims about scalability and the novelty of the implementations without accompanying empirical results in the manuscript itself. However, Bergson is presented as a software library paper whose primary contribution is the release of production-ready, scalable tooling rather than new methodological advances or performance benchmarks. The scalability claims are grounded in the library's explicit design choices (on-disk gradient stores and multi-node distributed training support), which directly address the engineering barriers described in the introduction. The implementations of MAGIC, SOURCE, and TrackStar are released as open source precisely so that faithfulness can be verified by direct code inspection against the source papers; we believe this is the appropriate form of validation for a library release. Adding an empirical section would shift the paper's focus away from its intended purpose as a tool description. We therefore do not plan to revise the manuscript to include benchmarks or timing results.","revision_made":"no","referee_comment":"[Abstract] Abstract: The central claims that Bergson 'provides a host of techniques that scale to very large language models and pre-training datasets' and supplies 'the first open-source implementations of three leading data attribution methods: MAGIC, SOURCE, and TrackStar' are asserted without any benchmarks, timing results, memory measurements, scaling curves, or side-by-side verification of faithfulness to the source papers. The manuscript provides no empirical section demonstrating these properties and instead points to the external repository."}],"tokens_in":1199,"tokens_out":344,"duration_ms":13337,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper announces an open-source library with the first public implementations of MAGIC, SOURCE, and TrackStar, along with on-disk gradient storage and multi-node training support. That fills a practical gap for people trying to run data attribution at LLM scale.\n\nThe work does what a tools paper should: it identifies the engineering barriers (gradient storage, distributed runs) and packages the methods so others do not have to start from scratch. The GitHub link is the actual deliverable, and the abstract is clear about what is new versus what already existed in closed form.\n\nThe soft spot is exactly what the stress-test flags. The manuscript asserts scalability and faithful re-implementations but supplies zero timing numbers, memory profiles, scaling plots, or side-by-side checks against the original papers. Readers are left to inspect the repo themselves. That is acceptable for a short announcement, but it means the central promise is not evidenced inside the document.\n\nThis is for interpretability researchers who need working code for data attribution experiments rather than for theorists looking for new proofs. A reader who already knows the three methods will mainly care whether the library runs without hidden bugs or excessive overhead.\n\nI would send it to peer review. The artifact is new and the field benefits from open implementations even if the write-up stays descriptive. A referee could usefully check the code against the cited originals and ask for minimal scaling data in a revision.","headline":"Bergson is a straightforward software release that ships the first public code for MAGIC, SOURCE, and TrackStar plus some scaling infrastructure, but the paper itself contains no benchmarks or verification of those claims.","tokens_in":2275,"tokens_out":376,"would_cite":false,"duration_ms":13661,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Bergson supplies the first open-source implementations of MAGIC, SOURCE, and TrackStar that scale to very large language models.","keywords":["data attribution","model interpretability","large language models","open source library","MAGIC","SOURCE","TrackStar","gradient stores"],"falsifier":"Running Bergson's implementations and the original method code side by side on one large model and dataset, then checking whether the attribution scores match within expected numerical tolerance.","tokens_in":2559,"feed_emoji":"🛠","tokens_out":620,"duration_ms":32511,"temperature":0.7,"pith_summary":"Data attribution traces how individual training examples shape a model's outputs, which can help identify sources of unwanted behavior or guide better dataset choices. The paper presents Bergson as a library built to run these techniques on the scale of current large language models and their pre-training data. It adds native support for storing gradients on disk and spreading work across multiple machines, plus everyday tools that reduce setup time. The release also ships the first public versions of three established methods named MAGIC, SOURCE, and TrackStar. A reader would care because prior work in this area stayed limited to small models or stayed behind closed implementations.","feed_headline":"Bergson brings data attribution methods to large language model scale","feed_subtitle":"The library supplies the first open implementations of MAGIC, SOURCE, and TrackStar with distributed training support.","key_machinery":"The Bergson library, which performs data attribution through on-disk gradient stores and multi-node distributed training to reach the scale of large language models and pre-training datasets.","core_discovery":"Bergson is an open source library that aims to enable faster progress in the field by providing a host of techniques that scale to very large language models and pre-training datasets. The library natively supports on-disk gradient stores and multi-node distributed training, and provides quality of life tools for researchers. Finally, it introduces the first open-source implementations of three leading data attribution methods: MAGIC, SOURCE, and TrackStar.","pith_inferences":["Routine use of the library could turn data attribution into a standard check applied during model development.","The shared code base may let groups compare different attribution methods on identical large-scale setups.","Community additions of further methods to the library could expand the set of available techniques over time."],"forward_implications":["Researchers gain immediate access to open code for MAGIC, SOURCE, and TrackStar without writing their own versions.","Data attribution work on full pre-training datasets becomes practical through the library's built-in distributed training and disk storage.","Quality of life tools in the library lower the engineering cost of running attribution experiments."],"fun_headline_variants":["Bergson library scales data attribution to large models","Bergson offers open MAGIC SOURCE TrackStar implementations","Bergson enables distributed data attribution for LLMs","Bergson supports on-disk gradients and multi-node training","Open source Bergson for scalable data attribution methods"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The library's versions of MAGIC, SOURCE, and TrackStar reproduce the original methods and retain their performance when applied to very large models and datasets.","fun_headline_variants_meta":{"raw":{"variants":["Bergson library scales data attribution to large models","Bergson offers open MAGIC SOURCE TrackStar implementations","Bergson enables distributed data attribution for LLMs","Bergson supports on-disk gradients and multi-node training","Open source Bergson for scalable data attribution methods"]},"model":"grok-4.3","cost_usd":0.002914,"raw_usage":{"total_tokens":1565,"prompt_tokens":584,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":29137000,"prompt_tokens_details":{"text_tokens":584,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":918,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":584,"tokens_out":63,"duration_ms":7617,"temperature":1.0,"reasoning_tokens":918,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T10:46:59.476927+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running Bergson's implementations and the original method code side by side on one large model and dataset, then checking whether the attribution scores match within expected numerical tolerance.","supporting_citations":[],"review_version":1}