{"id":"1550d517-4299-4f65-818c-64e34e478f6b","arxiv_id":"2412.00129","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A byte-level transformer trained from scratch on simulated collider data matches specialized jet-tagging models when given enough training examples.","lead":"This paper trains a 160-million-parameter transformer that reads particle physics jet data as raw bytes and shows it can identify jet types about as well as specialized physics models. The result suggests generic language-model architectures could become foundation models for experimental physics, but the scaling and 'emergence' claims need stronger evidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scaling/emergence claim confounds dataset size with total optimizer steps: fixed 30 epochs and batch size 512 mean compute grows with data, so the sharp jump may be an undertraining artifact.","rationale":"The reader identified the same load-bearing concern: a fixed training budget (30 epochs, batch size 512) applied across all dataset sizes means small-dataset runs may be undertrained. I agree and sharpen it: the fixed epoch count makes total gradient steps scale with dataset size, so dataset size and compute are perfectly confounded. This directly undermines the central scaling/emergence claim, which is the paper's most distinctive contribution and the basis for its foundational-model narrative. The performance comparison at the largest dataset size is secondary; even if the final accuracy claim holds, the emergence interpretation is unsupported without a compute-matched control or convergence checks. The concrete test I propose would settle the question: if a small dataset is trained with the same number of gradient steps as the large dataset, and performance rises smoothly, then the sharp S-curve is an optimization artifact. This is a feasible and decisive experiment. Given that the central claim is currently unverified but potentially salvageable, the reader's CONDITIONAL verdict remains appropriate; no verdict adjustment is needed.","tokens_in":13504,"tokens_out":4378,"duration_ms":54571,"concrete_test":"Train BBT-Neutron on a fixed small dataset (e.g., 10,000 events) while increasing the number of epochs so that the total number of seen samples matches the 10M dataset at 30 epochs (i.e., 30,000 epochs for 10k events, or equivalently match total gradient steps by reducing batch size). If the small-dataset model reaches performance comparable to the 10M model, the S-curve in Figures 4 and 5 is an artifact of dataset-size-compute confounding, not data-driven emergence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central scaling/emergence claim is that BBT-Neutron exhibits a sharp performance transition as a function of training dataset size (Figures 4 and 5). However, the training protocol in Section 4.1.3 fixes batch size at 512 and epochs at 30 for all dataset sizes. This means the total number of gradient steps grows linearly with dataset size: from roughly 6 steps for 100 events to roughly 586,000 steps for 10 million events. Thus the x-axis (dataset size) is perfectly confounded with optimizer steps (compute). The steep rise in BBT-Neutron's performance may simply reflect that models trained on larger datasets receive far more optimization updates, not that more data alone enables a phase transition. The authors provide no convergence checks, learning curves, or loss curves for any dataset size, so there is no evidence that the small-dataset runs have converged. For the same reason, the comparison with ParticleNet and Particle Transformer is not on an equal footing unless those models were trained with a matched compute budget, which is not reported. This confound is the load-bearing flaw in the emergence interpretation: if one holds compute constant (or trains to convergence), the sharp jump may vanish, reducing the central claim to a trivial statement that more training steps yield better accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BBT-Neutron, a decoder-only transformer that uses byte-level (\"binary\") tokenization and patch embeddings as a task-agnostic architecture for scientific numerical data. It is evaluated on Jet Origin Identification (JoI) using simulated e+e−→νν̄H, H→jj events at 240 GeV, and compared with the domain-specific models ParticleNet and Particle Transformer. The authors report that BBT-Neutron reaches comparable performance at 10 million training events and exhibits a sharp scaling transition (\"emergence\") in flavor-tagging efficiency and charge-flip rate as a function of training dataset size, which they interpret as evidence for a generalist-model phase transition and a step toward foundation models for particle physics.","tokens_in":13749,"tokens_out":8185,"duration_ms":72879,"significance":"If the main claims are supported, the result is of interest to both the ML-for-science and high-energy physics communities: it would demonstrate that a generic byte-level transformer with no physics-specific inductive biases can match specialized jet-tagging architectures, and that binary tokenization is a viable alternative to BPE for numerical data. The evaluation is a standard supervised train/test split against external baselines, and the code is open-sourced, so the central performance question is not circular. However, the current evidence is primarily visual and the scaling interpretation is confounded, so the significance is conditional on additional quantitative and controlled experiments.","major_comments":[{"comment":"Section 4.1.3 fixes batch size at 512 and training at 30 epochs for all dataset sizes. Consequently, the total number of optimizer steps grows roughly linearly with dataset size: about 30 steps for 100 events and about 5.9×10^5 steps for 10^7 events. The x-axis of the scaling plots (Figs. 4 and 5) therefore confounds dataset size with total compute/optimizer steps. The sharp jump in BBT-Neutron's curves may be an undertraining artifact for small dataset sizes, and no learning curves or convergence checks are reported for any dataset size. This is load-bearing for the emergence interpretation in Section 2. The authors should hold compute fixed, train to convergence, or otherwise separate the data-size effect from the optimizer-step effect, and report loss curves.","section":"4.1.3 and Figs. 4-5"},{"comment":"The central claims of comparable performance and of a phase-transition-like jump rest on visual inspection of confusion matrices and scaling curves. The manuscript gives no numeric values for the flavor-tagging efficiencies or charge-flip rates, no error bars, no repeated-seed variability, and no statistical test or fitted scaling law for the proposed critical data thresholds. Phrases such as \"strongly indicates the emergence of a generalist architecture\" go beyond what the plotted curves alone demonstrate. Add tables with point estimates and uncertainties, and a formal test (e.g., confidence intervals or a comparison of fitted models) for the differential scaling behavior between BBT-Neutron and the specialized baselines.","section":"Section 2 and Figs. 2-5"},{"comment":"To establish \"comparable performance\" to ParticleNet and Particle Transformer, the baselines need to be trained on the same data with a matched or at least reported training budget. The manuscript does not provide the training details, hyperparameters, parameter counts, or compute used for PN and ParT. Without this information, and given the fixed-budget issue above, the comparison in Figures 2-5 is not on an equal footing. Please report the baseline training protocol and, ideally, compute-matched scaling curves.","section":"4.2 and Figs. 2-5"},{"comment":"The Conclusion states that \"BBT-Neutron achieves state-of-the-art performance in Jet Origin Identification,\" but the Abstract and Results claim only that its performance is comparable to that of two specific models. Because no quantitative comparison with a broader set of methods is shown, this sentence overstates the evidence. Please either provide quantitative evidence for state-of-the-art status or soften the claim to comparability with the evaluated baselines.","section":"Section 5"}],"minor_comments":[{"comment":"Correct the typo \"Jet Original Identification\" to \"Jet Origin Identification,\" and replace \"spacial\" with \"spatial\" in the Abstract and Section 3.","section":"Keywords and Abstract"},{"comment":"The tokenization section calls the representation \"octal (base-8) byte sequences,\" but the worked examples show decimal byte arrays (e.g., [49, 50, 51, 52, 53]) and mention hexadecimal as an option. Please define the exact byte encoding used in the experiments, including endianness and width for the native numerical conversion, since this is the method's main contribution.","section":"4.1.2"},{"comment":"The abstract says BBT-Neutron uses binary tokenization \"to facilitate pretraining on a mixture of textual and large-scale numerical experimental data,\" but the experiments are end-to-end supervised training on JoI only and no pretraining on a multimodal mixture is reported. Please clarify whether any pretraining was performed or whether the current study only demonstrates the architecture's supervised use.","section":"Abstract and 4.1.3"},{"comment":"Reference [51] is incomplete: \"Vaswani, A.: Attention is all you need\" lacks co-authors and publication venue; cite the full NeurIPS 2017 paper or a version of record.","section":"References"},{"comment":"The panels in Figures 4 and 5 would be easier to read with explicit labels for the BBT-Neutron, ParticleNet, and Particle Transformer curves, since the caption text is the only guide in the grayscale version.","section":"Figures 4-5"}],"recommendation":"major_revision","confidential_remarks":"The strongest defensible claim after the suggested revisions is that a byte-tokenized generalist transformer matches specialized jet taggers at large training-data volume under the reported fixed-budget protocol. The \"emergence\" wording should be made conditional on compute-matched training or replaced by a more neutral description of the scaling curve. If the compute-matched experiments are not feasible, the authors should clearly relabel the effect as a property of their fixed 30-epoch schedule rather than of data volume alone."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read through carefully. The real news is narrow but real: a 160M-parameter causal transformer that eats raw bytes can, at 10M training jets, sit in the same ballpark as ParticleNet and Particle Transformer on jet origin identification. If that reproduces with solid numbers, it is a useful data point for the 'foundation model for HEP' conversation. The binary-tokenization idea is not new (bGPT, MEGABYTE, SpaceByte) but applying it to jet tagging with a scaling study is new, and the authors did the right thing by benchmarking against two strong domain-specific baselines.\n\nThe soft spots are serious, though, and they concentrate in the scaling/emergence story. Section 4.1.3 fixes batch size 512 and 30 epochs for every dataset size from 100 to 10 million. That means total gradient steps go from roughly 6 to 586,000. The paper's sharp performance jump as a function of dataset size is therefore perfectly confounded with compute; the small-dataset runs are almost certainly undertrained. The authors call the jump 'strongly indicative of emergence' without any convergence check, learning curve, or matched-compute experiment. That's a load-bearing flaw in the current draft, not a cosmetic one.\n\nThe other issues are smaller but real. There are no quantitative numbers anywhere - no tagging efficiencies, no charge-flip rates with uncertainties, no statistical test for the claimed threshold. The tokenization section is ambiguous: at one point it says 'binary tokenization,' then 'we opted for octal (base-8) byte sequences,' and the example of an integer encoding is not the same as the float attributes listed in Table 1. The GitHub link is a plus, but there's no hash or evidence of release. Finally, the Abstract says 'comparable,' the Conclusion says 'state-of-the-art' - minor inconsistency.\n\nWho is this for? Researchers working on ML for HEP and anyone thinking about byte-level models for scientific data. The paper deserves a serious referee: the core question is interesting and the result is plausible, but the current evidence does not support the emergence narrative. I'd send it to review with an explicit request for numbers, error bars, convergence controls, and a precise tokenization spec. My own verdict would be conditional pending those changes.","headline":"A generic byte-level transformer matching specialized jet taggers at 10M events is an interesting and plausible result, but the scaling/emergence claim is confounded with compute and needs major revision.","tokens_in":14301,"tokens_out":3024,"would_cite":false,"duration_ms":36802,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A task-agnostic byte-level transformer matches specialized jet-tagging models and jumps sharply past a data threshold.","keywords":["Large language models","Binary tokenization","Jet origin identification","Jet tagging","Scaling laws","Emergence","Numerical data","Particle physics"],"falsifier":"Retrain BBT-Neutron at dataset sizes from 100 to 10 million events with early stopping or with epochs scaled by dataset size, and plot final validation accuracy against data volume; if the sharp jump smooths away or shifts, the claimed emergence is an artifact of the fixed 30-epoch budget.","tokens_in":13299,"feed_emoji":"⚛️","tokens_out":6854,"duration_ms":58742,"temperature":0.7,"pith_summary":"This paper claims that a task-agnostic transformer trained on raw bytes can match specialist deep-learning models at Jet Origin Identification, the task of telling which quark, antiquark, or gluon initiated a jet in a particle collider. The proposed model, BBT-Neutron, uses binary tokenization, feeding numbers in as native byte sequences to preserve the quantitative structure that byte-pair encoding destroys. At ten million training events, its flavor-tagging efficiencies and charge-flip rates are comparable to ParticleNet and Particle Transformer. The paper also reports a sharp, S-shaped jump in BBT-Neutron's performance once the training set crosses a critical size, a behavior it reads as evidence for emergent, generalist capabilities in numerically intensive science.","feed_headline":"Byte-token transformer matches specialized jet taggers at scale","feed_subtitle":"A generic byte-level transformer matches physics-specific models on jet origin identification when data is large enough.","key_machinery":"The load-bearing mechanism is Binary Tokenization: every input, whether text, numbers, formulas, or images, is converted into byte sequences, with numbers optionally encoded as their native integer or float byte arrays rather than as character strings, and the bytes are grouped into fixed-length patches fed to a causal transformer decoder. Because each byte is a token and token IDs follow the byte layout, the same number always produces the same tokens and nearby numbers have nearby representations, which the paper argues preserves quantitative relationships that BPE fragments. The transformer uses rotary position embeddings and patch-level self-attention that also mixes information within a patch, letting a single left-to-right, sequence-to-sequence model act as a classifier without jet-specific structure.","core_discovery":"The central discovery is that domain-specific architecture is not required for state-of-the-art jet origin identification: a generic decoder-only transformer with causal attention, rotary position embeddings, and byte-level binary tokenization reaches the same 11-way classification quality as the leading specialist models once it sees enough data. The paper further claims that this generalist model shows a different scaling curve from the specialists. It performs near random guessing on small datasets, then jumps sharply above random after roughly ten thousand events for charge identification, and converges with ParticleNet and Particle Transformer around three million events. For bottom, charm, and strange quarks, BBT-Neutron needs roughly an order of magnitude more training data to match the specialists, and for bottom, strange, and up quarks its scaling curve trends above theirs at the largest sizes. The authors interpret the abrupt transition as the emergence of a generalist capability, analogous to emergent abilities in large language models, and attribute its absence in specialists to their built-in symmetry and interaction priors.","pith_inferences":["Inference: The emergence claim rests on an unverified fixed-budget assumption; the paper does not show convergence checks or learning curves, so the sharp jump could be small-dataset undertraining rather than a phase transition.","Inference: Re-plotting the scaling curves with early stopping or with training epochs scaled by dataset size would separate optimization effects from true emergent behavior; if the jump persists under matched compute, the generalist claim is much stronger.","Inference: Because binary tokenization claims to preserve numeric relations, a cheap transfer test is to benchmark BBT-Neutron on arithmetic and number-comparison tasks; standard LLM arithmetic failures would be expected to shrink or vanish if the mechanism works as described.","Inference: The comparison against permutation-invariant specialists suggests a testable hypothesis: adding permutation augmentation to BBT-Neutron's left-to-right input should reduce the data threshold, showing that symmetry prior, not architecture class, drives the gap."],"forward_implications":["At ten million events, BBT-Neutron's 11-category confusion matrix, flavor-tagging efficiencies, and charge-flip rates match ParticleNet and Particle Transformer, so specialist architecture is not necessary for strong JoI performance at that scale.","The sharp performance jump between ten and thirty thousand events means there is a minimum data volume below which a byte-level generalist is not competitive, making dataset size a key operating parameter.","The same byte representation works for text, numbers, formulas, and images, so the architecture can be pretrained on a mixed corpus and then applied to classification or regression with only an output-layer change.","For bottom, strange, and up quarks, BBT-Neutron's scaling curves continue to improve past the point where the specialists plateau, suggesting that further data may widen the generalist's advantage."],"supporting_citations":[{"why":"ParticleNet baseline: the specialized graph-network jet tagger whose performance BBT-Neutron claims to match.","marker":"[50]"},{"why":"Particle Transformer baseline: the specialized transformer jet tagger used for comparison and as an example of architecture without position encoding.","marker":"[46]"},{"why":"Emergent abilities of large language models: supplies the emergence interpretation the paper applies to its sharp scaling jump.","marker":"[40]"},{"why":"Byte-level model line of work: the byte-model approach BBT-Neutron builds on while training from scratch without patches-specific modules.","marker":"[42]"},{"why":"Rotary position embedding: the mechanism BBT-Neutron uses to encode relative position between numerical tokens.","marker":"[45]"},{"why":"Byte Pair Encoding: the tokenization method the paper argues fragments numbers and motivates binary tokenization.","marker":"[49]"},{"why":"Llama 3 single-digit tokenization: a contrast case for number representation that the paper says still breaks multi-digit continuity.","marker":"[17]"},{"why":"Data scaling laws in imitation learning: motivates the paper's focus on data-volume scaling rather than model-size scaling.","marker":"[41]"}],"fun_headline_variants":["No physics priors needed: byte transformer scales to match specialists","Generic transformer matches jet taggers without physics assumptions","Byte tokenization helps generic transformer rival jet taggers","Scaling a no-priors transformer to particle physics accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one fixed training budget (30 epochs, batch size 512) is a fair comparison at every dataset size; if the small-dataset runs are merely undertrained, the sharp jump is an optimization artifact rather than an emergent transition.","fun_headline_variants_meta":{"raw":{"variants":["No physics priors needed: byte transformer scales to match specialists","Generic transformer matches jet taggers without physics assumptions","Byte tokenization helps generic transformer rival jet taggers","Scaling a no-priors transformer to particle physics accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001105,"raw_usage":{"total_tokens":4625,"prompt_tokens":982,"completion_tokens":3643,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":3578}},"tokens_in":598,"tokens_out":3643,"duration_ms":27809,"temperature":1.0,"reasoning_tokens":3578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:30:05.255694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain BBT-Neutron at dataset sizes from 100 to 10 million events with early stopping or with epochs scaled by dataset size, and plot final validation accuracy against data volume; if the sharp jump smooths away or shifts, the claimed emergence is an artifact of the fixed 30-epoch budget.","supporting_citations":[{"cited_title":"https://arxiv.org/abs/2402","cited_arxiv_id":null,"evidence_quote":"Byte-level model line of work: the byte-model approach BBT-Neutron builds on while training from scratch without patches-specific modules."},{"cited_title":"https://arxiv.org/abs/2410","cited_arxiv_id":null,"evidence_quote":"Data scaling laws in imitation learning: motivates the paper's focus on data-volume scaling rather than model-size scaling."}],"review_version":1}