{"id":"3155c038-d8c2-42a8-be60-444b9309d060","arxiv_id":"2412.07867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An order-invariant BERT-style transformer pre-trained on masked particle 4-vectors improves dileptonic ttbar reconstruction and can be fine-tuned for toponium and initial-state classification.","lead":"Bumblebee is a transformer model trained to predict hidden particle 4-vectors in simulated top quark pair events, and it reports 10-20% better dileptonic top reconstruction resolution than a supervised transformer. The paper matters because it tests whether a BERT-style approach can be adapted to unordered particle physics events and then fine-tuned for new particle searches.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 10-20% reconstruction gain is attributed to pre-training, but the supervised Transformer baseline is unspecified and no from-scratch Bumblebee control is shown; the gain could be capacity or tuning rather than the masked-objective pre-training.","rationale":"The central claim is that masked pre-training produces a generalizable representation that improves reconstruction. But because the pre-training objective is essentially the same as the evaluation reconstruction task, and the baseline is under-specified, the observed improvement cannot be assigned to pre-training without a matched control. The paper does include a plausible mechanism and the architecture is clearly described, but the missing baseline and the absence of a from-scratch control leave the headline claim unverified. I agree with the reader's weakest assumption: the supervised Transformer comparison is the load-bearing point. The current CONDITIONAL verdict is appropriate; my concern does not change it. I do not see an internal contradiction that would justify REJECT; the issue is incomplete evidence. A secondary inconsistency is the toponium AUROC (0.877) being below the supervised Transformer (0.934), which undercuts the 'state-of-the-art' language but is not the central claim of the paper.","tokens_in":8094,"tokens_out":4178,"duration_ms":41274,"concrete_test":"Run a controlled comparison on the same 7M ttbar sample: (1) train a supervised Transformer with Bumblebee's exact architecture (L=8, dmodel=768, A=16, 57M) and the same optimizer schedule directly on the generator 4-vector MSE reconstruction loss; (2) train Bumblebee from random initialization on the same regression loss without the masked pre-training; (3) compare m(ttbar) P84-P16 resolution on the same test set. If Bumblebee retains the 10-20% gain over both matched controls, the pre-training attribution is supported. If not, the reported gain is due to capacity, initialization, or tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that pre-training improves dileptonic ttbar m(ttbar) reconstruction resolution by 10-20% (Section 5, Fig. 2C). This claim is not testable from the manuscript: the 'supervised Transformer' baseline has no stated architecture, parameter count, training data, loss, or hyperparameters. Bumblebee is (L=8, dmodel=768, A=16, 57M; Section 3.1), but it is unknown whether the baseline matches this capacity. The comparison is also confounded by the pre-training objective itself: Bumblebee is pre-trained with a masked-prediction MSE on exactly the generator-level 4-vectors that define the reconstruction task, so the only clean control would be a same-architecture model trained from random initialization on the same reconstruction loss. No such from-scratch ablation appears; Section 7 ablates input embeddings, not pre-training. Additionally, on toponium discrimination Bumblebee (AUC 0.877) is below the supervised Transformer (AUC 0.934) in Fig. 2A, so the paper's downstream 'state-of-the-art' framing is not consistent, and the pre-training benefit is not demonstrated across tasks. The reconstruction claim therefore rests on an uncontrolled baseline comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Bumblebee, a BERT-inspired bidirectional transformer that omits positional encodings and embeds particle 4-vectors (pT, eta, phi, m) together with b-tag scores, level-type, modified PDG ID, and mask-status embeddings. The model is pre-trained on a masked-vector prediction task using dileptonic ttbar Monte Carlo events, where both generator- and reconstruction-level objects are included. The authors claim that pre-training improves the resolution of the reconstructed ttbar invariant mass by 10-20% relative to a supervised transformer (Fig. 2C), that fine-tuning gives toponium discrimination AUROC of 0.877 (Fig. 2A) and initial-state classification AUROC of 0.625 (Fig. 2B), and that an ablation study shows the PDG ID embedding is the most important input component (Fig. 3).","tokens_in":8379,"tokens_out":2690,"duration_ms":27543,"significance":"If the central claims hold, the paper offers a conceptually interesting step toward a permutation-invariant, physics-aware foundation model for LHC analyses: the embedding design is simple, the pre-training task is well matched to a real reconstruction problem (dileptonic ttbar with two neutrinos), and the reported improvement in m(ttbar) resolution at high invariant mass is potentially valuable for BSM resonance searches. The paper also honestly states its limitations (focus on one process, no photons yet) and provides an ablation of input embeddings. However, the significance is currently limited by the lack of a fully specified baseline and the absence of statistical uncertainties, which prevents the reader from verifying that the 10-20% improvement is real and attributable to pre-training rather than to model capacity or training choices.","major_comments":[{"comment":"The central claim of a 10-20% improvement in m(ttbar) resolution is not testable as presented because the 'supervised Transformer' baseline is not specified: its architecture, parameter count, input representation, loss function, training data size, and hyperparameters are all absent. Bumblebee is (L=8, dmodel=768, A=16, 57M parameters), but the reader cannot verify whether the baseline has comparable capacity. More importantly, there is no same-architecture control trained from random initialization on the reconstruction task; without such a control, the improvement could be due to model capacity, optimization, or regularization rather than the masked pre-training objective. Please add a complete description of the baseline and, crucially, a from-scratch Bumblebee trained directly on the reconstruction loss.","section":"Section 5, Fig. 2C"},{"comment":"The abstract's statement that Bumblebee 'excels in downstream tasks, including toponium discrimination (AUROC 0.877)' is contradicted by the data in Fig. 2A, where the supervised Transformer achieves AUROC 0.934 and the DNN achieves 0.831. The paper should either temper the claim of excelling for toponium or provide an explanation (e.g., statistical uncertainty, different operating points, or a discussion of why the supervised Transformer is a stronger baseline for this task). As written, the downstream claim is inconsistent with the presented ROC curves.","section":"Abstract; Section 5, Fig. 2A"},{"comment":"No statistical uncertainties are given for the AUC values or for the P84-P16 resolution curves. With a 7M-event sample and a 15% test split, the resolution differences shown in Fig. 2C may be statistically meaningful, but the reader cannot assess whether the 10-20% improvement is significant, particularly in the high-mass bins where the curve appears to cross. The initial-state classification AUC of 0.625 versus 0.601 for the supervised Transformer is a small difference that could easily be within uncertainty; please add error bars or confidence intervals.","section":"Section 5, Fig. 2C"}],"minor_comments":[{"comment":"The masking probability is described as '(1/nparticles)%' which is dimensionally confusing; presumably each particle is masked with probability 1/n_particles, but this should be stated unambiguously as a fraction or percentage.","section":"Section 3.3"},{"comment":"The text states that b-tag scores indicate a likelihood, but the figure shows only binary values 0 and 1; clarify whether b-tag scores are continuous and, if so, how they are embedded. The figure caption also mixes PDG ID values and level-type values in a way that is hard to parse.","section":"Section 3.2, Fig. 1"},{"comment":"In the toponium discrimination description, the paper says a masked vector (1,0,0,0,0) is added for signal and (0,0,0,0,0) for background, and that the model is trained with weighted binary cross-entropy; it is unclear how a 5-vector target is used in a scalar classification loss. Please specify the loss computation on this vector.","section":"Section 5"},{"comment":"The toponium sample is generated with a toy model and the ttbar sample with POWHEG, but the description does not state whether the two samples are matched in kinematic acceptance or whether the same object/event selection is applied to both; this matters for interpreting the discrimination AUC. Please clarify.","section":"Section 4"},{"comment":"Reference [37] (Schick and Schütze) is cited for the 'Cloze task', but the original BERT paper (Devlin et al., 2018) introduced the masked-language-model objective; consider citing the original source or explaining why the Cloze-task reference is the relevant one.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a short workshop-style paper (NeurIPS ML4PS) and the authors should be given the opportunity to add the missing baseline details and a from-scratch control. If the authors cannot provide the baseline architecture or a same-architecture no-pre-training ablation, the central claim would be unsupported. In that case, a revision that reframes the paper as a proposal plus preliminary results might be more appropriate than the current 'outperforms state-of-the-art' framing. The discrepancy between the abstract's 'excels' and Fig. 2A should be resolved during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is clean: a BERT-style transformer with no positional encodings, fed joint generator- and reconstruction-level 4-vectors, pre-trained with a masked prediction objective. That combination is genuinely new relative to the cited MPM-on-sets work, and the application to dileptonic ttbar reconstruction is a sensible stress test. The writing is clear and the architecture choices are well motivated.\n\nThe real problem is the central claim. The 10-20% resolution improvement in m(ttbar) is measured against a 'supervised Transformer' whose architecture, parameter count, training data, loss, and hyperparameters are never given. We know Bumblebee is 57M params, L=8, d=768, A=16. Is the baseline the same size? Same training schedule? Without that, the gain could be capacity or tuning rather than pretraining. The clean control—a same-architecture model trained from scratch on the reconstruction loss—is absent. Section 7 ablates input embeddings, not pretraining. So the paper's headline result is currently uncontrolled.\n\nThe circularity concern is also fair. The reconstruction evaluation is essentially the same masked 4-vector prediction task used in pretraining, so good test-set performance is partly an in-distribution check of the objective. That does not kill the result—pretraining on a large corpus and then fine-tuning on the same task is a legitimate regime—but it undercuts the claim that Bumblebee learns a reusable representation.\n\nThere is also a consistency issue. The abstract says Bumblebee 'excels' in downstream tasks, but on toponium discrimination the supervised Transformer gets AUC 0.934 versus Bumblebee's 0.877 (Fig. 2A). That is a clear gap. The initial-state classification win (0.625 vs 0.601) is modest and the paper itself calls it a first attempt. So the downstream benefits are mixed, and the abstract overstates them.\n\nWhat is good: the permutation-invariance argument is physically sound, the embedding design is thoughtful, and the authors are transparent about the event-topology limitation. They also used a withheld test set. But no code or data are released, which makes the missing baseline harder to excuse.\n\nMy recommendation: this deserves a serious referee, but with the expectation of major revision. The authors need to specify the baseline fully, add a from-scratch ablation of Bumblebee itself, report statistical uncertainties on the AUCs and resolution numbers, and tone down the abstract to match Figure 2A. If those are fixed, the 10-20% claim could be credible. Right now it is not.\n\nI would bring this to a reading group as a case study in pre-training evaluation, but I would not cite it yet.","headline":"Bumblebee is a plausible permutation-invariant pretraining recipe, but the headline 10-20% gain is not yet attributable to pretraining because the baseline is unspecified and the from-scratch control is missing.","tokens_in":8883,"tokens_out":1126,"would_cite":false,"duration_ms":12270,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bumblebee, a transformer without positional encodings pre-trained on masked particle 4-vectors, improves top-pair mass reconstruction by 10-20% and fine-tunes to toponium (AUROC 0.877) and initial-state classification (AUROC 0.625).","keywords":["foundation model","transformer","masked pre-training","top quark reconstruction","permutation invariance","toponium","LHC","missing transverse energy"],"falsifier":"Train the same 8-layer, 57M-parameter transformer on the reconstruction task from random initialisation, with the same dataset, optimizer, and number of update steps as pre-training, and measure $m(t\\bar{t})$ resolution; if the randomly initialised model matches or beats the reported 10-20% gain, the improvement is not attributable to masked pre-training.","tokens_in":7940,"feed_emoji":"⚛️","tokens_out":12834,"duration_ms":122728,"temperature":0.7,"pith_summary":"The paper tries to establish that a transformer borrowed from language modeling, stripped of positional encodings and fed particle 4-vectors as tokens, can act as a reusable foundation model for particle physics. On dileptonic top-pair events, pre-training with a masked-vector objective is claimed to improve the invariant-mass resolution of the reconstructed $t\\bar{t}$ system by 10-20% relative to a supervised transformer. The same pre-trained weights then fine-tune to two downstream tasks: distinguishing toponium from ordinary top-pair production (AUROC 0.877) and classifying the initial state that produced the pair (AUROC 0.625). The motivation is that collision events have no natural ordering, so a permutation-invariant model that learns the mapping from reconstructed detector data to generator-level kinematics should transfer to new-particle searches. If true, this points toward reusable pre-trained models for LHC analyses rather than task-specific networks trained from scratch.","feed_headline":"Masked-particle transformer sharpens top-pair mass by 10-20%","feed_subtitle":"A BERT-style model trained on masked particle 4-vectors also finds toponium and classifies the initial state.","key_machinery":"The load-bearing mechanism is masked 4-vector pre-training on permutation-invariant set inputs. Bumblebee is an 8-layer, 57M-parameter bidirectional transformer encoder without positional encodings; each token is the unweighted sum of a linear embedding of $(p_T, \\eta, \\phi, m, \\text{b-tag})$, a modified PDG-ID embedding, a level-type embedding, and a mask-status embedding. The Cloze-style objective forces the model to learn the event's kinematic topology, including the relation between invisible neutrinos, which appear only as missing transverse energy, and visible leptons and jets, and to learn the detector-level transformation from reconstruction to generator quantities. This same objective is then repurposed as the reconstruction head, and fine-tuning replaces it with classification tokens to perform new-particle discrimination.","core_discovery":"Bumblebee's central claim is that masked pre-training on particle kinematics is a sufficient self-supervised objective for learning transferable LHC physics. The model embeds each particle's five-dimensional vector $(p_T, \\eta, \\phi, m, \\text{b-tag})$ at both generator and reconstruction level, adds learned embeddings for particle identity, generator-versus-reconstruction level, and mask status, and omits positional encodings so the input is order-invariant. Pre-training randomly masks particle 4-vectors, for half the training one particle per event at probability $1/n_{\\text{particles}}$ and for the other half all 4-vectors at one level, minimizing mean squared error on the missing vectors; validation and testing focus on predicting generator-level 4-vectors from reconstruction-level information, framed as dileptonic top reconstruction. The paper reports a 10-20% resolution gain over a supervised transformer, fine-tuned AUROC 0.877 for toponium-versus-$t\\bar{t}$ discrimination, and AUROC 0.625 for gluon-gluon versus quark-antiquark initial-state classification, the positive class being gluon-gluon.","pith_inferences":["A clean attribution test would be to train the same 8-layer, 57M-parameter architecture from random initialisation on the reconstruction task with identical data and compute; the current comparison does not specify the supervised transformer's size or training, so part of the reported gain could in principle come from capacity rather than pre-training.","If the representation is as general as claimed, the same pre-training recipe should transfer to other multi-neutrino topologies such as $t\\bar{t}H$ and four-top production, where missing momenta dominate the reconstruction challenge; the authors name these as future work.","The 0.625 initial-state AUROC suggests the model captures a weak but real kinematic handle on the production mechanism; combining its representation with spin-correlation observables might push this toward practical use."],"forward_implications":["The masked pre-training objective itself doubles as the reconstruction task: summing the predicted generator-level daughter 4-vectors gives a 10-20% improvement in $m(t\\bar{t})$ resolution over a supervised transformer, which is directly relevant for resonant $t\\bar{t}$ searches.","A single set of pre-trained weights transfers to two classification tasks, toponium-versus-$t\\bar{t}$ discrimination and initial-state classification, without changing the input representation.","Because the input representation treats each particle as an unordered token with no positional encoding, the model avoids imposing an arbitrary ordering convention on collision events.","The embedding procedure does not exclude photons, so the same architecture can be applied to final states with photons even though the demonstration uses dileptonic $t\\bar{t}$ events."],"supporting_citations":[{"why":"Supplies the bidirectional masked pre-training strategy and BERT-style foundation-model design that Bumblebee adapts to particle 4-vectors.","marker":"(19)"},{"why":"Supplies the Cloze task, the masked-prediction objective used to pre-train Bumblebee on particle 4-vectors.","marker":"(37)"},{"why":"Supplies the transformer encoder architecture that Bumblebee modifies by removing positional encodings.","marker":"(43)"},{"why":"Generates the 7M next-to-leading-order $t\\bar{t}$ Monte Carlo events used for pre-training, validation, and testing.","marker":"(23, 22, 34, 5)"},{"why":"Generates the toponium Monte Carlo sample used as the signal for the discrimination benchmark.","marker":"(6)"},{"why":"Performs parton showering and hadronization for both the $t\\bar{t}$ and toponium samples.","marker":"(40)"},{"why":"Provides the fast CMS detector simulation that turns generator-level events into reconstruction-level inputs.","marker":"(18)"},{"why":"Defines the object and event selection criteria from CMS dileptonic $t\\bar{t}$ analyses applied to all inputs.","marker":"(17, 13)"},{"why":"Marks the downstream motivation: improved $m(t\\bar{t})$ resolution at high mass is needed for resonant $t\\bar{t}$ searches.","marker":"(14)"},{"why":"Provides the toponium width of about 3 GeV that makes the toponium discrimination task a meaningful resolution benchmark.","marker":"(3)"}],"fun_headline_variants":["Bumblebee: BERT-style model sharpens top quark resolution by 10-20%","Masked-particle pretraining boosts toponium discrimination and top mass","Bumblebee: order-invariant transformer finds toponium and classifies initial states","Bumblebee improves top quark mass resolution by 10-20% and spots toponium","Self-supervised Bumblebee model finds toponium and sharpens top mass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 'supervised transformer' used for comparison in Fig. 2C is an equally capable model trained with the same data and effort; the paper never specifies that baseline's architecture, size, or training, so a weaker baseline could inflate the apparent pre-training gain.","fun_headline_variants_meta":{"raw":{"variants":["Bumblebee: BERT-style model sharpens top quark resolution by 10-20%","Masked-particle pretraining boosts toponium discrimination and top mass","Bumblebee: order-invariant transformer finds toponium and classifies initial states","Bumblebee improves top quark mass resolution by 10-20% and spots toponium","Self-supervised Bumblebee model finds toponium and sharpens top mass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001639,"raw_usage":{"total_tokens":6484,"prompt_tokens":881,"completion_tokens":5603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":5497}},"tokens_in":497,"tokens_out":5603,"duration_ms":37921,"temperature":1.0,"reasoning_tokens":5497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:27:59.564306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 8-layer, 57M-parameter transformer on the reconstruction task from random initialisation, with the same dataset, optimizer, and number of update steps as pre-training, and measure $m(t\\bar{t})$ resolution; if the randomly initialised model matches or beats the reported 10-20% gain, the improvement is not attributable to masked pre-training.","supporting_citations":[],"review_version":1}