{"id":"4c199e08-9c0f-4775-ada9-6df0b0229849","arxiv_id":"2412.08134","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 330k-parameter RetNet-based model tags b-jets with validation AUC 0.956 on CMS Open Data, beating an MLP baseline and approaching larger state-of-the-art taggers trained on 100x more data.","lead":"The paper applies Retentive Networks, a type of sequence model, to identify jets originating from bottom quarks in high-energy collisions. It shows that a compact 330k-parameter model performs competitively with a standard MLP on simulated CMS data, suggesting a light-weight b-tagging option for resource-limited settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out test set: reported AUC/F1 are validation-selected estimates, so the claim that JetRetNet is an efficient alternative rests on an optimistic and possibly biased performance measure.","rationale":"The reader's identified weakest assumption about causal masking is a legitimate conceptual concern, but it is not the most load-bearing for the stated claim. If pT ordering does not provide a physically meaningful causal signal, the retention mask still imposes a fixed ordering and the model can function as a set-based architecture with a particular inductive bias; the empirical performance claim could survive even if the causal interpretation is weak. The more direct vulnerability is the absence of a held-out test set and the use of validation-based model selection. The reader did mention the missing test set in the rationale, so there is partial agreement, but the reader chose the pT-ordering assumption as the weakest point. I elevate the test-set issue because it attacks the validity of the performance numbers themselves, which are the sole evidence for the central claim. The proposed concrete test would settle this by providing an unbiased estimate with seed-level uncertainty, and it is a straightforward revision that does not require new data or expensive retraining beyond what the authors already performed. Therefore, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":7268,"tokens_out":8017,"duration_ms":85782,"concrete_test":"Hold out a separate test set from the current 1M validation jets (e.g., 500k jets), use the remaining 500k as the validation set for early stopping, retrain JetRetNet and the MLP from scratch with the same hyperparameters and three random seeds, and report the mean and standard deviation of AUC and F1 on the untouched test set. If the test-set AUC of JetRetNet exceeds the MLP by more than the seed spread and the absolute AUC is consistent with the paper's validation number, then the central claim survives; if the test-set gap vanishes or reverses, the reported advantage was an artifact of validation-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that JetRetNet is an efficient alternative for b-jet tagging rests on the reported performance numbers (validation AUC 0.955, F1 0.913) and on the comparison with an MLP. These numbers come from the validation set, not from a held-out test set. Section 2.3 allocates 4M jets to training and 1M jets to validation; no separate test set is mentioned. Section 3.3 describes early stopping monitored on validation loss, and Figure 7 explicitly labels 'max val. AUC (0.955)', indicating that the displayed validation metric is the best over epochs. Selecting a model based on validation metrics makes the reported estimate optimistically biased. The bias need not cancel in the MLP comparison: JetRetNet has 330k trainable parameters, while the MLP parameter count is not reported and its architecture (four layers of 128 neurons) likely has considerably fewer parameters, so the two models have different capacities and different propensities to overfit the validation set. Thus the claimed advantage over the MLP could be an artifact of validation-set selection rather than a real generalization gain. Without an unbiased performance estimate on an untouched test set, the paper's suggestion that RetNet can be used as an efficient alternative for b-jet tagging is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies a Retentive Network (RetNet) to b-jet tagging on a simulated CMS Open Data t-tbar sample, using global jet, track, and secondary-vertex features. The proposed JetRetNet model is trained on 4 million jets and validated on 1 million jets, with reported validation AUC around 0.956 and F1 around 0.914. It is compared with a same-hyperparameter MLP baseline and with externally published DeepJet and Particle Transformer results, and the authors conclude that RetNet-based models are a promising low-parameter alternative. The central evidence is the validation-based comparison; no separate held-out test set is used.","tokens_in":7519,"tokens_out":7461,"duration_ms":78650,"significance":"If the reported performance estimates were unbiased, the paper would be a useful demonstration that a compact sequence model can approach state-of-the-art b-tagging performance with limited data. The use of public CMS Open Data, the clear description of the input features, and the controlled MLP baseline are commendable. The authors also correctly label the DeepJet/ParT comparison as unfair, which is an appropriate caveat. However, the headline numbers are selected from the validation set, and the manuscript lacks a held-out test evaluation, uncertainty estimates, and a test of the causal-ordering assumption that is central to the RetNet architecture. These gaps currently prevent the paper from supporting its main efficiency claim.","major_comments":[{"comment":"The paper uses only a train/validation split (4M/1M jets; §2.3) and selects the model by early stopping on validation loss (§3.3). Figure 7 explicitly quotes the maximum validation AUC (0.955) and F1 (0.913), so these are validation-selected estimates, not unbiased generalization estimates. Since the abstract's central claim rests on exactly these numbers, the manuscript needs an untouched test set and a final evaluation on it, with the model-selection procedure fixed, before the efficiency claim can be supported.","section":"§2.3, §3.3, Fig. 7"},{"comment":"The MLP baseline is described as using the same hyperparameters (four layers, hidden dimension 128), but its trainable parameter count is not reported, and no repeated training runs or error bars are given. Because JetRetNet has 330k parameters, a capacity mismatch could explain part of the ROC advantage. Please report the MLP parameter count, train both models over multiple seeds, and show mean and standard deviation for AUC, F1, and the ROC curves.","section":"§3.3, Fig. 8"},{"comment":"The retention mechanism applies causal masking through the matrix D in Eq. (2), while §2.3 orders tracks and SVs by decreasing pT as a proxy for time order. For jets, pT ordering is not a physically causal temporal sequence; if the causal mask simply imposes an arbitrary order, the RetNet-specific benefit over set-based aggregation is not demonstrated. A concrete test would be to compare against the same architecture with a bidirectional or no mask, or against a permutation-invariant baseline, and to report sensitivity to the track ordering.","section":"§2.3, Eqs. (1)–(2)"},{"comment":"The fixed maximum of 16 tracks and 5 SVs is applied by clipping or padding, yet the paper reports that only 2.63% of jets have fewer than 17 tracks and only 0.012% have fewer than 6 SVs. This means that the large majority of jets have at least 17 tracks and at least 6 SVs, so most jets are truncated and a substantial part of the low-level information is discarded in nearly every jet. The authors should report the fraction of jets with more than 16 tracks and more than 5 SVs, and investigate how performance changes with the maximum length.","section":"§2.3"},{"comment":"The conclusion that RetNet models are an efficient alternative for limited computational resources is supported only by the 330k parameter count; no training time, inference time, memory footprint, FLOPs, or throughput measurements are reported. An efficiency claim requires at least one compute metric beyond parameter count, especially because the model is compared with an MLP baseline whose computational cost is not characterized.","section":"Abstract, §4"}],"minor_comments":[{"comment":"The phrase \"alternative for b-jet with limited computational resources\" appears to be missing the word \"tagging\" after \"b-jet.\"","section":"Abstract"},{"comment":"The text above Eq. (1) contains the typo \"Querry\"; it should be \"Query.\"","section":"Eq. (1)"},{"comment":"The last two panels of Figure 3 are both labeled \"SV dxy\"; one is likely intended to be SV dxy significance and the other SV dlen significance, matching the features listed in §2.2.","section":"Fig. 3"},{"comment":"The caption of Figure 5 describes the panels as \"Number tracks (right) and number of SV's (left)\", but the surrounding text and figure layout suggest the left/right labels may be reversed; please confirm the panel order.","section":"Fig. 5"},{"comment":"The sentence \"Also, both performances are given to make sure there are no discrepancies between the training and validation datasets\" is ambiguous; it should state explicitly which curves correspond to training and which to validation, and whether the MLP is included in those figures.","section":"§4, Figs. 9–10"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about the unfair comparison with DeepJet and ParT and about its limitations, and I see no integrity concern. The main statistical obstacle is the absence of a held-out test set and uncertainty quantification; the missing order-sensitivity test for the RetNet causal mask is also important because it bears directly on the claimed novelty. These issues can be addressed within the scope of the manuscript, so major revision is appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuine new application: Retentive Networks for b-jet tagging, with a custom architecture (JetRetNet) that processes tracks, secondary vertices, and global jet features separately, trained on a public CMS simulation with 330k parameters. Second, the central claim that this is an efficient alternative for limited-resource use rests on validation-set metrics, not on a held-out test set. That is the soft spot to worry about.\n\nThe paper does real things well. The data preprocessing is documented thoroughly, the train/validation split is clear, and the MLP baseline is trained with the same hyperparameters as a controlled comparison. The authors also explicitly label the DeepJet/ParT comparison as unfair because of the dataset size difference, which is the right call. The architecture is a reasonable assembly of existing components, and the pT-ordered track/SV sequences are a defensible design choice.\n\nThe weaknesses are proportionate. The stress-test note is correct: Section 3.3 describes early stopping on validation loss, Figure 7 reports the max validation AUC (0.955) and F1, and there is no separate test set. That makes the reported performance optimistically biased. The MLP comparison is the key evidence for efficiency, but the MLP parameter count is not reported, so the capacity difference between the two models is unknown. If the MLP has substantially fewer parameters, its validation-selected performance could be worse for reasons unrelated to the RetNet mechanism. The paper needs a held-out test set and ideally repeated runs to give error bars. A second concern is the causal masking assumption: pT ordering is a proxy for time, but the retention mask imposes a one-directional decay. If that mask doesn't actually capture useful structure, the model effectively becomes a set-based model, and the RetNet-specific advantage is unsubstantiated. This isn't a fatal problem, but it should be tested or at least discussed.\n\nOverall, this is a solid engineering study with honest limitations. The novelty is modest, but the execution is careful and the dataset is public. The claimed advantage over an MLP and the efficiency story are not yet proven. Still, a good referee could get this into publishable shape with a proper test set and more careful model comparison.\n\nI would accept this for peer review. The topic is relevant, the work is reproducible in principle, and the flaws are addressable. My own verdict would be conditional: require the held-out test set, metric uncertainty, and a reported MLP parameter count before I'd trust the efficiency claim. I would not cite it in my own work in its current form, but I'd bring it to a reading group to discuss the pT-ordering assumption in sequence models for jets.","headline":"A clean, honest application of RetNet to b-jet tagging on CMS Open Data, but the headline performance numbers are validation-selected and the MLP comparison isn't yet controlled enough to support the efficiency claim.","tokens_in":8024,"tokens_out":1656,"would_cite":false,"duration_ms":19674,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 330k-parameter retentive network tags b-jets with 0.956 AUC, edging out an MLP twin.","keywords":["b-jet tagging","retentive networks","jet flavour identification","low-level jet features","particle jets","deep learning in high-energy physics","CMS Open Data","tagging efficiency"],"falsifier":"Retrain JetRetNet on the same data with the order of tracks and secondary vertices randomly shuffled for every jet. If the validation AUC stays at about 0.956, the retention mask contributes nothing through ordering, and the paper's architectural claim collapses to a permutation-invariant set model.","tokens_in":7080,"feed_emoji":"⚛️","tokens_out":6316,"duration_ms":60527,"temperature":0.7,"pith_summary":"This paper proposes JetRetNet, a b-jet tagging model built on retentive networks, and claims it offers an efficient alternative for flavor tagging when computational resources or training data are limited. On a simulated sample of semileptonic top-quark pair events, JetRetNet reaches a validation AUC of 0.956 and F1 of 0.914 with about 330,000 trainable parameters, outperforming a same-shaped MLP baseline at every misidentification rate. Against much larger state-of-the-art taggers, it does not win at any working point, but remains competitive at tight mistag rates despite training on two orders of magnitude fewer jets. The authors read this as evidence that retentive networks can capture the track and secondary-vertex structure needed for b-tagging at low computational cost.","feed_headline":"RetNet b-jet tagger hits 0.956 AUC with 330k parameters","feed_subtitle":"A compact retentive-network model beats an MLP twin and nearly matches much larger taggers on simulated data","key_machinery":"The load-bearing mechanism is the retention module from retentive networks, which computes Key, Query, and Value matrices and multiplies their product by an exponential-decay matrix with a causal mask, so earlier sequence positions fade while later ones remain visible. It is implemented as a multi-scale retention module with per-head decay scales, a group norm, and a swish gate. JetRetNet applies two such modules in parallel, one over sorted track features and one over sorted secondary-vertex features, then concatenates their outputs with global jet features before a final feed-forward classifier. Because the retention mask imposes an order, the paper must supply an ordering, and it does so by sorting tracks and secondary vertices by decreasing transverse momentum as a proxy for the experimentally inaccessible time order.","core_discovery":"The central discovery claimed is that a compact retentive-network architecture, JetRetNet, can learn b-jet discrimination from low-level jet constituents about as well as an MLP with the same hyperparameters and nearly as well as much larger models. The model processes tracks and secondary vertices through separate multi-scale retention blocks, appends global jet features, and classifies with a small feed-forward head. On the held-out validation sample it reaches AUC 0.955-0.956 and F1 0.913-0.914, with training and validation curves nearly overlapping, and it maintains an advantage over the MLP across loose, medium, and tight misidentification working points in both b-versus-light and b-versus-c comparisons. The authors explicitly call the comparison with DeepJet and Particle Transformer unfair because of the dataset-size gap, and report that JetRetNet does not surpass them at any working point, yet it comes closest at the tight working point, which is where a low-resource tagger would be most useful.","pith_inferences":["The causal mask is the fragile part: because jet constituents have no true time order, if randomly permuting the pT-sorted input leaves performance unchanged, JetRetNet's gain over the MLP would not come from the retention sequence at all, and the model would reduce to a permutation-invariant set architecture.","The same 330k-parameter design could be applied to other object-tagging tasks where constituent ordering is arbitrary, such as quark-gluon discrimination or boosted-W tagging, and a permutation-invariant variant would be a useful control benchmark.","Since the paper identifies dataset size rather than architecture as the main handicap, training the same model on the full public simulated dataset might close more of the gap to DeepJet and Particle Transformer without changing the network.","The near-overlap of training and validation curves suggests the model is capacity-limited rather than data-limited, so increasing the hidden dimension or number of RetNet blocks could raise the performance ceiling without immediately hurting generalization."],"forward_implications":["At the claimed 0.956 validation AUC, JetRetNet operates within reach of taggers used in physics analyses while using roughly 330k parameters and modest training data, suggesting low-resource experiments can deploy a performant tagger.","Because JetRetNet beats the same-hyperparameter MLP at all misidentification rates, the paper implies that the retention-style sequence processing, not raw model size, contributes the edge in b-jet discrimination.","At the tight working point of 0.001 misidentification rate, the gap to much larger models narrows, so the architecture may be most useful for high-purity b-jet selection with small statistics.","The training and validation curves nearly overlap, indicating that the model is not visibly overfitting the training sample even with early stopping, which supports the claim that its capacity is being used productively.","A model of this size can be trained on a data slice of four million jets, so the same architecture is a plausible starting point for experiments that lack the multi-hundred-million-jet datasets used by the larger taggers."],"supporting_citations":[{"why":"Supplies the retentive-network architecture and the retention mechanism with causal masking and exponential decay that JetRetNet is built on.","marker":"[12]"},{"why":"Defines the DeepJet tagger used as the state-of-the-art comparison in the ROC curves.","marker":"[9]"},{"why":"Defines the Particle Transformer tagger used as the state-of-the-art comparison in the ROC curves.","marker":"[11]"},{"why":"Provides the simulated semileptonic top-pair dataset on which JetRetNet and the MLP are trained and validated.","marker":"[13]"},{"why":"Motivates the choice to use low-level track and secondary-vertex features rather than high-level summaries alone.","marker":"[15]"},{"why":"Defines the anti-kT jet clustering algorithm and cone size used to build the jets fed into the model.","marker":"[14]"}],"fun_headline_variants":["Compact RetNet b-jet tagger rivals much larger models","330k-parameter RetNet keeps up with heavyweight taggers","Small RetNet b-jet tagger beats MLP twin at all working points","RetNet: a lightweight b-jet tagger competitive with big models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that sorting jet constituents by decreasing momentum gives the retention mask a meaningful order; if any ordering performs identically, the model's sequence machinery is not actually being used.","fun_headline_variants_meta":{"raw":{"variants":["Compact RetNet b-jet tagger rivals much larger models","330k-parameter RetNet keeps up with heavyweight taggers","Small RetNet b-jet tagger beats MLP twin at all working points","RetNet: a lightweight b-jet tagger competitive with big models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000931,"raw_usage":{"total_tokens":3974,"prompt_tokens":919,"completion_tokens":3055,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":2978}},"tokens_in":535,"tokens_out":3055,"duration_ms":23352,"temperature":1.0,"reasoning_tokens":2978,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:09:59.669500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain JetRetNet on the same data with the order of tracks and secondary vertices randomly shuffled for every jet. If the validation AUC stays at about 0.956, the retention mask contributes nothing through ordering, and the paper's architectural claim collapses to a permutation-invariant set model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DeepJet tagger used as the state-of-the-art comparison in the ROC curves."},{"cited_title":"Cms open data primary datasets, 2012","cited_arxiv_id":null,"evidence_quote":"Provides the simulated semileptonic top-pair dataset on which JetRetNet and the MLP are trained and validated."},{"cited_title":"Machine learning algorithms for b-jet tagging at the atlas experiment, 2017","cited_arxiv_id":null,"evidence_quote":"Motivates the choice to use low-level track and secondary-vertex features rather than high-level summaries alone."},{"cited_title":"The anti-ktjet clustering algorithm","cited_arxiv_id":null,"evidence_quote":"Defines the anti-kT jet clustering algorithm and cone size used to build the jets fed into the model."}],"review_version":1}