Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Learning Symmetry-Independent Jet Representations via Jet-Based Joint Embedding Predictive Architecture

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read J-JEPA, a jet-based joint embedding predictive architecture that trains without hand-crafted augmentations, learns jet representations whose finetuning outperforms training from scratch on top-quark tagging, especially when labeled data…

desk verdict A credible first JEPA for jets with real code, but the central finetuning claim is shakier than the text admits. read the letter →

arxiv 2412.05333 v1 pith:G4NZ7EAS submitted 2024-12-05 hep-ph cs.LGhep-exphysics.data-an

Subash Katel , Haoyang Li , Zihan Zhao , Raghav Kansal , Farouk Mokhtar , Javier Duarte This is my paper · ORCID
classification hep-phcs.LGhep-exphysics.data-an
keywords self-supervisedlearningjettaggingjointembeddingpredictivearchitecturemaskedsubjetpredictiontopquarktransformersparticlephysicspretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces J-JEPA, a self-supervised method that trains a transformer to predict the internal representations of randomly masked subjets inside a particle jet from the representations of the remaining context subjets, using the subjets' coordinates as joint information. The goal is to learn jet representations that transfer to downstream tasks without hand-crafted augmentations, which normally build task-specific symmetries into the model. The paper demonstrates that finetuning a J-JEPA-pretrained model on top-quark tagging outperforms training the same architecture from scratch, with the largest gains when only 10% of the labeled training data is used. If this holds, J-JEPA offers a route toward a single pretrained jet representation that can be adapted to multiple physics tasks without per-task augmentation design.

What carries the argument

The central object is the J-JEPA architecture itself: a context encoder and a target encoder, both subjet transformers (SjTs), plus a smaller predictor transformer. The context encoder processes only the unmasked subjets; the target encoder processes the full jet but only its outputs on the masked subjets serve as prediction targets; the predictor takes the context representations plus a spatial embedding of each target subjet's position (pseudorapidity and azimuth) and must match the target representations under an L2 loss. The target encoder's parameters are an exponential moving average (EMA) of the context encoder's parameters, which the paper identifies as the mechanism that prevents informational collapse. Two further design choices carry the argument: masking is applied at the encoder outputs rather than the inputs, so both encoders see the full jet, and the predictor includes an information bottleneck to force the context representations to be informative.

What would settle it

Measure the effective rank of the target encoder's output on a held-out sample of jets and compare the rank and the finetuned tagging accuracy for a model trained with EMA versus one trained with the target encoder updated directly from the context encoder; if removing EMA leaves both the representation rank and the finetuned accuracy essentially unchanged, the assertion that EMA is crucial for J-JEPA's success is refuted.

Watch

Extended reading notes

Core claim

The central claim is that a jet-based joint embedding predictive architecture (J-JEPA) learns useful, symmetry-independent jet representations by predicting the latent representations of randomly selected target subjets from context subjet representations, conditioned on the target subjets' positions. The paper shows that finetuning the target encoder of a J-JEPA-pretrained subjet transformer (SjT) for top-quark tagging outperforms training the same SjT from scratch on identical labeled data, in both accuracy and background rejection at 50% signal efficiency. The advantage is larger when the finetuning set is 10% of the labeled top-quark dataset (about 120k jets) than when the full set (785,767 jets after filtering) is used. This result is reported for two subjet-embedding designs (a plain MLP and one using class-attention blocks) and two ways of aggregating subjet representations (flattening and class attention), with standard deviations from five random initializations.

Load-bearing premise

The load-bearing premise is that the exponential-moving-average update to the target encoder prevents representation collapse, so that the L2 prediction loss actually drives the model to learn diverse, physically meaningful subjet embeddings rather than a trivial constant output.

Editorial extensions

If this is right

  • Finetuning a J-JEPA-pretrained encoder yields higher top-tagging accuracy and background rejection than the same architecture trained from scratch, with the improvement growing as the labeled sample size shrinks.
  • Because pretraining uses no data augmentations, the same pretrained checkpoints can be reused for downstream tasks that require different symmetries, without redesigning the pretraining stage.
  • Masking at the encoder-output level means both encoders see the full jet, so the learned representations are driven by full-jet semantics rather than by structural gaps from input masking.
  • Scaling pretraining to the full 100-million-jet unlabeled dataset is a direct next step that the paper expects to amplify these finetuning gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper does not run is whether the same pretrained encoder transfers to other jet tasks, such as quark-gluon discrimination or mass regression, with no change to the pretraining recipe.
  • The paper borrows the EMA-collapse-prevention argument from the image domain without reporting diagnostics of the learned jet representation space, so measuring its effective rank or nearest-neighbor structure would directly test that borrowed premise.
  • Because the attention-based subjet embedding outperforms the MLP embedding even from scratch, some of the reported gain may be attributable to the architecture rather than to pretraining; an ablation on a large labeled set with a fixed architecture would separate these factors.
  • Truly symmetry-independent pretraining does not remove the need for symmetry-specific supervision downstream: tasks requiring exact rotational or translational invariance would still need those invariances imposed during finetuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces J-JEPA, a self-supervised pretraining method for particle jet representations. A large-radius jet is reclustered into subjets; a random 30% of subjets are designated as targets, and a context encoder processes the remaining subjets. A predictor, conditioned on the positions of the target subjets, is trained to predict the target-subjet representations produced by a target encoder that is updated by exponential moving average. The authors pretrain on 1M JetClass top and QCD jets, then finetune on 10% or 100% of the Top Tagging dataset, comparing accuracy and background rejection against training the same architecture from scratch. They report that finetuning a pretrained model outperforms from-scratch training, especially with limited labeled data, and they ablate MLP-based versus attention-based subjet embeddings and flattening versus class-attention aggregation.

Significance. If the central empirical claim held, J-JEPA would provide a useful augmentation-free SSL initialization for jet tagging and a step toward a cross-task foundation model. The paper has several strengths: the code is publicly released, the evaluation is a direct comparison against from-scratch training on standard datasets, and the architecture ablations (embedding type and aggregation method) are informative. The paper is honest about reporting standard deviations and trial counts. However, the headline claim is an empirical performance comparison, and the data in Table 1 do not uniformly support it; the current evidence is mixed and in some configurations favors the from-scratch baseline. The contribution is therefore promising but needs a more careful and statistically disciplined statement of what is demonstrated.

major comments (3)
  1. The central claim that finetuning a J-JEPA-pretrained model outperforms training from scratch is not uniformly supported by Table 1. In the strongest architecture, AE-SjT-T with class-attention aggregation, the finetuned full-data rejection is 95.47 ± 1.83 versus 99.38 ± 2.80 from scratch, and the 10% accuracy is 88.82 ± 0.11 versus 88.84 ± 0.21, so the pretrained initialization is slightly worse on both of these entries. Several apparent gains are also within one standard deviation, for example AE-SjT-T Flatten 10% accuracy (88.94 ± 0.13 vs 88.92 ± 0.15) and AE-SjT-T Flatten full-data rejection (97.52 ± 1.71 vs 97.79 ± 3.90). I recommend reporting paired significance tests over the five random seeds and revising the claim to identify the configurations and dataset sizes for which the improvement is statistically meaningful.
  2. The phrase "especially for smaller dataset sizes" is not established by the reported results. For the two SjT-T configurations the full-data rejection gains are at least as large as the 10% gains (SjT-T Flatten: 19.36 at full versus 13.17 at 10%; SjT-T Cls Attn: 11.76 at full versus 8.76 at 10%), while for the AE-SjT-T configurations neither the 10% nor the full-data rejection gains are significant. The paper should either demonstrate a data-size interaction statistically or reframe the conclusion as configuration- and metric-dependent rather than a general small-data benefit.
  3. The paper states that EMA in the target encoder prevents informational collapse and that "this principle holds true for J-JEPA," but no diagnostic is provided to support that claim. Because the pretraining objective is an L2 regression into an EMA-updated representation space, a representation-collapse check (for example an effective-rank measure, a nearest-neighbor probe, or a linear-probe evaluation of the learned subjet embeddings) would substantially strengthen the mechanism invoked for why the method should work. This is not the primary weakness, but it is directly relevant to interpreting the downstream results.
minor comments (4)
  1. The caption contains a typo: "empbedding" should be "embedding."
  2. The text "the full 3 Top Tagging dataset" should be "the full Top Tagging dataset" with the footnote marker placed after the dataset name; the footnote should clarify whether the 785,767 jets are the training set only or include the validation set.
  3. The finetuning hyperparameters are not reported separately from the pretraining hyperparameters; please provide the finetuning learning rate, schedule, epoch count, and batch size, since these are needed to reproduce the central comparison.
  4. The "SEL" labels in Figure 1 are not defined in the caption or in the text; please define them or remove the unexplained label.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: J-JEPA finetuning comparison is an empirical measurement, not a derivation; no fitted input is renamed as a prediction, and no load-bearing self-citation chain is present.

full rationale

The paper's central claim—'We observe that finetuning a pretrained model outperforms training from scratch, especially for smaller dataset sizes' (Section 5)—is an empirical result obtained by comparing finetuned and from-scratch runs under fixed architectures in Table 1. There is no analytic derivation whose conclusion is equivalent to an input by construction. The pretraining objective minimizes an L2 loss between predicted and encoded target-subjet representations, while the downstream evaluation metrics (accuracy and 1/epsilon_B at epsilon_S = 0.5) are not the training loss; no fitted pretraining parameter is subsequently reported as the evaluation score. The EMA anti-collapse statement in Section 2 cites external Ref. [22] (Caron et al.) and is an empirical design assumption, not an imported uniqueness theorem; the paper's observation that 'this principle holds true for J-JEPA as well' is a testable empirical claim, not a circular one. References [15] and [16] are group/self citations but appear only as related work and software release and do not carry the argument. A possible concern is data overlap—pretraining used 500k top and 500k QCD JetClass jets and finetuning uses the Top Tagging dataset—but this is a dataset-selection choice, not a definitional reduction; it affects generalizability, not circularity. Whether Table 1 uniformly supports the headline (e.g., AE-SjT-T Cls Attn full-data rejection 95.47 +/- 1.83 vs 99.38 +/- 2.80 from scratch) is a correctness/statistical-claim issue, outside the circularity pass. Therefore no circular step is identified.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced; J-JEPA is a machine learning architecture. The main assumptions are about the EMA mechanism preventing collapse, the suitability of the pretraining data, and the behavior of standard transformers. The selected hyperparameters (mask fraction, subjet radius, number of subjets) are hand-chosen but not fitted to optimize the reported metrics.

free parameters (3)
  • Target subjet fraction = 0.3
    Randomly select 30% of subjets as targets. Chosen by hand following I-JEPA conventions; no sensitivity study is provided.
  • Subjet radius parameter = 0.2
    Cambridge-Aachen radius for reclustering. Chosen by hand; central claim does not depend on exact value.
  • Maximum number of subjets = 20
    Subjets are padded to 20 tokens. Chosen by hand; no ablation provided.
assumptions (3)
  • domain assumption EMA target encoder update prevents representation collapse
    Section 2 states that EMA is crucial to prevent informational collapse, citing Ref [22]. The paper observes it holds for J-JEPA but provides no formal proof or diagnostic.
  • domain assumption Pretraining on JetClass top and QCD jets is a suitable unlabeled distribution for downstream Top Tagging
    The pretraining set contains 500k top jets and 500k QCD jets, which are the same classes as the finetuning task. This is a reasonable domain match but is not justified beyond the experimental design.
  • standard math Transformer architectures and AdamW optimization behave as expected for subjet sequences
    The paper adapts ViT and I-JEPA designs without proving convergence properties; this is standard practice in deep learning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Symmetry-Independent Jet Representations via Jet-Based Joint Embedding Predictive Architecture." pith.science (2026). https://pith.science/paper/G4NZ7EAS

@misc{pith2026241205333,
  author       = {Pith},
  title        = {Pith review of: Learning Symmetry-Independent Jet Representations via Jet-Based Joint Embedding Predictive Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4NZ7EAS}},
  note         = {Machine review of arXiv:2412.05333}
}
read the original abstract

In high energy physics, self-supervised learning (SSL) methods have the potential to aid in the creation of machine learning models without the need for labeled datasets for a variety of tasks, including those related to jets -- narrow sprays of particles produced by quarks and gluons in high energy particle collisions. This study introduces an approach to learning jet representations without hand-crafted augmentations using a jet-based joint embedding predictive architecture (J-JEPA), which aims to predict various physical targets from an informative context. As our method does not require hand-crafted augmentation like other common SSL techniques, J-JEPA avoids introducing biases that could harm downstream tasks. Since different tasks generally require invariance under different augmentations, this training without hand-crafted augmentation enables versatile applications, offering a pathway toward a cross-task foundation model. We finetune the representations learned by J-JEPA for jet tagging and benchmark them against task-specific representations.

Figures

Figures reproduced from arXiv: 2412.05333 by the authors.

Figure 1
Figure 1. The J-JEPA architecture begins by splitting the large-radius jet (large black cone) into [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of the background rejection metric [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enhancing next token prediction based pre-training for jet foundation models

    hep-ph 2025-12 conditional novelty 6.0 of 10

    Using continuous particle features as input and combining next-token with masked-token pre-training markedly improves classification accuracy of the OmniJet jet foundation model without visibly hurting its generative quality.

  2. Pretrained Event Classification Model for High Energy Physics Analysis

    hep-ph 2024-12 unverdicted novelty 6.0 of 10

    A pretrained GNN on 120M simulated LHC events improves downstream event-classification accuracy when training data are scarce, with benefits shrinking as data grow.

  3. Discovering the Gell-Mann-Okubo Formula with Kolmogorov-Arnold Networks

    hep-ph 2026-01 reject novelty 3.0 of 10

    A KAN network's fitted polynomials are hand-rearranged into the known Gell-Mann-Okubo mass relations, so the claimed autonomous rediscovery is not demonstrated.

Reference graph

Works this paper leans on

29 extracted references · 7 canonical work pages · cited by 3 Pith papers

  1. [1]

    Fast inference of deep neural networks in FPGAs for particle physics

    J. Duarte et al., “Fast inference of deep neural networks in FPGAs for particle physics”, J. Instrum. 13 (2018) P07027, doi:10.1088/1748-0221/13/07/P07027, arXiv:1804.06913

    arXiv 2018

  2. [2]

    The Phase-2 upgrade of the CMS Level-1 trigger

    CMS Collaboration, “The Phase-2 upgrade of the CMS Level-1 trigger”, CMS Technical Design Report CERN-LHCC-2020-004. CMS-TDR-021, 2020. https://cds.cern.ch/record/2714892

    arXiv 2020

  3. [3]

    GNN-based end-to-end reconstruction in the CMS Phase 2 High-Granularity Calorimeter

    S. Bhattacharya et al., “GNN-based end-to-end reconstruction in the CMS Phase 2 High-Granularity Calorimeter”, in 20th International Workshop on Advanced Computing and Analysis Techniques in Physics Research. 2022. arXiv:2203.01189

    arXiv 2022

  4. [4]

    MLPF: Efficient machine-learned particle-flow reconstruction using graph neural networks

    J. Pata et al., “MLPF: Efficient machine-learned particle-flow reconstruction using graph neural networks”, Eur. Phys. J. C 81 (2021) 381, doi:10.1140/epjc/s10052-021-09158-w , arXiv:2101.08578

    work page Pith review arXiv 2021

  5. [5]

    Interaction networks for the identification of boosted H → bb decays

    E. A. Moreno et al., “Interaction networks for the identification of boosted H → bb decays”, Phys. Rev. D 102 (2020) 012010, doi:10.1103/PhysRevD.102.012010, arXiv:1909.12285

    arXiv 2020

  6. [6]

    ParticleNet: Jet tagging via particle clouds

    H. Qu and L. Gouskos, “ParticleNet: Jet tagging via particle clouds”, Phys. Rev. D 101 (2020) 056019, doi:10.1103/PhysRevD.101.056019, arXiv:1902.08570

    arXiv 2020

  7. [7]

    Particle Transformer for Jet Tagging

    H. Qu, C. Li, and S. Qian, “Particle Transformer for Jet Tagging”, in Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri et al., eds., volume 162 of Proceedings of Machine Learning Research, p. 18281. 2022. arXiv:2202.03772

    arXiv 2022

  8. [8]

    Self-supervised learning from images with a joint-embedding predictive architecture

    M. Assran et al., “Self-supervised learning from images with a joint-embedding predictive architecture”, 2023. https://arxiv.org/abs/2301.08243

    arXiv 2023

Show all 29 references
  1. [9]

    Masked autoencoders are scalable vision learners

    K. He et al., “Masked autoencoders are scalable vision learners”, in Proceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, p. 15979

    2022

  2. [10]

    Symmetries, safety, and self-supervision

    B. M. Dillon et al., “Symmetries, safety, and self-supervision”, SciPost Phys. 12 (2022) 188, doi:10.21468/SciPostPhys.12.6.188, arXiv:2108.04253

    2022 arXiv

  3. [11]

    Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models

    P. Harris et al., “Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models”, 2024. arXiv:2403.07066

    2024 arXiv

  4. [12]

    Masked particle modeling on sets: towards self-supervised high energy physics foundation models

    T. Golling et al., “Masked particle modeling on sets: towards self-supervised high energy physics foundation models”, Mach. Learn.: Sci. Technol. 5 (2024) 035074, doi:10.1088/2632-2153/ad64a8, arXiv:2401.13537. 6

    2024 arXiv

  5. [13]

    Is Tokenization Needed for Masked Particle Modelling?

    M. Leigh et al., “Is Tokenization Needed for Masked Particle Modelling?”, in NeurIPS 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges. 2024. arXiv:2409.12589

    2024 arXiv

  6. [14]

    OmniJet- α: the first cross-task foundation model for particle physics

    J. Birk, A. Hallin, and G. Kasieczka, “OmniJet- α: the first cross-task foundation model for particle physics”, Mach. Learn.: Sci. Technol. 5 (2024) 035031, doi:10.1088/2632-2153/ad66ad, arXiv:2403.05618

    2024 arXiv

  7. [15]

    Large-Scale Pretraining and Finetuning for Efficient Jet Classification in Particle Physics

    Z. Zhao et al., “Large-Scale Pretraining and Finetuning for Efficient Jet Classification in Particle Physics”, in 22nd International Workshop on Advanced Computing and Analysis Techniques in Physics Research. 2024. arXiv:2408.09343

    2024 arXiv

  8. [16]

    ucsd-hep-ex/j-jepa: v0.1.0

    Z. Zhao, S. Katel, and H. Li, “ucsd-hep-ex/j-jepa: v0.1.0”, 2024. doi:10.5281/zenodo.14251372, https://github.com/ucsd-hep-ex/J-JEPA

    2024 doi

  9. [17]

    The anti- kT jet clustering algorithm

    M. Cacciari, G. P. Salam, and G. Soyez, “The anti- kT jet clustering algorithm”, JHEP 04 (2008) 063, doi:10.1088/1126-6708/2008/04/063, arXiv:0802.1189

    2008 arXiv

  10. [18]

    Better jet clustering algorithms

    Y . L. Dokshitzer, G. D. Leder, S. Moretti, and B. R. Webber, “Better jet clustering algorithms”, JHEP 08 (1997) 001, doi:10.1088/1126-6708/1997/08/001, arXiv:hep-ph/9707323

    1997 arXiv

  11. [19]

    Hadronization corrections to jet cross-sections in deep inelastic scattering

    M. Wobisch and T. Wengler, “Hadronization corrections to jet cross-sections in deep inelastic scattering”, in Workshop on Monte Carlo Generators for HERA Physics (Plenary Starting Meeting), p. 270. 4, 1998. arXiv:hep-ph/9907280

    1998 arXiv

  12. [20]

    FastJet User Manual

    M. Cacciari, G. P. Salam, and G. Soyez, “FastJet User Manual”, Eur. Phys. J. C 72 (2012) 1896, doi:10.1140/epjc/s10052-012-1896-2 , arXiv:1111.6097

    2012 arXiv

  13. [21]

    An array-oriented Python interface for FastJet

    A. Roy, J. Pivarski, and C. W. Freer, “An array-oriented Python interface for FastJet”, J. Phys. Conf. Ser. 2438 (2023) 012011, doi:10.1088/1742-6596/2438/1/012011, arXiv:2202.03911

    2023 arXiv

  14. [22]

    Emerging properties in self-supervised vision transformers

    M. Caron et al., “Emerging properties in self-supervised vision transformers”, in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9650. October,

  15. [23]

    An image is worth 16x16 words: Transformers for image recognition at scale

    A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale”, in International Conference on Learning Representations. 2021. arXiv:2010.11929

    2021 arXiv

  16. [24]

    Gaussian error linear units (GELUs)

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs)”, 2023. arXiv:1606.08415

    2023 arXiv

  17. [25]

    Going deeper with Image Transformers

    H. Touvron et al., “Going deeper with Image Transformers”, in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 32. 2021. arXiv:2103.17239

    2021 arXiv

  18. [26]

    JetClass: A Large-Scale Dataset for Deep Learning in Jet Physics

    H. Qu, C. Li, and S. Qian, “JetClass: A Large-Scale Dataset for Deep Learning in Jet Physics”,

  19. [27]

    Top quark tagging reference dataset (v0 (2018_03_27))

    G. Kasieczka, T. Plehn, J. Thompson, and M. Russel, “Top quark tagging reference dataset (v0 (2018_03_27))”, 2019. doi:10.5281/zenodo.2603256. 7

    2019 doi

  20. [28]

    doi:10.5281/zenodo.6619768

    doi

  21. [2022]

    doi:10.1109/CVPR52688.2022.01553

    arXiv:2111.06377. doi:10.1109/CVPR52688.2022.01553

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.