{"id":"8fe97297-9ae9-4ebd-a8c0-abf7af70d67e","arxiv_id":"2411.19370","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On the L=4 2D Ising model, the best machine learning approach for detecting the phase transition depends on the indicator and on whether the Hamiltonian is known.","lead":"This paper compares two machine learning strategies, discriminative and generative, for locating the thermal phase transition in a small 2D Ising model. It finds that the best strategy depends on the indicator method used and on whether the underlying Hamiltonian is known.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fig. 3's envelope is built from unequal hyperparameter counts (4 PixelCNN settings vs 1 CNN setting), so the LBC 'generative favored' finding may be an artifact of search effort.","rationale":"The paper is otherwise a careful study: exact ground-truth indicators on the 4x4 Ising model, reproducible code, and a transparent statement that tuning cost is excluded. The strongest claim, however, is a practical recommendation about which modeling paradigm to choose under a computation-time budget. That recommendation is only as sound as the time-vs-error comparison. The asymmetry in the number of hyperparameter settings is a concrete, fixable flaw: it means the envelope comparison does not compare methods on equal footing. The reader correctly flagged tuning-cost exclusion and architecture representativeness; my concern is narrower and more specific—the unequal number of envelope constituents. Because this can be tested directly and may reverse the data-driven LBC ordering, the verdict should remain conditional pending the matched-search-effort re-run.","tokens_in":11126,"tokens_out":4780,"duration_ms":44696,"concrete_test":"Recompute the data-driven LBC comparison in Fig. 3(b) with matched search effort: train the CNN discriminative model on the same four epoch/learning-rate settings used for PixelCNN (10, 100, 1000, 10000 epochs; lr 0.001 and 0.0001), plot all runs, and compare per-configuration curves without envelope selection. If the best single CNN configuration achieves error equal to or below the best PixelCNN configuration at typical LBC time budgets, the 'generative favored' claim is an artifact of unequal hyperparameter counts; if PixelCNN still wins consistently, the claim survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central data-driven claim in Sec. 5—that NN-based generative modeling is most favored for LBC while discriminative modeling wins for SL/PBM—rests on Fig. 3's computation-time curves. The comparison protocol is asymmetric. Appendix B (Table B1) gives the generative PixelCNN four distinct hyperparameter configurations (epochs 10, 100, 1000, 10000 and learning rates 0.001 and 0.0001), while Table A1 lists a single fixed configuration for each discriminative CNN/MLP (100 epochs, lr=0.001, Adam). The bold 'envelope' lines in Fig. 3 are the lower envelope over these scattered runs. An envelope over four curves is guaranteed to be no worse than a single curve, so PixelCNN's apparent advantage for LBC may simply reflect four times more search effort. The authors note that the cost of hyperparameter tuning is not included, but the deeper issue is that the number of configurations searched is not matched across methods. This is not about whether the architectures are 'representative'; it is an internal inconsistency in the evaluation metric used to derive the headline ordering. The knowledge-driven setting is less affected because the nonparametric winner dominates there, but the data-driven ranking in Sec. 5 is directly vulnerable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a numerical case study on the 2D square-lattice ferromagnetic Ising model, comparing discriminative and generative approaches for the classification tasks underlying three phase-transition indicators: supervised learning (SL), learning by confusion (LBC), and the prediction-based method (PBM). For L=4, where ground-truth indicators are computed exactly, the authors compare histogram-based nonparametric generative models, MLP/CNN discriminative models, and PixelCNN generative models in both a data-driven setting (raw spin configurations, no Hamiltonian knowledge) and a knowledge-driven setting (Hamiltonian known, energy used as a sufficient statistic). They report mean-square errors as functions of dataset size and computation time, concluding that in the data-driven setting NN-based methods are favored over histogram binning, with the generative approach favored for LBC and the discriminative approach favored for SL and PBM, while in the knowledge-driven setting the nonparametric energy-histogram approach dominates except at very small time budgets. The code is publicly available.","tokens_in":11440,"tokens_out":6620,"duration_ms":60613,"significance":"If the rankings are robust, the paper would give concrete practical guidance for choosing between discriminative and generative models in automated phase-transition detection. The study has notable strengths: exact ground-truth indicators for L=4, errors averaged over five independent runs, public code, a clear reduction via the energy sufficient statistic in the knowledge-driven setting, and a connection to Fisher-information underapproximators. However, the central computational comparison rests on a single small system size and on evaluation envelopes built from unequal hyperparameter search budgets, so the contribution is currently a conditional case study rather than a general answer to the question posed in the abstract.","major_comments":[{"comment":"The central data-driven ranking is derived from envelopes over unequal hyperparameter search budgets. Table B1 gives the PixelCNN eight data-driven configurations (learning rates 0.001 and 0.0001, epochs 10/100/1000/10000), while Table A1 gives the CNN a single configuration (100 epochs, learning rate 0.001). The bold envelope in Fig. 3 is therefore a lower envelope over eight PixelCNN runs versus one CNN run; an envelope over more runs cannot do worse, so the claim that generative modeling is most favored for LBC may be an artifact of search effort rather than of the modeling paradigm. Please match the search budgets across methods, report the full set of individual curves for the discriminative models, and state or bound the cost of hyperparameter selection, which is currently excluded from the reported computation times.","section":"§5, Fig. 3, Tables A1 and B1"},{"comment":"All numerical evidence is for a single 4×4 lattice, where the transition is finite-size rounded and the state space has 65,536 configurations. The conclusion extrapolates to larger system sizes and to other thermal lattice models, but no larger-L experiment or scaling argument is provided. To support the practical recommendation stated in the abstract, the authors should either restrict the claims to the L=4 case study or add results for at least one or two larger system sizes to show that the rankings do not reverse.","section":"§5, Secs. 3 and 4"}],"minor_comments":[{"comment":"The KL-divergence labels appear to be reversed: Eq. (9) as written is D_KL(\\tilde P_θ || P), not KL(P || \\tilde P_θ), and Eq. (10) is D_KL(P || \\tilde P_θ), not KL(\\tilde P_θ || P). The subsequent forward/reverse KL discussion follows the standard formulas, so this is a notation issue rather than an error in the training losses, but it should be corrected for clarity.","section":"§2.2, Eqs. (9)-(10)"},{"comment":"In the caption for Fig. C4(a), the text says the small MLP has 610 parameters, but the legend and the data-driven setting use CNN models; this should read \"small CNN\".","section":"Fig. C4 caption"},{"comment":"The caption contains an extra parenthesis in \"prediction-based method (PBM))\".","section":"Fig. 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The main technical issue is the asymmetric hyperparameter search in the computation-time comparison, which directly affects the headline data-driven ranking; this is fixable with additional matched experiments. The paper also leans on several self-cited prior works for the sufficient-statistic reduction and Fisher-information relation, but those are published or posted results and I do not see them as a grounds for rejection. The single-system-size limitation is honestly stated, but the abstract's phrasing goes beyond the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful benchmark paper, not a general answer. The cleanest result is the knowledge-driven one: once you know the Hamiltonian, histogram binning on the energy (the sufficient statistic) beats the NNs at almost every time budget on this 4x4 Ising model. The data-driven rankings are interesting but less solid. The PixelCNN gets multiple hyperparameter configurations (epochs 10 to 10000 and two learning rates), while the discriminative CNN gets one fixed configuration. The bold envelopes in Fig. 3 are lower envelopes over those runs. An envelope over many curves will look better than a single curve even if the methods are equally good. The authors state that tuning cost is not included, but the deeper problem is that search effort is not matched. This specifically hits the LBC-generative-favored conclusion in Sec. 5. The SL/PBM discriminative-favored parts may survive, but the comparison as plotted cannot establish the ordering as stated.\n\nCredit where it is earned: ground-truth indicators are computed exactly for L=4, errors are averaged over five runs, hyperparameters are fully tabulated, and the code is openly available. That is the kind of transparency that makes a numerical comparison reproducible and worth engaging with. The paper also correctly places itself relative to the sufficient-statistic reduction and the Fisher-information connection, though those are self-cited from the same group.\n\nSoft spots, in proportion: the single system size L=4 is the main limitation. The conclusion that the trends hold for larger systems and other models is explicitly an expectation, not a demonstrated result. The choice of CNN and PixelCNN as representatives of their respective approaches is reasonable but not defended against alternatives. Minor: computation times come from a V-100 GPU with limited reporting on run-to-run timing variance, though the five-seed error bands help.\n\nWho this is for: practitioners choosing methods for automated phase-transition detection in small classical lattice models. It is not a general theory of when generative or discriminative models win. It deserves a serious referee, but the authors should be asked to match the number of hyperparameter trials across methods, report tuning-inclusive times or justify the envelope comparison, and either add evidence at larger L or soften the generalization claims. I would not cite it as evidence for the data-driven ranking until that is fixed, but the knowledge-driven comparison and the reproducible setup make it a legitimate paper to send to review.","headline":"A transparent, reproducible benchmark on the 4x4 Ising model, but the data-driven headline ranking is built on an asymmetric hyperparameter search, so the generative-vs-discriminative conclusion should not be taken as settled.","tokens_in":11914,"tokens_out":2744,"would_cite":false,"duration_ms":25778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The best phase detector on the Ising model is not one model class, but a ranking that shifts with the indicator and with available physics knowledge.","keywords":["Ising model","phase transition detection","discriminative vs generative models","supervised learning","learning by confusion","prediction-based method","PixelCNN","Fisher information"],"falsifier":"Run the same three indicators on an $L=8$ Ising lattice with raw spin snapshots, counting all hyperparameter-search time in the budget and using a deeper convolutional architecture; if histogram binning matches or beats the CNN for SL or PBM at any shared computation budget in the data-driven setting, the paper's central ranking is not robust. A cheaper check: on the $4\\times4$ system, include the time spent on the hyperparameter sweeps (the scattered points in Fig. 3) in the reported budgets and see whether the favored model for LBC remains the PixelCNN.","tokens_in":10943,"feed_emoji":"🧲","tokens_out":6164,"duration_ms":47522,"temperature":0.7,"pith_summary":"This numerical case study asks a practical design question: when machine learning is used to locate a phase transition, should the classifier be discriminative (learn $P(y|x)$ directly) or generative (model $P(x|y)$ and use Bayes' rule)? The paper answers the question for the thermal transition of the $4\\times4$ square-lattice ferromagnetic Ising model, using three indicator methods — supervised learning (SL), learning by confusion (LBC), and the prediction-based method (PBM). Its central finding is that the best choice is not one fixed approach: in the data-driven setting, neural-network discriminative models are favored for SL and PBM, the NN generative model is favored for LBC, while in the knowledge-driven setting a simple histogram over the energy sufficient statistic wins at almost every time budget. If the pattern generalizes, it gives practical guidance for automating phase-diagram mapping in other classical lattice models.","feed_headline":"Best Ising phase detector: discriminative, except for LBC","feed_subtitle":"Discriminative CNNs win for SL/PBM; PixelCNN wins for LBC; histograms win with known Hamiltonian.","key_machinery":"The argument runs through scalar phase-transition indicators that are underapproximators of the Fisher information: $I_{\\mathrm{SL}}(\\gamma)$ from supervised-learning posteriors, $I_{\\mathrm{LBC}}(\\gamma)$ from learning-by-confusion error probabilities, and $I_{\\mathrm{PBM}}(\\gamma)$ from a predictive model's output gradient and variance. Each indicator reduces phase detection to a classification task whose posterior $P(y|x)$ can be estimated either directly (discriminative, here a CNN or MLP) or through a learned density $P(x|y)$ combined with Bayes' rule (generative, here a PixelCNN or histogram binning). The decisive simplification in the knowledge-driven setting is that the energy $E$ is a minimal sufficient statistic for the inverse temperature, shrinking the state space from $2^{L^2}$ configurations to $L^2-1$ energy levels and making the nonparametric histogram accurate with few samples. Computation-time comparisons use envelopes over hyperparameter runs, with data-generation time excluded in the data-driven setting and included in the knowledge-driven setting.","core_discovery":"The paper claims that, for locating the Ising transition from raw spin configurations of a $4\\times4$ lattice, the relative efficiency of discriminative versus generative modeling is controlled by the indicator being computed and by how much physical knowledge is available. In the data-driven setting, comparing mean-squared error of the learned indicator versus the ground-truth indicator at fixed dataset size and fixed computation time, the discriminative CNN is the best choice for SL and PBM, whereas the generative PixelCNN wins for LBC. In the knowledge-driven setting, where the Hamiltonian is known and the energy is used as a minimal sufficient statistic, the nonparametric histogram-based generative approach outperforms both NN approaches in computation time for all but the smallest sample counts and largest error tolerances. The paper presents these as empirical findings on one model, with the expectation that the qualitative ranking extends to larger systems and other classical lattice models.","pith_inferences":["A broader reading is that no universal 'generative beats discriminative' answer exists for phase detection; the winning model is selected by the indicator's information geometry, with LBC needing the fuller conditional distribution while SL and PBM hinge on one-dimensional sufficient statistics.","A testable extension would be replacing the PixelCNN with an energy-based model or normalizing flow to see if the LBC advantage for generative models remains, or moving to the $L=8$ lattice where the histogram's state-space collapse weakens.","If hyperparameter-search cost were included in the time budgets, the discriminative NN ranking for SL/PBM could shift at short budgets, since the reported envelopes hide tuning overhead; this is an implicit limitation of the comparison as presented.","The sufficient-statistic argument suggests that in knowledge-driven settings, investing in learned or exact dimension reduction before classification may confer more benefit than choosing between discriminative and generative classifiers."],"forward_implications":["When raw spin snapshots are the only data and the goal is SL or PBM phase detection on the $4\\times4$ Ising model, a discriminative CNN reaches a given indicator accuracy faster than histogram binning or PixelCNN.","For learning-by-confusion on the same data, the generative PixelCNN is the most compute-efficient of the three model families.","When the Hamiltonian is known, histogram binning on energies outperforms both NN approaches for all but the smallest datasets because the energy sufficient statistic collapses the state space.","In the data-driven setting, NN-based methods stop beating the histogram once dataset size approaches the state-space size $|\\mathcal{X}|=2^{16}$; internal biases then dominate.","The qualitative ordering is expected by the authors to persist for larger systems, with the state-space size setting the threshold dataset size."],"supporting_citations":[{"why":"Supplies the result that energy is a minimum sufficient statistic for the Ising model and that optimal analytical predictors exist, underpinning the knowledge-driven histogram and ground-truth indicators.","marker":"[6]"},{"why":"Defines the three phase-transition indicators and the generative-classifier framework that this paper executes and compares.","marker":"[7]"},{"why":"Establishes that the three indicators underapproximate the Fisher information, grounding why these classification-based indicators locate phase boundaries.","marker":"[8]"},{"why":"Introduced supervised neural-network classification of phases, the baseline for the SL indicator used here.","marker":"[9]"},{"why":"Introduced the learning-by-confusion method used to define the LBC indicator.","marker":"[10]"},{"why":"Gives the exact critical temperature used to mark the transition and to define the ground-truth comparison.","marker":"[13]"},{"why":"Provides the PixelCNN autoregressive architecture used as the parametric generative model.","marker":"[16]"}],"fun_headline_variants":["Discriminative wins Ising detection, except for LBC","Best Ising detector? Depends on the indicator you use","Known Hamiltonian? Skip neural nets, use histograms","Ising transition: CNN beats PixelCNN for most tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings depend on the chosen CNN and PixelCNN architectures, fixed optimizers, and small hyperparameter grids being fair stand-ins for discriminative and generative modeling, and on excluding hyperparameter-tuning cost from reported runtimes; change those choices and the ordering could change.","fun_headline_variants_meta":{"raw":{"variants":["Discriminative wins Ising detection, except for LBC","Best Ising detector? Depends on the indicator you use","Known Hamiltonian? Skip neural nets, use histograms","Ising transition: CNN beats PixelCNN for most tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1309,"prompt_tokens":841,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":400}},"tokens_in":457,"tokens_out":468,"duration_ms":4970,"temperature":1.0,"reasoning_tokens":400,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:14:31.320559+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three indicators on an $L=8$ Ising lattice with raw spin snapshots, counting all hyperparameter-search time in the budget and using a deeper convolutional architecture; if histogram binning matches or beats the CNN for SL or PBM at any shared computation budget in the data-driven setting, the paper's central ranking is not robust. A cheaper check: on the $4\\times4$ system, include the time spent on the hyperparameter sweeps (the scattered points in Fig. 3) in the reported budgets and see whether the favored model for LBC remains the PixelCNN.","supporting_citations":[{"cited_title":"Replacing neural networks by optimal analytical predictors for the detection of phase transitions,","cited_arxiv_id":null,"evidence_quote":"Supplies the result that energy is a minimum sufficient statistic for the Ising model and that optimal analytical predictors exist, underpinning the knowledge-driven histogram and ground-truth indicators."},{"cited_title":"Mapping out phase diagrams with generative classifiers,","cited_arxiv_id":null,"evidence_quote":"Defines the three phase-transition indicators and the generative-classifier framework that this paper executes and compares."},{"cited_title":"Machine learning phases of matter,","cited_arxiv_id":null,"evidence_quote":"Introduced supervised neural-network classification of phases, the baseline for the SL indicator used here."},{"cited_title":"Learning phase transitions by confusion,","cited_arxiv_id":null,"evidence_quote":"Introduced the learning-by-confusion method used to define the LBC indicator."},{"cited_title":"Crystal Statistics. i. A Two-Dimensional Model with an Order-Disorder Transition,","cited_arxiv_id":null,"evidence_quote":"Gives the exact critical temperature used to mark the transition and to define the ground-truth comparison."}],"review_version":1}