{"id":"3532f063-7015-4e31-a866-247e6dbba1e2","arxiv_id":"2412.08841","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SEPC uses a maximizing-structural-entropy regularizer for probabilistic embeddings and a soft-label encoding tree for regression, reporting state-of-the-art results on 12 NLU tasks.","lead":"SEPC adds a structural entropy regularizer to probabilistic embedding models, pushing different classes to separate in the latent space. The method reports consistent improvements over prior probabilistic coding baselines on 12 natural language understanding tasks, including classification and regression.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Regression decoding rule is unspecified: the paper never defines how predicted bin probabilities are converted to a continuous label, so the Table 2 regression gains cannot be reproduced from the text.","rationale":"The paper has two independent contributions: the structural-entropy regularizer for classification (Eqs. 5-8) and the probabilistic encoding tree for regression (Eqs. 9-14). The classification regularizer is internally consistent: L_SE in Eq. 7 equals -H_T_C(G) from Eq. 6, and using L_SEPC = L_PC - gamma*L_SE indeed maximizes the structural entropy of the induced class partition, which penalizes cross-class edges and encourages class separation. The classification experiments with ablations and robustness support the claimed gains. The regression half, however, has a load-bearing gap: the method description ends at the regularizer. It does not state how predictions are produced, i.e., what the encoder outputs and how that output is mapped to a real number. Since the paper's second contribution is explicitly the probabilistic encoding tree for regression, and Table 2 reports SOTA regression results, this omission determines whether the central claim is substantiated. The reader identified this same weakness; I agree. The concern is not that the method is wrong, but that it is incompletely specified; the availability of code means a concrete check can resolve it. Thus I would keep the CONDITIONAL verdict, conditioned on the authors specifying (or the code confirming) the output head and decoding rule.","tokens_in":15861,"tokens_out":8745,"duration_ms":90876,"concrete_test":"Inspect the public repository at https://github.com/SELGroup/SEPC, specifically the regression training script and model definition. Identify (1) the output head for STS-B/Claire (e.g., a linear layer producing r logits versus a scalar), (2) the training loss (cross-entropy on soft labels Y' versus MSE on continuous labels), and (3) the decoding rule (e.g., y_hat = sum_i p_i * P_i). Then re-run the two regression datasets with the released code and five seeds; if the reproduced Pearson/Spearman values match Table 2 and the decoding rule is the expectation over bin centers, the missing-text gap is resolved. If the code uses a different decoding or an MSE head, the paper's description is inaccurate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SEPC achieves SOTA on regression (Table 2: STS-B and Claire) rests on the probabilistic encoding tree described in Eqs. 9-14. However, the paper only defines the structural-entropy regularizer L_SE (Eq. 14) for soft labels Y' = softmax(-D). It never specifies the model's output head or loss for regression: does the encoder output logits over the r bins (trained with cross-entropy against Y'), or is there a scalar regression head trained with MSE while L_SE is only an auxiliary regularizer? More importantly, the decoding rule mapping the predicted bin distribution to a continuous correlation score is absent (e.g., expectation over bin centers, argmax bin center, or a learned mapping). Without this, Table 2 cannot be reimplemented from the text, and the improvement attributed to the probabilistic encoding tree could be an artifact of an unstated decoding choice. This is not a minor exposition gap: the second contribution ('effective method to utilize structural entropy for regression') is precisely this mechanism, and the claimed 0.96%-2.65% average improvement over baselines rests on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEPC, a probabilistic coding model with a structural-entropy-based regularization loss. For classification, the model constructs a graph over latent embeddings, builds a three-tier encoding tree from class labels, and maximizes the structural entropy of the class partition (Eqs. 5-8). For regression, the paper proposes a 'probabilistic encoding tree': continuous labels are binned and softened into a distribution over bin centers (Eqs. 9-10), and the same structural-entropy loss is applied with soft assignments (Eq. 14). Experiments on 10 classification and 2 regression NLU tasks with BERT and RoBERTa backbones report consistent improvements over deterministic and probabilistic baselines, with ablations (w/o SE, w/o soft), label-noise robustness, limited-data generalization, and hyperparameter sensitivity studies.","tokens_in":16114,"tokens_out":6793,"duration_ms":72210,"significance":"The central idea---using the maximization of structural entropy as a regularizer for probabilistic embeddings---is novel and the empirical results are encouraging. The paper reports means and standard deviations over five seeds, includes ablations that isolate the structural-entropy term in classification, provides a robustness analysis, and links to public code, which are all strengths. If the regression pipeline is fully specified, the second contribution (probabilistic encoding tree for soft labels) would also be a useful practical technique. However, as written, the regression method is not reproducible from the text, and the claimed regression gains in Table 2 rest on an underspecified component.","major_comments":[{"comment":"The regression pipeline is underspecified. The paper defines soft labels Y' = softmax(-D) and the regularizer L_SE in Eq. (14), but it never states (i) the model's output head for regression (e.g., a linear layer with r logits), (ii) the task loss used to train against Y' (e.g., cross-entropy) or whether a scalar MSE head is retained with L_SE as an auxiliary regularizer, or (iii) the decoding rule from the predicted bin distribution to a continuous value (e.g., expectation over bin centers, argmax bin center, or a learned mapping). The bin centers P in Eq. (9) are also not specified beyond 'bin the entire regression label value space into r classes.' Consequently, Table 2 cannot be reimplemented from the text, and the second stated contribution ('effective method to utilize structural entropy for regression tasks') is not fully defined. Please specify these components explicitly, or state that the code is the only complete specification and summarize the details in the paper.","section":"Probabilistic Encoding Tree for Regression Tasks, Eqs. (9)-(14)"},{"comment":"The relationship between the verbal definition of the soft cut weight g'_alpha_j and the matrix expression in Eq. (14) is not derived. The text says the weight of cut edges 'should be multiplied by the probability of one vertex belonging to T_alpha_j and the other belonging to T^complement_alpha_j,' but Eq. (14) computes [(1-C)^T A C]_jj. For symmetric A this equals sum_{i,k} C_{ij} A_{ik}(1-C_{kj}), which is a particular soft-cut convention; please show that this matches the stated definition and discuss how the relaxation affects the encoding-tree properties from the Preliminaries (in particular, the single-parent assignment of leaf nodes). Without this derivation, the probabilistic encoding tree is presented as an ad hoc formula rather than a principled extension of structural entropy.","section":"Probabilistic Encoding Tree for Regression Tasks, Eqs. (11)-(14)"}],"minor_comments":[{"comment":"Figures 4 and 5 do not include error bars or confidence intervals, although the tables report standard deviations over five seeds; adding them would help assess the stability of the sensitivity and generalization results.","section":"Hyperparameter Sensitivity Analysis (Figure 4) and Generalization Analysis (Figure 5)"},{"comment":"The softmax in Eq. (10) implicitly uses unit temperature; the temperature controls the softness of the soft labels and is not mentioned as a tuned or fixed choice. Please state the temperature or discuss its effect.","section":"Eq. (10)"},{"comment":"The choice of bin centers P is not described. For STS-B (labels 0-5, r=5) and Claire (labels 1-5, r=4), please specify whether equal-width bins are used and how the centers are computed.","section":"Eq. (9) and Parameter Settings"},{"comment":"The text says SEPC w/o SE 'is the same as SPC w/o S model (Hu et al. 2024)' but then notes different hyperparameters and environments; please clarify the exact relationship so that the ablation cleanly isolates the contribution of L_SE rather than differences in training setup.","section":"Classification ablation (Table 1)"},{"comment":"There is a minor typo in the Related Work section: 'the conditional entropy bottleneck top improved robustness to adversarial examples' should likely read 'the conditional entropy bottleneck to improve robustness to adversarial examples.'","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The main scientific issue is the underspecified regression pipeline; the classification results and ablations are promising and suggest the method is sound, but the regression contribution cannot be independently assessed or reproduced from the text. The missing details are likely present in the released code, so the revision should be feasible. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SEPC is a solid, honest paper with a genuinely new regularizer for probabilistic embedding fine-tuning. The core idea—maximize the structural entropy of the graph induced by the latent embeddings, using labels as the partition—is new and clearly explained. The classification experiments are well done: 12 tasks, five seeds, means and standard deviations, an ablation that shows the regularizer matters, and a robustness study under label noise. The gains over SPC and VIB are consistent, not huge but credible. The authors also ship code and data, which is real evidence.\n\nThe soft spot is the regression half, and the stress-test note is right. The paper defines soft labels Y' and the structural entropy loss L_SE in Equations 9–14, but never says what the model outputs for a new input or how the predicted bin distribution is turned into a continuous score. There is no output head, no decoding rule, no mention of whether L_SE is a standalone loss or an auxiliary regularizer. Table 2's regression results therefore cannot be reproduced from the text. That matters because the probabilistic encoding tree is the paper's second contribution, not a minor detail. The code link may resolve it, but the paper as written is incomplete on exactly the mechanism it claims to introduce.\n\nLesser issues: no significance testing across the five seeds, and the hyperparameter search (beta, gamma) is reported only as ranges with no selected values per dataset. Those are minor.\n\nThe citation pattern is fine. Some related work is from the same group, but the central comparisons are independent baselines and the method does not reduce to its inputs.\n\nVerdict: this deserves a serious referee. The classification regularizer is novel, empirically supported, and likely citable. The regression part needs to be specified or removed before the paper is acceptable. I'd send it to review with a clear request for the missing decoding details.","headline":"Solid classification regularizer with a genuinely new structural-entropy objective, but the regression half is underspecified to the point of being unreproducible from the text.","tokens_in":16592,"tokens_out":2230,"would_cite":true,"duration_ms":23629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SEPC claims that maximizing the structural entropy of a label-induced partition, applied to probabilistic embeddings, improves classification and regression on 12 natural-language tasks.","keywords":["probabilistic coding","structural entropy","probabilistic embedding","information bottleneck","regularization","natural language understanding","regression as classification","label noise robustness"],"falsifier":"Re-run SEPC's regression experiments after fixing a concrete decoding rule, for example taking the expected bin center under the predicted soft distribution, and check whether the reported Pearson and Spearman gains over SPC on STS-B and Claire survive; if the gains vanish or cannot be reproduced without an arbitrary decoding choice, the regression claim fails. Separately, replacing the label-based intermediate partition with random class assignments in classification should destroy most of the reported improvements if the structural-entropy mechanism is the cause.","tokens_in":15656,"feed_emoji":"🧠","tokens_out":9362,"duration_ms":84953,"temperature":0.7,"pith_summary":"The paper proposes SEPC, a probabilistic coding method that adds a structural-entropy regularization term to the usual information-bottleneck objective. The claim is that maximizing the structural entropy of the partition induced by class labels separates the latent distributions of different classes and thereby improves generalization, robustness to label noise, and performance under scarce training data. For regression, the paper introduces a probabilistic encoding tree that turns continuous labels into soft probabilities over bin centers, allowing the same structural-entropy loss to be used. Experiments on 10 classification and 2 regression natural-language tasks report that SEPC outperforms deterministic and probabilistic baselines on both BERT and RoBERTa backbones. A sympathetic reader would take the paper to be establishing that inter-variable structural information, not just per-variable constraints, belongs in probabilistic embedding objectives.","feed_headline":"Structural entropy lifts probabilistic coding on 12 tasks","feed_subtitle":"New regularizer separates latent classes, with soft encoding trees carrying the same idea to regression.","key_machinery":"The load-bearing object is the structural-entropy regularization loss $L_{SE}$, built from the graph whose edge weights are sigmoid similarities between latent embeddings and whose assignment matrix $C$ encodes (soft) class membership. Structural entropy measures the information in a hierarchical partition of a graph by summing, over non-root nodes, cut-weight proportions times log volume ratios; here the three-tier encoding tree has classes as the intermediate layer. Maximizing the intermediate-layer structural entropy pulls different class distributions apart, and the loss is attached to the probabilistic coding loss as $L_{SEPC} = L_{PC} - \\gamma L_{SE}$. For regression, the probabilistic encoding tree replaces the hard assignment matrix with the soft-label matrix $Y' = \\operatorname{softmax}(-D)$, where $D$ holds distances to bin centers, so the same formula (Equation 14) applies to continuous labels.","core_discovery":"SEPC's central discovery is that structural entropy can serve as a regularizer in the opposite direction from prior usage: instead of minimizing the structural entropy of an encoding tree to compress task information, the model maximizes the structural entropy of a three-tier tree whose intermediate nodes are classes. This pushes the probabilistic embeddings of different classes apart in latent space while keeping the encoder-only probabilistic coding framework. The regularization loss is Equation (7) for hard labels and Equation (14) for soft labels, with both reducing to the same form once the assignment matrix is allowed to hold probabilities. The regression extension discretizes the label range into bin centers, forms soft labels by $Y' = \\operatorname{softmax}(-D)$ on distances to those centers, and relaxes the encoding tree so each leaf can belong to every intermediate node with varying probability; the paper reports that this beats hard discretization and improves Pearson and Spearman correlations on STS-B and Claire.","pith_inferences":["One extension the paper leaves open is replacing the label-derived intermediate partition with a cluster-derived partition; if the gains persist, the regularizer is not tied to label information.","A reader may test the same soft-bin recipe on other continuous outputs, such as age estimation or relevance scoring, where the bin-center decoding rule would have to be chosen explicitly.","The robustness experiments use symmetric random label flips; class-conditional or structured noise would better test whether structural entropy protects against realistic annotation errors.","The paper never specifies how predicted bin probabilities become a scalar, so any reimplementation must fix that rule; the reported regression gains should survive whichever reasonable rule is chosen."],"forward_implications":["If SEPC's results hold, probabilistic embeddings can be regularized with inter-variable structure at no architectural cost, since the loss is computed from embeddings and labels already in the pipeline.","Regression benefits from classification-style structural regularization once labels are softened, so hard binning is not required to apply structural information theory to continuous outputs.","The largest reported gains on the Hate and Irony datasets indicate the regularizer helps most where train-test topic imbalance or subtle semantics make the task hard.","The 10%, 20%, and 30% label-noise experiments imply the class-level structural objective tolerates individual label flips better than per-sample objectives.","The method introduces one hyperparameter $\\gamma$, and the reported sensitivity analysis says lower values are generally preferred and $\\gamma=10$ hurts, so deployment is cheap."],"supporting_citations":[{"why":"Defines structural entropy and the encoding tree, the mathematical object the regularization loss is built from.","marker":"Li and Pan 2016"},{"why":"Proposes the encoder-only structured probabilistic coding framework that SEPC adopts and supplies the strongest probabilistic baseline it must beat.","marker":"Hu et al. 2024"},{"why":"Introduces the variational information bottleneck objective that defines the base probabilistic coding loss.","marker":"Alemi et al. 2017"},{"why":"States the information bottleneck principle that probabilistic coding follows.","marker":"Tishby, Pereira, and Bialek 2000"},{"why":"Provides evidence that hard discretization hurts regression, motivating the soft-label probabilistic encoding tree.","marker":"Pintea et al. 2023"},{"why":"Supplies the STS-B semantic textual similarity dataset used for the regression evaluation.","marker":"Cer et al. 2017"},{"why":"Supplies the Claire dataset used alongside STS-B for regression evaluation.","marker":"Roth, Anthonio, and Sauer 2022"}],"fun_headline_variants":["Structural entropy maximizes class separation in probabilistic coding","SEPC: structural entropy regularizer separates latent classes","Max structural entropy to improve probabilistic coding","Structural entropy lifts coding on 12 NLU tasks","Structural entropy guides coding to beat 12-task SOTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The regression recipe rests on the assumption that replacing a continuous label with a soft distribution over a few bin centers and later decoding the predicted bin distribution back to a real number loses no information the model needs, but the paper never specifies the output head or decoding rule.","fun_headline_variants_meta":{"raw":{"variants":["Structural entropy maximizes class separation in probabilistic coding","SEPC: structural entropy regularizer separates latent classes","Max structural entropy to improve probabilistic coding","Structural entropy lifts coding on 12 NLU tasks","Structural entropy guides coding to beat 12-task SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2815,"prompt_tokens":903,"completion_tokens":1912,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1840}},"tokens_in":519,"tokens_out":1912,"duration_ms":14124,"temperature":1.0,"reasoning_tokens":1840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:30:33.898087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run SEPC's regression experiments after fixing a concrete decoding rule, for example taking the expected bin center under the predicted soft distribution, and check whether the reported Pearson and Spearman gains over SPC on STS-B and Claire survive; if the gains vanish or cannot be reproduced without an arbitrary decoding choice, the regression claim fails. Separately, replacing the label-based intermediate partition with random class assignments in classification should destroy most of the reported improvements if the structural-entropy mechanism is the cause.","supporting_citations":[{"cited_title":"A.; Fischer, I.; Dillon, J","cited_arxiv_id":null,"evidence_quote":"Introduces the variational information bottleneck objective that defines the base probabilistic coding loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the STS-B semantic textual similarity dataset used for the regression evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Claire dataset used alongside STS-B for regression evaluation."}],"review_version":1}