{"id":"a76a26ef-8551-4559-b5a3-29c94d82fdb0","arxiv_id":"2412.03458","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"For a kinetic consensus-based segmentation model, the choice of evaluation metric changes the optimized model parameters, with Surface Dice being the most representative and F-beta unreliable.","lead":"This paper compares how different image-segmentation scoring rules, Dice, Surface Dice, Jaccard, and F-beta, change the parameter tuning of a kinetic consensus-based segmentation model for brain tumor MRI. The authors find that Surface Dice is the most informative loss for their pipeline and that F-beta behaves erratically for this model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 reports loss values that are actually metric values (1=perfect) yet the text interprets smaller as better, contradicting Eq. (23); the Surface Dice recommendation is not supported by the reported numbers.","rationale":"The reader's weakest assumption (single slice, no variance) is a legitimate external-validity concern, but the most damaging issue is internal: the paper's quantitative evidence for preferring Surface Dice is self-contradictory. Section 3.2.1 defines Loss = 1 − Metric, but Section 4.3/Table 3 label the metric values as loss with 1=perfect. The text then interprets a smaller Surface Dice value as better, which only makes sense if those numbers were losses (1 − metric), not the reported metrics. The two leading entries in the Whole Tumor column even satisfy the Dice–Jaccard identity, proving they are raw metric values. Therefore the comparison 0.7447 < 0.9292 cannot be read as 'Surface Dice loss is smaller' in the sense of Eq. (23). This is not a matter of taste; it is a reversible error in the evidence chain. If corrected, the stated conclusion that Surface Dice is a better representation would need to be re-derived either from cross-metric evaluations or from a loss definition applied consistently. The weaker claim that metric choice affects optimized parameters (Δ1, Δ2, σ2) is still plausible from Table 3, though it too needs repeated-seed evidence; on this point I partially agree with the reader's weakest assumption. Since the problem is addressable by re-analysis and does not invalidate the kinetic model itself, the CONDITIONAL verdict stands.","tokens_in":19612,"tokens_out":6421,"duration_ms":59934,"concrete_test":"Recompute Table 3 using the definition in Eq. (23) so all rows report 1 − Metric, and also compute cross-metric evaluations: e.g., evaluate the Surface-Dice-optimized mask with the Volumetric Dice and Jaccard, and the Volumetric-Dice-optimized mask with Surface Dice. If the Surface-Dice-optimized mask does not achieve higher (or equal) Surface Dice than the Volumetric-Dice-optimized mask on the same image, the claim that Surface Dice is the better loss metric is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.1 defines Loss = 1 − Metric (Eq. 23), with lower loss better. Section 4.3 and Table 3 instead report numbers labeled 'loss' with '1 for perfect overlap and 0 for complete deviation' — i.e., they are the metric values themselves. This is confirmed by the Whole Tumor row: 0.9292 and 0.8672 satisfy JAC = DICE/(2−DICE) (Eq. 29), so they are Dice and Jaccard, not losses. The text then says 'the loss obtained with Surface Dice Coefficient is smaller than the one obtained with the Jaccard Index and the Volumetric Dice Coefficient' (Sec. 4.3), treating 0.7447 < 0.9292 as better. If these are metric values, smaller is worse; if they are losses, they should be 1 − metric and the reported numbers are wrong. Either way, the quantitative comparison in Table 3 does not support the conclusion that Surface Dice is 'more suitable' or 'a better representation'. The only remaining support is the qualitative statement that volumetric metrics 'do not represent the proximity between two surfaces', which is not an empirical demonstration. Consequently, the paper's headline claim about metric choice impacting optimization may survive, but the specific recommendation of Surface Dice over Dice/Jaccard is not evidenced by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends a kinetic consensus-based model for image segmentation, deriving a Fokker-Planck-type PDE via a quasi-invariant limit, and using a Direct Simulation Monte Carlo (DSMC) scheme to approximate its solution. The model interprets image pixels as interacting particles with spatial positions and static gray-level features, and the authors optimize the model parameters (Δ1, Δ2, σ2) by minimizing a loss defined as 1 − evaluation metric. The evaluation metrics considered are Volumetric Dice, Jaccard Index, Surface Dice, and the Fβ-measure. Experiments on synthetic shapes and two brain tumor MRI slices are used to study how the choice of optimization metric affects the resulting parameters and segmentation masks. The paper concludes that the choice of metric does affect the optimized parameters and that Surface Dice is a more suitable metric than Jaccard or Volumetric Dice, while the Fβ-measure is not reliable for this segmentation method.","tokens_in":19940,"tokens_out":3297,"duration_ms":32968,"significance":"The underlying kinetic modeling and DSMC pipeline are of interest to the community working on consensus-based and kinetic approaches to imaging problems, and the idea of linking parameter calibration to clinically used segmentation metrics is commendable and potentially useful. The paper also provides transparent algorithm descriptions and a clear statement of the optimization procedure. However, the central empirical claim that Surface Dice is more suitable than Jaccard or Volumetric Dice is not supported by the reported numbers because the values labeled 'loss' in Table 3 are actually metric values (1 = perfect overlap), yet the text interprets smaller values as better. This internal inconsistency directly affects the paper's headline recommendation. If corrected, and if the empirical evidence is strengthened with repeated runs and more than one slice per task, the manuscript could make a moderate contribution to the field of metric-aware parameter estimation for kinetic segmentation models.","major_comments":[{"comment":"The values labeled 'Loss' in Table 3 are, by the paper's own caption and by the relation in Eq. (29), metric values rather than losses. For example, the Whole Tumor row reports Vol. Dice = 0.9292 and JAC = 0.8672, and these numbers satisfy JAC = DICE/(2−DICE), so they are Dice and Jaccard similarity values, not 1−metric losses. The text then states that the 'loss' obtained with Surface Dice is smaller than that for Jaccard and Volumetric Dice, and uses this comparison as evidence that Surface Dice is 'more suitable.' Since a smaller metric value corresponds to worse overlap (for metrics that are 1 for perfect agreement), the reported numbers actually show that Surface Dice gives lower agreement than the other two metrics on both tasks. The quantitative comparison therefore contradicts, rather than supports, the conclusion that Surface Dice is a better representation for this model. The authors must either correct the definition and the reported numbers, or re-evaluate the suitability claim.","section":"Section 4.3, Table 3"},{"comment":"The generalization claim that the choice of evaluation metric determines the outcome of parameter optimization rests on experiments with a single axial MRI slice for the whole-tumor task and a single slice for the core-tumor task, with no identification of the slice, no repeated optimization runs with different random seeds, and no error bars. Because the optimization uses a fixed random seed (as stated in Section 3.2.1), the differences in optimized parameters across metrics in Tables 3 and 4 could be due to the particular random sampling of hyperparameters rather than to a reproducible property of the metrics. To make the claim load-bearing, the authors should report results over multiple slices and/or multiple seeds, or at least provide variance information from repeated runs.","section":"Section 3.2.1, Section 4.3"},{"comment":"The analysis of the Fβ-measure is internally inconsistent. The caption of Table 4 states that loss is 1 for perfect overlap and 0 for complete deviation, which defines the reported values as metric values, but Eq. (23) defines loss as 1 − metric. The text then says that for β=0.25 the loss is higher, 'meaning that this should be the most accurate result,' while also saying that 'the loss decreases for larger values of β' and that the resulting binary mask is less accurate for lower β. These statements mix the two interpretations: if the reported value is a metric, higher is better; if it is a loss, lower is better. The conclusion that Fβ is not a reliable metric is therefore not supported by the numbers as presented, because the direction of comparison has not been fixed. The authors should clarify which quantity is reported and re-derive the conclusions accordingly.","section":"Section 4.3, Table 4 and Figure 12"}],"minor_comments":[{"comment":"There are several typographical errors, including 'mode l' in the abstract, 'Euler-Maryuama' for Euler-Maruyama in Section 2.2, 'Coefficent' in the caption of Figure 10, and 'msethods' in reference [42]. A careful proofreading pass is recommended.","section":"Abstract and text throughout"},{"comment":"The variation index T is defined as an integral over the continuous density f, but in the DSMC context the density is a finite particle approximation. It would be helpful to specify how this integral is computed from the discrete particle set, for example via a histogram or kernel density estimate, to make the stopping criterion reproducible.","section":"Section 4.2, Eq. (35)"},{"comment":"The absolute approximation error bound of 3−2√2 and the relative error of 1 are stated without derivation. Since these bounds are used to argue that Jaccard and Dice are closely related, a brief proof or a reference to the specific derivation would strengthen this part.","section":"Section 3.3, Proposition 3.1"},{"comment":"The text says 'the loss is null in both cases' for the square image, but Table 2 shows parameters, not losses. The sentence is understandable, but it would be clearer to explicitly state that the resulting Surface Dice was 1 and hence the loss was 0 for all three metrics.","section":"Section 4.1, Table 2"},{"comment":"The relationship between Jaccard and Dice is correctly stated, but the notation in Eqs. (25) and (28) uses |S1g| for the number of pixels in the foreground; for consistency, the authors should define |·| as the cardinality of the set earlier, before first use.","section":"Section 3.3.2 and Eq. (29)"},{"comment":"The paper does not state the chosen tolerance τ for the Surface Dice metric in the section where it is used for optimization (Section 4.3 mentions τ equal to 1 pixel, but this is only in the caption of Figure 10 and in the surrounding text; the table itself does not note it). Including τ in the table caption would improve clarity.","section":"Section 4.3, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea—linking the choice of evaluation metric to parameter optimization in a kinetic segmentation model—is reasonable, and the mathematical framework is coherent. However, the empirical section has a fundamental sign/interpretation error in the reported 'loss' values, which directly undermines the main recommendation about Surface Dice. This is not a superficial issue: it affects the central claim of the paper. The authors need to correct the interpretation of Table 3 and Table 4, and likely re-run or re-analyze the experiments with proper loss definitions and statistical repetitions. With those fixes, the manuscript could be suitable for publication, but as it stands the evidence for the headline conclusion is invalid. I would not recommend rejection, as the errors are fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe main claim here does not survive a close reading of Table 3. The paper defines Loss = 1 - Metric (Eq. 23), then reports numbers in Table 3 as 'loss' but adds 'loss is equal to 1 for a perfect overlap' — that's a metric, not a loss. The values themselves confirm it: 0.9292 and 0.8672 satisfy JAC = DICE/(2−DICE), so they are Dice and Jaccard scores. The text then says the Surface Dice 'loss' is smaller (0.7447 vs 0.9292) and treats that as better. But a smaller metric means worse; if those numbers were losses, they'd be 1 − metric and the comparison inverts. Either way, Table 3 does not support the conclusion that Surface Dice is more suitable. This is a load-bearing error, not a typo, because the qualitative claim that volumetric metrics ignore boundary proximity is a property, not an empirical demonstration.\n\nWhat's actually new: very little. The kinetic model, Fokker-Planck derivation, and the DSMC pipeline come from the authors' prior paper [9]. The only genuinely new piece is the comparison of four known metrics on this model and the observation that optimized parameters shift with the metric. That observation is real — the tables show different (Δ1, Δ2, σ2) — and the F-beta analysis, despite the same loss/metric confusion, does show that F-beta is unstable for this task. Proposition 3.1 is a correct but elementary bound on the Jaccard-Dice relation.\n\nThe experimental base is also too thin: one axial slice per task, no identification of patient or scan, a fixed random seed, no repeated runs, no error bars. The 'more suitable' recommendation would be hard to support even with correct numbers. The paper reads like a competent write-up of the model and a hasty, underpowered evaluation.\n\nI would not send this to peer review in its current state. The loss/metric labeling is easily fixable, but the deeper issues — single slices, no variance, no external mask-quality check — require a rewrite of the evaluation. If the authors can show that metric choice changes the final mask meaningfully and that Surface Dice consistently tracks boundary quality across a few independent slices, there's a modest paper here. As it stands, it's a desk reject.","headline":"The paper's central claim that Surface Dice is more suitable is unsupported by its own mislabeled loss values, and the experiment is too thin to fix that.","tokens_in":20461,"tokens_out":4790,"would_cite":false,"duration_ms":42630,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["35Q84","35Q20","68U10","82C40","65C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The evaluation metric used to optimize a kinetic consensus-based segmentation model changes the MRI tumor masks it produces, and the boundary-aware Surface Dice proves the most suitable loss.","keywords":["consensus-based segmentation","kinetic equations","bounded confidence","evaluation metrics","Surface Dice","Jaccard index","brain tumor MRI","DSMC"],"falsifier":"Run the same 300-iteration optimization procedure on a set of at least twenty brain-tumor MRI slices from different patients, comparing Volumetric Dice, Jaccard, and Surface Dice for both whole-tumor and tumor-core tasks; if Volumetric Dice or Jaccard produces equal or better segmentation masks on a majority of slices, the paper's recommendation of Surface Dice would not generalize.","tokens_in":19423,"feed_emoji":"🧠","tokens_out":5295,"duration_ms":49621,"temperature":0.7,"pith_summary":"This paper studies how the evaluation metric chosen as a loss function affects the parameter optimization of a kinetic consensus-based segmentation model for brain-tumor MRI. It shows that optimizing the model against Volumetric Dice, Jaccard, or Surface Dice yields different optimal parameters and different final binary masks, while the scores of the overlap-based metrics remain linked by a known identity. The paper argues that Surface Dice, which compares boundary regions rather than raw pixel overlap, is a better loss for this type of segmentation, and that the Fβ-measure gives misleading results because low β values can reward inaccurate masks. If this is right, the practical lesson is that the choice of evaluation metric is not neutral: researchers calibrating such models need to match the loss to the geometrical property they actually care about.","feed_headline":"Surface Dice beats Dice and Jaccard for kinetic segmentation","feed_subtitle":"A kinetic particle model yields different tumor masks depending on the loss; boundary-aware scoring wins in tests.","key_machinery":"The load-bearing object is the bounded-confidence interaction kernel P_Δ₁,Δ₂((x,c),(x*,c*)) = χ(|x − x*| ≤ Δ1)χ(|c − c*| ≤ Δ2), which allows a pixel to influence another pixel only if they are sufficiently close in both space and gray level. Around this kernel the paper constructs a Boltzmann-type kinetic equation and its quasi-invariant Fokker-Planck limit, solved numerically by a Nanbu-Babovsky direct simulation Monte Carlo scheme. The optimization loop minimizes Loss(Sg, St) = 1 − Metric(Sg, St) over the parameters (Δ1, Δ2, σ2), and the identity JAC = DICE/(2 − DICE) ties the two overlap metrics together, while the Surface Dice coefficient replaces pixel intersection with the intersection of tolerance bands around the boundaries.","core_discovery":"The central claim is that the loss function used in the optimization loop is not a mere evaluation detail but actively determines which model parameters (Δ1, Δ2, σ2) are selected and what binary segmentation mask results. The paper demonstrates this by running the same 300-iteration hyperparameter search on whole-tumor and tumor-core MRI slices, with Volumetric Dice, Jaccard, Surface Dice, and Fβ-measure as the respective losses, and recording the resulting parameters and masks. It confirms the algebraic relationship JAC = DICE/(2 − DICE) between the two volume-based metrics, then shows that Surface Dice with a one-pixel tolerance yields numerically different parameters and visibly improved boundary fidelity, especially for the tumor-core task. The paper concludes that Surface Dice is the more suitable metric for this consensus-based segmentation pipeline, while the Fβ-measure is unreliable because its loss values correlate poorly with actual mask quality.","pith_inferences":["A natural extension the paper leaves implicit is to repeat the study across many patients and on full 3D volumes; the current evidence rests on a single axial slice per task, so the ranking of metrics could shift on a broader sample.","Boundary-aware losses such as Surface Dice may also be preferable for other particle-based or clustering segmentation methods, since volume-only scores reward any boundary of equal volume regardless of its position.","The observed parameter compensation (lower diffusion offsets by smaller Δ2) suggests the loss landscape contains nearly equivalent parameter manifolds, which would make the metric choice more influential than the exact parameter values for the final mask quality."],"forward_implications":["The optimal model parameters are not intrinsic to an image; they depend on which metric the optimizer is asked to maximize, so reporting parameters without stating the loss is incomplete.","Surface Dice should be preferred as the optimization loss for this consensus-based segmentation model when boundary accuracy matters, which is the case for tumor-core segmentation.","The identity JAC = DICE/(2 − DICE) lets practitioners convert between the two overlap scores without rerunning the experiment, making comparisons across studies that use different metrics straightforward.","The Fβ-measure is unsuitable as a loss for this model, because at low β it can assign a high score to a mask with many false negatives, decoupling the loss from the clinical objective."],"supporting_citations":[{"why":"Supplies the base kinetic consensus-based segmentation model, the mask generation procedure, and the biomedical imaging pipeline this paper extends.","marker":"[9]"},{"why":"Provides the deeper theoretical comparison between Jaccard and Volumetric Dice optimization, including the relationship used to interpret the overlap-based losses.","marker":"[7]"},{"why":"Introduces the Surface Dice similarity coefficient, which the paper uses as its boundary-aware loss and evaluation metric.","marker":"[39]"},{"why":"Defines the Jaccard index used as one of the optimization losses.","marker":"[29]"},{"why":"Defines the Volumetric Dice coefficient used as the baseline optimization and evaluation metric.","marker":"[18]"},{"why":"The Hyperopt library that implements the random-sampling parameter optimization loop (300 iterations).","marker":"[6]"},{"why":"The Nanbu-Babovsky direct simulation Monte Carlo scheme used to approximate the Boltzmann-type kinetic equation.","marker":"[41]"},{"why":"The mean-field model for large-data clustering that motivates interpreting pixels as interacting particles and cluster formation.","marker":"[26]"}],"fun_headline_variants":["Metric pick decides tumor mask in kinetic model","Surface Dice beats Dice for boundary fidelity","Evaluation loss steers consensus segmentation outcome","Which loss? It changes the whole segmentation mask"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that Surface Dice is the most suitable metric rests on experiments with just one axial MRI slice for the whole-tumor task and one for the tumor-core task, with a fixed random seed and no repeated-run variance, so if those slices are atypical the ranking of metrics could change.","fun_headline_variants_meta":{"raw":{"variants":["Metric pick decides tumor mask in kinetic model","Surface Dice beats Dice for boundary fidelity","Evaluation loss steers consensus segmentation outcome","Which loss? It changes the whole segmentation mask"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1149,"prompt_tokens":800,"completion_tokens":349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":295}},"tokens_in":416,"tokens_out":349,"duration_ms":4036,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:23:09.691392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 300-iteration optimization procedure on a set of at least twenty brain-tumor MRI slices from different patients, comparing Volumetric Dice, Jaccard, and Surface Dice for both whole-tumor and tumor-core tasks; if Volumetric Dice or Jaccard produces equal or better segmentation masks on a majority of slices, the paper's recommendation of Surface Dice would not generalize.","supporting_citations":[{"cited_title":"& Zanella, M","cited_arxiv_id":null,"evidence_quote":"Supplies the base kinetic consensus-based segmentation model, the mask generation procedure, and the biomedical imaging pipeline this paper extends."},{"cited_title":"The distribution of the flora in the alpine zone.1","cited_arxiv_id":null,"evidence_quote":"Defines the Jaccard index used as one of the optimization losses."},{"cited_title":"Measures of the Amount of Ecologic Association Betwee n Species","cited_arxiv_id":null,"evidence_quote":"Defines the Volumetric Dice coefficient used as the baseline optimization and evaluation metric."},{"cited_title":"& Cox, D","cited_arxiv_id":null,"evidence_quote":"The Hyperopt library that implements the random-sampling parameter optimization loop (300 iterations)."},{"cited_title":"An Introduction to Monte Carlo Metho ds for the Boltzmann Equation","cited_arxiv_id":null,"evidence_quote":"The Nanbu-Babovsky direct simulation Monte Carlo scheme used to approximate the Boltzmann-type kinetic equation."},{"cited_title":"& Visconti, G","cited_arxiv_id":null,"evidence_quote":"The mean-field model for large-data clustering that motivates interpreting pixels as interacting particles and cluster formation."}],"review_version":1}