{"id":"11f84b8c-87c9-486a-b44a-ca9948518455","arxiv_id":"2507.13887","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"A broad benchmark of intrinsic dimension estimators shows that no single method or set of hyperparameters works across datasets, and tuned benchmark scores frequently indicate overfitting rather than transferable accuracy.","lead":"This survey compares many methods that guess the hidden dimension of high-dimensional data, grouping them by the geometry they use. It finds that most methods only work well when their settings are tuned to the exact dataset, so their good results often do not transfer to new data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'overfitting is frequent' claim is inferred from oracle-based, in-sample hyperparameter selection, not from a held-out generalization test; the evidence supports sensitivity, not overfitting.","rationale":"The paper's practical contribution rests on two connected claims: no single estimator or hyperparameter set works across all settings, and tuning to benchmark datasets frequently overfits. The first claim is well supported by the benchmark tables and by sensitivity plots such as Figure 9, which shows lPCA's narrow optimal alpha range. The second claim, prominent in the abstract, is the load-bearing one: it is what turns a survey of sensitivities into a warning about generalization beyond the tested datasets. Under the Appendix C protocol, H_best is selected by minimizing error against the known intrinsic dimension d(M) on the same 20 point sets used for evaluation. This is an oracle-based, in-sample selection. It establishes sensitivity to H and heterogeneity of the optimal H across datasets; it does not establish that a hyperparameter tuned on one dataset fails on new samples or on new datasets. The abstract's inference therefore requires an additional experiment in which H is tuned on training data and evaluated on held-out data. The reader's CONDITIONAL verdict already flags hyperparameter ranges and benchmark representativeness as the weakest assumption; the concern here is more internal: the design does not measure generalization/overfitting at all. This does not overturn the survey's value, but it should be reflected in the verdict's conditions, so I would keep the CONDITIONAL verdict while extending its conditions to include a proper held-out evaluation of the overfitting claim.","tokens_in":72450,"tokens_out":5752,"duration_ms":72422,"concrete_test":"Re-run the Appendix C protocol with a train/test split. For each estimator E and manifold M, draw 20 training samples and 20 test samples. Select H_best using only the training samples (minimizing mean |dhat - d| over the training sets), then evaluate that H_best on the test sets. Also evaluate H_best(M) on every other benchmark manifold. If the test-set error of H_best(M) is close to its training-set error and its cross-manifold error is no worse than the fixed H_abs, then the 'overfitting is frequent' claim is an artifact of in-sample selection rather than a property of the estimators. If H_best(M) degrades on held-out samples or on other manifolds, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim — 'in identifying the best hyperparameters for benchmark datasets, overfitting is frequent' (Abstract, echoed in §3.3) — is not tested as overfitting. In Appendix C, for each estimator E and dataset M, the authors evaluate dhat(E,M,H) on 20 random point sets, then select H_best(E,M) = argmin_H |dhat(E,M,H) - d(M)| and report performance using the same 20 samples. This is in-sample, oracle-based selection: H_best is chosen using the true intrinsic dimension d(M) and evaluated on the same data used for selection. The comparison with a fixed hyperparameter H_abs (minimizing median absolute error across all benchmark manifolds) measures the sensitivity of the estimator to H and the tension between per-dataset and global choices; it does not measure whether H_best(E,M) degrades on new samples from M or on other manifolds. Consequently, the abstract's inference that 'many estimators may not generalise well beyond the datasets on which they have been tested' goes beyond the evidence. A benign alternative explanation — estimators are sensitive to H, and no single H is optimal everywhere — fully accounts for the reported gaps. To support the overfitting claim, the authors would need to (i) tune H on one collection of samples and evaluate on held-out samples from the same manifold, and (ii) apply H_best(E,M1) to the other benchmark manifolds and show it performs worse than a robust fixed H. Without such a test, the central practical warning is an interpretation, not a demonstrated result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper surveys intrinsic dimension estimation methods, organizing them into tangential, parametric, and topological/metric families, and extends the scikit-dimension package with new estimators. The empirical core benchmarks thirteen estimators on synthetic manifolds with intrinsic dimensions 1-70 and ambient dimensions 3-96, using four sample sizes, twenty repeated point sets per condition, and multiple hyperparameter choices. The authors report estimator-specific tables, noise and curvature experiments, a throttling analysis, and qualitative assessments. The central claims are that no single estimator or hyperparameter setting works across all settings and that frequent 'overfitting' in hyperparameter identification suggests limited generalization.","tokens_in":72640,"tokens_out":9256,"duration_ms":99124,"significance":"If the results are read as a sensitivity analysis of the tested estimators within the stated hyperparameter grids, the paper is a substantial and useful contribution. The benchmark is unusually careful: twenty repetitions, standard deviations, four sample sizes, and transparent tables; the authors also disclose in Appendix C that the 'best' hyperparameter is selected with knowledge of the true dimension. The code contribution to scikit-dimension is valuable. The main weakness is interpretive: the abstract and Section 3.3 present the oracle-based sensitivity comparison as evidence of 'overfitting' and poor generalization, which is not what the protocol measures.","major_comments":[{"comment":"The claim that 'overfitting is frequent' and that 'many estimators may not generalise well beyond the datasets on which they have been tested' is not supported by the experimental protocol. In Appendix C, the 'best' hyperparameter for estimator E on manifold M is selected as the minimizer of |d_hat(E,M,H) - d(M)| and the reported performance uses the same 20 point sets used for selection; the 'med abs' and 'med rel' columns use a hyperparameter chosen by minimizing median absolute or relative error against the true dimensions over the same benchmark manifolds. This measures hyperparameter sensitivity and the tension between per-dataset and global choices, not generalization to new samples or new manifolds. To support the abstract's wording, the authors should add a held-out evaluation: tune H on one collection of samples and evaluate on independent samples from the same manifold, and apply H_best(E,M1) to other benchmark manifolds. Without such a test, the evidence supports only 'no single hyperparameter setting is optimal across benchmarks'.","section":"Abstract; Section 3.3; Appendix C"},{"comment":"The qualitative column 'No tailoring of params' in Table 3 is based on the 'med abs' and 'med rel' hyperparameter choices, which are themselves obtained by minimizing the median absolute or relative error against the true dimensions across the benchmark manifolds (Appendix C). This is not an untuned or default setting; it is a globally oracle-tuned fixed parameter. The ticks in Table 3 therefore overstate the 'no tailoring' property. Please either use the scikit-dimension default parameters as the fixed baseline, or rename the column to something like 'insensitivity to a global hyperparameter choice' and adjust the discussion.","section":"Table 3; Appendix C"},{"comment":"The statement in Section 3.3 that lPCA 'appears to give the correct dimension every time' is contradicted by Table C1, where the best-hyperparameter estimate for M10d Cubic (true dimension 70) is 55.2 at every sample size. The caveat about hyperparameter tuning does not resolve the contradiction, because the quoted value is already the best over the grid. Please correct the claim or restrict it to the datasets on which lPCA actually succeeds.","section":"Section 3.3; Table C1"}],"minor_comments":[{"comment":"The abstract's generalizations should carry the qualifier that the empirical conclusions are relative to the hyperparameter grids listed in Appendix D, as the authors themselves acknowledge in Remark 3.2; as written, 'overfitting is frequent' reads as a statement about all hyperparameter ranges.","section":"Abstract; Remark 3.2"},{"comment":"The reference to 'eq. (14)' in the paragraph after the Costa and Hero discussion is a forward reference to Theorem 2.5; please renumber or point to eq. (10), where the Renyi-entropy connection is first visible.","section":"Section 2.3.1"},{"comment":"The WODCap throttling calculation would benefit from a cleaner derivation: the displayed expression for S(d) mixes Gamma-function notation and the resulting approximation for k is hard to parse; in particular, the notation involving Gamma((d+1)/2, 1/2) should presumably be Gamma((d+1)/2).","section":"Section 3.4.2"},{"comment":"There are several typographical errors: 'is is' in Section 4, 'Datas set' in Algorithm 1, 'M20 Norm' in the Table C10 caption, and 'Theorem 2.6' in Figure 6 and Section 3.3 where Remark 2.6 is meant.","section":"Section 4; Algorithm 1; Table C10"},{"comment":"The row for M2/M9 ('Affine 3to5 and M9 Affine 3, 20 5, 20') is hard to read; please reformat the table or caption to make the manifold, dimension, and ambient dimension columns unambiguous.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical survey and likely acceptable after revision. The main required change is interpretive: replace the unsupported 'overfitting/generalization' language with a sensitivity-based statement, and correct the lPCA claim. I would not require new experiments, though a small held-out test would strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful benchmark and survey of intrinsic dimension estimators, with a lot of new empirical work. Second, the central 'overfitting is frequent' claim is not actually tested: the paper measures hyperparameter sensitivity, not generalization.\n\nWhat's new and good: the benchmark covers 20+ estimators, four sample sizes, 20 repeats, multiple hyperparameter choices, plus separate noise and curvature experiments. The categorization into tangential, parametric, and topological families is fine but not novel; the real value is the comparison tables and the added implementations in scikit-dimension. The authors are honest about limitations (Remark 3.2), and they clearly flag when the best results use oracle tuning.\n\nThe soft spot is the overfitting language. In Appendix C, H_best is chosen using the true dimension on the same 20 samples used for evaluation. That is in-sample oracle selection; it measures sensitivity of the estimator to H, not whether H_best degrades on held-out data. The abstract's claim that 'many estimators may not generalise well beyond the datasets on which they have been tested' is an interpretation, not a demonstrated result. A benign alternative—estimators are sensitive to H, and no single H is optimal everywhere—fully explains the gaps. The authors should either add a held-out test (tune on some samples, evaluate on others; or apply H_best from one manifold to others) or soften the abstract. This is fixable; the underlying data are solid.\n\nMinor: the Github fork URL is missing, so the new implementations are not independently verifiable from the manuscript. Also the 'significant if true' is subfield-level, not field-level.\n\nWho it's for: practitioners who want guidance on choosing estimators, and researchers benchmarking new methods. It deserves a serious referee. The survey alone is worth publishing; the benchmark makes it useful. I'd accept for review and ask for a revised abstract plus the held-out experiment.","headline":"A useful, honest benchmark of dimension estimators whose 'overfitting' claim overstates the evidence: it measures hyperparameter sensitivity, not held-out generalization, but the empirical work is solid and deserves peer review.","tokens_in":73259,"tokens_out":1946,"would_cite":true,"duration_ms":24378,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No single dimension estimator works across all datasets; most need careful tuning and many overfit benchmarks.","keywords":["intrinsic dimension estimation","benchmark comparison","hyperparameter sensitivity","curvature and noise robustness","tangential estimators","parametric estimators","topological data analysis"],"falsifier":"A concrete falsifier would be to find a single estimator with a fixed hyperparameter setting that performs accurately on the same benchmark manifolds across a wide range of intrinsic dimensions, curvatures, noise levels, and out-of-distribution datasets, contradicting the paper's claim that no universal estimator or hyperparameter set exists.","tokens_in":72178,"feed_emoji":"📏","tokens_out":2444,"duration_ms":24830,"temperature":0.7,"pith_summary":"This survey asks a practical question: when a dataset lives on a low-dimensional structure inside a high-dimensional space, how reliably can existing estimators recover that intrinsic dimension in real conditions? The authors review and categorize the main estimation methods by the geometric information they exploit, then test them on benchmark manifolds. Their central finding is that there is no single estimator or set of hyperparameters that performs well across all settings, and that choosing the best hyperparameters for benchmark datasets frequently amounts to overfitting, so many estimators may not generalize beyond the data they were tuned on.","feed_headline":"Intrinsic dimension estimators: no winner across all data","feed_subtitle":"Benchmarking shows most estimators need case-by-case tuning and often overfit the test set.","key_machinery":"The central organizing framework is the categorization of estimators by the geometric information they use: tangential estimators that detect a local affine tangent structure, parametric estimators that rely on dimension-dependent probability distributions of distances or angles, and estimators based on topological or metric invariants such as minimum spanning trees, k-nearest-neighbour graphs, and magnitude. This categorization is what allows the authors to explain why different estimators fail differently on curvature, noise, boundary effects, and high dimensional throttling, where the neighbourhood size k sets a hard ceiling on the dimension an estimator can return.","core_discovery":"The paper's core claim is stated in its conclusion: \"There is no single estimator or set of hyperparameters that can perform well across all settings.\" This is established empirically by benchmarking a wide cohort of estimators on synthetic manifolds, and by showing that the performance of most estimators depends heavily on the choice of hyperparameters, with the best choice often varying from dataset to dataset. The authors also find that most estimators underestimate intrinsic dimension on high-dimensional datasets, while overestimation can occur on datasets such as SO(n), and that no estimator performs satisfactorily on non-linear manifolds of dimension above six.","pith_inferences":["The paper's conclusion implies that the apparent accuracy of an estimator on a benchmark should be read as a statement about that benchmark and that tuning process, not about the estimator alone.","A systematic rule for choosing hyperparameters, for instance based on features of the data before estimation, would be a direct extension of this work and could be tested on the same benchmark suite.","The finding that slightly positively curved surfaces can be easier for lPCA than flat ones suggests a testable extension: whether this bias generalizes to other tangential estimators or depends on the specific aggregation and thresholding.","The paper's evidence that estimators differ greatly in variance across random samples implies that reporting a single number for intrinsic dimension without uncertainty is misleading; interval estimates could be derived from running estimators across hyperparameters and subsamples."],"forward_implications":["Practitioners should not trust a single dimension estimator with default hyperparameters and should instead use a range of estimators and hyperparameters to assess how sensitive the inferred dimension is to modelling assumptions.","Benchmark results obtained by tuning hyperparameters to the test set should not be interpreted as a predictor of real-world performance, because the paper shows that overfitting to benchmark datasets is frequent.","On high-dimensional manifolds, tangential estimators face a throttling limit that grows with dimension, and slope-inference estimators can even produce negative estimates, so caution is required.","For practitioners with limited data, global estimators are generally more suitable, while local estimators require more samples because each neighbourhood must contain enough points.","Future work on automated or adaptive hyperparameter selection is needed, since the paper finds that no simple rule performs well universally."],"supporting_citations":[{"why":"Provides the scikit-dimension package and its benchmark manifolds that the authors use and extend with new estimators.","marker":"[6]"},{"why":"Supplies the standard benchmark framework and datasets that the paper uses for its empirical evaluation.","marker":"[18]"},{"why":"Establishes the maximum likelihood estimator for intrinsic dimension that many parametric estimators build on, and whose bias and aggregation are studied.","marker":"[67]"},{"why":"Introduces the TwoNN estimator, whose ratio distribution is used as a benchmark estimator and a basis for comparison.","marker":"[32]"},{"why":"Provides benchmark manifolds and high-dimensional estimators such as MiND ML that the paper evaluates.","marker":"[89]"},{"why":"Supplies the Euclidean functional theory that underlies the KNN and PH0 estimators.","marker":"[114]"},{"why":"Provides the persistent homology dimension recovery theorem used by the PH0 estimator.","marker":"[93]"},{"why":"Introduces the correlation dimension and correlation integral estimator used in the comparisons.","marker":"[42]"},{"why":"Introduces the DANCo estimator that performs well on high-dimensional datasets in the evaluation.","marker":"[20]"}],"fun_headline_variants":["No universal dimension estimator: survey finds all fail on some data","Dimension estimation survey: no one-size-fits-all approach","Intrinsic dimension estimators: none works everywhere, survey says","Dimension estimation: most methods overfit, need per-case tuning","Survey: no single dimension estimator succeeds across all data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the chosen benchmark manifolds and the range of hyperparameters tested represent how dimension estimators will actually be used in practice.","fun_headline_variants_meta":{"raw":{"variants":["No universal dimension estimator: survey finds all fail on some data","Dimension estimation survey: no one-size-fits-all approach","Intrinsic dimension estimators: none works everywhere, survey says","Dimension estimation: most methods overfit, need per-case tuning","Survey: no single dimension estimator succeeds across all data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1423,"prompt_tokens":838,"completion_tokens":585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":454,"tokens_out":585,"duration_ms":6188,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:14:06.863776+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier would be to find a single estimator with a fixed hyperparameter setting that performs accurately on the same benchmark manifolds across a wide range of intrinsic dimensions, curvatures, noise levels, and out-of-distribution datasets, contradicting the paper's claim that no universal estimator or hyperparameter set exists.","supporting_citations":[{"cited_title":"Maximum Likelihood Estimation of Intrinsic Dimension","cited_arxiv_id":null,"evidence_quote":"Establishes the maximum likelihood estimator for intrinsic dimension that many parametric estimators build on, and whose bias and aggregation are studied."},{"cited_title":"Rozza, G","cited_arxiv_id":null,"evidence_quote":"Provides benchmark manifolds and high-dimensional estimators such as MiND ML that the paper evaluates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Euclidean functional theory that underlies the KNN and PH0 estimators."},{"cited_title":"Fractal dimension and the persistent homology of random geometric complexes","cited_arxiv_id":null,"evidence_quote":"Provides the persistent homology dimension recovery theorem used by the PH0 estimator."},{"cited_title":"Measuring the strangeness of strange attractors","cited_arxiv_id":null,"evidence_quote":"Introduces the correlation dimension and correlation integral estimator used in the comparisons."}],"review_version":1}