{"id":"735e8e4f-d3ba-4df0-879d-de8ce83a691f","arxiv_id":"2505.19150","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new curated database, teMatDb272, filters thermoelectric data by requiring reported ZT to match ZT recalculated from digitized properties.","lead":"This paper introduces teMatDb272, a curated database of thermoelectric materials built by digitizing temperature-dependent property data from published figures. It also presents a self-consistent ZT filtering method that checks whether each paper's reported efficiency (ZT) matches the properties it reports, offering a quality-checked resource for machine learning and materials design.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sc-ZT filtering ensures internal consistency but not absolute accuracy; digitized TEPs are never checked against ground-truth values, so teMatDb272's reliability as an experimental benchmark is unverified.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the Sc-ZT filter treats agreement between ZTfig and ZTTEP as evidence of data quality, but self-consistency alone cannot validate absolute accuracy. If the underlying experiments are wrong in a way that affects both the reported ZT and the reported TEPs, the filter passes the sample. The paper provides real evidence of internal cleaning power: the Q-Q R2 improves from 0.6864 to 0.9324 after filtering (Table 4), and applying the protocol to Starrydata2 removes obvious unit errors. These results support the claim that the filter removes gross inconsistencies, but they do not establish that the surviving 272 samples are accurate benchmarks for machine learning or device modeling. The weakest link is therefore not in the arithmetic of the filter but in the external validity of the resulting dataset. The proposed synthetic false-acceptance test would directly quantify this risk: it uses known ground truth, so it can show whether samples with genuinely wrong TEPs can pass the Sc-ZT criteria. A REJECT verdict would be too strong because the dataset may well be accurate and the filter is a reasonable quality-control tool; however, the reliability claim should remain conditional until the filter's ability to control absolute error is demonstrated. For these reasons, I agree with the reader's CONDITIONAL verdict and recommend no change to the verdict category, while emphasizing that external validation is the decisive missing step.","tokens_in":14543,"tokens_out":4446,"duration_ms":46195,"concrete_test":"Synthetic end-to-end false-acceptance test: generate known ground-truth α(T), ρ(T), κ(T) curves for roughly 100 representative materials, render them as publication-style figures with axis labels, inject digitization noise at the levels measured in Tables S1–S2, and run the exact Sc-ZT pipeline. Then measure the false-acceptance rate among samples whose reconstructed ZTTEP differs from the true ZT by more than 10% (or by more than 0.1). If a substantial fraction of such inaccurate samples pass the filter, the self-consistency check does not guarantee accurate TEPs. A complementary check for a subset of teMatDb272 entries is to re-extract the data from original tables or high-resolution figures and compare the digitized values directly to the source numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that teMatDb272 is a reliable experimental benchmark rests on the assumption that a small δ(ZT)=ZTfig−ZTTEP implies accurate α, ρ, κ data. This is only a self-consistency check: both ZTfig and ZTTEP derive from the same published experimental data, and if the original measurements contain a systematic error (e.g., a biased κ measurement, an incorrect density correction, or a Seebeck contact artifact), the filter will pass the sample because the inconsistency is small. The paper's digitization-noise analysis (Section 2, Tables S1–S2) quantifies random extraction error, but there is no comparison against authoritative values, original data tables, or independent remeasurement for any of the 272 samples. The Starrydata2 application (Figure 5) shows the filter catches gross unit/label errors, but that only demonstrates sensitivity to large, self-inflicted inconsistencies, not sensitivity to subtle but real measurement errors. Consequently, the Sc-ZT pass criteria in Table 1 are necessary but not sufficient for the dataset to serve as a high-quality experimental benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces teMatDb, a database of temperature-dependent thermoelectric properties (Seebeck coefficient, electrical resistivity, thermal conductivity, and ZT) digitized from published figures. The central methodological contribution is a self-consistent ZT (Sc-ZT) filter that compares the reported ZT (ZTfig) with the ZT recalculated from the digitized TEPs (ZTTEP), defining δ(ZT) = ZTfig − ZTTEP. Six filters (Avg ZT, Peak ZT, Max error, RMS error, and two normalized versions) with default thresholds (0.1, 0.1, 0.1, 0.1, 0.2, 0.2) are applied to a parent database teMatDb v1.1.6, reducing 355 samples to teMatDb272 (272 samples from 262 publications). The same protocol is applied to Starrydata2 to generate starryz10840. The paper presents Q-Q plots and ZT-ZT comparisons as evidence of improved consistency, and provides code and data on GitHub/Zenodo. The authors claim the filtered dataset is a robust, high-quality experimental benchmark for machine learning and materials design.","tokens_in":14861,"tokens_out":7375,"duration_ms":58284,"significance":"If the dataset were validated for absolute accuracy, teMatDb272 would be a valuable benchmark: it covers a broad compositional space, includes self-consistency checks that catch unit errors and extrapolation artifacts, and ships with open code and data, which is a strength. The Sc-ZT filtering framework is a useful, physically motivated quality-control tool that could be applied to other experimental databases. However, the central claim of 'high-quality' is currently overreaching: the validation in Section 4 demonstrates only that the filter enforces internal consistency between digitized TEPs and reported ZT, not that the values are accurate. Because the paper emphasizes the dataset as a reliable experimental benchmark, the lack of external validation is a load-bearing issue that should be addressed before publication.","major_comments":[{"comment":"The validation of the Sc-ZT filter is self-referential: Table 4 reports that the Q-Q R2 of δ(ZT) improves from 0.6864 to 0.9324 after filtering, but this improvement is expected because the filter removes samples with large δ(ZT), truncating the distribution that the R2 is computed on. No independent validation is provided: the digitized TEPs are never compared against original data tables, authoritative measurements, or remeasured values for any sample in teMatDb272. Consequently, the claim that teMatDb272 constitutes a 'high-quality' experimental benchmark is not supported; the filter demonstrates internal consistency, not absolute accuracy. I recommend either adding an external validation of a subset of samples against the original publications' tables or tempering the claims to 'internally consistent' throughout the abstract and main text.","section":"Section 4, Table 4"},{"comment":"The default Sc-ZT filter thresholds (0.1, 0.1, 0.1, 0.1, 0.2, 0.2) in Table 1 are presented without derivation or sensitivity analysis. The digitization noise analysis in Tables S1-S2 shows that relative errors in reciprocals can reach 8% and the mean relative error in reciprocal values can be as large as 6.26%, yet the thresholds are stated in absolute ZT units (0.1) and in normalized units (0.2). For a low-ZT sample (e.g., ZT ~ 0.3), an absolute error of 0.1 is a 33% error, which may be acceptable or not depending on the application. The authors should justify the thresholds using the digitization uncertainty or provide a systematic sensitivity analysis showing how the retained dataset changes as thresholds vary, and discuss how threshold choices affect downstream machine-learning use.","section":"Table 1, Section 2"},{"comment":"The paper quantifies digitization noise on a synthetic test figure, but does not propagate this uncertainty into the individual δ(ZT) values used for filtering. Each sample is accepted or rejected based on point estimates without per-sample error bars. Given that reciprocal amplification can cause up to 8% error (Table S2), some borderline samples in teMatDb272 may have been erroneously retained or rejected. The authors should provide a per-sample uncertainty estimate for ZTTEP and δ(ZT) (e.g., via bootstrap or repeated digitization) and report how many samples lie near the filtering boundaries.","section":"Section 2, Tables S1-S2"},{"comment":"The Q-Q plot R2 against a normal distribution is used as a data-quality metric, but the paper does not justify why δ(ZT) should be normally distributed, nor does it test this assumption (e.g., with Shapiro-Wilk or Anderson-Darling tests). The increase in R2 after filtering (from 0.6864 to 0.9324) is partly a mechanical consequence of removing extreme values, which makes the remaining distribution more compact and closer to normal. The authors should either provide a physical argument for normality of δ(ZT) or replace the Q-Q R2 with a less assumption-dependent metric, such as the mean absolute error or the proportion of samples within a tolerance band.","section":"Section 2, Figure 3"},{"comment":"The Avg ZT and Peak ZT filters in Table 1 compare averages and peaks computed over potentially different temperature ranges: Avg(ZTfig) is integrated over ΔTfig while Avg(ZTTEP) is integrated over ΔTTEP, and the manuscript states that these are computed over the ZT or TEP temperature ranges, respectively. If ZTfig extends beyond the TEP measurement range (as in the extrapolation error discussed for sample_id = 113), the average over a wider interval will differ even when the curves coincide over the overlap, leading to spurious filter outcomes. The authors should clarify whether Avg and Peak filters are evaluated over the overlapping temperature range, as done for the Max and RMS filters, and if not, justify the use of different integration intervals.","section":"Table 1, Section 2"}],"minor_comments":[{"comment":"The abstract uses 'tMatDb272' while the rest of the text uses 'teMatDb272'; please make the name consistent throughout.","section":"Abstract"},{"comment":"The row 'Entries for κ in rawTEPs' appears to be incomplete; the text states 3,422 entries, but the value is missing from the table.","section":"Table 3"},{"comment":"The phrase 'temperature rannges' should be corrected to 'temperature ranges'.","section":"Section 2, Sc-ZT filtering protocol"},{"comment":"The limitations paragraph would benefit from an explicit statement that Sc-ZT filtering checks self-consistency and cannot detect systematic measurement errors common to all three TEPs (for example, a biased thermal conductivity measurement that leaves ZTTEP and ZTfig both shifted).","section":"Section 4, Limitations"},{"comment":"The logarithmic-scale plot in (a) and the linear-scale zoom in (b) would be easier to interpret if the axes were labeled with units (ZT is dimensionless) and if the three datasets were identified in a legend consistent with the text.","section":"Figure 5"},{"comment":"The field 'Composition_detailed' is described as 'Full stoichiometric composition'; consider providing an example of its format in the Usage Notes to help users parse the metadata.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a data-descriptor or materials-informatics journal. The main concern is the gap between the 'high-quality' label and the internal-consistency-only validation; I would encourage the editor to require either external validation or a revised, more modest framing. The open code and data availability are notable strengths. The threshold values and the Q-Q normality metric also need stronger justification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading. The real contribution is the Sc-ZT protocol: recompute ZT from digitized alpha, rho, kappa and compare to the ZT curve in the paper. That is a simple, physical, transparent way to catch unit errors, extrapolated peaks, and digitization noise. The paper does that carefully, with a sensible error taxonomy, double/triple digitization, and open code and data. The resulting teMatDb272 is a modest but genuinely useful cleaned dataset, and the application to Starrydata2 shows the filter catches real garbage. That deserves credit.\n\nThe soft spots are real but not fatal. The thresholds (0.1, 0.1, ...) are arbitrary; the authors acknowledge that and show how sample count vs R2 trades off, so that's fine. The Q-Q R2 values are the same metric the filter optimizes, so they don't independently validate quality. More substantively, the stress-test note is right: a small delta(ZT) only says the reported ZT agrees with the digitized TEPs; it says nothing about whether either is correct against true measurement. A biased kappa or a Seebeck contact artifact passes the filter. So teMatDb272 is best described as internally consistent, not absolutely accurate. The authors' language occasionally overreaches ('high-quality', 'robust dataset'), but they list limitations and don't hide the issue.\n\nOn the circularity point: yes, filtering and then reporting the filtered set as satisfying the filter is somewhat circular, but the criterion is physically grounded and applied consistently; it's not a contrived fit. I'd call it mild, not a load-bearing flaw.\n\nVerdict: worth a serious referee pass. The dataset and protocol will be useful to anyone doing ML or meta-analysis on thermoelectric data, and the explicit thresholds can be tuned. I'd ask the authors to either soften the 'high-quality' claim or provide spot-check validation against raw data tables or remeasurement for a handful of samples. As-is, I'd cite it for the filter method, not as a gold-standard benchmark.","headline":"Sc-ZT filtering is a genuinely useful curation protocol, but the dataset's claim to 'high quality' rests on internal consistency, not on verified absolute accuracy.","tokens_in":15344,"tokens_out":1854,"would_cite":true,"duration_ms":16963,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Comparing a paper's reported ZT against the ZT recomputed from its own Seebeck, resistivity, and conductivity curves yields a curated database of 272 internally consistent thermoelectric samples.","keywords":["thermoelectric materials","figure of merit ZT","data curation","self-consistency filter","digitization error","Seebeck coefficient","thermal conductivity","benchmark dataset"],"falsifier":"Take the samples that failed the Sc-ZT filter, obtain the original authors' raw tabulated measurements, and recompute ZT from those tables. If a large fraction of the rejected samples reproduce the reported ZT when computed from the original tables, then the filter is primarily removing accurate entries whose plotted curves are at fault, and $\\delta(\\mathrm{ZT})$ is not a reliable error metric.","tokens_in":14350,"feed_emoji":"⚡","tokens_out":13355,"duration_ms":74228,"temperature":0.7,"pith_summary":"The paper tries to establish that published thermoelectric performance data can be screened for internal consistency without any new measurements, and that the screened set is reliable enough to serve as a benchmark for machine learning and device modeling. Its central move is to recompute each sample's figure of merit, ZT, from the separately plotted Seebeck coefficient, electrical resistivity, and thermal conductivity curves, and compare that value with the ZT curve printed in the same publication. Samples whose reported and recomputed ZT disagree beyond calibrated thresholds are classified by error type and removed. The result is teMatDb272, with 14,717 temperature-property pairs from 272 samples across 262 publications, in which reported ZT and reconstructed ZT agree closely. If the filter works as claimed, inflated peak ZT values in the literature become detectable and avoidable, giving data-driven thermoelectric research a quantitatively vetted foundation.","feed_headline":"Self-consistent ZT filter cuts thermoelectric data to 272 sets","feed_subtitle":"The filter recomputes ZT from its three measured inputs and keeps only data that agree.","key_machinery":"The central object is the self-consistent ZT (Sc-ZT) filter, a six-threshold screening protocol whose core quantity is the ZT error $\\delta(\\mathrm{ZT}) = \\mathrm{ZT}_{\\mathrm{fig}} - \\mathrm{ZT}_{\\mathrm{TEP}}$, evaluated on curves collocated to a common 2 K temperature grid. The filter combines average-ZT, peak-ZT, maximum-error, root-mean-square-error, and two normalised maximum-error statistics, with default thresholds $(0.1, 0.1, 0.1, 0.1, 0.2, 0.2)$ that are user-tunable. This construction turns the vague notion of 'suspicious data' into a quantitative, repeatable test and simultaneously produces a taxonomy of the error types that make thermoelectric figures of merit unreliable.","core_discovery":"The central claim is that the ZT error, defined as $\\delta(\\mathrm{ZT}) = \\mathrm{ZT}_{\\mathrm{fig}} - \\mathrm{ZT}_{\\mathrm{TEP}}$ with $\\mathrm{ZT}_{\\mathrm{TEP}} = \\alpha^2 \\rho^{-1} \\kappa^{-1} T$, is a reliable diagnostic for whether a thermoelectric publication's performance claims are internally consistent. Applying six threshold filters built on this error, the authors reduce 355 candidate samples to 272, and the distribution of residual errors shifts from strongly non-normal (Q-Q $R^2 = 0.6864$) to essentially normal ($R^2 = 0.9324$). The same protocol applied to a large open digitised dataset removes thousands of self-inconsistent entries, including unit-label mistakes that can inflate ZT by a factor of $10^6$. The paper identifies six concrete error mechanisms that produce this inconsistency: resolution error, publication bias, ZT overestimation from curve fitting, extrapolation beyond the measured temperature range, interpolation across phase transitions, and digitisation noise.","pith_inferences":["If adopted as a reporting standard, the same comparison could be run by journals before publication: authors would submit the collocated curves and the $\\delta(\\mathrm{ZT})$ plot as supplementary material, preventing inflated figures of merit from entering the literature in the first place.","The discrepancy logic generalises to any derived material property that is a deterministic function of independently plotted curves, such as power factor, Lorenz number, or lattice thermal conductivity, so the protocol could become a template for other functional-materials databases.","The paper's validation assumes the digitised TEP curves are closer to ground truth than the reported ZT values; a targeted test against original experimental tables for a handful of rejected samples would settle whether the filter is removing publication errors or digitisation artefacts.","Because the same filter caught order-of-magnitude errors from unit confusion, it could also serve as an automated unit-sanity check in crowdsourced or legacy datasets where label mistakes are common."],"forward_implications":["Any publication reporting a ZT curve alongside its three constituent property curves can be checked automatically for internal consistency, so the protocol serves as a general data-quality screen rather than a one-off curation exercise.","The curated 272-sample database is internally consistent enough to act as a benchmark for machine-learning predictions of thermoelectric property curves, with residual scatter dominated by digitisation noise rather than systematic publication bias.","When the same filter is applied to a large community dataset, it reduces the usable sample count from roughly 15,500 to about 10,800, showing that many self-inconsistent entries are detectable and removable at scale.","Users can trade dataset size against strictness: the default thresholds keep 272 samples, stricter thresholds keep 187, and the strictest tested thresholds keep 71, with the Q-Q $R^2$ of the error distribution rising to about 0.985 when only the most self-consistent data remain."],"supporting_citations":[{"why":"It provides the large community-curated thermoelectric dataset to which the Sc-ZT protocol is applied for external validation.","marker":"6"},{"why":"It supplies the earlier 33-filter physics screening protocol used as the baseline for comparison.","marker":"13"},{"why":"It is one of the two digitisation tools used to extract temperature-property curves from published figures.","marker":"21"},{"why":"It is the complementary digitisation tool used to cross-check and repeat curve extraction.","marker":"22"}],"fun_headline_variants":["Self-consistent ZT filter trims thermoelectric data to 272","Recomputing ZT from its parts exposes bad thermoelectric data","Filter checks ZT consistency and keeps 272 reliable datasets","Self-consistent ZT filter prunes data to 272 reliable sets","ZT self-consistency check yields 272 clean thermoelectric datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The filter treats the ZT recomputed from the digitised Seebeck, resistivity, and thermal-conductivity curves as the trustworthy reference, so if those digitised curves are themselves systematically wrong, or if the reported ZT was computed from a different measurement set than the plotted curves, the filter's verdict will be biased.","fun_headline_variants_meta":{"raw":{"variants":["Self-consistent ZT filter trims thermoelectric data to 272","Recomputing ZT from its parts exposes bad thermoelectric data","Filter checks ZT consistency and keeps 272 reliable datasets","Self-consistent ZT filter prunes data to 272 reliable sets","ZT self-consistency check yields 272 clean thermoelectric datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001432,"raw_usage":{"total_tokens":5781,"prompt_tokens":957,"completion_tokens":4824,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":4733}},"tokens_in":573,"tokens_out":4824,"duration_ms":31000,"temperature":1.0,"reasoning_tokens":4733,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:19:14.429984+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the samples that failed the Sc-ZT filter, obtain the original authors' raw tabulated measurements, and recompute ZT from those tables. If a large fraction of the rejected samples reproduce the reported ZT when computed from the original tables, then the filter is primarily removing accurate entries whose plotted curves are at fault, and $\\delta(\\mathrm{ZT})$ is not a reliable error metric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the earlier 33-filter physics screening protocol used as the baseline for comparison."},{"cited_title":"WebPlotDigitizer","cited_arxiv_id":null,"evidence_quote":"It is one of the two digitisation tools used to extract temperature-property curves from published figures."}],"review_version":1}