{"id":"4192176a-414b-42a3-ba53-1445b4b453e7","arxiv_id":"2501.00241","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A regression study of 55 Hugging Face model cards reports correlations between hyperparameters and metrics, but does not establish comparability or causality.","lead":"This paper analyzes 55 fine-tuned DistilBERT models on Hugging Face and fits polynomial regressions to see how learning rate, batch size, and epochs relate to reported accuracy, F1, and loss. It suggests tuning goals conflict across metrics, but the data are scraped model cards, not controlled experiments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The regressions treat model-card metrics from different evaluation sets as directly comparable; the headline p-values can therefore reflect dataset confounds, not hyperparameter effects. Internal variable-label inconsistencies reinforce this.","rationale":"The paper's strongest claim is actionable only if a practitioner can transfer the regression coefficients to their own fine-tuning run. That transfer requires the scraped model-card metrics to be commensurable; otherwise the observed variability is dataset variability, not hyperparameter variability. The reader's weakest assumption identified exactly this comparability problem in Section III-A, and the manuscript text itself supports the concern: data are extracted from README.md and config.json with no record of evaluation set, split, or metric definition. I am not objecting to observational analysis as such; the fatal issue is an unobserved confounder (dataset identity) that is correlated with both the hyperparameters chosen by uploaders and the resulting metrics, and the paper provides no covariate, fixed effect, or stratification for it. The relative regression makes the problem worse by subtracting the SST-2 baseline metric from models evaluated on other datasets. Independently, the variable-label inconsistencies across Tables 1, 3, and 6 mean the numeric results cannot be interpreted consistently, so the specific p-values cited in the abstract are not reproducible even from the paper's own tables. These two problems together fully support the reader's REJECT verdict.","tokens_in":11362,"tokens_out":3413,"duration_ms":35125,"concrete_test":"Restrict the analysis to model cards whose README explicitly reports metrics on the same benchmark and split (e.g., SST-2 dev set), and re-estimate the Section IV polynomial regressions, including the X9 interaction. If the epochs-by-batch-size coefficient or its p-value changes materially or loses significance, the original result is an artifact of pooling across evaluation sets. As a secondary check, audit the codebook mapping X1-X9 to polynomial terms; if a single mapping cannot reproduce all three tables, the regressions are not a single coherent analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the accuracy, F1, and loss values scraped from 55 different Hugging Face model cards (Section III-A) are comparable measurements of the same underlying quantity. They are not. Each card reports evaluation on whatever dataset, split, and reporting convention the uploader chose, so a model trained with batch size 16 and evaluated on an easy split can look better than one trained with batch size 32 and evaluated on a hard split. The polynomial regressions in Section IV then mix dataset identity and task difficulty into the hyperparameter coefficients. The 'relative' regressions compound this: they subtract the SST-2 baseline metric from each model's metric, which is not an incremental fine-tuning effect unless every model was evaluated on the same benchmark as the baseline. A second, independent defect is internal: the variable index X2 is 'batch size' in Table 1 but 'learning rate' in Tables 3 and 5 and in Section IV.B; X7 is 'learning rate' in Table 1, a batch-size-related term in Table 3, and 'learning rate squared' in Table 6. The reported p=0.001 'epochs x batch size' interaction therefore cannot be trusted as a statement about that interaction until a single consistent codebook is supplied for X1-X9.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper scrapes Hugging Face model cards for 55 DistilBERT text-classification models, extracting reported hyperparameters (learning rate, batch size, epochs) and metrics (accuracy, F1, loss), and fits polynomial regressions in both absolute and relative (baseline-subtracted) forms to identify significant hyperparameter effects and interactions. The central claim is that hyperparameter configurations cause metric variability and trade-offs, with the epochs-by-batch-size interaction driving F1-score improvements (p=0.001), and that these findings inform practical fine-tuning strategies for DistilBERT and other models.","tokens_in":11616,"tokens_out":4513,"duration_ms":41726,"significance":"If the findings were valid, the paper would offer an inexpensive, data-driven route to fine-tuning guidance for DistilBERT. The authors are transparent about their data source and report full regression tables, which is a useful discipline. However, because the underlying data are uncontrolled model-card reports with no common evaluation dataset, and because the variable definitions are internally inconsistent, the empirical contribution cannot support the causal or practical claims made. The paper's significance is therefore limited to a cautionary demonstration of the hazards of meta-analyzing repository metadata.","major_comments":[{"comment":"The metrics and hyperparameters are extracted from README.md and config.json of 55 different Hugging Face models with no control for the evaluation dataset, data split, or reporting convention. Consequently, the regression coefficients in Section IV blend dataset identity and task difficulty with hyperparameter effects; statements such as 'batch size significantly impacts accuracy' (Section IV.A.1.a) cannot be interpreted as effects of the hyperparameter. This is a load-bearing confound that cannot be fixed within the manuscript's scope without recollecting data under a common evaluation protocol.","section":"III-A"},{"comment":"The mapping of variables X1-X9 is never defined in the paper, and the verbal labels shift across tables: Table 1 treats X2 as batch size and X7 as learning rate, Table 3's text treats X2 as learning rate and X3 as batch size, and Table 6 treats X7 as learning rate squared. Without a single consistent codebook, the reported p-values (including the headline 'epochs x batch size' interaction, p=0.001) cannot be assigned to the claimed hyperparameters, making the central results unverifiable and the abstract's attribution of p=0.028 to batch size for accuracy irreconcilable with Table 1's p=0.000 for X2.","section":"IV, Tables 1, 3, 5, 6"},{"comment":"The authors state that they 'focused on metrics and hyperparameters with a sufficient number of observations to allow for statistically significant fine-tuning strategies.' Selecting variables after inspecting significance invalidates the reported p-values as confirmatory evidence and renders the analysis circular: the 'findings' are the fitted coefficients of the same regressions that were used to decide which variables to include.","section":"III-D"},{"comment":"The relative regression subtracts the baseline model's metric and hyperparameter values from each fine-tuned model. This is only interpretable as an incremental fine-tuning effect if every model was evaluated on the same benchmark as the baseline (distilbert-base-uncased-finetuned-sst-2-english). Since the models are drawn from different model cards, the relative differences are artifacts of dataset choice, not of hyperparameter adjustments, so the relative-regression p-values (e.g., learning rate reducing loss, p=0.027) do not support the conclusions drawn.","section":"III-E"}],"minor_comments":[{"comment":"The abstract assigns p=0.028 to batch size for accuracy, but Table 1 reports p=0.000 for the variable labeled X2 (batch size) and Table 3 reports p=0.028 for X2, which the text labels 'learning rate'; the abstract conflates the two variables, and the inconsistency propagates into the strategy recommendations in Section IV.B.","section":"Abstract and IV.A.1.a"},{"comment":"The paper never writes out the polynomial regression equation or defines X1-X9 (their meanings, the polynomial degree, or the coding of interaction terms), making it impossible to reconstruct the analysis or interpret the magnitudes of coefficients such as 3.7e+08 in Table 1.","section":"III-E and IV"},{"comment":"The F1 absolute regression table omits the constant term that appears in Tables 1, 2, and 6, without any explanation; this inconsistency makes it difficult to assess the model specification across metrics.","section":"Table 3"},{"comment":"Figure references are internally inconsistent: the text cites 'Figures 1 and 2' and then 'Figures 2 and 4' for the accuracy results, which does not match the figure numbering in the manuscript and suggests errors in cross-referencing.","section":"IV.A.1.b"},{"comment":"Reference [3] is described as 'Fine-Tuning Large Language Models for Task-Specific Data' but the cited paper is actually about domain-specific machine translation; the related-work descriptions need to be checked against the actual references.","section":"II-C"},{"comment":"The regressions do not report sample size, R-squared, or any multiple-testing correction, which is particularly important because six separate regression models are fit on the same small set of 55 model cards and many coefficients are reported with p-values near 0.05.","section":"IV"}],"recommendation":"reject","confidential_remarks":"The paper's central flaw—comparing metrics from heterogeneous Hugging Face model cards—is intrinsic and cannot be remedied by revision; the authors would need to run a controlled fine-tuning study on a shared benchmark. The additional variable-label inconsistencies and significance-based variable selection further undermine the reported results. I recommend rejection, though I would encourage the authors to consider a properly controlled study if they wish to pursue this research question."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a reasonable motivation: take the 55 DistilBERT fine-tunes on Hugging Face, regress accuracy/F1/loss against learning rate, batch size, and epochs, and see which hyperparameters matter and how metrics trade off. The idea of looking at model-card metadata as a natural experiment is not crazy, and the paper is plainly written. But the central finding—that, say, the epochs-by-batch-size interaction drives F1 (p=0.001)—is not something you can rely on.\n\nTwo problems sink it. First, the data are not comparable. Each model card reports evaluation on whatever dataset and split the uploader chose. A batch size of 16 on an easy classification task will look better than batch size 32 on a hard task, and the regression has no way to separate hyperparameter effects from task difficulty. The relative analysis is meant to fix this by subtracting the SST-2 baseline metric, but that only works if every fine-tuned model was evaluated on SST-2. They were not. So the headline p-values likely reflect dataset confounds, not causal hyperparameter effects.\n\nSecond, the variable labels are internally inconsistent. In Table 1, X2 is batch size and X7 is learning rate. In Table 3, X2 is learning rate and X3 is batch size. In Table 5, X2 is learning rate again, X7 is something else. The interaction term X9 is described as epochs-by-batch-size in the F1 tables, but the paper never gives a single codebook for X1-X9. When the same symbol means different things in different tables, every reported significance is suspect.\n\nThere is also a selection-on-significance issue: the authors say they narrowed from nine hyperparameters and nine metrics to the three and three that had enough observations and significant trends. That is a garden path, and it means the remaining p-values are not reliable even within the paper's own assumptions.\n\nWhat is genuinely new? The particular dataset of 55 model cards, and the polynomial-regression setup with interactions. That could be a starting point for a descriptive study, but the paper draws prescriptive conclusions (\"set batch size and epochs first, tune learning rate later\") that the data cannot support. No artifact is released, so the regressions cannot be checked.\n\nWho is this for? Someone who wants a quick, admittedly noisy look at how DistilBERT fine-tunes vary across the HF hub, and who treats it as anecdote. It is not for someone who wants to know how to tune a model. I would not take this as actionable guidance, and I would not send it to a serious venue without major changes—at minimum, a consistent codebook, a dataset filter that restricts models evaluated on the same benchmark, and a preregistered variable selection.\n\nRecommendation: reject; do not invite revision in its current form. The kernel is worth a resubmission if the authors rework the analysis around comparable data.","headline":"A descriptive snapshot of 55 DistilBERT model cards that overreaches: the regressions mix incomparable metrics, and the variable labels are inconsistent, so the headline p-values are not trustworthy.","tokens_in":12103,"tokens_out":2832,"would_cite":false,"duration_ms":26413,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning DistilBERT metrics trade off against one another, and the epochs–batch-size interaction is the strongest driver of F1.","keywords":["DistilBERT","fine-tuning","text classification","hyperparameter variability","polynomial regression","metric trade-offs","F1-score","learning rate"],"falsifier":"Run a controlled fine-tuning experiment on a fixed text-classification dataset with a factorial design over learning rate, batch size, and epochs; if the interaction between epochs and batch size is not significant for F1, or if increasing learning rate does not reduce loss relative to baseline while lowering accuracy, the central claim would be contradicted.","tokens_in":11188,"feed_emoji":"🤖","tokens_out":4248,"duration_ms":37686,"temperature":0.7,"pith_summary":"This paper tries to establish that the effects of fine-tuning hyperparameters on DistilBERT text-classification models are metric-specific and interactive, so that optimizing one metric often trades off against another. Using polynomial regression on data from 55 publicly available fine-tuned DistilBERT models, the authors report that batch size consistently predicts accuracy and F1, learning rate mainly matters for reducing loss relative to baseline, and the interaction between epochs and batch size is the strongest significant predictor of F1 (p = 0.001). The practical upshot is that fine-tuning should be staged: establish a solid foundation with batch size and epochs, then refine learning rate while monitoring all metrics. If the claim is right, practitioners should expect non-linear hyperparameter interactions and plan for trade-offs rather than single-metric optimization.","feed_headline":"Epochs and batch size together decide DistilBERT F1","feed_subtitle":"A 55-model regression shows tuning one metric can hurt another, so balance matters.","key_machinery":"The central machinery is a set of polynomial regressions in two forms: absolute (raw metric values on raw hyperparameters) and relative (differences from the baseline DistilBERT variant). Each regression includes linear terms, quadratic terms, and interactions for learning rate, batch size, and epochs (variables x1–x9); the relative regressions are meant to isolate incremental gains from fine-tuning. These regressions identify which terms are statistically significant for each metric and are supplemented by heatmaps and scatter plots.","core_discovery":"The paper's central claim is that hyperparameter effects on fine-tuned DistilBERT are not uniform across metrics: in polynomial regressions of 55 model cards, batch size positively predicts accuracy and F1 in absolute terms, learning rate reduces loss in relative (difference-from-baseline) terms, and the interaction term between epochs and batch size is the most significant predictor of F1 (p = 0.001), while the quadratic batch-size term is negative, indicating diminishing returns. The authors interpret this as evidence that fine-tuning strategies must address non-linear interactions and trade-offs, and they propose an incremental approach: foundational tuning of batch size and epochs first, then careful learning-rate refinement.","pith_inferences":["Because the regressions use observational data from model cards that may be evaluated on different datasets, the reported effects mix hyperparameter influence with dataset and reporting differences; a controlled factorial fine-tuning study would be needed to confirm causality.","The interaction pattern between epochs and batch size could be tested on other transformer families (e.g., BERT or RoBERTa) to see whether it is a general property of fine-tuning or specific to DistilBERT.","The proposed two-stage strategy—fix batch size and epochs, then tune learning rate—could be operationalized as an automated hyperparameter search that first optimizes the epoch–batch interaction and then does a local learning-rate search."],"forward_implications":["Batch size and epochs should be chosen jointly, since their interaction is the strongest significant predictor of F1.","Learning-rate increases can lower loss relative to a baseline but may degrade accuracy, so learning rate should be tuned incrementally.","Because batch size has diminishing returns (negative quadratic term for F1), very large batch sizes can hurt performance.","Fine-tuning evaluation should report accuracy, F1, and loss together, since strategies that optimize one metric can harm another."],"supporting_citations":[],"fun_headline_variants":["Epoch-batch interaction is top predictor of DistilBERT F1","DistilBERT tuning: batch size and epochs set F1, learning rate sets loss","Fine-tuning DistilBERT: watch the epoch-batch size combo","Learning rate trims loss but disturbs accuracy in DistilBERT","Hyperparameter interplay beats single tweaks for DistilBERT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that accuracy, F1, and loss values reported on different model cards are directly comparable, even though each model may be evaluated on a different dataset, split, or reporting convention.","fun_headline_variants_meta":{"raw":{"variants":["Epoch-batch interaction is top predictor of DistilBERT F1","DistilBERT tuning: batch size and epochs set F1, learning rate sets loss","Fine-tuning DistilBERT: watch the epoch-batch size combo","Learning rate trims loss but disturbs accuracy in DistilBERT","Hyperparameter interplay beats single tweaks for DistilBERT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000497,"raw_usage":{"total_tokens":2418,"prompt_tokens":911,"completion_tokens":1507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1413}},"tokens_in":527,"tokens_out":1507,"duration_ms":12225,"temperature":1.0,"reasoning_tokens":1413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:55:19.372130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled fine-tuning experiment on a fixed text-classification dataset with a factorial design over learning rate, batch size, and epochs; if the interaction between epochs and batch size is not significant for F1, or if increasing learning rate does not reduce loss relative to baseline while lowering accuracy, the central claim would be contradicted.","supporting_citations":[],"review_version":1}