{"id":"5b0b9af5-44ae-4659-b050-5baa6202564f","arxiv_id":"2505.09792","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"A phased, human-guided multi-fidelity HPO framework with threshold meta-learning was applied to JEREX variants; Longformer with dynamic task loss weighting achieved the best relation F1 of about 49.6 on Re-DocRED.","lead":"This paper reports a case study that chains phased multi-fidelity Bayesian hyperparameter search with human pruning and a threshold-tuning meta-learner to compare variants of a joint entity and relation extraction model. The best variant, built on Longformer with dynamic task loss weighting and partitioned learning rates, reaches a relation F1 of about 49.6 on the Re-DocRED dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's '25E+Cal' column is labeled both 'test scores' and 'validation scores'; if the reported 49.64 is a validation score, the paper's headline result on Re-DocRED is unsupported, and no same-dataset baseline makes 'best improvement' untestable.","rationale":"The reader's weakest-assumption, Hypothesis 2 on cross-fidelity consistency, is legitimate, but it is not the most immediate threat to the central claim. Table 2's rank flips are real and the pruning step can in principle discard the eventual winner, yet the paper explicitly labels Hypothesis 2 as a hypothesis, and its own Table 2 shows the eventual top-2 configurations were not discarded. The more direct problem is that the reported headline number may not be a test score at all: the table title and caption directly contradict each other. The distinction between validation and test matters because configuration selection, threshold calibration, and trial pruning all use validation; only a held-out test score can support the abstract's claim of enhanced performance and Section 4's 'best improvement.' Additionally, without a same-dataset baseline, 'improvement' has no comparison point. These concerns are concrete and addressable with released logs or a rerun, so I keep a conditional verdict rather than moving to reject; the condition should be corrected, unambiguous reporting of test scores with variance and a baseline, not merely additional prose.","tokens_in":18684,"tokens_out":8833,"duration_ms":92619,"concrete_test":"Obtain or re-run the pipeline for the top configuration (Long-DTL LR0-L2) using Algorithm 2 on Re-DocRED with the stated hyperparameters. Record separately (1) the validation F1 at the end of the calibration loop and (2) the held-out test F1 from the final Test step, and repeat the full pipeline three times as Appendix B.3 says, reporting mean and per-repeat values. If the 49.64 entry corresponds to validation, re-run the final test and compare to the original JEREX run on the same Re-DocRED splits; the headline should be changed to the test number. This settles whether Table 2's central number is a test result or an artifact of selection and calibration.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline result of Section 4, \"The LongFormer-based model with DTL showed the best improvement reaching scores close to 50%,\" depends entirely on Table 2's '25E+Cal' column. The table is internally inconsistent about what that column contains. The table title says \"F1-micro test scores of 16 model configurations,\" but the caption says \"Columns 25E+Cal contain validation F1 scores of the full-fidelity 25-epoch training-calibration-validation cycle under an Optuna study of 9 trials with Hyperband, cold primed with the top-3 HP configurations of the Warmup sprint.\" Section 4 and the abstract treat these numbers as test performance on Re-DocRED. Appendix B.3 (Algorithm 2) does describe a final held-out Test after threshold calibration, and says the pipeline was repeated three times with the mean reported, but no column is labeled as that final test and no repeat-level variance is given. If the 49.64 value is a validation score used only for model selection and threshold calibration, then the central claim that JEREX-L \"enhances machine learning model performance\" (abstract) and the statement that the LongFormer-DTL variant reached \"best improvement\" have no demonstrated test-set support. The same column also has no same-dataset baseline, neither the original Eberts-Ulges JEREX on Re-DocRED nor a standard HPO comparator, so even a correctly labeled test score of 49.64 cannot establish \"improvement\" on its own. This is load-bearing because every downstream claim, the winner, the workflow's utility, and the comparison among the 16 variants, is read off this column.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes an interim case study of a phased, human-guided hyperparameter optimization (HPO) workflow applied to JEREX-L, a collection of variants of the Eberts and Ulges (2021) joint entity and relation extraction model. The workflow consists of three sprints: a low-fidelity single-epoch Gaussian-process sprint used to prune the hyperparameter space, a warmup-phase sprint run under the cyclic learning-rate schedule, and a full-fidelity Optuna/TPE sprint with Hyperband pruning that includes threshold calibration. A second contribution is a threshold meta-learner that tunes mention, coreference, and relation thresholds on validation data using SCut and hill-climbing. The paper reports F1-micro scores for 16 model configurations on Re-DocRED and claims that the Longformer-based model with dynamic task loss weighting (Long-DTL) reaches scores close to 50% and shows the best improvement.","tokens_in":19117,"tokens_out":3961,"duration_ms":40298,"significance":"If the reported results were properly supported, the work would be a useful practitioner-oriented demonstration of multi-fidelity HPO with human-in-the-loop space pruning, and the threshold-calibration component is a concrete, falsifiable mechanism that could benefit joint extraction models. The manuscript is strong on procedural detail: Section 3 and Appendix B describe the sprint protocol, the Hyperband budget parameters, the hill-climbing meta-learner equations, and Algorithm 2 with test-set evaluation. However, the significance is currently undercut by the empirical reporting issues detailed below: the headline numbers are presented ambiguously as validation or test scores, no same-dataset baseline is provided, and the three-run protocol described in Appendix B.3 is not reflected in the reported results. These issues must be resolved before the claimed enhancement of model performance can be accepted.","major_comments":[{"comment":"The central claim of Section 4, that Long-DTL shows the best improvement and reaches scores close to 50%, depends entirely on the '25E+Cal' column of Table 2. The table title calls these 'F1-micro test scores,' while the caption states that the same column contains 'validation F1 scores' from the full-fidelity training-calibration-validation cycle. The abstract and Section 4 treat these values as test performance on Re-DocRED, and Appendix B.3 describes a separate final test on Dtest after threshold calibration. A validation score used for model selection and threshold calibration does not support the test-set claim. The authors must clarify which quantity is reported, provide the actual test column, or revise the claims in the abstract and Section 4 accordingly.","section":"Table 2 and Section 4"},{"comment":"No same-dataset baseline is reported. The paper does not evaluate the original Eberts-Ulges JEREX model on Re-DocRED, does not report a random-search or default-configuration comparator, and the statement in Section 4 that 'all variants in JEREX-L benefited when compared with the Doc-Red benchmarks shown here' is not supported because no such benchmark numbers appear. Without a baseline, the abstract's claim that the methodology 'enhances machine learning model performance' and the phrase 'best improvement' cannot be evaluated. A minimal fix would be to report the original JEREX configuration trained and tested under the same pipeline on Re-DocRED, together with at least one standard HPO comparator.","section":"Section 4 and Table 2"},{"comment":"Appendix B.3 states that the fine-tuning and evaluation pipeline is repeated three times and that the mean of the three test scores is reported. Table 2 contains single F1 values with no variance or per-run values. This makes it impossible to assess whether the top difference (49.64 vs. 49.63) is meaningful, and it also obscures the stability of the multi-fidelity pruning decisions. The authors should report mean and standard deviation (or per-run values) for at least the top configurations and ideally for all rows.","section":"Appendix B.3 and Table 2"},{"comment":"The multi-fidelity pruning strategy relies on Hypothesis 2, which states that optimal hyperparameter distributions are more consistent across fidelity levels. This assumption is load-bearing because Phase 1 prunes the hyperparameter space to the top 10 low-fidelity iterations, and a mismatch between low- and full-fidelity optima could exclude the best configuration before expensive sprints run. The paper provides no direct evidence for the hypothesis, and the score rankings in Table 2 change substantially across fidelities (for example, Bert-ASL-POS GLOBAL goes from rank 1 at 1E to rank 8 at 25E+Cal). The authors should either verify the assumption directly by comparing hyperparameter distributions of top trials across fidelities, or temper the claims about the reliability of the pruning step.","section":"Section 3, Hypothesis 2 and Table 2"}],"minor_comments":[{"comment":"The text reads 'considering the two transformer architectures, i.e., Bert and Longfromer'; 'Longfromer' should be 'Longformer.'","section":"Section 3, Experiment A description"},{"comment":"Equations (16) and (17) are identical, which appears to be a typo; one of them should presumably define the per-class relation threshold vector explicitly, since equation (18) then selects per-class indices.","section":"Appendix B.1.3"},{"comment":"The line 'B = (smax + 1)R Toral Budget' contains a typo: 'Toral' should be 'Total Budget.'","section":"Algorithm 1"},{"comment":"The heading 'Mention Autho-threshold' should be 'Mention Auto-threshold.'","section":"Appendix B.1.1"},{"comment":"The paper states that a prototype visualization and space-pruning tool is published, but no repository, URL, or version is given in the text or references; this should be added to make the reproducibility claim concrete.","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"This is an explicitly interim report, and the empirical evidence is presently insufficient to support the stated claims. The main issues are fixable within the manuscript's scope: clarify whether the reported numbers are validation or test scores, add a same-dataset baseline and standard HPO comparator, report the repeated-run variance promised in Appendix B.3, and either test or substantially soften Hypothesis 2. I would not reject the paper outright because the procedural framework and threshold meta-learner are described in sufficient detail that a revised version could make a valid contribution, but acceptance in the current form would require overlooking the ambiguous and incomplete empirical support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuine interim report: it describes a phased, human-in-the-loop HPO workflow using sprints, data-subset multi-fidelity, Bayesian optimization, and threshold calibration, and it applies that workflow to 16 JEREX-L variants on Re-DocRED. The sprint/thread protocol with cold priming is a thoughtful recombination of existing tools (gp_minimize, TPE, Hyperband, SCut), and the class-specific relation threshold calibration is a practical contribution. The writing is also candid: the author flags aggressive pruning risk, distinguishes fidelity mechanisms, and labels the cross-fidelity consistency claim as a hypothesis rather than a result. Credit where due: the methodology section is coherent and the case study is honest about being a case study.\n\nThe soft spots are real and load-bearing. Table 2's title says \"test scores,\" but the caption says the 25E+Cal column contains \"validation F1 scores\" from the training-calibration-validation cycle. The abstract and Section 4 treat these numbers as the final result, and Appendix B.3 describes a separate held-out test repeated three times with the mean reported, yet no column is labeled that final test and no variance is given. If 49.64 is a validation score used for model selection, then the claim that JEREX-L \"enhances\" performance has no demonstrated test-set support. This is an internal inconsistency, not a minor presentation issue.\n\nThere is also no baseline: no original JEREX on Re-DocRED and no random search or standard HPO comparator. So even a correctly labeled test score of 49.64 cannot establish \"improvement\" of the workflow or of JEREX-L; it only ranks the 16 variants against each other. That is a legitimate but much weaker claim, and the framing should say so.\n\nThe cross-fidelity consistency hypothesis (Hypothesis 2) is , as stated, undemonstrated, and Table 2's rank flips across fidelities are a warning sign. The author acknowledges this as a hypothesis, so it is not a hidden circularity, but the Phase 1 top-10 pruning depends on it.\n\nWho this is for: practitioners who want a detailed, budget-conscious HPO recipe for multitask NLP models, and methodologists interested in human-guided multi-fidelity search. It is not yet a paper whose empirical claims should be cited.\n\nMy recommendation: do not desk-reject. Send it to review; a serious referee can push the authors to fix the test/validation labeling, add the missing baseline, report the three-run variance, and either release code/data or stop calling it reproducible. Revise-and-resubmit, not reject.","headline":"An honest interim HPO engineering report whose headline result is undermined by an internally inconsistent table and a missing same-dataset baseline; it deserves a serious referee, but major revision before the claims can stand.","tokens_in":19617,"tokens_out":1937,"would_cite":false,"duration_ms":22285,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a human-guided, multi-fidelity sprint workflow with post-training threshold meta-learning identifies a high-performing joint entity-relation extraction configuration, reporting about 49.64 relation F1 on Re-DocRED…","keywords":["hyperparameter optimization","multi-fidelity optimization","Bayesian optimization","joint entity and relation extraction","threshold calibration","multi-task learning","human-in-the-loop","Re-DocRED"],"falsifier":"Run full 25-epoch calibrated training for hyperparameter combinations that the Phase 1 top-10 pruning would have excluded; if any such combination scores above 49.64 relation F1 on Re-DocRED, the consistency assumption behind low-fidelity pruning is violated.","tokens_in":18484,"feed_emoji":"⚙️","tokens_out":7827,"duration_ms":68394,"temperature":0.7,"pith_summary":"This paper proposes and tests a practical workflow for tuning expensive multitask NLP models: run cheap low-fidelity Bayesian optimization sprints on rotating data subsets, let a human shrink the hyperparameter ranges based on the top results, then progress through warmer and finally full-fidelity sprints with early pruning, and finish by calibrating decision thresholds with a hill-climbing meta-learner. The claim is that this phased process finds strong configurations within a constrained compute budget, and that threshold calibration is a real performance lever. It demonstrates the workflow on sixteen variants of the JEREX-L joint entity and relation extraction model trained on Re-DocRED, reporting the best result of about 49.64 relation F1 for the long-document transformer variant with dynamic task-loss weighting. A sympathetic reader would care because the framework is reusable: it offers a blueprint for comparing model variants and tuning hyperparameters when full-fidelity training is too expensive to brute-force.","feed_headline":"Cheap Bayesian sprints plus human pruning hit 49.6 relation F1","feed_subtitle":"Low-fidelity runs prune hyperparameters before expensive training, then a threshold meta-learner locks in the score.","key_machinery":"The load-bearing mechanism is the sprint thread: a sequence of Bayesian optimization sessions at increasing fidelity, each with its own hyperparameter space and initialization. Phase 1 runs about 120 single-epoch trials on rotating one-sixth training and one-third validation subsets with the learning-rate scheduler disabled, then prunes each hyperparameter dimension to the range spanned by the top 10 trials plus a margin. Phase 2 runs 60–90 trials at the same data fidelity with the scheduler enabled and an early stop at the end of the warmup period, optimizing the scheduler's maximum learning rate and warmup length. Phase 3 cold-primes a full-fidelity tree-structured Parzen estimator study with Hyperband pruning using the top 3 Phase 2 configurations and runs nine 25-epoch trials plus a calibration phase. The calibration component is a hill-climbing meta-learner that perturbs mention, coreference, and relation thresholds, optionally per relation class, to maximize validation relation F1. Cold priming—sharing top hyperparameter sets without carrying over scores—is what lets knowledge cross a fidelity boundary without contamination.","core_discovery":"On the paper's own terms, the central discovery is that a three-sprint pipeline—single-epoch low-fidelity exploration, warmup-period tuning with the scheduler active, and a short full-fidelity study with Hyperband pruning—combined with post-training threshold meta-learning can identify a high-performing joint entity and relation extraction configuration. The best configuration found is the long-document transformer variant with Dynamic Task Loss Weighting and partitioned per-module optimizer rates, reaching 49.64 relation F1 on Re-DocRED; the same model with global parameter rates is essentially tied at 49.63. The paper also reports that dynamic task-loss weighting helps when parameters are partitioned and seems to hurt with global rates, and that score rankings are unstable across fidelity levels, with the top low-fidelity configuration dropping to rank 8 at full fidelity. The paper presents these as evidence that the human-guided multi-fidelity process, not just the model changes, contributes to the outcome.","pith_inferences":["If the cross-fidelity consistency of hyperparameter distributions holds, the same three-sprint pattern could be applied to other expensive fine-tuning workloads, such as long-document summarization or multimodal extraction, where data-subset fidelities are natural.","A direct test would compare top-10 pruning against random pruning under an equal full-fidelity budget; if random pruning matches or beats it, the human-guided pruning step is not the source of the gain.","The 49.64 figure is an interim case-study result with nine full-fidelity trials on a single GPU, so readers should treat the paper as an existence proof for the workflow rather than a head-to-head benchmark statement.","The per-class relation thresholds suggest a cheap deployment-time adaptation: re-calibrating thresholds on a new validation distribution may recover performance after domain shift without touching the trained weights."],"forward_implications":["Multi-fidelity data-subset sprints can shrink the hyperparameter search space before expensive full runs, lowering the compute cost of comparing model variants.","Post-training threshold calibration yields noticeable score gains and can be reused at inference time without retraining.","Ranking model configurations by low-fidelity scores is unreliable; the winning configuration at low fidelity can be mid-pack at full fidelity.","Dynamic task-loss weighting is most beneficial when the optimizer uses partitioned per-module rates, an interaction worth checking in other multitask models.","The sprint/thread structure gives practitioners a concrete protocol for mixing automated search with human judgment and checkpoint management."],"supporting_citations":[{"why":"Supplies the original JEREX joint entity and relation extraction model that JEREX-L extends.","marker":"(Eberts and Ulges, 2021)"},{"why":"Provides the long-document transformer encoder used by the best-performing variants.","marker":"(Beltagy et al., 2020)"},{"why":"Provides the Re-DocRED dataset whose corrected annotations define the benchmark for the 49.64 score.","marker":"(Tan et al., 2022)"},{"why":"Hyperband is the progressive-halving algorithm used for trial pruning in full-fidelity sprints.","marker":"(Li et al., 2018)"},{"why":"Supplies the tree-structured Parzen estimator sampler and Hyperband pruning integration used in Phase 3.","marker":"(Akiba et al., 2019)"},{"why":"SCut threshold selection for multi-label classification is the basis of the mention auto-threshold.","marker":"(Yang, 2001)"},{"why":"SCutFBR.1-inspired low-bound handling informs the threshold algorithm for underrepresented labels.","marker":"(Fan and Lin, 2007)"},{"why":"Two-step data-subset HPO is the direct antecedent and warning that motivates the sprint boundary design.","marker":"(Yu et al., 2023)"},{"why":"Uncertainty-based dynamic task loss weighting is the DTL option that defines the best configurations.","marker":"(Kendall et al., 2018)"}],"fun_headline_variants":["Human-guided sprints prune hyperparameters to 49.6 relation F1","Multi-fidelity Bayesian sprints with human input hit 49.6 F1","Human-in-the-loop hyperparameter sprints yield 49.6 relation F1","Adaptive sprints plus human pruning reach 49.6 relation F1","Three-sprint human-guided tuning hits 49.6 relation F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that optimal hyperparameter distributions stay consistent across fidelity levels, so pruning on cheap low-fidelity results will not discard the best full-fidelity configuration.","fun_headline_variants_meta":{"raw":{"variants":["Human-guided sprints prune hyperparameters to 49.6 relation F1","Multi-fidelity Bayesian sprints with human input hit 49.6 F1","Human-in-the-loop hyperparameter sprints yield 49.6 relation F1","Adaptive sprints plus human pruning reach 49.6 relation F1","Three-sprint human-guided tuning hits 49.6 relation F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000902,"raw_usage":{"total_tokens":3853,"prompt_tokens":887,"completion_tokens":2966,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2864}},"tokens_in":503,"tokens_out":2966,"duration_ms":19407,"temperature":1.0,"reasoning_tokens":2864,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:23:52.626328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run full 25-epoch calibrated training for hyperparameter combinations that the Phase 1 top-10 pruning would have excluded; if any such combination scores above 49.64 relation F1 on Re-DocRED, the consistency assumption behind low-fidelity pruning is violated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the original JEREX joint entity and relation extraction model that JEREX-L extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SCut threshold selection for multi-label classification is the basis of the mention auto-threshold."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SCutFBR.1-inspired low-bound handling informs the threshold algorithm for underrepresented labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Uncertainty-based dynamic task loss weighting is the DTL option that defines the best configurations."}],"review_version":1}