{"id":"8f6bf3fe-89a5-4642-b08d-23ab82cca7be","arxiv_id":"2501.07072","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A source-free domain adaptation method combining evidential deep learning with a per-dataset tuned calibrated softmax improves Domainnet40 and Office-Home slightly, but reduces average accuracy on Office31.","lead":"The paper adds evidential deep learning and a tuned calibrated softmax to source-free domain adaptation, where only a pretrained source model and unlabeled target data are available. The method shows small accuracy gains on some benchmarks but falls behind the baseline on Office31.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Office31 results in Table 4 already contradict the claim of consistent outperformance; per-dataset γ tuned on the test set may explain the remaining gains.","rationale":"The reader's weakest assumption and my own reading converge. The decisive issue is not a mathematical flaw in Eq. (9) or Eq. (13); it is that the paper's own numbers fail to support the abstract's blanket claim. Under UB(0.0), EKS loses to KSHOT by 0.5 points on Office31; under BR it loses by 0.1. The text (Sec. 6.1) cherry-picks task-level gains and asserts consistency. This is an internal inconsistency, not a matter of disagreement with community baselines. Additional support for the concern: Section 5's per-dataset γ values (1, 1.5, 0.2) are selected 'based on extensive experimentation' with no held-out split. Since the calibrated softmax (Eq. 9) changes only the confidence scale and not the argmax, γ can be chosen to inflate exactly the entropy/divergence terms reported. I do credit the paper's ablations (Tables 7, 8), ECE/NLL improvements (Table 3), and the E-TransDA extension (Table 6) as genuine evidence that the components can help. But the central claim must be narrowed, and the γ-selection protocol must be shown to be reproducible. These are addressable revisions, so keeping the conditional verdict is appropriate.","tokens_in":14189,"tokens_out":7378,"duration_ms":74590,"concrete_test":"Re-run EKS on Domainnet40, Office-Home, and Office31 with a pre-registered γ-selection rule (fix γ=1 across all datasets, or tune γ on a labeled source-domain validation split), and report mean±std over seeds. If Office31 remains below KSHOT or the Domainnet40/Office-Home gains shrink below ~1%, the paper's per-dataset test-set γ tuning is responsible for the headline improvements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'outperforms other state-of-the-art methods on benchmark datasets'—requires EKS/ES to improve over the corresponding baselines across benchmarks. That condition is already violated in the paper's own Table 4: on Office31, EKS under UB(0.0) averages 90.5 vs KSHOT's 91.0, and under BR averages 90.4 vs 90.5; ES's 88.67 is within noise of SHOT's 88.6. Section 6.1 nevertheless says EKS/ES 'consistently achieve better classification accuracies' and highlights only A→D (+0.5%) while omitting that the dataset average is lower. The other side of the same weakness is hyperparameter selection: Section 5 states γ = 1, 1.5, 0.2 for Domainnet40, Office-Home, Office31 'based on extensive experimentation,' with no held-out validation or sensitivity analysis. Because Eq. (9) reduces to standard softmax when γ=0 and otherwise shrinks confidences toward uniform, the reported gains on Domainnet40/Office-Home may partly reflect test-set-tuned γ rather than the EDL mechanism. The claim should be restricted to the two benchmarks and should be accompanied by a γ-selection protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two source-free domain adaptation (SFDA) methods, EKS and ES, which combine evidential deep learning (EDL) with information maximization (IM) and a calibrated softmax function. EKS incorporates unary-bound and binary-relationship prior knowledge following KSHOT, while ES is the prior-free analogue built on SHOT. The authors report classification accuracies on DomainNet40, Office-Home, Office-31, and digit datasets, together with ablations and ECE/NLL calibration measurements, and claim consistent improvements over state-of-the-art baselines.","tokens_in":14483,"tokens_out":4987,"duration_ms":48093,"significance":"If the reported gains were robust, the paper would offer a useful connection between evidential deep learning and pseudo-label refinement in SFDA, and it draws attention to a genuine limitation of the standard softmax (translation invariance). The manuscript provides a code link, ablations showing that adding IM to EDL helps (Table 7), and ECE/NLL improvements on DomainNet40. However, the current evidence is weakened by an inconsistency on Office-31 and by per-dataset selection of the key calibrated-softmax constant without a validation protocol; these issues must be addressed before the contribution can be assessed.","major_comments":[{"comment":"The statement that 'EKS and ES consistently achieve better classification accuracies' is not supported by Table 4. On Office-31, EKS under UB(0.0) averages 90.5 versus KSHOT's 91.0; under BR it averages 90.4 versus 90.5; ES averages 88.67 versus SHOT's 88.6. The text in §6.1 highlights only A→D (+0.5%) and omits the dataset-level deficit. Please either restrict the central claim to DomainNet40, Office-Home, and Digits, or analyze why Office-31 behaves differently; the abstract's 'outperforms other state-of-the-art methods on benchmark datasets' should be revised accordingly.","section":"§6.1 and Table 4"},{"comment":"The calibrated softmax constant γ is set separately per dataset (1.0 for DomainNet40, 1.5 for Office-Home, 0.2 for Office-31, 1.0 for Digits) 'based on extensive experimentation,' but no validation protocol, search range, or sensitivity analysis is given. Because Eq. (9) reduces to the standard softmax when γ=0, selecting γ on the same test data makes the reported gains partly a result of test-time tuning. The constant λ in Eq. (2) is also never assigned a value. Please specify how γ (and, if varied, β, w1, w2) are chosen on a validation split or report accuracy as a function of γ; otherwise the comparison to KSHOT/SHOT is not a fair apples-to-apples evaluation.","section":"§5, Implementation Details"},{"comment":"The paper reports means over three random seeds but no standard deviations, confidence intervals, or paired significance tests. Several reported advantages are small: ES vs SHOT on Office-Home is +0.05 points, ES vs SHOT on Office-31 is +0.07 points, and EKS vs KSHOT on Office-31 is negative by 0.1–0.5 points. Without variance estimates, these differences cannot be distinguished from seed noise, and the claim of consistent improvement is not statistically grounded. Please report per-seed results or error bars at least for the headline averages.","section":"§5 and Tables 1–6"},{"comment":"The ablation in Table 8 does not support 'consistent improvement' from calibrated softmax: under UB(0.5), UB(1.0), and UB(2.0) the calibrated variant is worse than the standard-softmax variant on Office-Home (72.72 vs 72.76, 72.01 vs 72.19, 71.66 vs 71.92). The gain appears only for UB(0.0), UB(0.1), and BR. The text should be revised to state which operating points benefit, and the analysis should discuss why the effect reverses as σ grows.","section":"§6.2.3 and Table 8"}],"minor_comments":[{"comment":"There are several typos and wording errors: 'drichilet' in §4.2, 'demonstarted' in §4.4, 'psedolabeling' in §2.1, and 'EKS ans ES' in §6.1.","section":"Throughout"},{"comment":"Algorithm 1 has a duplicate 'Output: Output:' line; the input/output formatting should be cleaned.","section":"Algorithm 1"},{"comment":"The tables use a 'κ' column that is never defined in the text or captions; define it or remove it. Also, several entries in Table 4 lack spacing (e.g., '75.3999.27').","section":"Tables 1–11"},{"comment":"Equation (1) uses both ρ_i and ρ_ik for the Dirichlet variable and its components; the notation should be made consistent.","section":"Eq. (1)"},{"comment":"The t-SNE visualizations in Figure 2 would be more informative with a quantitative cluster-quality metric; as presented they only support a qualitative claim.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's own Table 4 contradicts the abstract, and the per-dataset selection of γ without a validation protocol is a reproducibility problem. The methodological idea is interesting, but the current evaluation is not reliable enough for publication as-is. The authors should be asked to provide a proper validation protocol for γ, report variance, and soften or carefully scope the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a reasonable incremental paper that grafts evidential deep learning and a calibrated softmax onto existing SFDA baselines, but its headline claim—consistent gains over state-of-the-art—is undercut by its own Table 4, and the main gains elsewhere look partly like test-set tuning of one hyperparameter.\n\nWhat's actually new is the specific integration. Putting a Dirichlet prior over the target logits, training with the EDL loss, and replacing the softmax in the information-maximization loss with the calibrated softmax from Zong et al. [34] hasn't been done in SFDA before, as far as the cited literature goes. The paper also ships useful ablations: Table 7 shows the IM term is worth roughly +2 points on Office-Home; Table 8 shows the calibrated softmax gives a modest but consistent boost; Table 3 reports lower ECE and NLL for EKS on DomainNet40. Code is linked. That's real work.\n\nWhere it's soft: first, the Office31 result. Table 4 shows EKS at 90.5 under UB(0.0) versus KSHOT's 91.0, and 90.4 versus 90.5 under BR. ES lands at 88.67, essentially tied with SHOT's 88.6. Section 6.1 nonetheless says EKS/ES \"consistently achieve better classification accuracies.\" That's not supported by the table. The authors highlight the +0.5% on A→D and ignore the negative average. The abstract's \"outperforms other state-of-the-art methods\" is too strong.\n\nSecond, the calibrated softmax constant gamma is chosen per dataset—1, 1.5, 0.2 for DomainNet40, Office-Home, Office31—\"based on extensive experimentation.\" No held-out validation or sensitivity analysis is given. Since gamma=0 reverts to standard softmax and positive gamma shrinks confidences toward uniform, the DomainNet40/Office-Home gains could be partly an artifact of picking gamma on the test set. The paper does report three-seed means, but no error bars or significance tests, so it's hard to tell which differences are meaningful.\n\nOn the upside, the ablation structure is thoughtful and the ECE/NLL evidence is a genuine positive. The combination of EDL with IM is plausible, and the authors don't oversell the theory—there's no new mechanism, just an application-level integration.\n\nWho this is for: researchers working on SFDA with prior knowledge or uncertainty calibration. They'll find the ablations useful and the ECE numbers worth checking. But the headline numbers should be treated with caution until gamma selection is formalized and Office31 is reconciled.\n\nBottom line: it deserves referee time, but I'd want the authors to fix the claims, add validation, and present Office31 honestly. I wouldn't cite the accuracy gains as-is, but I might cite the integration idea. For a reading group, maybe.","headline":"A sensible integration of evidential deep learning and calibrated softmax into SFDA, but the paper's consistent-outperformance claim is contradicted by its own Office31 table and the per-dataset tuning of gamma.","tokens_in":14993,"tokens_out":3636,"would_cite":false,"duration_ms":31279,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By modeling prediction uncertainty with a Dirichlet prior and adding a constant to logits before softmax, this paper refines noisy pseudolabels in source-free domain adaptation, reporting gains over the SHOT and KSHOT baselines on…","keywords":["source-free domain adaptation","evidential deep learning","pseudo-label refinement","softmax calibration","information maximization","Dirichlet prior","uncertainty estimation","domain adaptation"],"falsifier":"Use the released code to run EKS on DomainNet40, Office-Home, and Office31 with a single constant $\\gamma$ (or with $\\gamma$ selected on a held-out validation split) and compare average accuracy to KSHOT; if the margin over the baseline disappears or reverses, the claimed improvement is an artifact of per-dataset tuning rather than of the evidential and calibrated-softmax losses.","tokens_in":14002,"feed_emoji":"🎯","tokens_out":6117,"duration_ms":54152,"temperature":0.7,"pith_summary":"This paper argues that source-free domain adaptation fails partly because pseudolabels generated by the frozen source model are noisy and miscalibrated under domain shift. It proposes to fix this by placing a Dirichlet distribution over the target network's class probabilities (evidential deep learning) and by replacing the standard softmax in the information-maximization loss with a calibrated softmax that adds a constant to the logits. The resulting methods, EKS for the prior-knowledge setting and ES for the no-prior setting, are tested on DomainNet40, Office-Home, Office31, and digit datasets. The authors report that EKS beats KSHOT by 2.74 points on DomainNet40 and ES beats SHOT by 1.14 points there, with smaller gains on Office-Home and mixed results on Office31. If correct, the work shows that uncertainty calibration and softmax calibration are cheap, effective upgrades for self-training SFDA pipelines.","feed_headline":"Calibrated pseudo-labels push SFDA accuracy up 2.74%","feed_subtitle":"Evidential deep learning and a logit-shifted softmax reduce pseudolabel noise, beating SHOT and KSHOT on DomainNet40.","key_machinery":"The load-bearing mechanism is the evidential Dirichlet head: instead of a single point-estimate softmax vector, the target network outputs evidence $\\alpha_{ik}=e^{o_{ik}}+\\lambda$ that parameterizes a Dirichlet distribution over class probabilities. The EDL loss penalizes both the negative log marginal likelihood $L_{nll}$ and the KL divergence to the uniform Dirichlet $L_{kl}$, which together suppress evidence for wrong pseudolabels. A second mechanism, the calibrated softmax $\\hat{\\delta}_{ik}=(e^{o_{ik}}+\\gamma)/\\sum_j(e^{o_{ij}}+\\gamma)$, breaks the translation invariance of standard softmax by making predictions depend on absolute logit magnitudes; it is inserted into the entropy and diversity terms of information maximization so that confident but miscalibrated predictions are not rewarded. The two mechanisms are combined in $L_{total}=w_1(L_{nll}+\\beta L_{kl})+w_2(\\hat{L}_{ent}+\\hat{L}_{div})$, with weights $w_1=0.3$, $w_2=1.0$ in both EKS and ES.","core_discovery":"The central claim is that predictive uncertainty and softmax miscalibration, not just domain shift, are responsible for noisy pseudolabels in SFDA. By training the target network to output evidence for a Dirichlet prior over class probabilities, the model can express uncertainty in a single forward pass; the EDL loss ($L_{nll}+\\beta L_{kl}$) then suppresses evidence for false labels. Replacing the translation-invariant softmax with the calibrated form $\\hat{\\delta}_{ik}=(e^{o_{ik}}+\\gamma)/\\sum_j(e^{o_{ij}}+\\gamma)$ in both entropy and diversity terms of information maximization prevents overconfident predictions that entropy minimization would otherwise encourage. The full objective $w_1 L_{edl}+w_2 \\hat{L}_{im}$ is applied either with prior knowledge (unary and binary bounds) in EKS or without it in ES. On the paper's reported benchmarks, this combination outperforms the SHOT and KSHOT baselines on DomainNet40 and Office-Home, with the largest margin on the large-domain-gap task C→P.","pith_inferences":["The per-dataset values of $\\gamma$ (1 for DomainNet40, 1.5 for Office-Home, 0.2 for Office31) suggest the method would benefit from an automatic or validation-based selection of the calibration constant; the paper leaves this as manual tuning.","If the uncertainty estimates from the evidential head are reliable, they could also support active SFDA, sample selection, or out-of-distribution rejection on the target domain, applications not explored in the paper.","The translation-invariance argument applies to any softmax-based self-training, so the calibrated softmax may transfer to semi-supervised learning or noisy-label learning beyond domain adaptation.","A direct extension would test whether the improvement scales with the number of classes, since the Dirichlet prior and the KL regularization term both depend on $K$; the paper does not report such an analysis."],"forward_implications":["Existing SFDA methods that rely on pseudolabel self-training can adopt the evidential head and calibrated softmax without accessing source data, since both operate entirely on target logits.","The calibrated softmax should make entropy minimization safer when the source model is trained on small or far-distribution data, because it prevents confidence from being built on logit differences alone.","The EDL loss gives a per-sample uncertainty estimate at inference that can be used to filter or weight pseudolabels, complementing the prior-knowledge bounds in EKS.","Because the approach is a loss and head modification rather than a new architecture, it can be stacked with other SFDA methods, as the paper demonstrates by turning TransDA into E-TransDA."],"supporting_citations":[{"why":"Supplies the evidential deep learning loss that places a Dirichlet prior over predictions to quantify uncertainty.","marker":"[26]"},{"why":"Provides the Dirichlet-based prediction calibration and calibrated softmax used to counter translation invariance with noisy labels.","marker":"[34]"},{"why":"SHOT is the non-prior baseline whose pseudo-labeling and information maximization loss ES extends.","marker":"[15]"},{"why":"KSHOT is the prior-knowledge baseline that supplies the unary/binary bound formulation and hyperparameters EKS builds on.","marker":"[27]"},{"why":"TransDA is the transformer-based SFDA method that E-TransDA extends, used to test generalizability of the calibration.","marker":"[33]"}],"fun_headline_variants":["Calibrated softmax + evidential learning fix noisy pseudolabels","Uncertainty-guided calibration improves source-free domain adaptation","EDL and calibrated softmax beat SHOT and KSHOT","Pseudolabel noise tamed by evidential deep learning","Calibrated pseudolabels lift SFDA accuracy 2.74%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains hinge on selecting the calibrated softmax constant $\\gamma$ separately for each dataset through extensive experimentation; if $\\gamma$ must be fixed in advance or chosen without test data, the improvements over the baselines may shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Calibrated softmax + evidential learning fix noisy pseudolabels","Uncertainty-guided calibration improves source-free domain adaptation","EDL and calibrated softmax beat SHOT and KSHOT","Pseudolabel noise tamed by evidential deep learning","Calibrated pseudolabels lift SFDA accuracy 2.74%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00079,"raw_usage":{"total_tokens":3481,"prompt_tokens":944,"completion_tokens":2537,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2448}},"tokens_in":560,"tokens_out":2537,"duration_ms":20991,"temperature":1.0,"reasoning_tokens":2448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:48:43.574752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the released code to run EKS on DomainNet40, Office-Home, and Office31 with a single constant $\\gamma$ (or with $\\gamma$ selected on a held-out validation split) and compare average accuracy to KSHOT; if the margin over the baseline disappears or reverses, the claimed improvement is an artifact of per-dataset tuning rather than of the evidential and calibrated-softmax losses.","supporting_citations":[{"cited_title":"Evidential deep learning to quantify classification un- certainty","cited_arxiv_id":null,"evidence_quote":"Supplies the evidential deep learning loss that places a Dirichlet prior over predictions to quantify uncertainty."},{"cited_title":"Dirichlet-based prediction calibra- tion for learning with noisy labels","cited_arxiv_id":null,"evidence_quote":"Provides the Dirichlet-based prediction calibration and calibrated softmax used to counter translation invariance with noisy labels."},{"cited_title":"Do we re- ally need to access the source data? source hypoth- esis transfer for unsupervised domain adaptation","cited_arxiv_id":null,"evidence_quote":"SHOT is the non-prior baseline whose pseudo-labeling and information maximization loss ES extends."},{"cited_title":"Prior knowl- edge guided unsupervised domain adaptation","cited_arxiv_id":null,"evidence_quote":"KSHOT is the prior-knowledge baseline that supplies the unary/binary bound formulation and hyperparameters EKS builds on."}],"review_version":1}