{"id":"a7545d4c-b0bb-4bcc-b394-7a03b8391717","arxiv_id":"2505.01851","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FVL-FP combines cross-layer fair prompts, orthogonal projection off demographic subspaces, and fairness-weighted prompt fusion to reduce group bias in federated vision-language models.","lead":"This paper proposes FVL-FP, a federated learning method that tunes fairness prompts in vision-language models to reduce demographic bias across client data. The authors report roughly 45% lower demographic disparity than standard federated averaging while keeping accuracy close to state-of-the-art.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline fairness metric is undefined and inconsistent with Eq. (1): Table 1 reports Φ_A=1.829 for age detection although a TPR difference is bounded by 1, so the 45% reduction claim lacks a verified referent.","rationale":"The reader's weakest assumption focused on DSOP's linear-subspace hypothesis. That is a legitimate modeling risk, but it is an assumption about mechanisms and is partially addressed by the ablation, where removing DSOP causes a 7.3% fairness deterioration. It could be tested with additional probes. The more load-bearing problem is that the paper's headline quantitative claim is built on a fairness metric that is never defined and that takes values impossible under the only formal definition given in the manuscript. If Φ_A can be 1.829, then Eq. (1) is not the quantity being reported, and the '45% reduction' has no verifiable meaning as written. This is an internal consistency problem rather than a matter of consensus or taste, so it should take priority in a revision. The additional mismatch between 'four benchmark datasets' in the abstract and two datasets in Section 5, plus the client-number discrepancy between Section 5.1 and Table 5, reinforce that the empirical basis needs substantial clarification. I recommend CONDITIONAL rather than REJECT because the issues may be resolvable: the authors could define Φ_A precisely, reconcile the values with Eq. (1), release code and data, and rerun the comparisons under a single stated client configuration. If those clarifications are not provided, the central claim should not be accepted as stated.","tokens_in":16310,"tokens_out":6860,"duration_ms":75847,"concrete_test":"Require the authors to provide the exact definition and computation script for Φ_A. Then recompute the Age Detection (CelebA) CLIP zero-shot and FVL-FP rows using the TPR-difference definition in Eq. (1) and verify that every reported entry lies in [0,1]. Independently recompute the claimed average 45% reduction from these corrected entries. If entries such as 1.829 persist, the reported metric is not EOD, and the reduction claim must be re-derived using a named, bounded fairness metric. Also check whether Table 1 corresponds to N=5, N=20, or N=4 by matching its FVL-FP entries against Table 5 and the stated FL setup.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim—an average 45% reduction in demographic disparity—depends entirely on the fairness metrics reported in Tables 1, 3, 4, and 5, especially Φ_A. The paper never defines Φ_A, and the values reported for age detection violate the only formal definition given in the paper. Eq. (1) defines global group fairness as an absolute difference of true positive rates, which is mathematically bounded between 0 and 1. Yet Table 1 reports Φ_A = 1.829 for CLIP zero-shot, 1.898 for FedAvg, and 1.738 for CLIP zero-shot on FairFace. These entries cannot be EOD values under Eq. (1). Either Φ_A is a different, unnamed quantity, or the table entries are not what Eq. (1) defines. Either way, the headline '45% reduction' is computed on an undefined or mislabeled metric. This is an internal inconsistency, not a disagreement about design choices, and it is load-bearing because the paper's central claim is precisely a quantitative fairness improvement. A secondary but related problem is that the abstract and conclusion claim evaluation on 'four benchmark datasets,' while Section 5.1 states 'We use CelebA and FairFace' and no fourth dataset appears in any table. Additionally, Section 5.1 says comparisons use 5 clients, Section 5.1's implementation details say four nodes, and Table 1's FVL-FP entries exactly match Table 5's N=20 row, so the comparison conditions for the headline numbers are unclear. These issues do not necessarily mean the method is ineffective, but they mean the central claim, as stated, is not currently supported by a well-defined, internally consistent evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FVL-FP, a federated prompt-tuning framework for group fairness in vision-language models. It combines three components: Cross-Layer Demographic Fair Prompting (CDFP), Demographic Subspace Orthogonal Projection (DSOP), and Fair-aware Prompt Fusion (FPF). The authors claim an average 45% reduction in demographic disparity relative to standard FL methods while maintaining task performance within 6% of state-of-the-art results, based on experiments on CelebA and FairFace for smiling and age detection tasks, together with ablation, data-heterogeneity, and client-count studies.","tokens_in":16724,"tokens_out":6492,"duration_ms":63837,"significance":"If the claims were substantiated, FVL-FP would be a useful parameter-efficient contribution to federated VLM fairness, since it avoids full model retraining and provides a geometric, interpretable debiasing mechanism. The paper includes mean/std statistics, comparisons to several FL fairness baselines, ablations of each module, and robustness studies over heterogeneity and number of clients. However, the headline quantitative claim is currently not verifiable because the main fairness metric used in the tables is undefined and the dataset count is misstated; the significance is therefore conditional on a corrected and reproducible evaluation.","major_comments":[{"comment":"The fairness metric Φ_A is never defined, and the values reported for age detection violate the only formal definition in the paper. Eq. (1) defines global fairness as an absolute difference of true positive rates, which is bounded by 1; Table 1 reports Φ_A = 1.829 for CLIP zero-shot and 1.898 for FedAvg on CelebA age detection, and Table 3 reports values up to 2.195. Because the abstract's 45% reduction is computed from these Φ_A values, the central claim lacks a verified referent. The authors must define Φ_A (and Φ_demo, Φ_eq) and either correct the values or replace them with a metric consistent with Eq. (1).","section":"Section 3, Eq. (1) and Tables 1, 3, 5"},{"comment":"The experimental conditions for the headline results are inconsistent. Section 5.1 says all methods are compared under 5 clients, while Implementation Details says the system consists of four nodes and the default number of nodes is 4; moreover, Table 1's FVL-FP entries exactly match Table 5's N=20 row (e.g., A_B=0.915, Φ_A=0.139, Φ_demo=0.008, Φ_eq=0.023 for smiling detection). Please state precisely which client count produced Table 1 and reconcile the text.","section":"Section 5.1, Implementation Details, Tables 1 and 5"},{"comment":"The implementation details needed to reproduce the method are missing from the manuscript. The referenced appendices are not present, and the main text does not give values for K (number of fair prompt basis vectors), k (top singular directions), the margin μ in Eq. (11), the trade-off weights λ and λ1, or the client data distributions. Since the method's behavior depends on these choices, the experiments are not reproducible as written.","section":"Section 5.1 and Appendix D"},{"comment":"It is unclear whether the validation set used for FPF weighting and hyperparameter selection is disjoint from the balanced test set used to report Table 1. If the same data are used for both, then the reported fairness improvements are in part optimized on the evaluation set. Please specify the split and, if they are not disjoint, re-evaluate on a held-out test set.","section":"Section 4.4, Eqs. (15)-(16) and Section 5.2"}],"minor_comments":[{"comment":"The phrase \"As mentioned in,\" before the claim about smiling detection lacks a citation or reference.","section":"Section 5.1"},{"comment":"The abstract and conclusion claim evaluation on \"four benchmark datasets,\" but Section 5.1 states that only CelebA and FairFace are used, and no fourth dataset appears in any table.","section":"Abstract and Conclusion"},{"comment":"The section heading \"Methology\" should be \"Methodology.\"","section":"Section 4 heading"},{"comment":"Table 2 uses the acronym \"FAPF\" while Section 4.4 and elsewhere use \"FPF\"; please make the acronyms consistent.","section":"Table 2 and Section 4.4"},{"comment":"The table captions should define Φ_A, Φ_demo, and Φ_eq; currently these metrics are never defined in the text.","section":"Tables 1, 3, 4, 5"},{"comment":"The \"FVL-FP (centralized)\" column in Table 1 is not described in the experimental setup; please state how the centralized version is trained and how it differs from the federated variant.","section":"Table 1"},{"comment":"Table 1 reports A_B=0.915 for FVL-FP on smiling detection, while Table 5 reports A_B=0.924 for N=5 and A_B=0.915 for N=20; the relationship between these two tables should be explained explicitly.","section":"Tables 1 and 5"}],"recommendation":"major_revision","confidential_remarks":"The metric-definition problem in Tables 1, 3, and 5 is the main gate for this paper. If the authors cannot supply a definition of Φ_A that reproduces their numbers and satisfies the bound implied by Eq. (1), the central claim fails. I would also ask the editor to verify that the submitted version includes the appendices referenced in Section 5.1, since their absence makes the experimental setup unverifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — read this one with a metric red flag. The method is plausible and the ablations are reasonably honest, but the headline number is computed on an undefined quantity. Table 1 reports Φ_A values above 1.8 for age detection, while Eq. (1) defines global fairness as a TPR difference, which cannot exceed 1. The paper never defines Φ_A. So the abstract's '45% reduction in demographic disparity' has no verified referent.\n\nWhat is actually new: the combination of cross-layer demographic fair prompts (CDFP), nullspace-style orthogonal projection (DSOP), and fairness-weighted prompt fusion (FPF) for federated VLMs. Each ingredient exists in some form—prompt tuning, INLP, FairFed-style aggregation—but the specific package is new and the design choices are explained. The ablation table shows each component contributes, and the comparison set includes FairFed and FF-DVP, which is the right neighborhood. Means and standard deviations are reported.\n\nThe soft spots, in proportion. The undefined metric is load-bearing: it is the basis for the central claim and for the 'up to 87%' statements. That is a serious problem, not a copyedit. The abstract and conclusion say four benchmark datasets; Section 5.1 names exactly two (CelebA, FairFace) and only those appear in tables. The setup says comparisons use 5 clients, implementation details say four nodes, and the Table 1 FVL-FP entries exactly match Table 5's N=20 row, so the conditions underlying the headline comparison are ambiguous. Hyperparameters are grid-searched without sensitivity analysis, and no code or data are provided. Also, the fairness gains come from optimizing fairness losses and validation-based weighting, so the reported reductions are the optimized objective, not an independent prediction—fine to report, but the framing should not imply otherwise.\n\nNone of that makes the method worthless. The approach is parameter-efficient and the empirical pattern is consistent. But right now the paper is not ready to have its central quantitative claim taken at face value.\n\nThe right move: send it to peer review, but the reviewers should insist that Φ_A (and the other Φ metrics) be defined, the dataset and client-count inconsistencies be resolved, and the headline numbers be recomputed on a defined metric. A serious referee could get a solid paper out of this. I would not cite it in its current form.","headline":"Plausible method and honest ablation work, but the headline fairness metric is undefined and internally inconsistent, so the central claim is not yet supported.","tokens_in":17237,"tokens_out":3474,"would_cite":false,"duration_ms":31961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FVL-FP reduces demographic disparity in federated vision-language models by an average of 45% while keeping task accuracy within 6% of state-of-the-art results.","keywords":["federated learning","group fairness","vision-language models","prompt tuning","orthogonal projection","demographic bias","non-IID data"],"falsifier":"Train or evaluate the framework on data where bias is generated by a nonlinear function of the protected attribute (for example, gender affects only high-order feature interactions), or swap the demographic attribute in paired images and check whether the projected representations of the two versions remain identical. If DSOP's fairness gains disappear or accuracy collapses under nonlinear bias, the linear-subspace assumption is the failure point.","tokens_in":16160,"feed_emoji":"⚖️","tokens_out":4724,"duration_ms":44539,"temperature":0.7,"pith_summary":"The paper proposes FVL-FP, a prompt-tuning framework that tackles group-level fairness in federated vision-language models without centralizing user data. It claims to cut demographic disparity by an average of 45% compared with standard federated learning, while keeping task accuracy within 6% of state-of-the-art results. The method works by training small fairness-aware prompts locally, projecting demographic bias out of image representations, and fusing client prompts with weights that reward both accuracy and fairness. A sympathetic reader should see this as evidence that fairness in privacy-preserving multimodal systems can be improved by a parameter-efficient, interpretable mechanism rather than costly retraining.","feed_headline":"Fair prompts cut demographic bias in federated VLMs by 45%","feed_subtitle":"A three-part prompt-tuning framework keeps accuracy within 6% of state-of-the-art results.","key_machinery":"The load-bearing mechanism is the Demographic Subspace Orthogonal Projection (DSOP). It encodes demographic prompts such as “a photo of a man” and “a photo of a woman” through the frozen text encoder, collects the embeddings into a matrix, takes the top-k singular vectors as the demographic subspace $V_a$, and computes the debiased image representation as $z_{\\text{debiased}} = z - \\mathrm{Proj}_{V_a}(z)$. This carries the argument because it gives an interpretable geometric way to strip protected-attribute information while retaining task-relevant content; a fairness-aware contrastive loss then penalizes any residual cosine similarity to demographic prompts. The other two components support this core: CDFP supplies fair prompts locally, and FPF reweights client contributions during aggregation.","core_discovery":"On the paper's own terms, the central discovery is that demographic bias in federated vision-language models can be substantially reduced by prompt tuning alone, without updating the underlying vision-language model. The framework combines three mechanisms: CDFP inserts demographic fair prompts across transformer layers with adaptive residual connections to neutralize biased attention; DSOP constructs a low-dimensional demographic subspace from text embeddings of demographic prompts and orthogonally projects image representations away from it; FPF aggregates client prompts by weighting each client by accuracy times one minus measured bias. Together these yield the reported 45% average reduction in demographic disparity across CelebA and FairFace smiling and age detection tasks, with balanced accuracy within about 6% of state-of-the-art federated baselines.","pith_inferences":["If demographic bias in the embeddings really is near-linear, DSOP could be used as an inference-time post-hoc debiaser on off-the-shelf models, which would extend the contribution beyond federated settings.","The subspace is built from a fixed set of demographic prompt texts, so results are likely sensitive to prompt wording and attribute coverage; testing many prompt templates would reveal how much of the gain depends on that choice.","The paper evaluates single protected attributes (gender) on face datasets; extending to intersecting attributes such as gender × race would stress whether one low-rank subspace can capture multiple entangled biases.","A direct test of the mechanism's specificity would be to replace the demographic subspace with a random subspace of the same rank; if fairness gains persist, the gains would come from regularization rather than identification of bias directions."],"forward_implications":["Only prompt vectors are updated and exchanged, so the fairness gains avoid retraining the billion-parameter backbone and add minimal communication overhead.","Demographic disparity falls by an average of 45% versus standard federated approaches while balanced accuracy stays within about 6% of state-of-the-art baselines.","The method remains effective as data heterogeneity increases (Dirichlet $\\alpha$ down to 0.1) and as the number of clients grows to 40, retaining roughly 96–98% of centralized-training accuracy.","Improvements are larger on harder tasks: on age detection, fairness disparity drops by up to 87% on CelebA and 83% on FairFace."],"supporting_citations":[{"why":"Supplies the frozen vision-language backbone whose prompt embeddings and image representations FVL-FP debias.","marker":"[24]"},{"why":"Standard federated averaging baseline whose demographic disparity FVL-FP is claimed to reduce by 45%.","marker":"[20]"},{"why":"Federated baseline compared in experiments, representing heterogeneity-robust aggregation.","marker":"[16]"},{"why":"Strongest fairness-focused federated vision-language baseline that FVL-FP must beat.","marker":"[37]"},{"why":"Fairness-aware federated baseline using client reweighting that FVL-FP extends with prompt fusion.","marker":"[9]"},{"why":"The nullspace-projection idea that DSOP adapts to remove demographic directions from representations.","marker":"[25]"}],"fun_headline_variants":["Prompt tuning cuts federated VLM bias by 45%","No VLM update: prompts slash group bias in federated models","Fair prompts: 45% less demographic bias in federated VLMs","Federated VLM fairness boost: prompt-only, 45% lower bias","Three-part prompt method cuts demographic gap in federated VLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that demographic bias in the model's image representations lies in a low-dimensional linear subspace spanned by text embeddings of demographic prompts, so subtracting that subspace removes bias without removing task-relevant information.","fun_headline_variants_meta":{"raw":{"variants":["Prompt tuning cuts federated VLM bias by 45%","No VLM update: prompts slash group bias in federated models","Fair prompts: 45% less demographic bias in federated VLMs","Federated VLM fairness boost: prompt-only, 45% lower bias","Three-part prompt method cuts demographic gap in federated VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1799,"prompt_tokens":920,"completion_tokens":879,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":786}},"tokens_in":536,"tokens_out":879,"duration_ms":8487,"temperature":1.0,"reasoning_tokens":786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:07:53.768591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train or evaluate the framework on data where bias is generated by a nonlinear function of the protected attribute (for example, gender affects only high-order feature interactions), or swap the demographic attribute in paired images and check whether the projected representations of the two versions remain identical. If DSOP's fairness gains disappear or accuracy collapses under nonlinear bias, the linear-subspace assumption is the failure point.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the frozen vision-language backbone whose prompt embeddings and image representations FVL-FP debias."},{"cited_title":"Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas","cited_arxiv_id":null,"evidence_quote":"Standard federated averaging baseline whose demographic disparity FVL-FP is claimed to reduce by 45%."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Strongest fairness-focused federated vision-language baseline that FVL-FP must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fairness-aware federated baseline using client reweighting that FVL-FP extends with prompt fusion."}],"review_version":1}