{"id":"1d9f2bbb-d3fa-4f60-8d88-e9a0892a6499","arxiv_id":"2505.08325","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A realistic 135-client federated learning benchmark for remote sensing, assembled from eight datasets, with baseline results for ten FL algorithms.","lead":"FedRS-Bench introduces a 135-client federated learning benchmark for remote sensing, built by merging eight existing image datasets and splitting them by data source. The paper benchmarks ten federated learning algorithms and finds collaborative training substantially outperforms isolated local training on this testbed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's validity hinges on TI/TB being disjoint from client training, but the paper never states that official test splits were excluded when building the 135 clients; if not, all reported FL-vs-local gains are invalid.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern I find: the test sets TI and TB may overlap with client training data because the paper does not document that official test splits were excluded when assigning images to the 135 clients. This is the single most load-bearing issue because, if true, every quantitative comparison in the benchmark—including the headline claim that FL consistently outperforms local training—is invalidated by leakage. The other concerns raised by the reader (structurally unfavorable local baseline, best-over-rounds reporting, missing error bars, unreported Dirichlet parameter) affect the strength or interpretability of the conclusions but do not undermine basic validity in the same way. I agree with the reader's conditional verdict: the resource itself is valuable and likely acceptable once the construction is clarified and verified, but the empirical claims should not be taken as final until the disjointness is checked. My concrete test directly settles the question by comparing image identifiers between test splits and client partitions using the released artifacts. If the test finds no overlap, the concern is resolved and the paper's claims stand; if it finds overlap, Tables 3 and 4 must be regenerated.","tokens_in":37733,"tokens_out":3515,"duration_ms":35443,"concrete_test":"Download the released FedRS dataset and preprocessing scripts from the provided URL. Reconstruct the client assignment and the TI/TB splits from the code, then compute the intersection between the set of image identifiers (or perceptual hashes) in TI∪TB and the union of the 135 client training sets. Report the count of overlapping images per source dataset. If the intersection is non-empty, Tables 3 and 4 are invalid and must be regenerated after removing test images from client partitions; if empty, the leakage concern is resolved. Also verify the claim 'derived from the original test sets' by checking whether the source datasets' official train/test splits were used to select TI/TB.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (FL > local, Tables 3–4) depends on TI and TB being disjoint from every client's training data. Section 3.2 says 'Each client has a non-overlapping subset of the data' and describes partitioning the eight source datasets into 135 clients, with no mention of reserving official test splits before client construction. Section 4.1 then says TI is 'extracted 20% from each source dataset' and 'derived from the original test sets.' If clients were built from the full source datasets (rather than only the training splits), then the images in TI and TB—which are subsets of the original test sets, and hence of the full source datasets—are already assigned to clients, and the reported accuracies are inflated by train/test leakage. The paper never states that client construction excluded the official test splits of EuroSAT, AID, NWPU-RESISC45, UCM, OPTIMAL-31, RSD46-WHU, WHU-RS19, or NaSC-TG2, nor that the source datasets' test sets were removed before partitioning. Appendix B.1 only reasserts 'no data overlap between [the eight datasets]', which is about inter-dataset duplication, not train/test separation. This is not a minor implementation detail: a single overlapping image shared between a client and TI/TB directly transfers label information to the global model, and with 20% of each source in the test set, even modest overlap could account for a large fraction of the gap between FL and Local Avg (e.g., 62.26 vs 20.57 in Table 3).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FedRS, a federated learning dataset for remote sensing built from eight existing datasets (EuroSAT, AID, NWPU-RESISC45, UCM, NaSC-TG2, WHU-RS19, OPTIMAL-31, RSD46-WHU), with 135 clients where each client holds data from a single source and labels are remapped into 15 unified semantic categories. The authors also construct a simpler five-class subset, FedRS-5, and evaluate 10 federated learning algorithms plus local and centralized baselines under two partition schemes (NIID-1 and NIID-2) and two test sets (imbalanced TI and balanced TB). The main empirical claim is that federated learning consistently outperforms isolated local training, while no single FL algorithm dominates across settings.","tokens_in":38086,"tokens_out":4959,"duration_ms":47337,"significance":"If the dataset construction is sound and the generated partitions are released with the tooling, FedRS-Bench would fill a genuine gap: a realistic, multi-source, large-client-count federated benchmark for remote sensing, complementing existing benchmarks like FLamby and FLAIR. The source-based partitioning is a principled way to produce realistic heterogeneity, and the paper includes substantial supporting documentation (label mappings, client statistics, t-SNE analyses). The authors also provide code and a dataset release, which supports reproducibility. However, the quantitative claims about FL superiority are currently undermined by evaluation-protocol choices that structurally disadvantage the local baseline, and by unresolved ambiguity about train/test separation. The dataset contribution itself is potentially valuable, but the benchmark conclusions need to be placed on a sounder footing before the paper can be accepted.","major_comments":[{"comment":"The Local Avg baseline is evaluated on the global test sets TI and TB, which contain all 15 unified categories, but many clients have training data for only a subset of those classes (e.g., EuroSAT clients cover 8 classes, NaSC-TG2 clients 7 classes, as stated in §3.3). A local model cannot classify classes it never observed, so its accuracy on the global test set is artificially bounded below regardless of its true quality. The reported gap between FL and Local Avg (e.g., 62.26% vs 20.57% for ResNet18 on FedRS NIID-1 TI) is therefore inflated by construction. The paper should evaluate each local model only on the test images whose classes are present in that client's training data (and average over clients with appropriate weighting), or report a separate local baseline evaluated on per-client test splits. Without this, the headline claim that federation is consistently superior is not established.","section":"§4.1, Table 3"},{"comment":"The paper does not explicitly state that the official test splits of the eight source datasets were excluded before the 135 clients were built. Section 3.2 says 'Each client has a non-overlapping subset of the data,' and Section 4.1 says TI and TB are 'derived from the original test sets,' but it is never stated that client construction used only the training portions of EuroSAT, AID, NWPU-RESISC45, UCM, OPTIMAL-31, RSD46-WHU, WHU-RS19, and NaSC-TG2. If any test image was assigned to a client, then training data and test data overlap, and every reported accuracy is invalid because label information leaks into training. The authors must state the exact reservation procedure for test splits, publish the client-to-image mapping, and ideally include a hash-based verification that the intersection between client training sets and TI/TB is empty.","section":"§3.2 and §4.1"},{"comment":"The client count is internally inconsistent. The text reads: 'EuroSAT contributes 45 clients, AID 15 clients, RSD46-WHU 35 clients, NaSC-TG2 34 clients, NWPU-RESISC45 12 clients, AID 5 clients, UCM 2 clients and the remaining 2 clients are from WHU-RS19 and OPTIMAL-31.' This lists AID twice (15 and 5) and sums to 150 clients, not 135. Table 2 lists AID with 5 clients, WHU-RS19 with 1, and OPTIMAL-31 with 1, which sums to 135. This discrepancy must be corrected, because the number of clients is a central claim of the benchmark.","section":"§3.2"},{"comment":"The claim that each client holds data from a single 'source' is not verifiable for the datasets that are said to be split by known metadata. The paper says 'For datasets that are inherently multi-source, we try to use known metadata, including regions/cities/countries to split by source,' but AID, NWPU-RESISC45, and RSD46-WHU do not publicly provide per-image source metadata. Without describing how the authors determined each image's source (or providing the actual source labels), the central property of FedRS—that each client is a single coherent source—cannot be checked. Please specify the exact algorithm and data used for these splits, and release the source labels for each client.","section":"§3.2"},{"comment":"The text states that Table 3 reports 'the final best accuracy of each method.' If 'best accuracy' means the maximum test accuracy over the 500 communication rounds, then the reported numbers are not the performance of the final model and can differ substantially from the model that a user would actually deploy at round 500. This cherry-picking of the best round overstates method performance and makes comparative claims unreliable, especially for unstable methods such as FedDC and FedDyn, whose curves collapse in some scenarios. Report the accuracy at the final round (or the mean over a final window of rounds) in addition to any best-round value, and state clearly which quantity is tabulated.","section":"§4.2 and Table 3"}],"minor_comments":[{"comment":"The description of TI construction is quantitatively unclear. The paper says 'we extracted 20% from each source dataset to compose TI,' then gives examples of 'UCM images (300 images, 15 classes, 20 per class)' and 'EuroSAT images (4,800 images, 7 classes, 400–1,000 per class).' For UCM (2,100 images total, 21 classes), 20% of the full dataset would be 420 images, not 300; for EuroSAT (27,000 images total), 20% would be 5,400, not 4,800. Please clarify whether the 20% is taken from the official test splits, from each class, or from the whole dataset, and provide the exact sampling procedure and resulting image counts.","section":"§4.1"},{"comment":"Several source datasets are marked '-' for some unified categories (e.g., Airport for NaSC-TG2 and UCM). The paper explains that a manual review selected 15 conflict-free categories, but it does not say how many images from the original datasets were excluded because their labels did not map to these categories. Reporting the discarded counts would help readers understand the extent of label harmonization and its effect on the final dataset.","section":"Table 5"},{"comment":"The t-SNE axis labels render as long strings of hexadecimal-like tokens (e.g., '/uni00000019/uni00000013'). This is likely a font/encoding artifact, but as displayed in the PDF the figures are unreadable. The x- and y-axis tick labels should be removed or properly rendered.","section":"Figures 2 and 6"},{"comment":"Appendix B.1 reasserts that there is 'no data overlap between' the eight datasets, but this statement addresses inter-dataset duplication only. It does not address whether the client partitions overlap with the test sets, which is the more critical separation for the benchmark. Please add an explicit sentence confirming that client training partitions are disjoint from TI and TB, or provide the verification I requested in the major comments.","section":"Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty claim of being the 'first realistic FL dataset for RS' is worth scrutinizing against recent concurrent work (e.g., FedRSClip, Fed-RSIC), which the authors do cite; the contribution is more accurately framed as a dedicated large-scale benchmark rather than a fundamentally new concept. The dataset release is a clear positive, but the evaluation protocol issues are serious enough that the quantitative conclusions should not be propagated until the authors fix the local-baseline evaluation and explicitly confirm test-set separation. I would not reject because the dataset and its documentation, once corrected, are likely to be useful to the community; but the current submission should not be accepted as-is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the dataset contribution here is real and worth refereeing, but the empirical claims as written need substantial tightening before the numbers can be trusted.\n\nWhat is actually new: FedRS is the first large, multi-source federated benchmark for remote sensing, with 135 clients built from eight existing datasets and a manually unified 15-class taxonomy. Keeping each client source-consistent is the right call—it mimics real deployment where different stations or satellites hold naturally heterogeneous imagery. The construction documentation is detailed, including label mappings, per-source statistics, and t-SNE diagnostics. That portion is solid and serves the FL-RS community well, especially compared to prior ad-hoc single-dataset splits.\n\nWhere it gets soft: the evaluation protocol overclaims in several ways. The Local Avg baseline is tested on the global test set, including classes a client never trained on, so the central FL-vs-local comparison is structurally favorable to FL. The paper reports best accuracy over rounds rather than the final-round accuracy, and there are no run-to-run error bars—only variation across the four settings. The Dirichlet concentration for NIID-1 is not reported, which hurts reproducibility. These are fixable, but they all push in the direction of making the quantitative results look more decisive than the evidence supports.\n\nThe bigger flag is train/test disjointness. Section 4.1 says TI and TB are \"derived from the original test sets,\" and Section 3.2 says clients are built by partitioning the eight source datasets. Nowhere does the paper state that the original test splits were excluded before client construction. If they were not, the benchmark is invalid. I do not see obvious numerical evidence of gross leakage—FL accuracy is far from ceiling—but for a benchmark paper, an explicit statement and a check in the released code are mandatory. Appendix B.1 only confirms no inter-dataset overlap, not train/test separation. So this is a required clarification, not proof of fraud, but it is the first thing I would ask the authors to nail down.\n\nWho this is for: anyone working on federated learning for remote sensing, or benchmarking FL methods on domain-heterogeneous data. The dataset and construction are the value; the baseline numbers should be re-reported with a fair local baseline, proper seeds, and final-round accuracy.\n\nRecommendation: send it to serious peer review. The resource deserves referee time, but the evaluation section will likely need a major revision before the numbers are citable.","headline":"FedRS is a genuinely useful FL benchmark for remote sensing, but the paper's evaluation protocol and an unstated train/test disjointness assumption need fixing before the headline numbers can be trusted.","tokens_in":38546,"tokens_out":6078,"would_cite":true,"duration_ms":58383,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedRS-Bench claims the first realistic federated benchmark for remote sensing, with 135 single-source clients, and reports that federated training consistently beats isolated local training.","keywords":["federated learning","remote sensing","benchmark dataset","data heterogeneity","domain shift","partial client participation","scene classification","FedRS"],"falsifier":"Run a perceptual-hash or exact-duplicate scan across all FedRS client training images and both test sets; if any image appears on both sides, recompute the headline FL-versus-local numbers with that image removed, because duplicates would inflate the reported margins.","tokens_in":37562,"feed_emoji":"🛰️","tokens_out":6994,"duration_ms":66084,"temperature":0.7,"pith_summary":"The paper introduces FedRS, a federated-learning dataset assembled from eight existing remote-sensing scene-classification corpora, repartitioned into 135 clients with a single data source per client and a unified 15-class label scheme. The authors claim this is the first realistic FL benchmark for remote sensing, because prior work split a single dataset artificially, whereas FedRS inherits skewed label distributions, imbalanced client sizes, and sensor- and region-level domain shifts. They benchmark ten FL algorithms against local-only and centralized training, reporting that every federated method beats the average local model in every tested configuration while no single method wins everywhere. The benchmark is positioned as a standard testbed so future FL-for-RS work can be compared fairly.","feed_headline":"135-client federated benchmark: FL beats local-only training","feed_subtitle":"Each client keeps one data source, and across all settings federated models outscore local silos.","key_machinery":"The load-bearing construction is the 'same client, same source' partition: each of the 135 clients receives images from exactly one of eight source datasets (NaSC-TG2, WHU-RS19, EuroSAT, AID, NWPU-RESISC45, UCM, OPTIMAL-31, RSD46-WHU), after all labels are mapped to 15 unified semantic categories and all images are resized to 64x64. Two training partitions are derived, NIID-1 via Dirichlet allocation and NIID-2 via uniform source-based allocation, and two test sets are sampled, an imbalanced set TI reflecting source proportions and a balanced set TB with 10 images per category-source cell. This machinery creates realistic client skew and domain shift in a controlled, reproducible way, and the benchmark protocols, with 10 clients sampled per round, 500 rounds, CNN and ResNet18 models, and hyperparameters held at published values, turn those properties into comparable accuracy numbers.","core_discovery":"On the paper's own terms, the central discovery is that a global federated model trained on genuinely multi-source remote-sensing data consistently outperforms models trained independently on each client's silo, despite severe heterogeneity: on FedRS with ResNet18, FedAvg reaches 62.26% on the imbalanced test set versus 20.57% for the local average, and similar gaps appear across CNN and ResNet18, both partitions, and both test sets. A second discovery is that no algorithm dominates: SCAFFOLD is best with ResNet18 but not with CNN; MOON and FedAvg lead with CNN; and FedDC and FedDyn collapse with ResNet18, showing that methods proposed under milder heterogeneity can lose stability on realistic data. The paper therefore positions FedRS-Bench as the first standardized realistic federation testbed for remote sensing, with two non-IID partitions and an imbalanced and a balanced test set.","pith_inferences":["A testable extension the authors do not run: vary the participation rate from 10-of-135 to full participation to see whether SCAFFOLD, FedDC, and FedDyn collapse is caused by sparse client sampling or by label and domain skew alone.","Because each client is single-source, FedRS can be reused as a domain-adaptation testbed: one can treat each source as a domain and directly measure how much of FL's gain comes from bridging sensor and region shift rather than from label averaging.","The paper's no-overlap claim could be audited by publishing image hashes; the current test-set description as 20% of each source does not explicitly exclude client-training images, so the benchmark would be strengthened by an explicit train-test disjointness certificate.","If the dataset is adopted, personalized federated learning methods should be a natural target: clients within the same source could form a testbed for clustering or personalized aggregation under known source membership."],"forward_implications":["A researcher evaluating a new FL method on FedRS can compare against ten baseline algorithms under identical data partitions and test sets, removing the apples-to-oranges problem that currently blocks comparison across FL-in-RS papers.","Federated collaboration is worthwhile even under extreme heterogeneity: every benchmarked FL method beat the average locally trained model in every setting, so data-sharing constraints need not force isolated silo models.","Method rankings are not transferable across architectures: a method can be the best with ResNet18 and near-worst with a CNN, so benchmark claims need to report both.","Advanced optimization methods such as SCAFFOLD, FedDC, and FedDyn can fail to converge on this realistic heterogeneity, warning that convergence behavior seen in simpler settings does not carry over.","Differential privacy costs accuracy: user-level DP on FedAvg shows the largest gap below non-private methods, quantifying the price of provable privacy on remote-sensing data."],"supporting_citations":[{"why":"Defines FedAvg, the base federated algorithm that all benchmark methods are compared against.","marker":"[5]"},{"why":"Supplies SCAFFOLD, whose instability under heterogeneity is one of the paper's notable findings.","marker":"[30]"},{"why":"NaSC-TG2, one of the eight source datasets, contributing single-source China TG-2 clients.","marker":"[43]"},{"why":"WHU-RS19, the single-region source dataset contributing one client.","marker":"[44]"},{"why":"EuroSAT, the largest source dataset, providing 45 clients with coarse-resolution satellite imagery.","marker":"[45]"},{"why":"AID, a multi-source aerial dataset whose metadata supports source-based client partitioning.","marker":"[46]"},{"why":"NWPU-RESISC45, one of the heterogeneous Google-Earth-sourced datasets used to build clients.","marker":"[47]"},{"why":"UCM, the USGS-sourced dataset contributing two clients.","marker":"[25]"},{"why":"OPTIMAL-31, the VHR scene-classification source dataset contributing one client.","marker":"[24]"},{"why":"RSD46-WHU, a multi-source dataset contributing 35 clients.","marker":"[48]"}],"fun_headline_variants":["Federated learning beats local silos on 135-client RS benchmark","No single FL algorithm wins on realistic federated remote sensing","Federated outranks local silos across 135 realistic clients","Realistic federated benchmark: global model beats per-client training","Heterogeneous FL wins on 135-client remote sensing benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's accuracy numbers are only as valid as the claim that no image appears both in a client's training data and in the TI or TB test sets, and that the eight source datasets contain no duplicate images.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning beats local silos on 135-client RS benchmark","No single FL algorithm wins on realistic federated remote sensing","Federated outranks local silos across 135 realistic clients","Realistic federated benchmark: global model beats per-client training","Heterogeneous FL wins on 135-client remote sensing benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001027,"raw_usage":{"total_tokens":4366,"prompt_tokens":1019,"completion_tokens":3347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":3258}},"tokens_in":635,"tokens_out":3347,"duration_ms":23486,"temperature":1.0,"reasoning_tokens":3258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:57:14.247876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a perceptual-hash or exact-duplicate scan across all FedRS client training images and both test sets; if any image appears on both sides, recompute the headline FL-versus-local numbers with that image removed, because duplicates would inflate the reported margins.","supporting_citations":[{"cited_title":"Reddi, Sebastian U","cited_arxiv_id":null,"evidence_quote":"Supplies SCAFFOLD, whose instability under heterogeneity is one of the paper's notable findings."},{"cited_title":"Nasc-tg2: Natural scene classification with tiangong-2 remotely sensed imagery","cited_arxiv_id":null,"evidence_quote":"NaSC-TG2, one of the eight source datasets, contributing single-source China TG-2 clients."},{"cited_title":"Satellite image classification via two-layer sparse coding with biased image representation","cited_arxiv_id":null,"evidence_quote":"WHU-RS19, the single-region source dataset contributing one client."},{"cited_title":"Bag-of-visual-words and spatial extensions for land-use classification","cited_arxiv_id":null,"evidence_quote":"UCM, the USGS-sourced dataset contributing two clients."},{"cited_title":"Scene classification with recurrent attention of vhr remote sensing images","cited_arxiv_id":null,"evidence_quote":"OPTIMAL-31, the VHR scene-classification source dataset contributing one client."},{"cited_title":"Accurate object localization in remote sensing images based on convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"RSD46-WHU, a multi-source dataset contributing 35 clients."}],"review_version":1}