REVIEW 4 major objections 5 minor 2 cited by
Any2Any: Unified Arbitrary Modality Translation for Remote Sensing
T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read A single latent-diffusion model can translate any remote-sensing modality into any other, beating per-pair models.
desk verdict Useful dataset and unified framework; the 'geometrically aligned latent space' claim is overreaching, but the paper deserves serious peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The shared latent space Z: each of the five modalities is projected by a separately trained VAE into a common 4×64×64 tensor space, regularized toward a standard normal and scaled by hand-set per-modality factors. The Latent Anchor mechanism then treats the paired target's encoded latent zj as a delta-distribution supervision target, so the shared Diffusion Transformer (DiT) can be trained to regress the clean target latent directly (x0-prediction) instead of predicting noise. Target-indexed residual adapters, zero-initialized and trained with stop-gradient, add a learned correction outside the denoising loop; modality tokens enter through AdaLN conditioning. This trio—unified latent, anchor
What would settle it
Take a held-out set of co-registered SAR–PAN and PAN–MS pairs (or split the RST-1M RGB sources by satellite) and compare Any2Any's zero-shot outputs against ground truth with PSNR/SSIM; if the zero-shot pairs are no better than a constant-colour baseline, or if performance drops sharply when the RGB pivot comes from a different satellite than the one used at training, the shared-latent alignment claim fails.
Extended reading notes
Core claim
Any2Any's central claim is that once heterogeneous sensor images are embedded into a common latent space by per-modality autoencoders, a single diffusion transformer can predict the target modality's latent from any source modality's latent. The paired target latent acts as a deterministic 'anchor' that turns generation into anchored regression, while target-specific residual adapters, applied once after denoising, correct the systematic mismatches left by independently trained autoencoders. Trained on 14 directed tasks from the connected supervision graph of RST-1M, the model claims state-of-the-art PSNR/SSIM/RMSE on all evaluated pairs and zero-shot generalization to six unseen modality pa
Load-bearing premise
The load-bearing premise is that RGB images from very different satellites and resolutions are interchangeable in the dataset, and that independently trained VAE latents, nudged only by a small KL penalty and manual scaling factors, really do lie in one geometrically aligned shared space; if that alignment fails, the shared backbone is not learning one coherent scene map and the zero-shot results lose their grounding.
Editorial extensions
If this is right
- A single trained model can synthesize any missing modality for an observed scene, including pairs never seen together in training.
- Adding a new sensor modality requires only training one new VAE and one small adapter; the existing backbone and all other adapters stay frozen.
- Supervision can be sparse and transitive: training on a connected graph of modality pairs transfers along paths (e.g., SAR→RGB and RGB→PAN support SAR→PAN).
- Inference cost is independent of the number of modalities: one forward pass of the shared backbone plus one adapter.
- The O(N²) storage and training burden of direction-specific translators is replaced by O(N) modality encoders/decoders plus one shared network.
Reading between the lines
- If the shared latent is truly scene-aligned, zero-shot translations are compositions of learned pairwise paths; a direct test is whether SAR→PAN output accuracy on held-out co-registered pairs matches SAR→RGB→PAN composition quality.
- The dataset pools RGB from Sentinel-based 10 m sources with WorldView pan-sharpened sub-metre RGB under one modality label; if these are not physically interchangeable, the 'shared' space may encode two clusters, and zero-shot results would degrade when source and pivot resolutions differ. This is testable by splitting the RGB test set by source satellite.
- The KL weight is tiny and latent alignment relies heavily on manual scaling factors; a stronger alignment check would be whether nearest-neighbour latents across modalities correspond to the same geographic scene more often than chance.
- A practical extension the paper leaves implicit: the learned latent anchors could provide synthetic paired supervision for modality pairs that are physically unavailable, effectively growing the supervision graph iteratively.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces Any2Any, a unified latent diffusion framework for translating among five remote sensing modalities (RGB, SAR, NIR, MS, PAN). The method consists of: (i) independently trained per-modality VAEs that project each modality into a common latent dimension; (ii) a shared Diffusion Transformer backbone that, conditioned on source/target modality embeddings, directly regresses the target latent from a concatenation of the source latent and a noisy target latent; and (iii) lightweight target-specific residual adapters that correct latent mismatches. The authors also construct RST-1M, an aggregated dataset of about 1.2M pairwise aligned images drawn from SEN1-2, SEN12MS, CACo, SpaceNet-3, and SpaceNet-5, covering seven paired modality combinations and supporting 14 directed translation tasks. Experiments report improved PSNR/SSIM/RMSE over generic image-to-image baselines on these 14 tasks, and qualitative results on six unseen directions are presented as evidence of zero-shot generalization. The paper claims to reduce cross-modal translation complexity from O(N^2) to O(1).
Significance. If the empirical and architectural claims hold, the paper would be a useful step toward scalable multi-modal remote sensing translation: it provides a large benchmark dataset, a unified model that avoids per-pair training, and promising qualitative generalization. The dataset itself, if released, would be a valuable resource. The paper also incorporates sensible design choices: x0-prediction with a shared backbone, zero-initialized residual adapters, and a clear staged training protocol. The ablations, though limited, are informative. However, the central claims rest on the existence of a 'geometrically aligned shared latent space', which is asserted but not demonstrated, and the zero-shot evaluation is only qualitative. These gaps prevent the paper from being accepted in its current form.
major comments (4)
- [§4.3, Eq. (3); §4.5, Eq. (8)] The central claim of a 'geometrically aligned shared latent manifold Z' is not established. Stage I trains independent per-modality VAEs, each regularized only toward N(0,I) via its own KL loss. This constrains each marginal but imposes no cross-modal correspondence: there is no paired latent loss, no shared encoder/decoder, and no canonical coordinate alignment. The scaling factors in §5.1 normalize latent magnitudes, not directions. Consequently, after Stage I the latent spaces are merely equal-dimensional, approximately normal distributions, not a common semantic manifold. The Stage II backbone learns mappings from each source latent space to each target latent space under paired supervision; such mappings need not compose transitively. The residual adapters are trained only for seen target-source combinations, so for an unseen pair such as SAR→PAN there is no grounded basis for using
- [§5.1/Table 2] The comparison omits recent remote-sensing-specific translation methods that are discussed in Related Work (§2.2), specifically S3OIL, RLIDM, and DOGAN. These are direct state-of-the-art baselines for SAR→optical translation, a central task in Table 2. Without them, the claim that Any2Any 'consistently outperforms existing methods' is not fully supported. Please add these methods (or explain why they are excluded, e.g., unavailability of code/results) and report their numbers on the same test sets.
- [§5.2, 'Zero-shot Experiments', Fig. 5] Zero-shot generalization is a principal contribution, but it is evaluated only qualitatively. Figure 5 shows 'reasonable' outputs without any quantitative metric or comparison. Since the six unseen directions lack paired ground truth, a natural protocol is to hold out a subset of existing paired directions during training and evaluate quantitatively on those, or to compare against chaining pairwise models (e.g., SAR→RGB→PAN). As written, the claim of 'strong zero-shot generalization' is not quantitatively supported.
- [§3, Appendix B.2/B.3] The construction of RST-1M relies on 'RGB' as a pivot, but the physical content of this modality is heterogeneous: RGB from SEN12MS/SEN1-2/CACo is Sentinel-2 at ~10 m resolution, while RGB from SpaceNet is pan-sharpened WorldView at sub-meter resolution. These are treated as the same modality and encoded by a single RGB VAE. If the RGB encoder spans two different domains, the 'shared' latent space may contain a domain split that undermines the connectedness of the modality graph. The paper should analyze whether the RGB encoder is truly unified (e.g., by visualizing latent embeddings or evaluating RGB reconstruction on both sources) or at minimum discuss this limitation.
minor comments (5)
- [Abstract] 'First million-scale dataset with paired observations across five sensing modalities' is overstrong: RST-1M is an aggregation of pairwise datasets, not a collection of five-modality aligned tuples. Clarify that it provides connected pairwise supervision.
- [Table 2] No error bars, standard deviations, or significance tests are reported. Even if the improvements are large, a reader cannot assess run-to-run variability. Please report at least means and stds over multiple seeds.
- [Table 3] The layout is very hard to parse: the checkmarks and column headers are ambiguous, especially for Settings 5–6. Please present each setting on a separate row with explicit column names.
- [Eq. (1)] Treating p(z|x_i) as a Dirac delta is a conceptual idealization; in practice the conditional distribution is not degenerate. This is acceptable as motivation but should be phrased as an approximation rather than a literal statement.
- [§5.2] The text says SSIM is 'slightly lower than BBDM in a few cases' but the exact directions are not listed. Please specify which tasks and by how much; this would help readers interpret the trade-off.
Circularity Check
No significant circularity: the central translation results are trained and evaluated against external pixel-space ground truth, and the alignment assumption is an architectural premise rather than a self-referential prediction.
full rationale
The paper's derivation chain is not circular. Stage I trains independent per-modality VAEs with a reconstruction plus KL objective, and Stage II regresses the shared DiT backbone to latent anchors derived from ground-truth target images through Eq. (10). The latent anchors are not fitted constants that are later renamed as predictions; they are the regression targets themselves, and the final evaluation is done in pixel space against external baselines using PSNR, SSIM, and RMSE. The 'geometrically aligned latent space' is an architectural assumption: the paper regularizes each VAE toward N(0,I) and then asserts alignment, but no target result is encoded into the training objective. If the assumption fails, the zero-shot composition would be empirically incorrect, not true by construction. The residual adapters are trained with a stop-gradient latent reconstruction loss against ground-truth latent anchors and ablated on held-out test splits. The claimed O(1) scalability follows directly from the architecture of one shared DiT backbone plus per-modality encoders, decoders, and adapters, and is not produced by fitting a parameter that already contains the reported metrics. The RST-1M dataset is assembled from public sources, and the model is compared against external, independently published methods. Self-citations appear only in background and related-work references and are not load-bearing for the central claims. The concern about whether independent VAEs actually yield a common latent manifold is a correctness or robustness limitation, not a circular reduction of the outputs to the inputs.
Assumptions & free parameters
free parameters (5)
- Modality latent scaling factors s_SAR, s_RGB, s_MS, s_NIR, s_PAN =
0.422003, 0.387068, 0.484645, 0.568811, 0.447582
- VAE perceptual weight γ =
1.0 for RGB, 0 for others
- KL weight β =
1e-5
- Calibration weight λ =
1.0
- Per-modality standardization resolutions =
PAN 512×512, RGB 256×256, SAR/NIR 256×256, MS 128×128
assumptions (4)
- standard math Diffusion forward noising and x0 reparameterization (Eq. 6) are valid; standard VAE and KL objectives apply.
- domain assumption Spatially aligned pairs from different public datasets are co-registered and can be pivoted through RGB without material geometric or photometric mismatch.
- domain assumption Independently trained per-modality VAEs, regularized only by KL and scaling factors, yield a geometrically aligned shared latent space Z.
- domain assumption The conditional target distribution collapses to a Dirac delta centered on the paired latent anchor (Eq. 1).
invented entities (1)
-
Unified geometrically aligned shared latent space Z
Cite this review
Pith. "Pith review of Any2Any: Unified Arbitrary Modality Translation for Remote Sensing." pith.science (2026). https://pith.science/paper/P2BVGE5A
@misc{pith2026260304114,
author = {Pith},
title = {Pith review of: Any2Any: Unified Arbitrary Modality Translation for Remote Sensing},
year = {2026},
howpublished = {\url{https://pith.science/paper/P2BVGE5A}},
note = {Machine review of arXiv:2603.04114}
}
read the original abstract
Multi-modal remote sensing imagery provides complementary observations of the same geographic scene, yet such observations are frequently incomplete in practice. Existing cross-modal translation methods treat each modality pair as an independent task, resulting in quadratic complexity and limited generalization to unseen modality combinations. We formulate Any-to-Any translation as inference over a shared latent representation of the scene, where different modalities correspond to partial observations of the same underlying semantics. Based on this formulation, we propose Any2Any, a unified latent diffusion framework that projects heterogeneous inputs into a geometrically aligned latent space. Such structure performs anchored latent regression with a shared backbone, decoupling modality-specific representation learning from semantic mapping. Moreover, lightweight target-specific residual adapters are used to correct systematic latent mismatches without increasing inference complexity. To support learning under sparse but connected supervision, we introduce RST-1M, the first million-scale remote sensing dataset with paired observations across five sensing modalities, providing supervision anchors for any-to-any translation. Experiments across 14 translation tasks show that Any2Any consistently outperforms pairwise translation methods and exhibits strong zero-shot generalization to unseen modality pairs. Code and models are available at https://github.com/MiliLab/Any2Any.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
UniTriGen: Unified Triplet Generation of Aligned Visible-Infrared-Label for Few-Shot RGB-T Semantic Segmentation
UniTriGen uses unified diffusion in a shared latent space plus lightweight adapters and scene-balanced sampling to produce high-quality aligned VIS-IR-Label triplets from limited paired data, improving few-shot RGB-T ...
-
CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration
CoRE-UIR achieves state-of-the-art all-in-one remote sensing image restoration with a common dense expert plus low-rank routed residual experts, improving PSNR by 1.05 dB over BaryIR at 11.83x lower latency.
Reference graph
Works this paper leans on
-
[1]
For what purpose was the dataset created?
“For what purpose was the dataset created?” A: Existing remote sensing image-to-image translation datasets typically include only 1–3 modalities and a small number of modality pairs, with limited image scale and coverage. Such constraints make them insufficient for studying arbitrary cross-modal translation in realistic remote sensing scenarios. To addres...
-
[2]
Who created the dataset (e.g., which team, research group) and on behalf of which entity?
“Who created the dataset (e.g., which team, research group) and on behalf of which entity?” A: This dataset was created collaboratively by all authors of this paper
-
[3]
Who funded the creation of the dataset?
“Who funded the creation of the dataset?” A: The development of the dataset was supported by the authors’ affiliated institutions. B.5.2. COMPOSITION The majority of questions in this section are intended to equip dataset users with the information necessary to make informed decisions regarding the dataset’s suitability for their tasks. Some questions spe...
-
[4]
What do the instances that comprise our datasets represent (e.g., documents, photos, people, countries)?
“What do the instances that comprise our datasets represent (e.g., documents, photos, people, countries)?” A: Our RST-1M is built upon publicly available datasets via systematic preprocessing and standardization, yielding seven groups of spatially aligned modality pairs and a unified dataset covering five sensing modalities
-
[5]
How many instances are there in total (of each type, if appropriate)?
“How many instances are there in total (of each type, if appropriate)?” A: RST-1M consists of 1,175,000 remote sensing images and 1,200,000 spatially aligned modality pairs, covering five remote sensing modalities: RGB, SAR, NIR, MS, and PAN
-
[6]
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
“Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?” A: The images in RST-1M are sourced from existing remote sensing modality translation datasets, including SEN1- 2 (Schmitt et al., 2018), SEN12MS (Schmitt et al., 2019), CACo (Mall et al., 2023), SpaceNet-3 (Van Etten et al., 2018),...
2018
-
[7]
Is there a label or target associated with each instance?
“Is there a label or target associated with each instance?” A: Yes. For each remote sensing image, we provide a corresponding spatially aligned paired image from another modality
-
[8]
Is any information missing from individual instances?
“Is any information missing from individual instances?” A: No, each individual instance is complete
Show all 31 references
-
[9]
Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?
“Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?” A: Yes, the relationship between individual instances is explicit
-
[10]
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
“Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?” A: RST-1M is a self-contained dataset that will be publicly released on platforms such as Hugging Face to facilitate easy access and use
-
[11]
“Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor–patient confidentiality, data that includes the content of individuals’ non-public communications)?” A: No, all data are clearly licensed
-
[12]
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
“Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?” A: No, RST-1M does not contain any data with negative information. B.5.3. COLLECTIONPROCESS Beyond the objectives described in the previous s...
-
[13]
How was the data associated with each instance acquired?
“How was the data associated with each instance acquired?” A: Our RST-1M is derived from an aggregation of five publicly available repositories: SEN1-2 (Schmitt et al., 2018), SEN12MS (Schmitt et al., 2019), CACo (Mall et al., 2023), SpaceNet-3 (Van Etten et al., 2018), and Sp...
2018
-
[14]
“Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?” A: Yes. To expand modality coverage, we preprocess raw Sentinel-2 ...
-
[15]
Was the ‘raw’ data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?
“Was the ‘raw’ data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?” A: Yes, raw data is accessible. B.5.5. USES The questions in this section aim to prompt dataset creators to consider the intended and unintended uses o...
-
[16]
Has the dataset been used for any tasks already?
“Has the dataset been used for any tasks already?” A: No
-
[17]
Is there a repository that links to any or all papers or systems that use the dataset?
“Is there a repository that links to any or all papers or systems that use the dataset?” A: Yes, we will provide such links on GitHub and the Huggingface repository
-
[18]
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
“Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?” A: No
-
[19]
Are there tasks for which the dataset should not be used?
“Are there tasks for which the dataset should not be used?” A: N/A. B.5.6. DISTRIBUTION Dataset creators are expected to answer these questions before releasing the dataset, whether for internal use within the originating organization or for external distribution to third parties
-
[20]
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
“Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?” A: No. Our RST-1M dataset will be made publicly accessible to the research community
-
[21]
How will the dataset be distributed (e.g., tarball on website, API, GitHub)?
“How will the dataset be distributed (e.g., tarball on website, API, GitHub)?” A: We will provide RST-1M in the GitHub and the Huggingface repository
-
[22]
When will the dataset be distributed?
“When will the dataset be distributed?” A: We plan to release the dataset through a public repository upon official publication of the paper, while maintaining 16 Any2Any: Unified Arbitrary Modality Translation for Remote Sensing compliance with anonymity policies
-
[23]
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
“Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?” A: Yes, the dataset will be released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License
-
[24]
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
“Have any third parties imposed IP-based or other restrictions on the data associated with the instances?” A: No
-
[25]
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
“Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?” A: No. B.5.7. MAINTENANCE Similar to the previous section, these questions should be addressed by dataset creators before dataset release. They are intended to encourage ...
-
[26]
Who will be supporting/hosting/maintaining the dataset?
“Who will be supporting/hosting/maintaining the dataset?” A: The dataset will be supported, hosted, and maintained by the authors of this work
-
[27]
How can the owner/curator/manager of the dataset be contacted (e.g., email address)?
“How can the owner/curator/manager of the dataset be contacted (e.g., email address)?” A: Upon acceptance of the paper, the authors’ email addresses will be provided in the paper or on the project website, through which the dataset curators can be contacted
-
[28]
Is there an erratum?
“Is there an erratum?” A: No separate erratum is planned; known issues and corrections will be documented in subsequent dataset releases
-
[29]
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
“Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?” A: Any future updates will be announced and documented on the dataset website
-
[30]
Will older versions of the dataset continue to be supported/hosted/maintained?
“Will older versions of the dataset continue to be supported/hosted/maintained?” A: Yes. This initial release may be updated over time, with earlier versions superseded by newer releases
-
[31]
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?
“If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?” A: Yes. We will provide clear and detailed guidelines to support future dataset extensions. 17 Any2Any: Unified Arbitrary Modality Translation for Remote Sensing MS(C...
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.