REVIEW 3 major objections 4 minor 47 references
ColorFM: An Optimization-to-Learning Framework for Color Transfer via Flow Matching
T0 review · 3 major / 4 minor · reviewed 2026-07-09 · glm-5.2
Pith's one-line read One-step color transfer distills slow optimization into real-time inference
desk verdict Solid optimization-to-learning framework for color transfer; the hierarchical color coupling strategy is the real novelty, but the evaluation is thin enough to weaken the central SOTA claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The framework rests on three load-bearing components. First, explicit semantic alignment: pixels from matching semantic regions (sky-to-sky, skin-to-skin) are paired for distribution transport, but instead of processing each region independently (which causes seam artifacts), all pairs are fed into a single unified velocity field that is optimized jointly. Second, hierarchical color coupling (HCC): a recursive algorithm that partitions source and target color distributions into octants of RGB space, aligning their relative positions at each level before pairing points at leaf nodes. This preserves locality in color space, yielding nearly straight flow paths. Third, bidirectional linearized:Θ
What would settle it
Find a content-style pair where the optimization-stage trajectory has a path-length ratio significantly above 1.0 (say, 1.2 or higher), then show that the one-step bidirectional transport in ColorFM-L produces visibly incorrect or banded color output compared to the full multi-step ODE integration of ColorFM-O. If such pairs are common rather than rare, the core premise — that hierarchical coupling reliably yields straight paths that can be collapsed to a single step — fails.
Extended reading notes
Core claim
The central mechanism is the discovery that carefully constructed pixel pairings — specifically, recursive octant-based hierarchical coupling of color distributions within semantic regions — produce flow trajectories in RGB space that are quasi-linear (path-length ratio ~1.009). This geometric regularity means the entire color transport can be approximated by a single bidirectional Euler step: push the content image forward to an implicitly learned intermediate color state, then pull it backward toward the style distribution. The feed-forward model predicts the velocity-field parameters that define this two-step transport, inheriting the precision of per-instance optimization without its per
Load-bearing premise
The entire learning pipeline depends on the quasi-linearity of the optimization-stage trajectories. If this near-straightness breaks down for complex or out-of-distribution content-style pairs — which the paper itself acknowledges as a risk — the single-step Euler approximation in the feed-forward model will produce inaccurate color mappings, since there is no iterative correction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes ColorFM, a two-stage framework for color transfer that bridges optimization-based and learning-based paradigms via Flow Matching (FM). The first stage, ColorFM-O, performs instance-specific optimization of a velocity field using hierarchical color coupling (HCC) and semantic priors to generate pseudo-ground truth. The second stage, ColorFM-L, is a feed-forward model trained on this pseudo-supervision that predicts flow parameters for a bidirectional, one-step linearized transport. The core claim is that ColorFM-L achieves state-of-the-art performance in visual quality and structural fidelity while enabling real-time inference. The approach is technically coherent, and the optimization-to-learning pipeline is a reasonable way to transfer the fidelity of iterative methods to a fast inference model.
Significance. Color transfer is an established problem, and the formulation of color distribution transport via Flow Matching is a meaningful contribution to the literature. The hierarchical color coupling (HCC) strategy (Algorithm 1) is a practical heuristic for constructing geometrically coherent couplings without the cubic cost of exact Optimal Transport. The bidirectional linearized transport in ColorFM-L (Eq. 4) is an efficient approximation enabled by the quasi-linearity of the HCC-induced trajectories. The authors provide code and a large-scale dataset, which adds value. The core claim of combining optimization accuracy with feed-forward speed is well-motivated and supported by the efficiency gains reported in Table 1 (0.016s inference).
major comments (3)
- §4.1 and Table 1: The evaluation is based on a test set of 40 images exhaustively permuted into 1,560 pairs. Because each image appears in approximately 78 pairs, these pairs are not statistically independent. Treating 1,560 as the sample size without accounting for this clustered structure inflates the apparent statistical power. The margin between ColorFM-L (0.320) and ColorFM-O (0.338) in 'Distance to Ideal' is small. The authors should either report standard errors or confidence intervals adjusted for the clustered design, or evaluate on a larger, independent test set to confirm that the reported differences are statistically meaningful.
- §4.2 and Table 1: The Style Similarity metric relies on a discriminator pretrained by Neural Preset [14]. Since ColorFM-L is trained on pseudo-ground truth generated by ColorFM-O, and the discriminator may systematically prefer outputs resembling its own training distribution, there is a risk of evaluation bias. The authors should either validate this metric against human judgments or include an alternative, standard metric (e.g., FID or feature statistics matching using a standard backbone like VGG or CLIP) to corroborate the style transfer quality independently.
- §3.4 and §4.3: The one-step bidirectional Euler approximation in ColorFM-L (Eq. 4) relies on the quasi-linearity of ColorFM-O trajectories, supported by a path length ratio of 1.009 (§4.3). However, this ratio is computed on ColorFM-O outputs, which are the training distribution for ColorFM-L. The paper acknowledges in §6 that ColorFM-L may face generalization challenges for out-of-distribution styles. The manuscript would benefit from an explicit evaluation of trajectory linearity on held-out or extreme style pairs to validate that the one-step approximation holds for the cases where ColorFM-L is applied without ColorFM-O's direct supervision.
minor comments (4)
- §3.3, Algorithm 1: The notation 'O(v)' for the octant index is introduced but could be clarified for readers unfamiliar with octant partitioning. A brief inline definition would improve readability.
- Table 2: The 'Time (s)' column for Mini-Batch OT (51.48s) is significantly higher than other methods. It would be useful to clarify whether this includes the coupling construction time or only the optimization, to ensure a fair comparison.
- Fig. 2: The diagram is dense. The distinction between the 'Param Generator' and the 'pixel-wise MLP' in ColorFM-L could be labeled more explicitly to clarify that the generator predicts weights for the MLP rather than processing pixels directly.
- §4.1: The training dataset composition (237,408 triplets) is described, but the diversity of the style images is not detailed. Given the reliance on semantic categorization, a brief note on the style distribution would help contextualize the generalization claims.
Circularity Check
No significant circularity found; the derivation chain is self-contained.
full rationale
The paper's derivation chain is straightforward and non-circular. ColorFM-O reformulates color transfer as Flow Matching (citing external works [2,21,23]) and introduces a novel Hierarchical Color Coupling strategy (Algorithm 1) to construct source-target pairs, then optimizes a velocity field via the standard FM objective (Eq. 2). The resulting outputs serve as pseudo-ground truth for training ColorFM-L. This is explicitly acknowledged self-supervision — the core design of the optimization-to-learning framework — not hidden circularity. ColorFM-L is then evaluated on a separate test set (40 images, no overlap with training) using external metrics: Style Similarity (discriminator from Neural Preset [14], an external work), Content Similarity (SSIM on edge maps), and Lipschitz Constant. The quasi-linearity assumption (path length ratio 1.009) that justifies the one-step Euler approximation in Eq. 4 is empirically verified, not imported via self-citation. No self-citations are load-bearing for the central claims. No step in the derivation reduces to its inputs by construction. The concern that the Style Similarity discriminator might prefer ColorFM-O-like outputs is a metric-bias concern (correctness risk), not a circularity in the derivation chain.
Assumptions & free parameters
free parameters (9)
- Dmax (max tree depth for HCC) =
3
- lambda (LPIPS loss weight) =
0.1
- ODE integration steps (ColorFM-O inference) =
5
- ColorFM-O optimization steps =
700
- ColorFM-O learning rate =
5e-4
- ColorFM-L learning rate =
1e-5
- Batch size (ColorFM-O) =
4096
- Batch size (ColorFM-L) =
48
- Input resolution (ColorFM-L) =
256x256
assumptions (5)
- domain assumption Color transfer can be formulated as transport of pixel distributions in RGB color space via Flow Matching (§3.2).
- ad hoc to paper Preserving relative locality in color space suffices to eliminate banding and ensure flow coherence (§3.3, HCC insight).
- ad hoc to paper The quasi-linearity of ColorFM-O trajectories holds sufficiently for one-step Euler approximation in ColorFM-L (§3.4).
- domain assumption Semantic segmentation masks from SegFormer-B5 on ADE20K provide adequate semantic correspondence for color transfer (§3.3).
- domain assumption The Style Similarity discriminator from Neural Preset [14] is a valid evaluator for color transfer quality (§4.2).
invented entities (2)
-
Implicit intermediate state z
-
Hierarchical Color Coupling (HCC) plan
independent evidence
Cite this review
Pith. "Pith review of ColorFM: An Optimization-to-Learning Framework for Color Transfer via Flow Matching." pith.science (2026). https://pith.science/paper/QAVDTH62
@misc{pith2026260707119,
author = {Pith},
title = {Pith review of: ColorFM: An Optimization-to-Learning Framework for Color Transfer via Flow Matching},
year = {2026},
howpublished = {\url{https://pith.science/paper/QAVDTH62}},
note = {Machine review of arXiv:2607.07119}
}
read the original abstract
Color transfer aims to align the color distribution of a source image with that of a reference image while preserving structural and semantic consistency. However, existing methods often suffer from inaccurate global mapping, semantic misalignment, and visual artifacts. To address these issues, we propose ColorFM, an optimization-to-learning framework. ColorFM connects online optimization to offline inference by reformulating color transfer as the transport of pixel distributions along velocity fields via Flow Matching. Specifically, we introduce ColorFM-O, an instance-specific optimization scheme that fits the velocity field through hierarchical color coupling guided by semantic priors. By numerically integrating the induced flow trajectories, ColorFM-O produces precise and semantically consistent color transfer results, while generating high-quality paired data as pseudo-supervision. Building upon this, we design ColorFM-L, an efficient feed-forward model trained on the generated pairs. Through implicit state modeling, ColorFM-L extracts deep semantic features to predict flow parameters for bidirectional linearized transport, ensuring accurate color transfer. Extensive experiments demonstrate that ColorFM-L outperforms state-of-the-art methods in visual quality, structural fidelity, and semantic consistency, successfully combining the accuracy of optimization with the speed of feed-forward inference.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
- [14]
- [1]
-
[2]
Albergo, M.S., Vanden-Eijnden, E.: Building normalizing flows with stochastic interpolants. In: ICLR (2023) 2, 4
work page 2023
- [3]
-
[4]
Chen, Y., Yang, H., Yang, Y., Liu, Y., Wang, W., Wen, X., Xie, C.: NLUT: Neural- based 3D lookup tables for video photorealistic style transfer. arXiv (2023) 2, 3, 9, 10
work page 2023
- [5]
- [6]
-
[7]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021) 9
work page 2021
Show all 47 references
-
[8]
In: AISTATS
Fatras, K., Zine, Y., Flamary, R., Gribonval, R., Courty, N.: Learning with mini- batch wasserstein : asymptotic and gradient properties. In: AISTATS. vol. 108, pp. 2131–2141 (2020) 4 16 Y. He et al
2020
-
[9]
In: CVPR
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR. pp. 2414–2423 (2016) 3
2016
-
[10]
In: ICCV
Gong, Z., Wu, Z., Tao, Q., Li, Q., Loy, C.C.: SA-LUT: Spatial adaptive 4d look-up table for photorealistic style transfer. In: ICCV. pp. 18294–18303 (2025) 2, 3, 4, 9, 10
2025
-
[11]
In: WACV
Ho, M.M., Zhou, J.: Deep preset: Blending and retouching photos with color style transfer. In: WACV. pp. 2113–2121 (2021) 2, 3, 9, 10
2021
-
[12]
In: ICCV
Hong, K., Jeon, S., Yang, H., Fu, J., Byun, H.: Domain-aware universal style trans- fer. In: ICCV. pp. 14609–14617 (2021) 3
2021
-
[13]
In: ICCV
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV. pp. 1501–1510 (2017) 3
2017
-
[15]
In: ICLR (2015) 9
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015) 9
2015
-
[16]
In: AAAI
Larchenko, M., Lobashev, A., Guskov, D., Palyulin, V.V.: Color transfer with mod- ulated flows. In: AAAI. pp. 4464–4472 (2025) 2, 3, 4, 7, 9, 10
2025
-
[17]
In: WACV
Li, M., Wang, G., Zhang, X., Liao, Q., Xiao, C.: D-LUT: Photorealistic style trans- fer via diffusion process. In: WACV. pp. 9188–9196 (2025) 2, 3, 9, 10
2025
-
[18]
NeurIPS30(2017) 3
Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Universal style transfer via feature transforms. NeurIPS30(2017) 3
2017
-
[19]
In: ECCV
Li, Y., Liu, M.Y., Li, X., Yang, M.H., Kautz, J.: A closed-form solution to photo- realistic image stylization. In: ECCV. pp. 453–468 (2018) 2, 3
2018
-
[20]
In: AAAI
Lin, T., Lin, H., Li, F., He, D., Wu, W., Wang, M., Li, X., Liu, Y.: AdaCM: adaptive colormlp for real-time universal photo-realistic style transfer. In: AAAI. pp. 1613–1621 (2023) 2, 3
2023
-
[21]
In: ICLR (2023) 2, 4, 6
Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. In: ICLR (2023) 2, 4, 6
2023
-
[22]
IEEE TIP32, 4742–4756 (2023) 2
Liu, C., Yang, H., Fu, J., Qian, X.: 4D LUT: learnable context-aware 4d lookup table for image enhancement. IEEE TIP32, 4742–4756 (2023) 2
2023
-
[23]
In: ICLR (2023) 2, 4, 6
Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow. In: ICLR (2023) 2, 4, 6
2023
-
[24]
In: ICLR (2024) 4
Liu, X., Zhang, X., Ma, J., Peng, J., et al.: Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In: ICLR (2024) 4
2024
-
[25]
In: CVPR
Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: CVPR. pp. 4990–4998 (2017) 2, 3, 6
2017
-
[26]
TMLR (2024) 9
Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P., Li, S., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., ...
2024
-
[27]
In: CVPR
Park, D.Y., Lee, K.H.: Arbitrary style transfer with style-attentional networks. In: CVPR. pp. 5880–5888 (2019)
2019
-
[28]
In: CVMP
Pitié, F., Kokaram, A.: The linear monge-kantorovitch linear colour mapping for example-based colour transfer. In: CVMP. pp. 1–9 (2007) 2, 3
2007
-
[29]
In: ICCV
Pitie, F., Kokaram, A.C., Dahyot, R.: N-dimensional probability density function transfer and its application to color transfer. In: ICCV. pp. 1434–1439 (2005) 2, 3
2005
-
[30]
CVIU107(1-2), 123–137 (2007) 2, 3 ColorFM for Color Transfer 17
Pitié, F., Kokaram, A.C., Dahyot, R.: Automated colour grading using colour dis- tribution transfer. CVIU107(1-2), 123–137 (2007) 2, 3 ColorFM for Color Transfer 17
2007
-
[31]
In: ICML
Pooladian,A.A.,Ben-Hamu,H.,Domingo-Enrich,C.,Amos,B.,Lipman,Y.,Chen, R.T.: Multisample flow matching: straightening flows with minibatch couplings. In: ICML. pp. 28100–28127 (2023) 4, 6
2023
-
[32]
In: ICML
Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., Courville, A.: On the spectral bias of neural networks. In: ICML. pp. 5301–5310 (2019) 11
2019
-
[33]
In: ICLR Workshop (2018) 9
Ramachandran, P., Zoph, B., Le, Q.V.: Searching for activation functions. In: ICLR Workshop (2018) 9
2018
-
[34]
NeurIPS32(2019) 3
Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. NeurIPS32(2019) 3
2019
-
[35]
IEEE Access10, 68281–68290 (2022) 10
Soria, X., Pomboza-Junez, G., Sappa, A.D.: LDC: Lightweight dense cnn for edge detection. IEEE Access10, 68281–68290 (2022) 10
2022
-
[36]
TMLR (2024) 4, 6
Tong, A., Fatras, K., Malkin, N., Huguet, G., Zhang, Y., Rector-Brooks, J., Wolf, G.,Bengio,Y.:Improvingandgeneralizingflow-basedgenerativemodelswithmini- batch optimal transport. TMLR (2024) 4, 6
2024
-
[37]
In: CVPR
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: CVPR. pp. 9446– 9454 (2018) 11
2018
-
[38]
arXiv preprint arXiv:2404.02733 (2024) 3
Wang, H., Spinelli, M., Wang, Q., Bai, X., Qin, Z., Chen, A.: InstantStyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.02733 (2024) 3
2024 arXiv
-
[39]
IEEE TIP13(4), 600–612 (2004) 10
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE TIP13(4), 600–612 (2004) 10
2004
-
[40]
In: CVPR
Wen, L., Gao, C., Zou, C.: CAP-VSTNet: Content affinity preserved versatile style transfer. In: CVPR. pp. 18300–18309 (2023) 2, 3, 6, 9, 10
2023
-
[41]
In: ECCV
Xia, X., Zhang, M., Xue, T., Sun, Z., Fang, H., Kulis, B., Chen, J.: Joint bilateral learning for real-time universal photorealistic style transfer. In: ECCV. pp. 327–342 (2020) 2, 3
2020
-
[42]
NeurIPS 34(2021) 9
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS 34(2021) 9
2021
-
[43]
In: CVPR
Ye, Z., Huang, H., Wang, X., Wan, P., Zhang, D., Luo, W.: Stylemaster: Styl- ize your video with artistic generation and translation. In: CVPR. pp. 2630–2640 (2025) 3
2025
-
[44]
In: ICCV
Yoo, J., Uh, Y., Chun, S., Kang, B., Ha, J.W.: Photorealistic style transfer via wavelet transforms. In: ICCV. pp. 9036–9045 (2019) 2, 3, 6, 9, 10
2019
-
[45]
IEEE TPAMI44(4), 2058–2073 (2022) 2, 3
Zeng, H., Cai, J., Li, L., Cao, Z., Zhang, L.: Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time. IEEE TPAMI44(4), 2058–2073 (2022) 2, 3
-
[46]
In: CVPR
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR. pp. 586–595 (2018) 8
2018
-
[47]
In: CVPR
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: CVPR. pp. 633–641 (2017) 9
2017
Reviewed July 9, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.