REVIEW 3 major objections 6 minor 1 cited by
Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondence
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that an expert's brief functional-correspondence tip, decoded into segment-level matches and used to average in-distribution latent embeddings, lets a vision-based diffusion policy succeed on out-of-distribution objects…
desk verdict Useful system paper with a plausible central claim; the empirical evidence is thinner than the headline, but the integration is novel enough to deserve a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the functional correspondence map Φ(i,î,l), which pairs image segments of the OOD and ID observations according to an expert language description, with alignment scored as summed IoU over matched segments. This is used to rank and retrieve in-distribution observations; the executed intervention then averages the latent embeddings of the top-M retrieved observations, z_hat = (1/M) Σ E(o), and passes this average to the diffusion policy as the conditioning observation. The averaging step is what converts a set of analogous training scenes into a single behavior-bearing input without retraining.
What would settle it
On the object-in-cup task, give the expert description that maps the OOD pencil to the marker instead of the pen (the wrong functional correspondence). If task success remains at the same level as the correct correspondence, then the retrieved functional alignment is not the causal mechanism behind ABA's improvement.
Extended reading notes
Core claim
The central discovery is that behavior transfer across object categories can be driven by functional, not visual, correspondence: a pencil is treated like a pen for the task of placing it in a cup. ABA operationalizes this by grounding an expert's textual correspondence description into segment-level matches (Eq. 1), scoring each training observation by total IoU of corresponding segments (Eq. 2), and then intervening on the policy by replacing the OOD observation's latent embedding with the mean embedding of the top-M matching ID observations. The paper shows this intervention recovers task success under novel objects and backgrounds on a Franka manipulator, and that retrieval based on the policy's own embeddings or DINOv2 features is not sufficient—successful baseline rollouts retrieve the same functionally corresponding observations ABA selects.
Load-bearing premise
The policy's embedding space is assumed to be closed under averaging, so the mean of several good in-distribution observation encodings is still a valid conditioning input that generates the intended behavior.
Editorial extensions
If this is right
- If a diffusion policy's embedding space tolerates averaging of functionally aligned observations, then any task whose OOD objects share affordances with training objects can be adapted at deployment time with a few expert labels.
- Functional correspondence retrieval outperforms both policy-embedding and DINOv2-based visual retrieval for novel-object transfer, indicating that semantic and visual similarity are not sufficient.
- Intervening with functional correspondences also lifts in-distribution success by disambiguating behavior modes, so the same mechanism can act as a general policy improver, not only an OOD fix.
- Expert feedback demand is small relative to rollout length (about 6% of timesteps on sweep-trash), making the method usable in practice.
- Because the policy is not fine-tuned, the approach is compatible with any pretrained diffusion policy and requires only access to the training observations.
Reading between the lines
- An unstated consequence is that the method's ceiling is set by the geometry of the policy's latent space; if the average of retrieved embeddings lands outside the support of the training distribution, the intervention will fail regardless of correspondence quality.
- The same analogy mechanism could be automated by having a vision-language model propose the correspondence description, removing the human expert and opening the door to closed-loop self-adaptation.
- The framework transfers to other embodied domains wherever an affordance-alignment can be defined between seen and unseen conditions, such as terrain traversal or tool use with novel implements.
- A cheap testable extension is to replace the mean-embedding intervention with a weighted average by alignment score; whether that improves or degrades success would directly probe the averaging assumption.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Adapting by Analogy (ABA), a test-time intervention method for visuomotor policies. Given an out-of-distribution (OOD) observation, ABA first detects OOD via cosine similarity between the observation embedding and training embeddings (Sec. 4.1). If OOD, it asks an expert for a natural-language functional correspondence (e.g., "treat a pencil as a pen"), decodes this into semantic segment pairs using Grounded SAM, scores every in-distribution (ID) observation by the total IoU of functionally corresponding segments (Eq. 2), and retrieves the top-M ranked ID observations (Sec. 4.2). The expert instruction is interactively refined until the entropy over K-means-clustered action modes of the retrieved observations is low (Sec. 4.3). Finally, the policy is conditioned on the average embedding of the top-M functionally corresponding ID observations, z_hat = (1/M) * sum E(o), and the resulting action plan is executed (Sec. 4.4). The method is evaluated on a Franka Research 3 with a diffusion policy on two tasks, sweep-trash and object-in-cup, across ID conditions, novel backgrounds, and novel object categories, comparing against Vanilla, PolicyEmbed, and DINOEmbed baselines with 10 rollouts per condition. The headline result is that ABA improves over the vanilla policy by 76% on OOD objects in both tasks (Sec. 5.1).
Significance. The central idea is timely and practically motivated: instead of collecting new demonstrations and retraining at every OOD encounter, ABA reuses existing ID behaviors by establishing functional correspondences with minimal expert input. If the reported results are robust, the method would be a useful low-feedback alternative in settings where ID and OOD environments share geometric and functional structure. The paper deserves credit for testing on real hardware across two tasks and multiple OOD conditions, for comparing against embedding-based intervention baselines (PolicyEmbed and DINOEmbed), and for including subgoal-level and retrieval-level analyses that go beyond aggregate success rates. The expert-language interface and iterative refinement mechanism are clearly described. However, the empirical claims rest on small rollout counts without statistical quantification, and the key intervention step in Sec. 4.4 relies on an unvalidated assumption that averaged ID embeddings are valid conditioning inputs for the diffusion policy. These issues are local and addressable, but they currently limit the strength of the central claim.
major comments (3)
- [Sec. 4.4] The intervention step computes z_hat = (1/M) * sum_{o in O_f} E(o) and feeds this averaged embedding to the diffusion policy. This is the only mechanism that converts retrieved functional correspondences into an action plan, yet the paper provides no evidence that the mean of several top-M ID embeddings lies in the support of the policy's training-time conditioning distribution or that it generates the intended behavior mode rather than a blended or invalid one. The value of M is never reported, no comparison against using the single top-ranked embedding is given, no distance of z_hat to the training embedding manifold is measured, and the averaging operation is not ablated. If the top-M observations straddle distinct behavior modes (e.g., pen-from-front vs. marker-from-top), their mean can fall in a low-density region where the denoiser was not trained, so even a perfect expert correspondence would fail to transfer behavior. I request an ablation or validation of this assumption, at minimum reporting M, comparing top-1 vs. top-M averaging, and measuring the embedding-space distance of z_hat to the ID embedding distribution.
- [Sec. 5.1, Fig. 3] All success rates are computed from 10 rollouts per condition, and no confidence intervals, per-condition counts, or significance tests are reported. The headline claim that ABA improves over vanilla by 76% on OOD objects is therefore ambiguous: with 10 rollouts, a difference between 2/10 and 8/10 would correspond to a 300% relative improvement, while a difference between 5/10 and 9/10 would be 80%; neither is statistically distinguishable at conventional levels. Given that this is the paper's central quantitative claim, I ask for per-condition success counts, binomial confidence intervals, or a paired significance test (e.g., exact McNemar test across matched rollouts), and a clarification of exactly which aggregate the 76% figure refers to.
- [Sec. 4.3 and Sec. 5.3] The feedback-efficiency claim depends on several unspecified thresholds and hyperparameters: the OOD detection threshold lambda, the proprioceptive distance threshold lambda_q, the action-mode entropy threshold, and the number of K-means clusters n_c. None of these values are reported in the main text or supplementary material. Without these, the feedback counts in Fig. 4 cannot be reproduced, and it is unclear how sensitive the method is to these thresholds. I also note that no ablation is shown for the refinement loop itself; comparing ABA with and without the entropy-based refinement step would clarify whether the reported feedback counts and success rates are attributable to the refinement mechanism.
minor comments (6)
- [Sec. 3 and Sec. B (Supplementary)] There is a direct contradiction in the description of the grasp behavior: Sec. 3 (or the main experiment section) states that pens are grasped above their center-of-mass and need to be dropped from the bottom, while markers are grasped below and dropped from the front, but Supplementary Sec. B states the opposite. Please correct this inconsistency.
- [Sec. 5, OOD Environments paragraph] Typo: "we e evaluate" should be "we evaluate." There is also a repeated misspelling of "correspondance" (e.g., Eq. (1) and surrounding text) which should read "correspondence."
- [Sec. 5.3] Typo: "feedback about the pend was requested" should be "feedback about the pen was requested."
- [Sec. 5.1 and Conclusion] The conclusion states that the method improves success rate "by 76% to new objects and backgrounds," but Sec. 5.1 reports 76% only for OOD objects, with 20% and 90% improvements for backgrounds on the two tasks. Please make the conclusion consistent with the results.
- [Sec. 5.4 and Fig. 5] The retrieval-precision analysis is limited to the object-in-cup task in ID and OOD-background conditions, with no data for OOD objects, even though OOD objects are the central claim. Please clarify whether this analysis was also performed for OOD-object rollouts and, if not, why it was omitted.
- [Eq. (2)] The summation index in Eq. (2) starts at i=0 while K is defined as the number of corresponding segments; please make the indexing consistent with the definition of K (e.g., sum over j=1..K or clarify that K counts segments including a zero-indexed segment).
Circularity Check
No significant circularity: ABA's generalization claim is an empirical result; the expert-supplied correspondence is the method's intended input, not a hidden restatement of the output.
full rationale
The derivation chain is empirical throughout. ABA's only 'prediction' is closed-loop task success on OOD objects and backgrounds, measured by physical rollouts. The expert's language input l is decoded into correspondence features phi_l that select segmentation labels (e.g., 'pencil' with 'pen'), and Eq. (2) ranks ID observations by IoU of those labels. This is the intended mechanism: the expert supplies the analogy, and the method retrieves ID observations consistent with it. There is no equation in which the claimed output (generalization improvement) is defined in terms of the retrieved observations; success is scored independently by whether the object lands in the cup or trash is swept to the correct goal. The latent-averaging step in Sec. 4.4 is an unverified assumption about embedding-space closure, but assuming a mechanism is not circularity; the paper does not present it as a derived theorem. Citations to prior work are for standard components (Diffusion Policy, Grounded SAM, DINOv2) and one related-work paper by a co-author ([13]) that is not load-bearing. No self-citation supplies the core premise; no fitted parameter is relabeled as a prediction. Therefore the appropriate score is 0.
Assumptions & free parameters
free parameters (5)
- OOD detection threshold lambda =
not reported
- Proprioceptive distance threshold lambda_q =
not reported
- Action-mode entropy threshold =
not reported
- Number of K-means clusters n_c =
not reported
- Number of retrieved observations M =
not reported
assumptions (5)
- domain assumption The diffusion policy's latent embedding space is closed under averaging, so the mean embedding of top-M ID observations is a valid conditioning input.
- domain assumption ID and OOD environments differ only by objects and background color, with environment geometry unchanged.
- domain assumption The expert's templated language description correctly and unambiguously specifies the functional correspondence.
- domain assumption The OOD detector based on cosine similarity reliably separates OOD from ID observations.
- domain assumption Entropy over K-means action-mode labels is a reliable signal that the functional correspondence is well-specified.
Cite this review
Pith. "Pith review of Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondence." pith.science (2026). https://pith.science/paper/AFDWPEUM
@misc{pith2026250612678,
author = {Pith},
title = {Pith review of: Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondence},
year = {2026},
howpublished = {\url{https://pith.science/paper/AFDWPEUM}},
note = {Machine review of arXiv:2506.12678}
}
read the original abstract
End-to-end visuomotor policies trained using behavior cloning have shown a remarkable ability to generate complex, multi-modal low-level robot behaviors. However, at deployment time, these policies still struggle to act reliably when faced with out-of-distribution (OOD) visuals induced by objects, backgrounds, or environment changes. Prior works in interactive imitation learning solicit corrective expert demonstrations under the OOD conditions -- but this can be costly and inefficient. We observe that task success under OOD conditions does not always warrant novel robot behaviors. In-distribution (ID) behaviors can directly be transferred to OOD conditions that share functional similarities with ID conditions. For example, behaviors trained to interact with in-distribution (ID) pens can apply to interacting with a visually-OOD pencil. The key challenge lies in disambiguating which ID observations functionally correspond to the OOD observation for the task at hand. We propose that an expert can provide this OOD-to-ID functional correspondence. Thus, instead of collecting new demonstrations and re-training at every OOD encounter, our method: (1) detects the need for feedback by first checking if current observations are OOD and then identifying whether the most similar training observations show divergent behaviors, (2) solicits functional correspondence feedback to disambiguate between those behaviors, and (3) intervenes on the OOD observations with the functionally corresponding ID observations to perform deployment-time generalization. We validate our method across diverse real-world robotic manipulation tasks with a Franka Panda robotic manipulator. Our results show that test-time functional correspondences can improve the generalization of a vision-based diffusion policy to OOD objects and environment conditions with low feedback.
Figures
Figures from the paper (8 more)
Forward citations
Cited by 1 Pith paper
-
SemAnCorr: Semantic Anchored Correspondence for Zero-Shot Manipulation Skill Transfer
Training-free semantic anchors plus functional maps yield dense correspondences that are both part-accurate and geometrically coherent, improving zero-shot robot skill transfer over nearest-neighbor 3D feature matching.
Reference graph
Works this paper leans on
-
[1]
A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024
arXiv 2024
-
[2]
A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y . Lu, H. Michalewski, I. Mordatch, K. Pe...
arXiv 2023
-
[3]
A. W. C. contributors. Agibot world colosseum. https://github.com/OpenDriveLab/ AgiBot-World, 2024
work page 2024
-
[4]
H.-S. Fang, H. Fang, Z. Tang, J. Liu, J. Wang, H. Zhu, and C. Lu. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023
2023
-
[5]
O. X.-E. Team. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), 2024
work page 2024
-
[6]
C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2024. 9
work page 2024
-
[7]
T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023. URL https://arxiv.org/abs/2304.13705
arXiv 2023
-
[8]
M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model. Conference on robot learning (CoRL), 2024
work page 2024
Show all 22 references
-
[9]
S. Lee, Y . Wang, H. Etukuru, H. J. Kim, N. M. M. Shafiullah, and L. Pinto. Behavior generation with latent actions. In Forty-first International Conference on Machine Learning, 2024
2024
-
[10]
Reuss, ¨O
M. Reuss, ¨O. E. Ya˘gmurlu, F. Wenzel, and R. Lioutikov. Multimodal diffusion transformer: Learning versatile behavior from multimodal goals. In Robotics: Science and Systems, 2024
2024
-
[11]
Y . Wang, L. Wang, Y . Du, B. Sundaralingam, X. Yang, Y .-W. Chao, C. Perez-D’Arpino, D. Fox, and J. Shah. Inference-time policy steering through human interactions.arXiv preprint arXiv:2411.16627, 2024
2024 arXiv
-
[12]
Nakamoto, O
M. Nakamoto, O. Mees, A. Kumar, and S. Levine. Steering your generalists: Improving robotic foundation models via value guidance. Conference on Robot Learning (CoRL), 2024
2024
-
[13]
Y . Wu, R. Tian, G. Swamy, and A. Bajcsy. From foresight to forethought: Vlm-in-the-loop policy steering via latent alignment. arXiv preprint arXiv:2502.01828, 2025
2025 arXiv
-
[14]
H. Qi, H. Yin, Y . Du, and H. Yang. Strengthening generative robot policies through predictive world modeling. arXiv preprint arXiv:2502.00622, 2025
2025
-
[15]
A. J. Hancock, A. Z. Ren, and A. Majumdar. Run-time observation interventions make vision- language-action models more visually robust. arXiv preprint arXiv:2410.01971, 2024
2024 arXiv
-
[16]
Z. Lai, S. Purushwalkam, and A. Gupta. The functional correspondence problem. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision , pages 15772–15781, 2021
2021
-
[17]
C. Tang, A. Xiao, Y . Deng, T. Hu, W. Dong, H. Zhang, D. Hsu, and H. Zhang. Functo: Function-centric one-shot imitation learning for tool manipulation. arXiv preprint arXiv:2502.11744, 2025
2025 arXiv
-
[18]
Y . Liu, J. Mao, J. Tenenbaum, T. Lozano-P ´erez, and L. P. Kaelbling. One-shot manipulation strategy learning by making contact analogies. arXiv preprint arXiv:2411.09627, 2024
2024 arXiv
-
[19]
Y . Ju, K. Hu, G. Zhang, G. Zhang, M. Jiang, and H. Xu. Robo-abc: Affordance general- ization beyond categories via semantic correspondence for robot manipulation. In European Conference on Computer Vision, pages 222–239. Springer, 2024
2024
-
[20]
T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y . Chen, F. Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024
2024 arXiv
-
[21]
Oquab, T
M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2023
2023
-
[22]
C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS), 2024. 10 Supplementary This is the supplementary...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.