{"id":"bf4d1940-24c3-4808-8861-25d5c2f2887b","arxiv_id":"2504.17695","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PICO introduces a dataset with dense bijective body-object contact annotations on natural images and a retrieval-plus-optimization method that reconstructs 3D human-object interaction, generalizing to novel object classes.","lead":"This paper builds a dataset that links natural images of people touching objects to dense 3D contact maps on both the body and the object, and adds an optimization method that uses those maps to reconstruct 3D human-object interactions from a single photo. It claims the approach works for object categories, like couches and bananas, that prior methods could not handle.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-the-wild generalization claim is not actually tested for automatic PICO-fit: the perceptual study uses PICO-fit* (GT contact) on images sampled from the same DAMON pool used to build PICO-db, with no stated exclusion from the retrieval index.","rationale":"The paper's central claim is that automatic PICO-fit scales to object categories no existing method handles. For that claim to hold, the retrieval step must work on novel images, and the evaluation must demonstrate this. The reader's DECO-F1 concern addresses the first condition, and the authors' failure analysis (S.3.5) shows it is a real risk. I think the more load-bearing issue is that the paper's main in-the-wild evidence does not test the automatic pipeline at all: it uses PICO-fit* (GT contact) on images sampled from DAMON, the same source as PICO-db. Because PICO-db was built from DAMON images, any of the 75 perceptual-study images may literally be in the retrieval database. If so, 'retrieval' is memorization, not generalization. Even if the images are not in PICO-db, using PICO-fit* means the DECO+GPT-4V contact-estimation and retrieval stages—the components that make the method scale to new classes—are bypassed in the perceptual study. The only quantitative automatic evaluation, InterCap, covers 10 objects and is in-lab; it supports robustness of the fitting stages but not the 'many object categories' claim. This is not an accusation; the dataset, the annotation tool, and the ablations are useful, and the authors are candid about failure modes. But the headline generalization needs a held-out, fully automatic evaluation. The proposed check—rerun the perceptual study with PICO-fit (no star) on images verified to be absent from PICO-db—would directly settle whether the concern lands. If preference rates hold, the central claim stands; if not, the claim should be narrowed to reconstruction from known correspondences, which is still a contribution but not the stated one.","tokens_in":22940,"tokens_out":10465,"duration_ms":102141,"concrete_test":"Rerun the Sec. 5.2 perceptual study with automatic PICO-fit (no GT contact) on 75 images drawn from DAMON test images that are verified (by image ID) to be absent from PICO-db and removed from the retrieval index; compare preference rates against PHOSA*/CONTHO* and against the PICO-fit* numbers in Table 1. If automatic PICO-fit does not stay near the reported preference rates, or if the original 75 images turn out to be in PICO-db, the in-the-wild generalization claim should be restricted to fitting with oracle contact correspondences pending a true held-out evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PICO-db is built from DAMON images (Sec. 3, S.1.5), and the in-the-wild perceptual study (Sec. 5.2) randomly selects 75 images from DAMON. The paper never states that these 75 images are excluded from PICO-db, or that the query image is removed from the OpenShape/PICO-db retrieval index before nearest-neighbor lookup. If a query image is in PICO-db, the retrieval step can return the very annotation created for that image (object mesh, object contact, correspondences), reducing 'retrieval' to lookup. Moreover, Table 1 right column and Sec. 5.2 evaluate PICO-fit* (GT contact), not the automatic PICO-fit; Fig. 7, the evidence for previously untackled classes, is explicitly PICO-fit*. Thus the quantitative in-the-wild support for the headline claim does not exercise the automatic DECO+GPT-4V contact inference and retrieval that the claim depends on. The only quantitative automatic evaluation is InterCap (Sec. 5.1), a 10-object in-lab dataset. The DECO F1=0.35 (S.2) and 105/500 failures due to contact prediction/retrieval (S.3.5) are real, but they are downstream of this evaluation gap: even perfect DECO and retrieval would not make the current experiments establish the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PICO, a framework for reconstructing 3D human-object interactions from a single RGB image. It contributes PICO-db, a dataset of 4,123 in-the-wild images (drawn from DAMON) with dense vertex-level contact annotations on both body and object, plus bijective correspondences, obtained by retrieving object meshes via OpenShape and transferring body contact patches through a crowd-sourced two-click axis-based method. The second contribution is PICO-fit, an optimization pipeline that initializes body pose and shape with OSX, retrieves an object mesh and contact correspondences from PICO-db (using DECO+GPT-4V body contact as the query), and then fits the body and object meshes to image evidence in three stages using contact, mask, penetration, and regularization losses. The paper evaluates PICO-fit on InterCap (quantitative) and on in-the-wild images via a perceptual study, reporting improvements over PHOSA, CONTHO, and HDM.","tokens_in":23269,"tokens_out":5063,"duration_ms":44074,"significance":"If the automatic pipeline works as claimed, PICO would be a meaningful step toward scaling HOI reconstruction to arbitrary object categories without a-priori object models, and PICO-db would be a valuable resource with dense bijective contact correspondences. The paper also contains an unusually honest failure analysis (Sec. S.3.5) and provides quantitative InterCap results showing that the automatic PICO-fit is competitive with baselines that use ground-truth contact. However, the central generalization claim is currently supported mainly by PICO-fit* (with ground-truth contact), and the in-the-wild perceptual study appears to evaluate images that are included in PICO-db, raising a circularity concern. The automatic pipeline's in-the-wild performance is not quantitatively established. These issues are fixable with additional experiments, so the work is potentially significant but needs revision.","major_comments":[{"comment":"The in-the-wild perceptual study in Sec. 5.2 selects 75 images 'randomly' from DAMON, and PICO-db is built from DAMON images (Sec. 3, Sec. S.1.5). The paper never states that these 75 images are excluded from PICO-db or from the retrieval index. Since Sec. 4.1 says that 'for PICO-db images, we use the associated annotations,' PICO-fit* on these images can use the exact object mesh, object contact, and correspondences that were created for that same image; the claimed 'retrieval' is then a lookup rather than a test of generalization. This compromises the central claim that PICO-fit generalizes to in-the-wild images, and it makes the comparison with PHOSA*/CONTHO* (which do not have access to PICO-db annotations) unfair. The authors should either exclude the 75 images from PICO-db before evaluation, or explicitly report how many of them are in PICO-db and rerun the study under an exclusion protocol.","section":"Sec. 5.2, Sec. 3, Sec. S.1.5"},{"comment":"The abstract and Sec. 4 claim that PICO-fit 'works well for many object categories that no existing method can tackle,' but the quantitative evidence for the automatic pipeline is limited to InterCap, a 10-object in-lab dataset (Sec. 5.1). Table 1's right column and Fig. 7 use PICO-fit* with ground-truth contact, not the automatic DECO+GPT-4V contact inference and PICO-db retrieval that the method is supposed to contribute. The only in-the-wild automatic results are qualitative (Fig. 5) and the failure analysis (Sec. S.3.5), which reports 85/500 failures from incorrect DECO contact and 20/500 from invalid retrieved object contact. The authors should provide a quantitative evaluation of the full automatic PICO-fit on a held-out in-the-wild set, for example the DAMON test split with query images explicitly excluded from PICO-db, and report success rates or reconstruction errors rather than only preference rates.","section":"Sec. 5.1, Sec. 5.2, Tab. 1, Fig. 7, Sec. S.3.5"},{"comment":"The perceptual study reports preference rates (e.g., 62.7% vs 37.3% for PHOSA) without error bars, confidence intervals, or significance tests, and the number of participants after filtering (73 out of 100 completions) and the number of images (75) are modest. Since this study is the only in-the-wild quantitative evidence in the paper, the authors should report per-image variance and run a statistical test (e.g., a permutation test or Wilcoxon signed-rank test) to support the claim that PICO-fit* is preferred 'much more' often. This is also needed to assess whether the 62.7% preference over PHOSA is distinguishable from chance after accounting for image difficulty.","section":"Sec. 5.2"}],"minor_comments":[{"comment":"The caption contains a stray LaTeX comment '/searcZoom in to see details' that should be removed.","section":"Fig. 4 caption"},{"comment":"The statement 'we use the same train, validation and test splits as the DAMON dataset' is ambiguous for the 75-image subset used in Sec. 5.2; the authors should clarify whether the perceptual study draws from the test split and whether those images are included in PICO-db.","section":"Sec. S.1.5"},{"comment":"The GPT-4V-based object scale initialization is described only in the supplement; a one-sentence description in the main text would improve readability.","section":"Sec. 4.1"},{"comment":"The term 'chamfer distance' is used inconsistently with capitalization; the authors should standardize to 'Chamfer distance' throughout.","section":"Sec. S.3.2"},{"comment":"The failure analysis would be easier to interpret if the 500-sample study reported the total success count and the category breakdown of failures, rather than only the failure-mode counts.","section":"Sec. S.3.5"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the overlap between the in-the-wild evaluation images and PICO-db, which could turn retrieval into lookup; the authors should be asked to clarify and, if necessary, re-run the perceptual study with explicit exclusion. I would also encourage the editor to ask for a quantitative automatic in-the-wild evaluation, since the current evidence does not directly test the method as described in the abstract. The lack of significance testing in the perceptual study should be addressed. The paper's honest failure analysis and the external InterCap evaluation are strengths, but the generalization claim needs stronger support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: PICO-db is a real contribution — the first natural-image dataset pairing dense body-object contact with bijective correspondences — and the two-click contact transfer is a genuinely clever piece of work. But the paper's headline claim that PICO-fit generalizes in the wild to object classes no existing method can tackle is not actually tested for the automatic pipeline. Read the experiments carefully before citing the claim.\n\nWhat's new and good: The dataset construction is thoughtful. Using ContactEdit-style axes with an automatic PCA axis and only two annotator clicks per patch makes object contact annotation scalable; the proxy mesh for webbed fingers is a sensible fix for non-convex hands. The ablations on InterCap show each stage contributes, and the failure analysis (500 samples categorized, 85/500 due to DECO contact errors, 20/500 invalid object contact retrieval) is more honest than most. PICO-fit without GT contact gets PA-CD h+o 10.33 on InterCap, better than CONTHO*/PHOSA* with GT contact — that's a believable result because it's on external data.\n\nWhere it's soft: The in-the-wild evaluation is the problem. Sec. 5.2 and Fig. 7 use PICO-fit* — GT contact — not automatic PICO-fit. The 75 perceptual-study images are randomly drawn from DAMON, the same pool PICO-db is built from, and the paper never says those images are excluded from the retrieval index. If a query image is in PICO-db, retrieval can return the annotation created for that image, making the 'retrieval' step a lookup. Also, no error bars or significance tests on any of the numbers, and the validation split for the tuned loss weights isn't stated. These are fixable: state exclusion, evaluate the automatic pipeline on held-out DAMON images or another in-the-wild set, and report variance. The DECO F1 of 0.35 is low, but with proper evaluation the automatic pipeline might still do OK; the point is we don't know from the current experiments.\n\nWho it's for: anyone working on 3D HOI, especially dataset builders. The dataset and annotation tool are likely to be used regardless of what happens to PICO-fit.\n\nRecommendation: deserves a serious referee. I'd ask for the overlap/exclusion check and a real test of the automatic pipeline before accepting the generalization claim, but the dataset alone justifies peer review.","headline":"The dataset and the two-click contact transfer are genuinely valuable, but the in-the-wild generalization claim for the automatic pipeline is not actually tested in the paper as written.","tokens_in":23806,"tokens_out":1713,"would_cite":true,"duration_ms":16675,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PICO claims that dense body-object contact correspondences — gathered at scale through a two-click annotation tool — make a single image enough to reconstruct 3D humans and objects in contact, across object classes no existing method…","keywords":["3D human-object interaction","contact correspondences","single-image 3D reconstruction","mesh retrieval","contact-aware fitting","crowdsourced annotation","in-the-wild generalization"],"falsifier":"Take an image set with known ground-truth human-object meshes, run PICO-fit twice per image — once with the original DECO+GPT-4V contact prediction and once with that prediction deliberately corrupted — and measure the endpoint Chamfer distance gap; a large gap would confirm that retrieval error, not the fitting stages, bounds accuracy. A second direct test: images whose object class appears nowhere in PICO-db should visibly degrade, showing that cross-class generalization is bounded by database coverage rather than by the fitting machinery.","tokens_in":22786,"feed_emoji":"🤝","tokens_out":16187,"duration_ms":129629,"temperature":0.7,"pith_summary":"This paper claims that dense, vertex-level contact correspondences between a person and the object they touch can be obtained for natural images at scale, and that these correspondences are enough to reconstruct both the 3D body mesh and the 3D object mesh from a single photo, including object classes that no existing method handles. To support the claim, the authors build PICO-db, a dataset of 4,123 images that pairs existing body-contact annotations with retrieved object meshes and object-side contacts, linked by bijective body-object correspondences established through a two-click, axis-based transfer tool. They then build PICO-fit, a three-stage optimization that uses retrieved contact correspondences to register the object to the body, align both to image masks, and refine the contacting limbs of the human pose. If the claim is right, joint 3D human-object reconstruction no longer needs pre-known object shapes or category-specific contact rules, which is what has confined the field to controlled settings.","feed_headline":"One photo now yields 3D humans and objects in contact","feed_subtitle":"PICO-fit recovers body and object meshes for classes no prior method could handle, using dense contact correspondences.","key_machinery":"The argument is carried by one geometric device, the contact axis. Each body contact patch, a cluster of neighboring vertices, is reduced to an axis — an open curve computed by principal component analysis on the patch vertices — so that the whole patch can be re-parameterized on any other surface by specifying only the axis start and orientation, which takes two clicks. This is what makes PICO-db's bijective body-object correspondences cheap to produce and what lets PICO-fit treat contact as a transferable constraint rather than a per-category rule. Around that device sits the retrieval loop, which uses OpenShape's joint image-and-shape embedding space to fetch a matching mesh from the Objaverse-derived database, and the three-stage optimization whose losses are the contact distance, an SDF-based penetration term, IoU mask alignment, and a pose-proximity regularizer applied only to the kinematic chain leading to the contacting limb.","core_discovery":"On the paper's own terms, the discovery is that a nearest-neighbor lookup in a database of contact examples, followed by render-and-compare optimization, recovers 3D human and object meshes in interaction from a single image, and does so for object categories no existing method can tackle. The pipeline turns body contact into a query: a contact predictor (DECO) augmented by a vision-language model (GPT-4V) decides which body parts touch the object; the highest-IoU match in PICO-db returns the object's 3D mesh, the object-side contact, and the dense body-object point correspondences without further annotation; and a three-stage fit uses those correspondences to place the object against the body, align the object to the image mask, and refine the pose of the contacting limb. The authors report that the full system beats prior optimization- and regression-based methods on both an out-of-domain lab dataset and a perceptual study on in-the-wild images, and that it extends to object classes — couches, bananas, frisbees — that lie outside all existing baselines.","pith_inferences":["If the central claim is right, the real ceiling on generalization is database density rather than model capacity: success on a given object class should track how many similar interactions already sit in PICO-db, a prediction testable by plotting per-class reconstruction accuracy against per-class image counts.","The two-click axis projection is a general shape-to-shape contact-transfer mechanism, not a body-specific one; it could be reused for grasp transfer to robot hands, functional-part labeling, or annotating contacts on deformable objects, none of which the paper pursues.","The paper's failure counts — 85 of 500 failures traced to contact prediction and 12 to wrong object retrieval — suggest that improving the contact estimator would buy more accuracy than further tuning the fitting stages; the feed-forward contact regressor the paper proposes as future work is the direct test of that hypothesis.","Because object retrieval runs in a joint image-text-shape embedding space, the same pipeline could be driven by a text description of the object instead of the image crop with no architectural change, opening the door to text-conditioned human-object interaction reconstruction."],"forward_implications":["3D human-object interaction recovery no longer needs a known object shape or category: any object that can be retrieved from a growing mesh database becomes a candidate for joint 3D reconstruction from a single photo.","The two-click, axis-based contact transfer makes dense body-object contact annotation cheap enough to scale, so contact supervision can be extended to new object classes without redesigning the annotation tool.","Contact correspondences act as the geometric prior that resolves depth and occlusion ambiguity during fitting; the supplement's ablation shows object Chamfer distance worsens from 13.3 cm to 34.4 cm when the contact-registration stage is removed.","PICO-db's paired contact data makes a feed-forward contact and correspondence regressor feasible, which the paper names as the path to replacing the nearest-neighbor lookup and scaling further.","Because object scale is estimated per instance from the image rather than per class, the method adapts to atypical object sizes instead of assuming a category-average scale."],"supporting_citations":[{"why":"Supplies the body-contact annotations that PICO-db repurposes onto objects, and the DECO regressor that infers body contact on unlabeled images.","marker":"[77]"},{"why":"OpenShape's joint image-and-shape embedding space is what makes automatic 3D object mesh retrieval from an image crop possible.","marker":"[53]"},{"why":"Objaverse-LVIS is the mesh database that PICO retrieves objects from, so it determines the object categories and shapes the method can cover.","marker":"[15]"},{"why":"ContactEdit's contact-axis parameterization is the mechanism that transfers body patches onto objects with two clicks and yields bijective correspondences.","marker":"[44]"},{"why":"OSX provides the initial SMPL-X body pose and shape that PICO-fit refines in its third stage.","marker":"[50]"},{"why":"Grounded SAM supplies the object and human masks used by the render-and-compare alignment stages.","marker":"[66]"},{"why":"PHOSA is the optimization-based in-the-wild baseline whose hand-crafted category contact constraints PICO-fit replaces with image-grounded retrieved contacts.","marker":"[97]"},{"why":"InterCap is the out-of-domain lab dataset on which the quantitative comparisons and stage ablations are run.","marker":"[32]"},{"why":"Provides the retrieval-over-regression design choice that the contact lookup from PICO-db is built on.","marker":"[83]"}],"fun_headline_variants":["Single image yields 3D human-object contact","New method recovers 3D humans and objects from one photo","Contact-based fitting recovers 3D human-object interactions","Single photo recovers 3D body and object in contact","PICO-fit: 3D humans and objects in contact from one image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the body-contact prediction from DECO plus GPT-4V — whose F1 score the paper reports as only 0.35 — is accurate enough that the nearest-neighbor lookup in PICO-db returns the right object mesh, the right object-side contact, and the right correspondences; when the contact prediction is wrong the retrieved object and contact are wrong and the three-stage fit has no way to recover, a failure the paper's own analysis attributes to incorrect contact prediction in 85 of 500 sampled cases.","fun_headline_variants_meta":{"raw":{"variants":["Single image yields 3D human-object contact","New method recovers 3D humans and objects from one photo","Contact-based fitting recovers 3D human-object interactions","Single photo recovers 3D body and object in contact","PICO-fit: 3D humans and objects in contact from one image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3813,"prompt_tokens":1116,"completion_tokens":2697,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":2611}},"tokens_in":732,"tokens_out":2697,"duration_ms":17631,"temperature":1.0,"reasoning_tokens":2611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:33:25.247670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an image set with known ground-truth human-object meshes, run PICO-fit twice per image — once with the original DECO+GPT-4V contact prediction and once with that prediction deliberately corrupted — and measure the endpoint Chamfer distance gap; a large gap would confirm that retrieval error, not the fitting stages, bounds accuracy. A second direct test: images whose object class appears nowhere in PICO-db should visibly degrade, showing that cross-class generalization is bounded by database coverage rather than by the fitting machinery.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the body-contact annotations that PICO-db repurposes onto objects, and the DECO regressor that infers body contact on unlabeled images."},{"cited_title":"OpenShape: Scaling up 3D shape representation towards open-world understanding","cited_arxiv_id":null,"evidence_quote":"OpenShape's joint image-and-shape embedding space is what makes automatic 3D object mesh retrieval from an image crop possible."},{"cited_title":"Lakshmipathy, Nicole Feng, Yu Xi Lee, Moshe Mahler, and Nancy S","cited_arxiv_id":null,"evidence_quote":"ContactEdit's contact-axis parameterization is the mechanism that transfers body patches onto objects with two clicks and yields bijective correspondences."},{"cited_title":"One-stage 3D whole-body mesh recovery with component aware transformer","cited_arxiv_id":null,"evidence_quote":"OSX provides the initial SMPL-X body pose and shape that PICO-fit refines in its third stage."},{"cited_title":"Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa","cited_arxiv_id":null,"evidence_quote":"PHOSA is the optimization-based in-the-wild baseline whose hand-crafted category contact constraints PICO-fit replaces with image-grounded retrieved contacts."},{"cited_title":"Black, and Dim- itrios Tzionas","cited_arxiv_id":null,"evidence_quote":"InterCap is the out-of-domain lab dataset on which the quantitative comparisons and stage ablations are run."},{"cited_title":"PlaNet - Photo geolocation with convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the retrieval-over-regression design choice that the contact lookup from PICO-db is built on."}],"review_version":1}