{"id":"0cb4eece-876b-47f9-91da-68a23ff46ea1","arxiv_id":"2608.06827","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Coupling a simulator-rollout robot proxy with capture-anchored geometry to select and generate ego-view pseudo-observations improves sparse-capture scene rendering and real-G1 sitting transfer.","lead":"R2S-EGO refines a simulation scene from only six real camera images by generating extra training views only where the robot's own behavior would look, guided by geometry anchored to the captured scene. The refined scene renders robot ego views far better than existing baselines (19.1 dB versus 14.2 dB PSNR) and lifts real-humanoid sitting success from 10.0% to 82.5%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VGGT pose calibration for pseudo-observations is never validated; if biased for video-diffusion frames, the registered refinement is misdirected and the measured gains depend on an unmeasured calibration error.","rationale":"The reader's weakest assumption matches the most load-bearing unresolved point: the pseudo-observation registration step is essential to the central claim that the gains come from targeted, capture-anchored refinement, and it is the only stage of the benchmark that uses unvalidated calibration. The concern is concrete and testable because the Replica benchmark provides exact ground-truth poses, so the authors could report VGGT pose error and an ablation with GT-registered pseudo-observations. The paper otherwise has strong independent support: evaluation views were fixed in advance, budgets were matched across conditions, the hardware comparison shares the collision mesh and chair assets, and five paired policy seeds show a consistent ordering. The hardware success on real G1 is substantial evidence that the full pipeline works, but it does not by itself isolate the VGGT calibration step. I therefore agree with the reader's conditional verdict: the central claim is plausible and well supported in most components, but this specific load-bearing assumption should be verified before full acceptance. No adjustment to the verdict is needed because the concern is already captured by the conditional recommendation and can be resolved by a focused experiment.","tokens_in":15935,"tokens_out":8823,"duration_ms":92845,"concrete_test":"Use the Replica benchmark, where GT poses are known for all six inputs and all 48 evaluation views, to settle whether VGGT calibration is accurate enough to support the targeting claim. First, for every generated pseudo-observation across all scenes and rounds, render the GT terminal frame at the known query pose v and compute its PSNR against the generated terminal frame; independently compute the VGGT-estimated pose π̂_v and report median and 90th-percentile translation/rotation error relative to v, plus error versus distance from the reference capture. Second, re-run the full pipeline with pseudo-observations registered at the GT query poses v instead of at π̂_v, holding all other settings fixed, and compare the 48-view PSNR to 19.062 dB.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that behavior-scoped selection and capture-anchored assimilation, not generation alone, produce the reported gains. The mechanism routes every generated frame through VGGT to obtain its training pose (Eq. 3, Sec. III-C), and the query pose is deliberately discarded: 'VGGT then estimates that generated frame's intrinsics and pose relative to the fixed real reference before it enters the visual-asset update; the query pose itself is not used as its training pose.' In the Replica benchmark (Sec. IV.A), input views and evaluation cameras use ground-truth calibration, while only generated observations use VGGT calibration. The paper therefore never measures whether the estimated pose π̂_v equals the selected target pose v, and it reports no pose-error statistics for VGGT on video-diffusion output frames. If VGGT relative poses are systematically biased — for example, drifting with distance from the reference capture or reacting to the diffusion model's low-level statistics — pseudo-observations are inserted at locations other than the deficit that was scored. The 3DGS update would then bake misregistered appearance into the asset, and the 19.062 dB result would depend on an unvalidated calibration step rather than on deficit-targeted refinement. The ablation 'w/o robot-proxy allocation' (16.182 dB) would also no longer isolate targeting if VGGT pose noise distributes frames regardless of the intended query. The paper's own limitation statement (Sec. V) acknowledges that repeated pseudo-observation updates may propagate errors in generated appearance or prior-completed structure, which is exactly the failure mode at issue. This is not an internal inconsistency, but it is an unmeasured load-bearing assumption that a targeted-refinement account must secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents R2S-EGO, a method for refining a sparse-capture real-to-sim scene along robot ego views without collecting additional real images. A simulator-derived robot proxy enumerates behavior-scoped executable camera poses and scores them by current visual-asset deficit combined with geometry-proxy availability; a capture-anchored geometry proxy, built from triangulated fragments, SAM 3D priors, and NKSR, supplies structural conditioning for a video-diffusion generator. Generated terminal frames are registered by VGGT, added to a 3D Gaussian splatting asset as lower-weight pseudo-observations, and the fused geometry is reinstalled as the collision surface between refinement rounds. The authors report 19.062 dB PSNR on 48 frozen G1 ego views across three Replica scenes versus 14.226 dB for the strongest R2S baseline, and 82.5% versus 10.0% real-G1 sitting success across five paired policy-training seeds, with a matched-budget evaluation protocol and component ablations.","tokens_in":16113,"tokens_out":5941,"duration_ms":63906,"significance":"If the reported results hold, R2S-EGO is a meaningful step toward making sparse-capture real-to-sim useful for robot learning: it shows that a small number of real images plus targeted, behavior-scoped pseudo-observation refinement can improve frozen robot-view rendering and downstream sim-to-real policy transfer. The paper's experimental hygiene is a real strength: the six inputs, 48 targets, and qualitative views were fixed before running comparisons; development and evaluation views are disjoint; the five pipeline seeds are pre-specified and paired; iterative 3DGS conditions share a 15k-step budget; the hardware comparison shares the same collision mesh and simulator stack and varies only the visual asset; and the authors transparently label descriptive statistics and scope limits. The main risk is that the pseudo-observation registration step, which is load-bearing for the targeting claim, is never validated on the actual video-diffusion output frames.","major_comments":[{"comment":"The training pose of every generated pseudo-observation is the VGGT estimate π̂_v, and the query pose is deliberately discarded ('the query pose itself is not used as its training pose'). In the Replica benchmark, the real inputs and evaluation cameras use ground-truth calibration while only the generated observations use VGGT (Sec. IV.A). The paper therefore never measures how close π̂_v is to the intended target v, nor whether VGGT's error is biased for video-diffusion output frames. If VGGT drifts with distance from the reference capture or reacts to diffusion-specific image statistics, pseudo-observations are inserted at positions other than the scored deficit; the 19.062 dB result and the 2.880 dB allocation gap (Table III) would then depend on an unmeasured calibration property rather than on deficit-targeted refinement. Please add a Replica-side calibration experiment: compare VGGT-estimated poses of the retained generated frames with the known target poses (rotation and translation errors, ideally binned by distance from the reference), and/or rerun the full pipeline using the query pose as the training pose as a sensitivity check.","section":"Sec. III-C, Eq. (3); Sec. IV.A"},{"comment":"The 'w/o iterative scene refinement' row changes multiple properties at once: it selects all 36 targets once from the initial proxy, performs a single 15k-step update, and has no intermediate NKSR/collision refresh, controller rerollout, or reranking. The 1.554 dB gap relative to full R2S-EGO therefore conflates the benefit of repeated budgeted reallocation with the benefit of refreshed geometry/collision and rerollouts. Because this row is the main evidence for the refinement-loop contribution, please add a variant that holds one of these dimensions fixed (for example, one-shot target selection but with proxy refresh and rerollout, or multi-round updates without reallocation) so the source of the improvement is identifiable.","section":"Sec. IV.C, Table III"}],"minor_comments":[{"comment":"The paired seed counts (0,0,1,2,1) versus (7,7,7,6,6) are visually decisive, but since the authors explicitly describe the pooled 4/40 and 33/40 as descriptive, a paired nonparametric test or a bootstrap over seeds would make the cross-seed claim easier to quantify.","section":"Sec. IV.D, Table IV"},{"comment":"The protocol states that 'the operator is aware of the condition.' Given that success is defined by no intervention, no load-bearing tether, and 3 s stable support, the bias risk is limited, but a blinded or automated success assessment would further strengthen the hardware result.","section":"Sec. IV.A, hardware protocol"},{"comment":"The word 'frozen' is used both for pre-registered evaluation views and for fixed model checkpoints; distinguishing 'pre-registered' from 'frozen parameters' in the text would remove a source of ambiguity.","section":"Throughout"},{"comment":"The figure caption and the text already clarify that error bars are SEM over 16 target views and not seed uncertainty; consider stating in the main text why the error bars are not used to compare the R2S-EGO six-view point with baseline curves.","section":"Sec. IV.B, Fig. 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this about R2S-EGO: it's a well-scoped, honestly reported system paper that reports a large real-to-sim improvement, and the claim is mostly believable because the authors went out of their way to control the evaluation. The new content is the coupling of two existing ideas—simulator rollouts that define which ego views are behavior-relevant, and capture-anchored geometry that conditions synthetic view generation—into a fixed-budget refinement loop for 3DGS. The paper doesn't claim novelty for the components, and the related work is honest about that.\n\nWhat it does well: the fixed-before protocol (six inputs, 48 targets, qualitative views frozen before running), matched reconstruction/generation budgets, five paired seeds, a disjoint development split, and a hardware comparison that shares the collision mesh and chair assets. The ablations are sensible: dropping robot-proxy allocation costs 2.88 dB, one-shot refinement costs 1.55 dB, and removing SAM 3D grounding costs 2.68 dB. The paper repeatedly states what it doesn't claim, including that the hardware counts are descriptive. That discipline makes the 19.062 dB vs 14.226 dB and 33/40 vs 4/40 numbers more credible than the usual pipeline paper.\n\nThe soft spot that matters most is the VGGT calibration of generated frames. Every generated pseudo-observation enters the 3DGS update at a pose estimated by VGGT; the query pose is deliberately not used. In the Replica benchmark, real inputs and evaluation cameras use ground-truth calibration, so the paper never measures whether these estimated poses are accurate for video-diffusion output frames. If VGGT drifts with distance from the reference capture or reacts to generated-image statistics, the pseudo-observations land where they weren't intended, and the PSNR gain could be partly an artifact of pose noise rather than deficit-targeted refinement. The paper's own limitation section (Sec. V) mentions error propagation in generated appearance and prior-completed structure but never mentions calibration. That's a real gap, and it's the first thing I'd ask an author to address—either report pose error against ground truth in Replica or run an ablation that uses the known query poses as training poses.\n\nMinor concerns: baselines like Vanilla 3DGS and GaussGym are single fixed runs with no variance, which makes the visual gap look larger than it may be; the hardware evaluation is small and unblinded, though the paired-seed design helps; no artifacts are released, so independent verification is harder; and the visual benchmark uses Replica renders rather than sensor images, which the authors disclose. None of these sink the paper. The central claim holds up as plausible, conditional on the calibration check.\n\nWho's this for: anyone working on real-to-sim, sparse-view reconstruction, or sim-to-real transfer with humanoids. It deserves a real peer review. I'd send it out, and I'd tell the reviewers to focus on the VGGT calibration question and the single-run baselines. If the authors can close the calibration gap, this is a strong result.","headline":"Honest, well-controlled systems paper with a big R2S gain; the main open question is whether VGGT really registers the generated frames at the poses the method claims.","tokens_in":16865,"tokens_out":2405,"would_cite":false,"duration_ms":23352,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"R2S-EGO claims that a six-photo real-to-sim scene refined with 36 behavior-targeted, capture-anchored synthetic ego views outperforms baselines using up to 45 real photos, raising held-out robot-view PSNR from 14.226 to 19.062 dB and…","keywords":["real-to-sim","sparse-capture reconstruction","ego-view synthesis","3D Gaussian splatting","pseudo-observation refinement","behavior-scoped view selection","sim-to-real policy transfer","geometry proxy"],"falsifier":"Render the same 36 generated terminal frames into two refined scenes, one registered with VGGT poses and one inserted at the known query poses; if the VGGT version does not match or beat the known-pose version on the 48 frozen ego views, the reported gains depend on calibration accuracy rather than on the dual-proxy selection. A cheaper check is to measure VGGT's relative pose error on generated frames against the query poses as a function of distance from the reference capture; if the error grows past a few pixels at the farthest targets, mis-registered pseudo-observations are being baked into the very views the method claims to improve.","tokens_in":15600,"feed_emoji":"🤖","tokens_out":8329,"duration_ms":75295,"temperature":0.7,"pith_summary":"This paper is trying to show that a sparse real-to-sim scene can be made usable for robot training without dense multi-view capture, by generating a small number of synthetic camera views that are chosen and placed according to the robot's own behavior. The proposed method, R2S-EGO, takes six registered photos of a room, uses a simulator rollout to find the ego-camera poses a robot would actually occupy during a target behavior, renders a capture-anchored geometry proxy at those poses, and asks a video-diffusion generator to produce the missing views. Those generated views are then registered and assimilated into a 3D Gaussian splatting asset as low-weight pseudo-observations, while the real photos stay as anchors. On 48 frozen Unitree G1 ego views across three Replica scenes, the refined scene renders at 19.062 dB PSNR versus 14.226 dB for the strongest reported baseline, and policies trained in it succeed at 82.5% on a real G1 sitting task versus 10.0% for the GaussGym baseline. If these results hold, sparse capture combined with targeted synthesis could replace much denser real-image capture for building robot-ready simulation scenes.","feed_headline":"Six photos plus 36 synthetic views beat 45 real photos for robot sims","feed_subtitle":"R2S-EGO lifts held-out G1 ego-view PSNR to 19.062 dB and real sitting success to 82.5%.","key_machinery":"The load-bearing mechanism is the dual-proxy coupling and its selection score $s_l(t)=u_l(t)\\,g_l(t)$, where $u_l(t)$ is the fraction of pixels the current visual asset cannot render at candidate ego pose $t$ and $g_l(t)$ is the fraction the geometry proxy can render there; temporal non-maximum suppression and a TopK budget turn this score into a fixed set of target views each round. The robot proxy is a simulator-derived stream of executable ego-camera poses obtained by rerunning the fixed behavior controller with resampled initial conditions, which guarantees the query pose belongs to the behavior scope. The geometry proxy is composed as $M^l_{\\mathrm{scaf}} = \\mathrm{Compose}(\\{T^*_m(P^m_{\\mathrm{prior}})\\}_m, \\mathrm{NKSR}(P^{\\neg\\mathrm{prior}}_{\\mathrm{obs}} \\cup P(A^l)^{\\neg\\mathrm{prior}}))$, i.e. SAM 3D prior surfaces aligned to real captures by a similarity transform and fused with a neural-kernel surface reconstruction of the observed and current back-end points; it supplies the structural condition rendered along the generation path and the refreshed collision surface. The assimilation rule keeps real captures at unit weight and generated pseudo-observations at weight $w_{\\mathrm{syn}}=0.5$, so the real images remain coordinate and appearance anchors while the generated frames extend renderable support at behavior-scoped views.","core_discovery":"On its own terms, R2S-EGO's finding is that the usefulness of generated observations in real-to-sim refinement is governed less by the generator than by query selection and grounding. The paper defines a robot proxy: by rerunning a fixed behavior controller in the existing simulator, it obtains the set of ego-camera poses the embodied robot can actually reach while performing the declared behavior. A geometry proxy, formed by aligning SAM 3D shape priors to triangulated capture fragments and completing the remainder with NKSR, supplies structure at those poses. The method scores candidate poses by the product of current visual-asset deficit and geometry availability, keeps temporal local maxima up to a fixed budget, generates ego-view videos with ViewCrafter conditioned on a real reference, the relative camera motion, and the rendered geometry, then registers the terminal frames with VGGT and adds them to the 3DGS asset at half weight. The central claim is that this targeted, capture-anchored assimilation, not the generation itself, produces the reported gains: without robot-proxy allocation the PSNR drops by 2.880 dB, and one-shot refinement without round-by-round reassessment drops by 1.554 dB.","pith_inferences":["A direct stress test of the registration assumption would compare the VGGT-calibrated pseudo-observations against the same frames inserted at the known query poses; if the gap to full R2S-EGO narrows or reverses, pose calibration would be the real bottleneck and a stronger pose estimator would immediately improve the results.","The method's components are modular; any generator that accepts a reference image, relative camera motion, and a structural condition could replace ViewCrafter, and any shape prior could replace SAM 3D, so the central claim about targeted assimilation is separable from the specific public checkpoints used.","The hardware evaluation covers only sitting; if the same behavior-scoped allocation logic transfers to walking or manipulation, the capture-count savings would generalize, but that requires redefining the behavior scope and re-running the protocol, which the paper explicitly leaves to future work.","Since the evaluation uses Replica ground-truth intrinsics and extrinsics for real captures and evaluation views, the strongest numbers may not hold in physical rooms where VGGT must also estimate the real-capture calibration; the paper notes this boundary but does not quantify it."],"forward_implications":["At the measured office-2 capture-count curve, no baseline setting up to 45 real input views reaches the six-view R2S-EGO PSNR, so the method's claim is not merely about matching image density but about replacing it.","The ablation pattern implies the gain decomposes: robot-proxy allocation contributes the largest single drop (2.880 dB), SAM 3D grounding contributes 2.678 dB, and iterative round-by-round reassessment contributes 1.554 dB, all under matched generation and optimization budgets.","The refined scene keeps the simulator's robot dynamics and control stack fixed, so the method is an asset-construction layer that plugs into an existing simulator rather than a new simulator or policy learner.","Policies trained on the refined asset succeed on the real G1 in 33/40 pooled trials versus 4/40 for GaussGym, with the same direction across all five paired seeds; the paper reports this as evidence that the refined ego-view asset, not in-simulation performance alone, tracks transfer."],"supporting_citations":[{"why":"ViewCrafter is the camera-controlled video-diffusion generator that produces the ego-view clips whose terminal frames become the pseudo-observations.","marker":"[10]"},{"why":"SAM 3D supplies the shape-prior object surfaces that the geometry proxy aligns to captured fragments, extending structure into sparsely observed regions.","marker":"[27]"},{"why":"NKSR reconstructs the non-prior regions of the geometry proxy and thereby provides the scene collision surface refreshed each round.","marker":"[11]"},{"why":"VGGT estimates each generated frame's intrinsics and pose relative to the fixed real reference, the registration step that turns generated images into calibrated pseudo-observations.","marker":"[45]"},{"why":"GaussGym is the main real-to-sim baseline and the policy-training comparison condition; its scene and 3DGS renderer with collision physics define the protocol R2S-EGO is measured against.","marker":"[2]"},{"why":"GRAIL supplies the egocentric sitting teacher-student distillation and SONIC control stack used for the downstream real-G1 policy comparison.","marker":"[20]"},{"why":"3D Gaussian splatting is the visual-asset representation that the real captures and pseudo-observations jointly refine.","marker":"[5]"},{"why":"DUSt3R-family feed-forward reconstruction provides the initial sparse asset A0 that R2S-EGO refines and the baseline row without generated ego video reduces to.","marker":"[23]"},{"why":"GenFusion is the component-level baseline that closes the loop between video generation and reconstruction, isolating what behavior-scoped allocation adds over generic generation-reconstruction feedback.","marker":"[35]"}],"fun_headline_variants":["Six real photos plus targeted synthetic views beat dense capture","Robotic query selection enables sparse-capture real-to-sim","Real-to-sim boost comes from view selection, not generation","R2S-EGO: Sparse inputs, smarter queries, better sims","Choose camera queries wisely: 6 views beat 45 for robot sims"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gain rests on the assumption that the pose-estimation software places each generated synthetic camera view accurately enough that adding those images at the estimated positions improves exactly the robot views the method targets, even though the method refuses to use the known camera positions and never measures the software's error.","fun_headline_variants_meta":{"raw":{"variants":["Six real photos plus targeted synthetic views beat dense capture","Robotic query selection enables sparse-capture real-to-sim","Real-to-sim boost comes from view selection, not generation","R2S-EGO: Sparse inputs, smarter queries, better sims","Choose camera queries wisely: 6 views beat 45 for robot sims"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1764,"prompt_tokens":1082,"completion_tokens":682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":590}},"tokens_in":698,"tokens_out":682,"duration_ms":7714,"temperature":1.0,"reasoning_tokens":590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:07:44.817804+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render the same 36 generated terminal frames into two refined scenes, one registered with VGGT poses and one inserted at the known query poses; if the VGGT version does not match or beat the known-pose version on the 48 frozen ego views, the reported gains depend on calibration accuracy rather than on the dual-proxy selection. A cheaper check is to measure VGGT's relative pose error on generated frames against the query poses as a function of distance from the reference capture; if the error grows past a few pixels at the farthest targets, mis-registered pseudo-observations are being baked into the very views the method claims to improve.","supporting_citations":[{"cited_title":"ViewCrafter: Taming video diffusion models for high-fidelity novel view synthesis,","cited_arxiv_id":null,"evidence_quote":"ViewCrafter is the camera-controlled video-diffusion generator that produces the ego-view clips whose terminal frames become the pseudo-observations."},{"cited_title":"SAM 3D: 3Dfy anything in images,","cited_arxiv_id":null,"evidence_quote":"SAM 3D supplies the shape-prior object surfaces that the geometry proxy aligns to captured fragments, extending structure into sparsely observed regions."},{"cited_title":"Neural kernel surface reconstruction,","cited_arxiv_id":null,"evidence_quote":"NKSR reconstructs the non-prior regions of the geometry proxy and thereby provides the scene collision surface refreshed each round."},{"cited_title":"VGGT: Visual geometry grounded transformer,","cited_arxiv_id":null,"evidence_quote":"VGGT estimates each generated frame's intrinsics and pose relative to the fixed real reference, the registration step that turns generated images into calibrated pseudo-observations."},{"cited_title":"3D Gaussian splatting for real-time radiance field rendering,","cited_arxiv_id":null,"evidence_quote":"3D Gaussian splatting is the visual-asset representation that the real captures and pseudo-observations jointly refine."},{"cited_title":"DUSt3R: Geometric 3D vision made easy,","cited_arxiv_id":null,"evidence_quote":"DUSt3R-family feed-forward reconstruction provides the initial sparse asset A0 that R2S-EGO refines and the baseline row without generated ego video reduces to."},{"cited_title":"GenFusion: Closing the loop between reconstruction and generation via videos,","cited_arxiv_id":null,"evidence_quote":"GenFusion is the component-level baseline that closes the loop between video generation and reconstruction, isolating what behavior-scoped allocation adds over generic generation-reconstruction feedback."}],"review_version":1}