{"id":"1b490748-07d1-441c-824c-d2b4e25d120b","arxiv_id":"1908.01887","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DoorGym is a domain-randomized door-opening simulator with PPO and SAC baselines, reaching up to 95% simulation success and 59% success in a real-world transfer test.","lead":"DoorGym is an open-source simulation environment for training robots to open many kinds of doors under randomized conditions, with baseline PPO and SAC policies. A trained policy opened a real door in 59% of trials, which makes the environment a useful testbed for sim-to-real manipulation research.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5's real-world transfer uses a Baxter robot, but DoorGym's described environment and all Section 4 experiments use BLUE arms; no Baxter model or embodiment mapping is documented, so the transfer claim is not reproducible.","rationale":"The reader's weakest assumption correctly identifies the missing robot embodiment in Section 5. I examined Section 2.2, Section 4.4, Section 5, and Tables 8/9. The paper's environment is BLUE-only in the main text, while the real-world experiment is on Baxter. This is not merely a missing detail: it breaks the continuity of the observation and action spaces, so the reported 0.59 success rate cannot be reproduced or even interpreted without additional information. The concern is not about disagreement with domain randomization or with the benchmark's usefulness; it is about the internal consistency of the transfer claim. A conditional verdict remains appropriate because the repository may contain a Baxter model and training config that the paper omitted; if so, the concern is addressable. If the concrete test finds no such model, the real-world transfer claim should be removed or the verdict should move toward rejection. I therefore keep the reader's verdict unchanged but sharpen the reason: the transfer experiment's connection to DoorGym is unsupported as written. Other issues, such as missing error bars and table inconsistencies, are real but secondary; the embodiment gap is the load-bearing one.","tokens_in":13135,"tokens_out":3554,"duration_ms":38897,"concrete_test":"Check the DoorGym repository for a Baxter robot definition (e.g., XML/MJCF or URDF under robots/ or environments/) and for the exact PPO checkpoint and configuration used in Section 5. Confirm whether the checkpoint was trained on Baxter or BLUE and whether the action/observation dimensions match the deployed Baxter arm. If no Baxter training config or checkpoint metadata exists, rerun the sim-to-real protocol with a simulator-matched Baxter model and report both simulated and real success rates. This settles whether the 59% transfer result belongs to DoorGym or to an undocumented separate pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DoorGym's central practical claim is zero-shot sim-to-real transfer. Section 5 reports a 59% real-world success rate on a Baxter robot, stating only that the policy was trained purely in simulation using PPO with the same hyperparameters as in Section 4.4. However, Section 2.2 describes only the Berkeley BLUE Robot arm, and all experiments in Section 4 use BLUE platforms (floating hook, gripper, mobile, etc.); no Baxter model appears in Tables 8/9 or in the environment description. Since observations are per-joint positions/velocities and actions are per-DoF forces/torques (Section 3), a BLUE-trained policy has different observation and action dimensionality and different kinematics than a Baxter right arm. Direct deployment would require either a Baxter model in the simulator or an explicit action/observation mapping; neither is described. If the policy was actually trained on an undocumented Baxter model, the real-world result is not reproducible from the paper and the DoorGym-based transfer narrative is unsupported. If it was trained on BLUE, the experiment as written is physically under-specified. This gap is load-bearing for the paper's strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DoorGym, an open-source MuJoCo/Gym-based door-opening environment designed for domain randomization (DR), and provides PPO and SAC baselines. The environment includes multiple doorknob types (pull, lever, round), randomized door and robot parameters, and a vision network for estimating doorknob position. The authors report success rates on three tasks (pull knob with floating hook, lever with floating hook, pull knob with gripper), an ablation showing that DR improves policy and vision-network transfer, and a zero-shot sim-to-real transfer experiment on a Baxter robot that achieves a 59% real-world success rate. The central claims are that DoorGym provides a usable DR benchmark and that a policy trained purely in simulation can transfer to a real door.","tokens_in":13403,"tokens_out":4429,"duration_ms":45354,"significance":"If the claims hold, DoorGym fills a useful niche as an open-source, domain-randomized benchmark for a practical manipulation task, complementing existing sim-to-real environments. The paper provides several concrete strengths: the environment code is released, the randomization ranges are tabulated (Table 5), the baseline PPO/SAC implementations are described, the DR ablation supports the qualitative conclusion that DR improves transfer, and the real-world vision-network and policy experiments attempt to validate sim-to-real transfer. The PPO results (up to 95% success on a pull-knob task with ground-truth position) indicate the environment is solvable by a standard algorithm. However, the real-world transfer section does not specify which simulated robot the deployed policy was trained on, which is load-bearing for the paper's strongest claim. In addition, all quantitative results are single-seed point estimates, so the statistical reliability of the reported differences is unknown.","major_comments":[{"comment":"The real-world transfer experiment uses a Baxter robot, but Section 2.2 describes only the Berkeley BLUE Robot arm, and all Section 4 experiments and Tables 8/9 use BLUE arm variants (hook, gripper, floating, mobile). No Baxter model in DoorGym is described, and no action/observation mapping from BLUE to Baxter is given. Because observations are per-joint positions/velocities and actions are per-DoF forces/torques (Section 3), a BLUE-trained policy has different dimensionality and kinematics than a Baxter right arm, so direct deployment requires either a documented Baxter simulation model or an explicit retargeting scheme. Neither is present. Moreover, the simulator success rate of 0.70 in Table 3 does not match any condition in Table 1 (PPO with vision is 0.48–0.57 on the three tasks, and PPO with ground-truth position is 0.71–0.95), so even the simulator baseline for the deployed policy is not identifiable. This gap makes the real-world transfer claim unreproducible and undermines the paper's central contribution of zero-shot sim-to-real transfer.","section":"Section 5, Table 3"},{"comment":"All reported success rates and average opening times are point estimates from a single training run, with no variance, confidence intervals, or number of random seeds. For a benchmark intended to support comparisons (PPO vs. SAC, DR vs. no-DR, simulator vs. real), the absence of uncertainty makes it impossible to assess whether differences such as 0.95 vs. 0.68 (Task 1 vs. Task 2 PPO) or 0.71 vs. 0.59 (Table 3) are meaningful or within run-to-run noise. This limits the strength of the quantitative claims and is particularly important for the DR ablation, where the conclusion depends on the gap between single-environment and randomized training.","section":"Tables 1, 2, 3, 8, 9"},{"comment":"The DR ablation compares a policy trained on 'env1' with a policy trained on randomized environments, but env1's parameters are not specified, the test conditions are only 'env1' and 'randomized env', and no statistical uncertainty is reported. The paper concludes that 'DR is required' for both policy and vision-network robustness, yet the evidence is limited to two conditions per comparison and single seeds. While the qualitative direction (single-env policies generalize poorly) is plausible and consistent with prior work, the quantitative support is weaker than the text implies. Please report the randomization parameters of env1, multiple seeds, and standard deviations, or soften the claim to 'DR improves transfer in these experiments.'","section":"Section 4.5, Table 2"}],"minor_comments":[{"comment":"The text states that 'All experiments are evaluated using the unshaped reward as defined in section 4.1,' but Eq. (1) is a shaped reward with weights a0–a5; 'unshaped' appears to be a typo for 'shaped' or the sentence should be rephrased.","section":"Section 4.4, paragraph 1"},{"comment":"The SAC row for 'Ground Truth Position + N(0, σ)' is filled with dashes, and the text does not explain whether this condition was not run or the results were omitted; please clarify.","section":"Table 1"},{"comment":"The text says '6 different arm/gripper combinations' but Figure 3 shows four arm types (8DoF BLUE with grippers, 7DoF BLUE with hook, 7DoF floating gripper, 6DoF floating hook), with mobile variants mentioned in the text; the count and the distinction between fixed and mobile platforms should be made explicit.","section":"Section 2.2, Figure 3"},{"comment":"The sentence 'the arm starts from the hanging position' is ambiguous; specify whether this is a known joint configuration and whether it matches the initialization distribution used in simulation.","section":"Section 5, paragraph 1"},{"comment":"Reference [2] is incomplete: the venue is listed as 'In None, 2015,' and the title appears truncated; please provide the full citation.","section":"References"},{"comment":"The caption contains a typo: 'ramdomizable' should be 'randomizable'.","section":"Figure 1 caption"},{"comment":"The real-world success criterion is the same as simulation (door opening >0.2 rad within 20 s), but the control-loop frequency and the number of real-world evaluations are reported; please also state whether the 100 evaluations used the same door and whether the robot was reset to the same initial configuration each time.","section":"Section 5, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The missing Baxter documentation is the central issue. If the authors can add a Baxter model to DoorGym, describe the training setup, and show that the simulator result in Table 3 corresponds to a clearly defined condition, the real-world transfer claim would become credible. As written, the claim is not reproducible. The single-seed issue is pervasive but fixable with additional runs or explicit statements of variance. I would not reject the paper, since the environment and ablation are useful contributions, but the real-world transfer section needs substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DoorGym is a genuinely useful resource. The open-source benchmark combines MuJoCo physics with Unity rendering, multiple knob types, several robot arm variants, and a domain randomizer that randomizes both visual and physical parameters. The baseline PPO and SAC results across the full combination table are a real service to the community, even if many entries are 0% — that is still information. The DR ablation is the strongest part: it shows clearly that training on a single environment does not transfer, while DR-trained policies and vision networks generalize much better, both in simulation and to real doorknob images. That is a clean, reproducible finding.\n\nThe novelty is modest but real. Prior door-opening environments exist (Gu, Rajeswaran, Nemec), and DR is established. What is new is the specific packaged combination: a configurable, open-source benchmark with knobs, robots, randomized door physics, and baseline agents. For a paper whose primary contribution is an artifact, that is enough to justify serious review.\n\nThe soft spots are real but addressable. Most important: Section 5 reports a 59% real-world success rate on a Baxter robot, but the entire paper describes only BLUE arms. No Baxter model exists in the environment description or in Tables 8/9. The reader is left to guess whether the policy was trained on an undocumented Baxter model or directly deployed from BLUE, which would change observation and action dimensions and kinematics. Either way, the transfer experiment is not reproducible from the paper. This is a heavy gap for the paper's headline claim, though it does not undercut the benchmark itself.\n\nSecond, all success rates are single-seed point estimates with no variance. For an empirical RL paper that is a notable omission, especially since the reported numbers include outliers like 0.00 versus 1.00 across similar settings. Third, there are small internal inconsistencies: Table 1 omits SAC GT+Noise rows, the ablation section refers to \"Table 1\" when it means Table 2, and the writing contains scattered typographical errors that suggest a rushed revision.\n\nOverall: the benchmark is worth having and the DR ablation is a solid contribution. The real-world transfer section needs either full documentation of the Baxter model or a more cautious claim. This paper deserves peer review, not desk rejection, but it should come back with mandatory revisions on reporting standards and transfer details.","headline":"A useful open-source door-opening benchmark whose baseline results and DR ablation are worth having, but the real-world transfer claim is under-specified and the evaluation reporting needs tightening.","tokens_in":13899,"tokens_out":1321,"would_cite":true,"duration_ms":51690,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DoorGym claims a domain-randomized door-opening simulator can train a policy that transfers to a real door.","keywords":["DoorGym","door opening","domain randomization","sim-to-real transfer","reinforcement learning","robot manipulation","Proximal Policy Optimization","Soft Actor-Critic"],"falsifier":"Inspect the released DoorGym code for a Baxter arm model and reproduce the Section 5 experiment with the arm used in training; if no Baxter model exists, or if deploying the trained policy requires converting its action space, the sim-to-real transfer claim fails as reported. A simpler check is to rerun the transfer with the same arm inside and outside simulation and compare the 70%/59% gap.","tokens_in":12966,"feed_emoji":"🚪","tokens_out":7497,"duration_ms":76082,"temperature":0.7,"pith_summary":"This paper introduces DoorGym, a simulated door-opening environment built for domain randomization, and argues that policies trained across its randomized door worlds become robust enough to open unseen simulated doors and, in one zero-shot experiment, a real wooden door. The authors report a PPO baseline achieving success rates between 0% and 95% depending on knob type and robot configuration, with the hardest cases (round knobs, lever knobs with certain arms) often failing completely. The transfer experiment reports 70% success in simulation and 59% on a real Baxter robot, which the paper presents as evidence that the randomized training recipe carries over to reality. The aim is to give the reinforcement-learning community an accessible benchmark that sits at the intersection of domain transfer, practical manipulator tasks, and realism.","feed_headline":"Sim-trained policy opens a real door at 59%","feed_subtitle":"DoorGym randomizes door shape, physics, and lighting so a PPO policy generalizes from simulation to reality.","key_machinery":"The load-bearing object is DoorGym's randomized door-world generator, which resamples a new simulated world for every episode: door dimensions, mass, hinge side, opening direction, knob type (pull, lever, or round), knob geometry from a CAD-generated dataset, robot joint damping, lighting, and material appearance all draw from specified uniform ranges. Behind it sits a shaped reward that drives the end-effector toward the knob, aligns fingertip orientation, penalizes control effort, and rewards door and knob rotation, plus a vision network that estimates the knob's 3D position from top and front camera views and feeds a direction vector into a small policy network. PPO and SAC serve as the baseline algorithms, with PPO the stronger exploitative policy in these experiments. The generator is what the transfer claim rests on: it is the mechanism that produces a distribution broad enough to overlap the real door.","core_discovery":"The paper's central claim is that domain randomization over door geometry, knob type, physical properties, and visuals is sufficient to train a single reinforcement-learning policy that generalizes across a distribution of doors. With ground-truth knob position, a PPO agent trained on randomized worlds reaches 0.95 success on a pull-knob task with a floating hook, 0.68 on a lever knob with a floating hook, and 0.71 on a pull knob with a gripper, all on 100 unseen test worlds. Replacing ground truth with a learned vision estimator lowers these numbers but keeps some tasks viable, and the same training recipe transfers zero-shot to a real pull-knob door at 0.59 success. The paper also claims that domain randomization is necessary for this transfer: policies and vision networks trained on a single environment collapse on new domains.","pith_inferences":["The paper's Section 5 uses a Baxter robot without stating whether DoorGym was given a Baxter model; if the deployed policy was trained on the BLUE arm, the 59% result suggests an unstated cross-embodiment transfer that would be worth isolating in a controlled experiment.","The reported 'about 3 cm localization error makes opening hard, above 5 cm impossible' threshold suggests a concrete design target: a perception system good to within a few centimeters is a prerequisite for door-open RL, and DoorGym could be used to score that target directly.","The same randomized-world generator could be repurposed for other articulated household objects such as cabinets or drawers, since the knob, hinge, and door parameterization generalizes, though the paper does not claim this.","A natural next experiment the paper leaves implicit is to combine DoorGym with automatic domain-randomization methods that adapt randomization ranges based on task success; that would test whether the current fixed ranges are near-optimal or merely sufficient."],"forward_implications":["If the claims hold, a single PPO policy checkpoint can open a range of randomized doors without resetting or retraining, making DoorGym a ready benchmark for comparing door-opening algorithms.","The large gap between ground-truth and vision-estimated success rates implies that improvements in 3D knob perception, not control, are the clearest route to raising overall performance.","The ablation showing single-environment training collapses from 46% to 0% vision-based success on randomized worlds indicates that domain randomization is not optional for this task's sim-to-real goals.","SAC's near-zero success on lever and round knobs means the baseline results are algorithm-dependent; reporting both on-policy and off-policy baselines lets users choose according to their sample-efficiency needs."],"supporting_citations":[{"why":"Establishes the domain-randomization sim-to-real recipe for dexterous manipulation that this work adapts to doors.","marker":"[17]"},{"why":"Supplies the core domain-randomization principle for transferring networks from simulation to reality.","marker":"[14]"},{"why":"Supports zero-shot policy transfer via dynamics randomization, a precedent for DoorGym's physical randomization.","marker":"[16]"},{"why":"Provides the PPO algorithm used as the stronger baseline in the experiments.","marker":"[28]"},{"why":"Provides the SAC algorithm used as the off-policy baseline.","marker":"[29]"},{"why":"Defines the default robot arm and gripper configurations used in the environment.","marker":"[26]"},{"why":"Supplies the environment interface through which DoorGym exposes its randomized worlds.","marker":"[21]"},{"why":"Supplies the physics engine used for all contact and dynamics simulation.","marker":"[23]"},{"why":"The codebase the PPO training pipeline is structured on, including eight-worker rollouts.","marker":"[30]"},{"why":"Motivates the log-distance reward term that sharpens accuracy near the doorknob.","marker":"[27]"}],"fun_headline_variants":["Randomized sim door training hits 95%, transfers to real at 59%","DoorGym: one policy opens varied doors, 95% sim, 59% real","Sim-to-real door opening via domain randomization: 95% to 59%","One RL policy opens many doors: 95% sim, 59% real","DoorGym: domain randomization enables 95% sim, 59% real transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-world experiment assumes the policy trained in the simulator can be run on the Baxter robot even though the paper does not state which robot model was trained for that deployment; if the simulator arm and the real arm do not share the same action space and kinematics, the 59% transfer number is not interpretable as stated.","fun_headline_variants_meta":{"raw":{"variants":["Randomized sim door training hits 95%, transfers to real at 59%","DoorGym: one policy opens varied doors, 95% sim, 59% real","Sim-to-real door opening via domain randomization: 95% to 59%","One RL policy opens many doors: 95% sim, 59% real","DoorGym: domain randomization enables 95% sim, 59% real transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000712,"raw_usage":{"total_tokens":3168,"prompt_tokens":876,"completion_tokens":2292,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":2182}},"tokens_in":492,"tokens_out":2292,"duration_ms":16333,"temperature":1.0,"reasoning_tokens":2182,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:59:55.084287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released DoorGym code for a Baxter arm model and reproduce the Section 5 experiment with the arm used in training; if no Baxter model exists, or if deploying the trained policy requires converting its action space, the sim-to-real transfer claim fails as reported. A simpler check is to rerun the transfer with the same arm inside and outside simulation and compare the 70%/59% gap.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PPO algorithm used as the stronger baseline in the experiments."},{"cited_title":"Quasi-Direct Drive for Low-Cost Compliant Robotic Manipulation","cited_arxiv_id":"1904.03815","evidence_quote":"Defines the default robot arm and gripper configurations used in the environment."},{"cited_title":"Todorov, T","cited_arxiv_id":null,"evidence_quote":"Supplies the physics engine used for all contact and dynamics simulation."},{"cited_title":"Kostrikov","cited_arxiv_id":null,"evidence_quote":"The codebase the PPO training pipeline is structured on, including eight-worker rollouts."}],"review_version":1}