{"id":"d7ef352b-9b53-405d-9992-f84f304d4dc2","arxiv_id":"2412.18774","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Images are labeled by downstream robot task reward, yielding a 12,500-image benchmark on which human-oriented quality metrics fail and robot quality preferences diverge sharply from human ones.","lead":"Robots were given corrupted camera images while performing simulated pick-and-push tasks, and each image was labeled by how well the robot still completed its task rather than by how it looked. The paper releases a 12,500-image benchmark, a quality-prediction network, and evidence that robot and human image-quality preferences barely correlate (about 0.21).","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward-derived EPD labels may measure policy robustness to OOD corruption rather than a stable robot-centric image quality axis; without cross-policy reliability, the low human-robot PLCC does not establish the claimed divergence.","rationale":"The strongest claim is the divergence between robot and human quality judgments. The only quantitative evidence is the low PLCC between human MOS and EPD reward labels. If the reward labels are not valid as a stable, task-relevant quality measure, the divergence claim is not established. My proposed test is feasible: the pipeline already runs multiple RL algorithms; adding seeds and reporting per-image inter-policy correlations is a modest extension. I do not claim the labels are necessarily invalid; a high inter-policy correlation would support the dataset. I also do not think the mechanical defects or the small MA-EIQA gain change the central assessment. The reader's CONDITIONAL verdict already identifies label validity as the weak point; I refine it: the first-frame attachment is less damning than the policy-specificity of the reward, because all frames share the distortion. Thus agreement is partial. Verdict recommendation unchanged: conditional on the stability check (or equivalent construct-validity evidence), the paper's central claim can be accepted; absent that, it should not be treated as establishing a general embodied quality axis.","tokens_in":24919,"tokens_out":7184,"duration_ms":69349,"concrete_test":"Regenerate EPD labels for a fixed set of 100 initial frames and at least one full distortion/level grid using three fresh random seeds for each of PPO, SAC, and TDMPC2 policies trained on clean images. Compute the pairwise Spearman correlation between the per-image reward labels across all nine policies (or all six original participants if per-image scores can be released). If the median pairwise SRCC is below about 0.6, or if the averaged label's split-half reliability falls below 0.7, the reward labels are dominated by policy-specific variance and the dataset does not define a stable robot-centric image quality axis; the human-robot divergence claim would then need to be reframed as 'human aesthetic scores are uncorrelated with the robustness of these specific agents.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim ('quality assessment of embodied images is different from that of humans') rests on EPD labels being a valid measure of image quality from a robot's perspective. According to Sec. III-B, labels are the cumulative reward of an episode in which a policy trained on clean images is evaluated under a consistently applied distortion. That scalar is not an intrinsic property of the distorted image; it is a measure of how robust a particular PPO, SAC, or TDMPC2 policy is to an out-of-distribution corruption. Because these algorithms differ in inductive bias and training stochasticity, the same image can receive very different 'quality' scores depending on which policy is used. The paper averages six embodied agents (Table III), but reports only distortion-category means; the per-image agreement across those agents is never given. If the per-image reward labels are not stable across independently trained policies, then (i) the EPD label conflates policy fragility with image quality, (ii) the PLCC=0.2116 against human MOS is an attenuated correlation between two policy-specific measurements rather than evidence of a distinct quality axis, and (iii) the claim of a human-robot gap outruns the data. The first-frame attachment issue noted by the reader is secondary: because the corruption is applied consistently across all frames, the episode reward is a legitimate label for the distortion condition; the unresolved question is whether that condition label is policy-independent enough to be called embodied image quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EPD, an 'Embodied Preference Database' for image quality assessment (IQA) in robotic settings, with 12,500 distorted-image annotations obtained by running RL policies (PPO, SAC, TDMPC2) on push and pick tasks in SAPIEN and attaching the episode reward as the quality score. It also proposes MA-EIQA, a no-reference IQA network combining a multi-scale feature encoder with a CBAM-style attention module, and benchmarks 16 existing IQA methods on EPD. The central claim is that embodied-image quality assessment differs from human perception, supported primarily by a low measured correlation (PLCC 0.2116) between robot reward-derived scores and human MOS. The paper also includes a real-world UR5 experiment and an ablation study.","tokens_in":25042,"tokens_out":4439,"duration_ms":42995,"significance":"If the reward-derived labels are accepted as a valid robot-centric quality axis, the paper provides a useful new resource and a plausible demonstration that human-oriented IQA metrics are poor predictors of robotic task-relevant image quality. The EPD dataset, the benchmark of 16 methods, the released code, and the lightweight MA-EIQA model are concrete contributions. It is also to the paper's credit that the low human-robot PLCC is an external benchmark measurement, so the headline correlation is not purely circular. However, the interpretation of that correlation as evidence of a distinct embodied quality axis depends on a load-bearing premise that the RL episode reward is a stable, image-attributable quality label; this premise is not yet established, and the paper's scope is also narrower than the title suggests.","major_comments":[{"comment":"The robot quality label is defined as the cumulative RL episode reward, which is a property of a particular policy interacting with a dynamical environment, not an intrinsic property of the distorted image. A policy trained on clean images and evaluated under a consistent corruption may fail because of OOD policy fragility, exploration noise, or simulator dynamics rather than because the first-frame image loses task-relevant information. This makes the low PLCC of 0.2116 against human MOS (Fig. 5) substantially entailed by the label construction. To support the claim that a distinct embodied quality axis exists, the paper should demonstrate that reward-derived labels are stable across independently trained policies and correlate with task-relevant image information. Table III reports only distortion-category means for six agents; the missing per-image agreement across those agents (e.g., per-image ICC or rank correlations) is the key evidence needed. Without it, the low human-robot correlation could be an attenuated correlation between policy-specific robustness measurements rather than evidence of a robot-specific image quality axis.","section":"Section III-B, Eqs. (1)-(3); Table III"},{"comment":"Section III-A states that 'only the image of the initial frame is selected as the image to be evaluated,' while the reward label is accumulated over an episode of 50 action steps (Section V-A). Because the distortion is applied consistently across all frames, the episode reward is a legitimate label for the distortion condition, but it is not a label for the specific initial-frame image that is scored. The dataset is described as containing '12,500 distorted image annotations,' yet the annotation is episode-level. The paper should either validate that reward scores are insensitive to which frame is selected, or reformulate EPD as containing condition-level annotations attached to representative first frames. This distinction matters for the central claim because the correlation with human MOS is computed on single images.","section":"Section III-A; Section V-A"},{"comment":"All benchmark results in Table I are reported without uncertainty, despite the fact that the train/val split is a single 8:2 partition and several reported differences are small (e.g., MA-EIQA exceeds MANIQA by 2.29% SRCC on All Tasks). Table IV reports ablation means over 10 runs but does not report standard deviations or significance tests, and its caption incorrectly says 'Comparison of 14 IQA methods' when it compares four model variants. The state-of-the-art claim for MA-EIQA and the component contributions from the multi-scale encoder and attention module cannot be assessed without confidence intervals, multiple seeds, or paired significance tests. Given that the central claim rests on the divergence between robot and human quality assessment, the benchmark table should also include bootstrap or permutation-based intervals for the low PLCC values in Fig. 5, since the current 95% prediction intervals describe the fitting curve, not the uncertainty of the correlation estimate.","section":"Table I; Section V-A; Table IV"},{"comment":"The human subjective experiment used to obtain MOS is described only as involving '15 experienced experts.' No information is provided about viewing conditions, display device, viewing distance, rating scale, instruction, outlier rejection, or inter-observer agreement, and no per-image variance of the human MOS is reported. Since the headline result is the low correlation between robot scores and this human MOS, the reliability of the human labels is load-bearing. The paper should report standard IQA subjective-testing details and at least one inter-observer consistency measure (e.g., average SRCC between each expert and the mean) so that an attenuated PLCC can be separated from noisy human labels or an unreliable protocol.","section":"Section III-C; Fig. 5"}],"minor_comments":[{"comment":"The abstract contains a typo: 'does the the image quality of Robot-Generated Content' should be 'does the image quality.' The term 'Moravec paradox' is invoked but never explained or cited; either define it or remove the reference.","section":"Abstract; Section I"},{"comment":"The heading 'PROPOSED METHED' should read 'PROPOSED METHOD.'","section":"Section IV heading"},{"comment":"The simulator name is misspelled as 'Issac Sim' in 'The Issac Sim [39]'; it should be 'Isaac Sim.'","section":"Section II-A"},{"comment":"Terminology is inconsistent: the text introduces 'embodied Differential Mean Opinion Score (DMOS)' and later uses 'EMOS' in Fig. 5, which is not defined. The normalized score should be given one name and one definition throughout.","section":"Section III-B; Fig. 3; Fig. 5"},{"comment":"The TDMPC2 reward in Eq. (3) contains an unspecified weight factor lambda, and no value or sensitivity analysis is given. If the final EPD labels depend on this hyperparameter, its value should be reported.","section":"Section III-B, Eq. (3)"},{"comment":"The real-world experiment is reported as a set of illustrative cases with no systematic quantitative evaluation, sample-size information, or comparison against a baseline. If it is intended as evidence for the human-robot divergence, it needs a protocol and results table; otherwise it should be framed explicitly as a qualitative demonstration.","section":"Section V-E; Fig. 9"},{"comment":"The title and abstract claim a general conclusion about 'embodied image quality assessment,' but the dataset covers only two fixed-base robotic arm tasks in a single simulator. The paper acknowledges this in Section II-B, but the abstract and conclusion should carry a qualifier such as 'for the tested manipulation tasks' to avoid overgeneralization.","section":"General scope"}],"recommendation":"major_revision","confidential_remarks":"The EPD resource and benchmark are potentially valuable to the IQA and embodied-AI communities, and the external PLCC measurement is a useful sanity check. However, the central conceptual claim currently outruns the evidence because the reward-derived labels conflate policy robustness with image quality, and the per-image stability of those labels is not reported. I would ask for a validation experiment with independently trained policies and per-image cross-policy reliability, plus uncertainty quantification in the benchmark tables, before considering acceptance. The two-task, single-simulator scope also argues for a more modest framing of the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EPD is a genuinely new resource and the low human-robot correlation (PLCC ≈ 0.21) is a real empirical result, but the reward-based label definition and the missing cross-policy reliability check mean the central 'embodied image quality' claim is not yet nailed down.\n\nWhat's actually new: the idea of using RL task rewards as quality labels for distorted images, the EPD database (12.5k images, 25 distortions at 5 levels, two SAPIEN manipulation tasks, six agent-task combinations), and the direct comparison showing that these labels barely correlate with human MOS. That comparison is valuable regardless of how you interpret the label. Benchmarking 16 existing IQA methods on EPD is also useful, and MA-EIQA is an honest lightweight baseline even if its components are all cited (ResNet50 + PANet-style fusion + CBAM-style attention).\n\nThe soft spots are real but mostly fixable. The load-bearing one: the label is the cumulative reward of a policy trained on clean images and evaluated under a consistently applied distortion. That is a measure of policy robustness to out-of-distribution corruption, not necessarily a stable property of the image. The paper averages six embodied agents but never reports per-image agreement across them. Without that, the low PLCC against human MOS could be an attenuated correlation between two policy-specific measurements rather than evidence of a distinct robot quality axis. The first-frame attachment is, as the stress-test note says, secondary, because the distortion is constant across the episode; the reward is legitimately a condition-level label. But it's still worth stating that the image being scored is one frame of a 50-step episode.\n\nOther issues are minor: the claimed 2.2% SRCC improvement over MANIQA has no error bars or significance test (the ablation shows run-to-run variance); the human MOS collection is underdescribed (15 experts, no protocol details); and generalization from two simulated box tasks to 'embodied AI' outruns the evidence, even with the small real-world UR5 demo. The manuscript also has mechanical defects: a URL with a space, inconsistent image counts, garbled figure labels. None of these refute the core finding, but they need cleaning.\n\nWho this is for: people in IQA who care about new task-driven quality axes, and embodied-AI researchers who need to think about sensor degradation. It deserves a serious referee. I'd send it to review with a clear request to add cross-policy label agreement, human MOS details, and significance testing, and to soften the generalization claims.","headline":"EPD is a genuinely new resource and the low human-robot correlation is worth taking seriously, but the reward-based label definition and missing cross-policy reliability check mean the central divergence claim is not yet nailed down.","tokens_in":25763,"tokens_out":3237,"would_cite":true,"duration_ms":53401,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an embodied robot's image quality is defined by how well the image supports its task, not by human visual appeal, and that the two views are nearly uncorrelated.","keywords":["image quality assessment","embodied AI","robotic perception","reinforcement learning reward","no-reference image quality assessment","benchmark dataset","human visual system","robotic manipulation"],"falsifier":"Run many episodes that start from the exact same distorted first frame, changing only the random seed: if the spread in accumulated rewards across seeds is as large as the spread across distortion levels, the EPD scores are not stable properties of the image and the claimed human-robot gap would need to be recomputed from per-image medians.","tokens_in":24548,"feed_emoji":"🤖","tokens_out":10049,"duration_ms":86530,"temperature":0.7,"pith_summary":"The paper tries to establish that image quality for an embodied robot is a task-dependent property: a distorted image is 'good' to the extent that a robot acting on it still earns high task reward, and this is not the same as what a human finds attractive. To make this concrete, it builds the Embodied Preference Database (EPD), 12,500 distorted images from robotic-arm episodes, each labeled by the normalized cumulative reward the robot accumulated while performing push and pick tasks with that image as input. A human-rating study on the same images gives a correlation of only 0.2116 with the robot labels, and sixteen human-centered IQA algorithms perform poorly on EPD, which the paper reads as evidence that human-visual-system metrics are not valid proxies for robotic perception. It also proposes MA-EIQA, a lightweight no-reference model with multi-scale feature fusion and channel-spatial attention, and reports it as the best performer on EPD.","feed_headline":"Robot and human image-quality judgments barely correlate","feed_subtitle":"New EPD benchmark scores 12,500 distorted images by how well a robot arm finishes its task, not by appearance.","key_machinery":"The carrying mechanism is the EPD annotation protocol: a first-frame RGB image from a robotic-arm episode is corrupted by one of 25 distortions at five intensity levels, the episode is then executed by reinforcement-learning policies, and the normalized cumulative reward of the episode is attached to that image as its embodied DMOS. This protocol converts 'quality for the robot' into a measurable scalar without any human opinion. The paper's model, MA-EIQA, is a no-reference network built from a pretrained residual backbone, a bidirectional multi-scale feature encoder that fuses high-level semantics with low-level texture, and an attention module that weights channels and spatial regions; it regresses the reward-derived score with mean squared error loss.","core_discovery":"On the paper's terms, the discovery is that quality assessment of embodied images is different from that of humans. Robots and humans assign nearly unrelated quality scores to the same distorted images, with PLCC values of 0.2116 overall, 0.1778 on the push subtask, and 0.1297 on the pick subtask; existing IQA methods designed for the human visual system, even deep Transformer-based ones, plateau below 0.6 SRCC/PLCC on the EPD benchmark. The paper explains the gap by the different feature budgets of the two systems: embodied agents depend on low-level texture, edge, and structural information needed for manipulation, while human judgment keys on semantic content and aesthetics. It then introduces MA-EIQA, a lightweight no-reference network that fuses multi-scale features and applies channel-spatial attention, achieving higher correlation with the reward-derived labels than any of the compared human-centered methods.","pith_inferences":["If task reward is the quality axis, then 'good image' becomes policy- and task-dependent: the same frame could be high-quality for one robot policy and low-quality for another, so future databases may need per-policy scores rather than a single averaged MOS.","A natural extension the paper does not test is to close the loop: have the robot predict embodied quality online and trigger restoration, replanning, or a request for a new viewpoint before executing a manipulation.","The divergence claim should be stress-tested beyond two box-manipulation tasks in one simulator; navigation, locomotion, and real-robot experiments with different cameras and policies would show whether the 0.21 correlation is a general property of embodied vision or an artifact of this task family.","An explicit prediction follows from the paper's low-level-feature account: distortions that destroy global semantics but preserve local edges and textures should hurt robot rewards less than they hurt human scores, while the opposite should hold for noise that destroys structure; paired human-and-robot rating experiments could test this directly."],"forward_implications":["Human-centered IQA metrics should not be treated as proxies for robotic perception; any robot system that uses them to filter or select images may be optimizing the wrong quantity.","No-reference models are a better starting point for embodied quality than full-reference ones: on EPD they outperform reference-based methods, consistent with the idea that the robot's own task defines quality rather than comparison with a pristine image.","Lightweight CNN designs can compete with much larger Transformer models on this quality axis, which matters for onboard, real-time use in robots with limited compute.","EPD-style reward labels give a robot a training signal to predict, before acting, which images will hurt its task, enabling selective restoration, reacquisition, or refusal.","Because the human-robot correlation is so low, embodied-AI benchmarks that evaluate perception should include task-derived image quality labels rather than relying on human subjective ratings."],"supporting_citations":[{"why":"Supplies the physics-simulation environment in which the robotic-arm episodes are rendered and executed.","marker":"[44]"},{"why":"Supplies one reinforcement-learning policy whose episode rewards become quality labels.","marker":"[45]"},{"why":"Supplies the second reinforcement-learning policy used for label generation.","marker":"[46]"},{"why":"Supplies the third, model-based policy used for label generation.","marker":"[47]"},{"why":"Provides the manipulation task definitions and per-step reward structure behind the EPD labels.","marker":"[48]"},{"why":"Provides the pretrained residual backbone that MA-EIQA uses for multi-scale feature extraction.","marker":"[49]"},{"why":"Provides the bidirectional top-down/bottom-up fusion design adopted by MA-EIQA's feature encoder.","marker":"[50]"},{"why":"Provides the channel-and-spatial attention design adopted by MA-EIQA's embodied attention module.","marker":"[51]"},{"why":"The strongest compared no-reference baseline; its performance frames MA-EIQA's improvement.","marker":"[61]"},{"why":"Provides the shared training and evaluation platform on which all compared methods are benchmarked.","marker":"[65]"}],"fun_headline_variants":["Robots and humans disagree on image quality","For robots, image quality means task success, not looks","New benchmark reveals robot-specific image quality standards","MA-EIQA: first IQA model for embodied robot perception","Robot image quality judged by task performance, not aesthetics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark depends on treating the total reward a robot earns during a 50-step episode as a fair quality score for the single starting image; if that reward is driven mainly by chance variation in the robot's behavior or by the simulator rather than by what is visible in that one frame, the labels measure how robust the robot is to damage, not image quality.","fun_headline_variants_meta":{"raw":{"variants":["Robots and humans disagree on image quality","For robots, image quality means task success, not looks","New benchmark reveals robot-specific image quality standards","MA-EIQA: first IQA model for embodied robot perception","Robot image quality judged by task performance, not aesthetics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00034,"raw_usage":{"total_tokens":1892,"prompt_tokens":980,"completion_tokens":912,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":836}},"tokens_in":596,"tokens_out":912,"duration_ms":13555,"temperature":1.0,"reasoning_tokens":836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:29:39.352681+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run many episodes that start from the exact same distorted first frame, changing only the random seed: if the spread in accumulated rewards across seeds is as large as the spread across distortion levels, the EPD scores are not stable properties of the image and the claimed human-robot gap would need to be recomputed from per-image medians.","supporting_citations":[{"cited_title":"Sapien: A simulated part-based interactive environment,","cited_arxiv_id":null,"evidence_quote":"Supplies the physics-simulation environment in which the robotic-arm episodes are rendered and executed."},{"cited_title":"Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,","cited_arxiv_id":null,"evidence_quote":"Supplies the second reinforcement-learning policy used for label generation."},{"cited_title":"Path aggregation network for instance segmentation,","cited_arxiv_id":null,"evidence_quote":"Provides the bidirectional top-down/bottom-up fusion design adopted by MA-EIQA's feature encoder."},{"cited_title":"Cbam: Convolutional block attention module,","cited_arxiv_id":null,"evidence_quote":"Provides the channel-and-spatial attention design adopted by MA-EIQA's embodied attention module."},{"cited_title":"Maniqa: Multi-dimension attention network for no-reference image quality assessment,","cited_arxiv_id":null,"evidence_quote":"The strongest compared no-reference baseline; its performance frames MA-EIQA's improvement."},{"cited_title":"IQA-PyTorch: Pytorch toolbox for image quality assessment,","cited_arxiv_id":null,"evidence_quote":"Provides the shared training and evaluation platform on which all compared methods are benchmarked."}],"review_version":1}