Pith. sign in

REVIEW 10 cited by

PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.11802 v3 pith:G5AGJ5GV submitted 2024-06-17 cs.CV

Fanqing Meng , Wenqi Shao , Lixin Luo , Yahong Wang , Yiran Chen , Quanfeng Lu , Yue Yang , Tianshuo Yang
show 3 more authors
Kaipeng Zhang Yu Qiao Ping Luo
This is my paper · ORCID
classification cs.CV
keywords modelsphysicalcommonsenseevaluationimagespromptstext-to-imagecurrent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Text-to-image (T2I) models have made substantial progress in generating images from textual prompts. However, they frequently fail to produce images consistent with physical commonsense, a vital capability for applications in world simulation and everyday tasks. Current T2I evaluation benchmarks focus on metrics such as accuracy, bias, and safety, neglecting the evaluation of models' internal knowledge, particularly physical commonsense. To address this issue, we introduce PhyBench, a comprehensive T2I evaluation dataset comprising 700 prompts across 4 primary categories: mechanics, optics, thermodynamics, and material properties, encompassing 31 distinct physical scenarios. We assess 6 prominent T2I models, including proprietary models DALLE3 and Gemini, and demonstrate that incorporating physical principles into prompts enhances the models' ability to generate physically accurate images. Our findings reveal that: (1) even advanced models frequently err in various physical scenarios, except for optics; (2) GPT-4o, with item-specific scoring instructions, effectively evaluates the models' understanding of physical commonsense, closely aligning with human assessments; and (3) current T2I models are primarily focused on text-to-image translation, lacking profound reasoning regarding physical commonsense. We advocate for increased attention to the inherent knowledge within T2I models, beyond their utility as mere image generation tools. The data will be available soon.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do Image Editing Models Understand Lighting?

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    New 3DLP benchmark with real-world 1K HDR pairs shows state-of-the-art image editing models vary in physical lighting consistency, with best models close to reality but error-prone in low-light regions.

  2. GenSpace: Benchmarking Spatially-Aware Image Generation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    GenSpace benchmarks spatial awareness in image generation with a 3D reconstruction-based evaluator, showing models struggle with allocentric relations and metric measurements.

  3. PhysMirror: Physics-Aware Mirror Object Generation

    cs.CV 2026-07 conditional novelty 6.5 of 10

    An end-to-end pipeline lifts text objects to 3D meshes, constructs exact planar-mirror scenes, extracts depth/segmentation priors, and conditions diffusion models to generate physically consistent reflections, measure...

  4. OmniPhys: Knowledge-Graph-Driven Benchmarking and Collective Optimization for Physical Commonsense in Text-to-Image Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A knowledge-graph benchmark (OmniPhys, 1,551 prompts, 14 physics knowledge points) and a batch-feedback prompt optimizer (OmniPrompt) improve measured physical consistency of text-to-image models by 0.01–0.03 Joint Sc...

  5. TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models

    cs.SD 2025-09 conditional novelty 6.0 of 10

    TTA-Bench offers a seven-dimension, 2,999-prompt evaluation of ten text-to-audio models with 118,000 human ratings, covering quality, robustness, fairness, bias, and toxicity.

  6. AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    AIGI-Holmes combines visual expert pretraining, SFT on explanation data, and direct preference optimization to deliver human-verifiable explanations and top detection accuracy on unseen AI generators.

  7. Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ABP evaluates and improves how well text-to-image models render implicit real-world knowledge.

  8. Test-time Prompt Refinement for Text-to-Image Models

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A training-free closed loop, in which a multimodal LLM rewrites a text prompt after inspecting the generated image, improves overall text-to-image alignment but degrades some attribute and spatial categories.

  9. A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.

  10. T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generation

    cs.CV 2025-07 reject novelty 4.0 of 10

    A 1,200-prompt benchmark across six world-knowledge domains reports that ten state-of-the-art text-to-video models average below 0.70 on a 0 to 1 scale for producing videos consistent with real-world knowledge.

Pith tools