Pith. sign in

REVIEW 4 major objections 5 minor 75 references

Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes an LLM-driven evaluation agent that turns a natural-language question into a small number of targeted generation-and-tool rounds, and claims it reproduces benchmark-level capability judgments at about a tenth of the…

desk verdict Useful evaluation-agent framework with a solid open release, but the 'comparable results' claim needs qualification per dimension before it's trustworthy. read the letter →

arxiv 2608.09666 v1 pith:NV754SGV submitted 2026-08-10 cs.AI

Shulin Tian , Ziqi Huang , Fan Zhang , Hongyuan Zhu , Yu Qiao , Ziwei Liu This is my paper · ORCID
classification cs.AI
keywords evaluationagentvisualgenerativemodelstext-to-videotext-to-imagesmall-sampletierestimationLLMasjudgeinstructiontuningtool-augmentedreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that evaluating visual generative models can be done the way a human reviewer does it—quickly, adaptively, and with a small number of samples—rather than by running fixed benchmarks that require hundreds or thousands of generations. It proposes the Evaluation Agent, an LLM-driven planner that turns a natural-language question into sub-aspects, generates targeted prompts, samples a few dozen images or videos per dimension, invokes benchmark or VQA tools on each batch, and uses the observations to decide what to probe next. The agent maps raw scores into five capability tiers and claims that its tier predictions agree with full benchmark results within one tier most of the time, while cutting evaluation time to about 10% of the standard pipeline. The paper also claims that this protocol can be distilled into a compact open-source 3-billion-parameter planner trained on 10,042 trajectory records, preserving the reasoning, tool-use, and summary behavior and transferring partially to generator families never seen in training. A sympathetic reader would care because the framework promises cheaper, user-tailored, explainable evaluation of the fast-moving field of image and video generation.

What carries the argument

The load-bearing mechanism is the propose–sample–evaluate–observe loop run by two cooperating LLM agents: a Plan Agent that selects the next sub-aspect, decides when evidence is sufficient, and writes the final summary, and a PromptGen Agent that designs or selects the generation prompts and VQA questions for each round. Each round samples a small batch of visuals (roughly 3–9 prompts, about 23–26 total per dimension), passes them through a modular Evaluation Toolkit of benchmark or VQA tools, and returns structured observations that condition the next decision. Recorded rollouts are then unfolded into history-conditioned step-level instruction records—tagged as thoughts, tool calls, information, and summaries—forming a 10,042-record corpus that supervises a compact 3-billion-parameter local planner, so the same protocol runs without a proprietary API.

What would settle it

Run the agent many times on the same generator and dimension with the default budget, and compare the distribution of predicted tiers against the tier from the full benchmark; if, for a semantic dimension such as human action, the predicted tier jumps across tiers across repetitions, the small-sample tier estimate is not stable enough to support the tenfold-cost-reduction claim.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that dynamic multi-round evaluation with a few samples per round is enough to reproduce the conclusions of full benchmarks. The agent's loop is propose–sample–evaluate–observe: for each user query it selects a sub-aspect, has a prompt-design agent create or select prompts, generates roughly 3–9 visuals per round (about 23–26 per dimension in total), runs an evaluation tool, and updates its plan. On a 15-dimension text-to-video benchmark, the API-based agent achieves 44.8% exact-tier and 81.3% within-one-tier accuracy with one proprietary LLM backbone, and 41.5% / 77.2% with another, while using 23–25 samples per dimension versus 290 for the benchmark average; on a four-dimension compositional text-to-image benchmark it reaches 53.8% / 96.3% with about 25 samples versus 3000. The prompt-budget experiment shows that raising the budget to 30 prompts improves within-one-tier accuracy by 15 points on average on percentage-based dimensions, implying a direct trade-off between samples and calibration. The paper further claims that a 3-billion-parameter local planner trained on 10,042 step-level records from recorded rollouts follows the same protocol and retains partial cross-family transfer (41.3% / 77.3% in-domain, 31.1% / 64.9% out-of-domain on exact / within-one-tier accuracy).

Load-bearing premise

The whole efficiency claim rests on the assumption that a few rounds of 3 to 9 prompts per dimension produce a stable estimate of a model's true capability tier; if that estimate is noisy, the 'comparable results' conclusion weakens.

Editorial extensions

If this is right

  • A single user query about a model dimension can be answered from about 23–26 generated samples in minutes, rather than 290–3000 samples taking hours.
  • Because prompts and tools are chosen dynamically, the same framework handles open-ended queries (e.g., how precisely a model can place objects) and returns a detailed textual rationale, not just a score.
  • The protocol is not tied to one LLM: swapping the planning backbone changes aggregate accuracy by only a few points, so the method can run on whichever capable model is available.
  • A 3-billion-parameter local planner trained from recorded rollouts preserves the evaluation protocol and transfers partially to unseen generator families, lowering the bar for reproducible, low-cost evaluation.
  • There is a tunable cost-accuracy knob: raising the per-dimension prompt budget from 3–9 to 30 prompts increases within-one-tier agreement by about 15 points on average, so users can trade time for calibration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The near-zero exact-tier accuracy on semantic dimensions such as object class, human action, and spatial relationship suggests the bottleneck is concept coverage rather than planning; a testable extension would be to make prompt diversity per semantic dimension a training feature of the local planner.
  • The tier-based protocol fits naturally into continuous model monitoring: cheap agent evaluations on each release could flag capability regressions long before a full benchmark run is affordable.
  • If small-sample tiering is trusted, an adaptive sampler that escalates the prompt budget only when the agent reports borderline evidence would retain accuracy while using even fewer samples than the fixed budget.
  • The paper hints at a recommendation database built from accumulated evaluations; turning evaluation output into a queryable model-selection service would be a direct next step that the current experiments do not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes the Evaluation Agent, an LLM-driven multi-round framework that evaluates text-to-image and text-to-video generative models by decomposing a natural-language request into sub-aspects, generating a small number of prompts, sampling generated outputs, invoking benchmark or VQA-based evaluation tools, and iteratively updating its plan from the observed evidence. It claims to reduce evaluation time and sample count to roughly 10% of standard VBench and T2I-CompBench pipelines while delivering comparable tier-level conclusions. The paper also introduces EA-CoT-10K, a history-conditioned instruction-tuning corpus derived from rollout trajectories, and EA-3B, a 3B-parameter local planning backbone fine-tuned from Qwen2.5-3B-Instruct, with experiments on four in-domain and three out-of-domain T2V generator families plus a set of open-ended user queries.

Significance. If the efficiency and tier-accuracy claims hold, this is a valuable contribution to generative-model evaluation: it reframes evaluation as adaptive evidence acquisition rather than a fixed pipeline, releases a training corpus and a local open-source agent, and explicitly tests transfer to unseen generator families. The paper is strongest where it is concrete: the multi-round protocol, data filtering and training details, cost comparison in Table 4, prompt-budget ablation in Figure 3, and backbone-robustness check are described in sufficient detail for reproduction, and the authors are transparent about the distinction between API-based and local end-to-end costs. The central 'comparable results' claim, however, is currently supported only by aggregate point estimates over ten trials without confidence intervals, and several per-dimension entries in Tables 2, 3, and 5 are at or below the uniform-random baseline. The contribution is therefore promising but needs additional statistical support and a more qualified statement of the accuracy claim before it can be accepted.

major comments (4)
  1. [Section 4.1.2, Tables 2, 3, 5] The 'comparable results' claim is load-bearing but is reported only as Exact/Within-1 point estimates over ten trials, without confidence intervals or significance tests. This matters because several per-dimension entries are at or below the uniform-random baseline for five tiers (20% exact, 52% within-1). For example, with ten trials a 0/10 exact score has a 95% binomial confidence interval of roughly 0-31%, and a 1/10 within-1 score has an interval of roughly 0.3-44.5%. Entries such as Table 5's Object Class 0%/0% for Latte-1, Table 5's Spatial Relationship 0%/0% for ModelScope, and Table 2's Human Action 10%/10% for Latte-1 are therefore statistically indistinguishable from chance. Since these are precisely the fine-grained semantic dimensions that distinguish generators, the aggregate Exact/Within-1 numbers cannot support the unqualified 'comparable results' claim in the abstract without per-dimension uncertainty quantification and a statement of which dimensions are reliably above baseline.
  2. [Section 4.1.2, Figure 3] The prompt-budget ablation shows that the default budget of 3-9 prompts per round is not stable for percentage-based dimensions. Raising the budget to 30 prompts improves Within-1 accuracy by 15.0 percentage points on average across Latte-1 and ModelScope, while Exact accuracy improves by only 7.5 points. This indicates tier-boundary noise rather than convergence, and it directly weakens the claim that roughly 25 samples per user question deliver 'comparable results' for fine-grained semantic evaluation. The paper should either use the larger budget in the headline comparisons, report accuracy separately for each budget, or explicitly restrict the comparability claim to dimensions where stability has been demonstrated.
  3. [Section 4.1.1] The T2I-CompBench comparison excludes the Spatial Relationships dimension, with the paper stating that its 'score distribution and statistical interpretation are not directly comparable under our small-sample setting.' This selective exclusion narrows the benchmark claim, yet the abstract and Section 4.1.2 state 'comparable results' and 'high accuracy' without this qualification. Please state explicitly in the abstract and conclusion that the T2I comparison covers four of the five T2I-CompBench dimensions and that the excluded dimension is one for which comparability is not claimed.
  4. [Section 4.2, Table 5] The in-domain Open-EA results are not an independent validation of the learned policy because EA-CoT-10K training traces are derived from rollouts on the same four generators (Latte-1, ModelScope, VideoCrafter-0.9, VideoCrafter-2) that Table 5 labels as in-domain. The paper acknowledges this overlap in the text, but the in-domain rows should be clearly labeled as train-distribution matches, and claims about Open-EA generalization should rest on the out-of-domain rows only. Moreover, many out-of-domain rows are at chance level (e.g., Show-1 Spatial Relationship 0%/0%, CogVideoX-2B Spatial Relationship 0%/0%, CogVideoX-5B Spatial Relationship 0%/0%), so the 'partial cross-family transfer' conclusion needs the same uncertainty quantification and per-dimension baseline analysis as the API-based experiments.
minor comments (5)
  1. [Section 3.1] The notation in Eq. (1) writes y_j = e_k(v_j, c_j), but the dynamic selection of the tool k across rounds is not formalized; define k as a function of the round index and history H_t to match the rollout description in Section 3.2.3.
  2. [Section 3.3] The data-filtering thresholds are described qualitatively ('extremely low aggregate scores', 'lowest-quality range'); please provide the concrete thresholds or point to the code that implements them so the corpus construction is reproducible.
  3. [Figure 3] Figure 3 is difficult to read because the light/dark bars and hatched/solid regions carry two separate distinctions; add direct value labels or a legend that states both the budget and the exact-versus-within-one distinction for each bar.
  4. [Section 4.2, Table 6] Table 6 reports median and mean runtimes over saved traces, but the row labels do not state how many successful saved traces each row contains; adding the trace counts would clarify the stability of the runtime estimates.
  5. [Section 3.3] The paper notes that the 986-record T2I companion set is not mixed into the current EA-3B training run; please state explicitly whether this companion set is intended for future T2I Open-EA training or is released only as a resource.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark comparisons are a subsampling compression test, and the trained local agent is tested on held-out generator families.

full rationale

The paper's central claims are empirical rather than derived from a fitted identity. The Evaluation Agent is validated by restricting it to the official VBench/T2I-CompBench prompt lists and metrics, having it sample a small subset of prompts, and then comparing its tier conclusion against the full-benchmark tier. This is a compression test, not a circular derivation: the agent never sees the full-benchmark scores or the reference tier, and its prediction is not statistically forced by any fitted parameter. The tier thresholds are defined from benchmark density statistics, but the agent's outputs are independent of those thresholds. The self-citation of the ACL 2025 preliminary version [2] and the overlapping authorship of VBench do not make the comparison circular, because VBench is an externally published benchmark with reproducible scoring, and the cited prior work is not invoked as an unverified uniqueness or existence theorem. Open-EA is trained on rollout trajectories and then evaluated on four in-domain and three out-of-domain generator families, so the cross-family transfer claim has independent content. The paper's own ablations and discussion acknowledge that several semantic dimensions perform poorly and that larger prompt budgets improve stability; these are correctness and statistical-power concerns, not evidence that the conclusions reduce to their inputs by construction. No circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on benchmark reference validity, small-sample representativeness, VLM observation accuracy, and the quality of GPT-4o rollouts as supervision. None of these are formally proven in the paper, and the prompt-budget ablation partially undermines the small-sample representativeness assumption.

free parameters (3)
  • Tier boundaries for five-level categorization = Constructed from leaderboard statistics
    These thresholds define the ground-truth labels used to measure the agent's accuracy; they are not derived from first principles and are applied per dimension.
  • Prompt budget per round = 3-9 prompts per round, 30 in budget ablation
    Chosen by the authors; the ablation shows accuracy depends on this choice.
  • Data filtering thresholds for EA-CoT = Unspecified cutoffs for prompt-visual misalignment
    Trajectories with severe misalignment are removed; threshold values are not reported, and this curates the training distribution.
assumptions (4)
  • domain assumption VBench and T2I-CompBench scores provide a valid reference for visual generation quality.
    The paper uses these benchmarks as ground truth for tier accuracy in Section 4.1.
  • domain assumption A small number of generated samples can represent a generator's capability on a dimension.
    Central to the efficiency claim; the paper's own ablation in Section 4.1.2 shows this is fragile for some dimensions.
  • domain assumption The VLM used for VQA provides accurate enough descriptions to ground open-ended evaluation.
    Open-ended queries rely on VQA answers as evidence in Section 4.3.2; no independent verification of VLM accuracy is provided.
  • domain assumption GPT-4o rollouts are high-quality supervision for training EA-3B.
    EA-CoT-10K is generated by the API agent in Section 3.3; if those rollouts are flawed, the distilled model inherits the flaws.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models." pith.science (2026). https://pith.science/paper/NV754SGV

@misc{pith2026260809666,
  author       = {Pith},
  title        = {Pith review of: Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NV754SGV}},
  note         = {Machine review of arXiv:2608.09666}
}
read the original abstract

Recent advances in visual generative models have enabled high-quality image and video generation, but evaluating these models often demands sampling hundreds or thousands of images or videos, which is computationally expensive. Existing evaluation methods also rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations. Mimicking how humans quickly form impressions of a model's capabilities from only a few samples, we propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations, offering detailed, user-tailored analyses. Given a natural-language evaluation request, the agent decomposes it into sub-aspects, generates targeted prompts, samples images or videos from the evaluated model, invokes suitable evaluation tools, and iteratively updates its plan from the observed evidence, covering both predefined benchmark dimensions and open-ended user concerns. The framework is thus efficient, promptable, explainable, and scalable across models and tools. Experiments show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results. We further introduce Open Evaluation Agent (Open-EA) by constructing EA-CoT-10K, a corpus of history-conditioned step-level instruction-tuning records derived from multi-round evaluation rollouts, and training EA-3B from Qwen2.5-3B-Instruct as a local planning backbone that preserves the structured reasoning, tool invocation, and summary protocol of the API-based agent while reducing dependence on proprietary backbones. Experiments validate the API-based agent on established T2I/T2V benchmarks and open-ended queries, and evaluate Open-EA on four in-domain and three out-of-domain T2V generator families, showing partial cross-family transfer of the learned policy.

Figures

Figures reproduced from arXiv: 2608.09666 by the authors.

Figure 1
Figure 1. Overview of the Evaluation Agent Framework. The upper rollout pipeline contains a proposal stage, where the Plan Agent decomposes the user request and the PromptGen Agent prepares evaluation prompts, and an execution stage, where visual outputs are sampled and assessed by selected tools. These stages form an iterative observation loop for dynamic, user-conditioned evaluation. The bottom training pipeline converts co… view at source ↗
Figure 2
Figure 2. EA-CoT-10K Construction and Data Composition. (a) A representative T2V evaluation rollout is unfolded into history-conditioned step-level instruction-tuning records (training samples). Under a shared system protocol, each record retains preceding turns in its history and supervises either the next reasoning-and-tool decision or, after sufficient evidence has been collected, the terminal summary; text is abbreviated … view at source ↗
Figure 3
Figure 3. Effect of Prompt Budget on Percentage-Based VBench Dimensions. Lighter bars use the original prompt budget and darker bars use 30 prompts. Hatched portions denote exact-tier predictions and solid portions denote predictions within one neighboring tier. results, showcasing high accuracy within an error margin of one range. Effect of the Prompt Budget. The lower accuracy on Human Action, Scene, Color, and Object Class… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Data Distribution of Open-Ended User Query Dataset. We analyze the constructed open-ended user query dataset from three aspects: General/Specific, Ability, and Specific Domain. The distribution shows that the collected queries cover both broad capability questions and …
Figure 5
Figure 5. Figure 5: A Complete Case of Open-Ended User Query Evaluation. Given the query “How precisely can the user specify object relation￾ships?”, the Evaluation Agent progressively probes simple two-object relations, complex relations among three or more objects, and non￾standard imag…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 49 canonical work pages

  1. [1]

    Qwen2.5 technical report,

    Qwen Team, “Qwen2.5 technical report,”arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  2. [2]

    Evaluation Agent: Efficient and promptable evaluation framework for visual generative models,

    F. Zhang, S. Tian, Z. Huang, Y . Qiao, and Z. Liu, “Evaluation Agent: Efficient and promptable evaluation framework for visual generative models,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2025, pp. 7561–7582

    work page 2025

  3. [3]

    Towards accurate generative models of video: A new metric & challenges,

    T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, “Towards accurate generative models of video: A new metric & challenges,”arXiv preprint arXiv:1812.01717, 2018

    arXiv 2018

  4. [4]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,”Advances in neural information processing systems, vol. 30, 2017

    2017

  5. [5]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation,

    K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu, “T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation,” inAdvances in Neural Information Processing Systems, vol. 36, 2023

    work page 2023

  6. [6]

    VBench: Comprehensive benchmark suite for video generative models,

    Z. Huang, Y . He, J. Yu, F. Zhang, C. Si, Y . Jiang, Y . Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y . Wang, X. Chen, L. Wang, D. Lin, Y . Qiao, and Z. Liu, “VBench: Comprehensive benchmark suite for video generative models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  7. [7]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” inAdvances in Neural Information Processing Systems, vol. 33, 2020

    work page 2020

  8. [8]

    Frozen in time: A joint video and image encoder for end-to-end retrieval,

    M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” inIEEE International Conference on Computer Vision, 2021

    2021

Show all 75 references
  1. [9]

    Panda- 70m: Captioning 70m videos with multiple cross-modality teachers,

    T.-S. Chen, A. Siarohin, W. Menapace, E. Deyneka, H.-w. Chao, B. E. Jeon, Y . Fang, H.-Y . Lee, J. Ren, M.-H. Yang, and S. Tulyakov, “Panda- 70m: Captioning 70m videos with multiple cross-modality teachers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  2. [10]

    Video enhancement with task-oriented flow,

    T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,”International Journal of Computer Vision (IJCV), vol. 127, no. 8, pp. 1106–1125, 2019

    2019

  3. [11]

    Vbench++: Comprehensive and versatile benchmark suite for video generative models,

    Z. Huang, F. Zhang, X. Xu, Y . He, J. Yu, Z. Dong, Q. Ma, N. Chanpaisit, C. Si, Y . Jiang, Y . Wang, X. Chen, Y .-C. Chen, L. Wang, D. Lin, Y . Qiao, and Z. Liu, “Vbench++: Comprehensive and versatile benchmark suite for video generative models,”IEEE Transactions on Pattern An...

    2026

  4. [12]

    Evalcrafter: Benchmarking and evaluating large video generation models,

    Y . Liu, X. Cun, X. Liu, X. Wang, Y . Zhang, H. Chen, Y . Liu, T. Zeng, R. Chan, and Y . Shan, “Evalcrafter: Benchmarking and evaluating large video generation models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 139–22 149

    2024

  5. [13]

    Llamafactory: Unified efficient fine-tuning of 100+ language models,

    Y . Zheng, R. Zhang, J. Zhang, Y . Ye, and Z. Luo, “Llamafactory: Unified efficient fine-tuning of 100+ language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2024, pp. 400–410

    2024

  6. [14]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  7. [15]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis,

    D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” inInternational Conference on Learning Representations, 2024

    2024

  8. [16]

    Scaling rectified flow transformers for high-resolution image synthesis,

    P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boeselet al., “Scaling rectified flow transformers for high-resolution image synthesis,” inForty-first International Conference on Machine Learning, 2024

    2024

  9. [17]

    Latte: Latent diffusion transformer for video generation,

    X. Ma, Y . Wang, G. Jia, X. Chen, Z. Liu, Y .-F. Li, C. Chen, and Y . Qiao, “Latte: Latent diffusion transformer for video generation,”Transactions on Machine Learning Research, 2025

    2025

  10. [18]

    Mod- elscope text-to-video technical report,

    J. Wang, H. Yuan, D. Chen, Y . Zhang, X. Wang, and S. Zhang, “Mod- elscope text-to-video technical report,”arXiv preprint arXiv:2308.06571, 2023

    2023 arXiv

  11. [19]

    Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models,

    H. Chen, Y . Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y . Shan, “Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7310–7320

    2024

  12. [20]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths,

    Y . He, T. Yang, Y . Zhang, Y . Shan, and Q. Chen, “Latent video diffusion models for high-fidelity video generation with arbitrary lengths,”arXiv preprint arXiv:2211.13221, 2022

    2022 arXiv

  13. [21]

    Lavie: High-quality video generation with cascaded latent diffusion models,

    Y . Wang, X. Chen, X. Ma, S. Zhou, Z. Huang, Y . Wang, C. Yang, Y . He, J. Yu, P. Yanget al., “Lavie: High-quality video generation with cascaded latent diffusion models,”International Journal of Computer Vision, vol. 133, no. 5, pp. 3059–3078, 2025

    2025

  14. [22]

    VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness,

    D. Zheng, Z. Huang, H. Liu, K. Zou, Y . He, F. Zhang, Y . Zhang, J. He, W.-S. Zheng, Y . Qiao, and Z. Liu, “VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness,”arXiv preprint arXiv:2503.21755, 2025

    2025 arXiv

  15. [23]

    Holistic evaluation of text- to-image models,

    T. Lee, M. Yasunaga, C. Meng, Y . Mai, J. S. Park, A. Gupta, Y . Zhang, D. Narayanan, H. Teufel, M. Bellagenteet al., “Holistic evaluation of text- to-image models,”Advances in Neural Information Processing Systems, vol. 36, pp. 69 981–70 011, 2023

    2023

  16. [24]

    T2v- compbench: A comprehensive benchmark for compositional text-to-video generation,

    K. Sun, K. Huang, X. Liu, Y . Wu, Z. Xu, Z. Li, and X. Liu, “T2v- compbench: A comprehensive benchmark for compositional text-to-video generation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 8406–8416

    2025

  17. [25]

    RealUnify: Do unified models truly benefit from unification? A comprehensive benchmark,

    Y . Shi, Y . Dong, Y . Ding, Y . Wang, X. Zhu, S. Zhou, W. Liu, H. Tian, R. Wang, H. Wang, Z. Liu, B. Zeng, R. Chen, Q. Wang, Z. Zhang, X. Chen, C. Tong, B. Li, Q. Liu, H. Wang, W. Yang, Y . Zhang, P. Wan, Y .-F. Zhang, and Z. Liu, “RealUnify: Do unified models truly benefit f...

    2026

  18. [26]

    Uni-MMMU: A massive multi-discipline multimodal unified benchmark,

    K. Zou, Z. Huang, Y . Dong, S. Tian, D. Zheng, H. Liu, J. He, B. Liu, Y . Qiao, and Z. Liu, “Uni-MMMU: A massive multi-discipline multimodal unified benchmark,” inProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2...

    2026

  19. [27]

    Unison: Benchmarking unified multimodal models via synergistic understanding and generation,

    J. Liu, X. Shuai, H. Ding, and Y .-G. Jiang, “Unison: Benchmarking unified multimodal models via synergistic understanding and generation,” inInternational Conference on Machine Learning (ICML), 2026

    2026

  20. [28]

    Multi-dimensional evaluation of text summarization with in-context learning,

    S. Jain, V . Keshava, S. Mysore Sathyendra, P. Fernandes, P. Liu, G. Neubig, and C. Zhou, “Multi-dimensional evaluation of text summarization with in-context learning,” inFindings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguis...

    2023 doi

  21. [29]

    Can large language models be an alternative to human evaluations?

    C.-H. Chiang and H. yi Lee, “Can large language models be an alternative to human evaluations?” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. [Online]. Available: https://arxiv.org/abs/2305.01937

    2023 arXiv

  22. [30]

    Gptscore: Evaluate as you desire,

    J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “Gptscore: Evaluate as you desire,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024, pp. 6556–6576. [Online]. Available: https://aclant...

    2024

  23. [31]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  24. [32]

    Exploring the reliability of large language models as customized evaluators for diverse nlp tasks,

    Q. Li, L. Cui, L. Kong, and W. Bi, “Exploring the reliability of large language models as customized evaluators for diverse nlp tasks,” inProceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 10 325–10 344. [Online]. Available: https://aclant...

    2025

  25. [33]

    Au- tonomous evaluation and refinement of digital agents,

    J. Pan, Y . Zhang, N. Tomlin, Y . Zhou, S. Levine, and A. Suhr, “Au- tonomous evaluation and refinement of digital agents,” inProceedings of the First Conference on Language Modeling, 2024

    2024

  26. [34]

    A survey on large language model based autonomous agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Linet al., “A survey on large language model based autonomous agents,”Frontiers of Computer Science, vol. 18, no. 6, p. 186345, 2024

    2024

  27. [35]

    From generation to judgment: Opportunities and challenges of LLM-as-a-judge,

    D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y . Jiang, C. Chen, T. Wu, K. Shu, L. Cheng, and H. Liu, “From generation to judgment: Opportunities and challenges of LLM-as-a-judge,” inProceedings of the 2025 Conference on Empirical Methods in Natural ...

    2025

  28. [36]

    Agent-as-a-Judge,

    R. You, H. Cai, C. Zhang, Q. Xu, M. Liu, T. Yu, Y . Li, and W. Li, “Agent-as-a-Judge,”arXiv preprint arXiv:2601.05111, 2026

    2026

  29. [37]

    Agent-as-a-Judge: Evaluate agents with agents,

    M. Zhuge, C. Zhao, D. R. Ashley, W. Wang, D. Khizbullin, Y . Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y . Tian, Y . Shi, V . Chandra, and J. Schmidhuber, “Agent-as-a-Judge: Evaluate agents with agents,” in Proceedings of the 42nd International Conference on Machine Learning...

    2025

  30. [38]

    Adaptively profiling models with task elicitation,

    D. Brown, P. Balehannina, H. Jin, S. Havaldar, H. Hassani, and E. Wong, “Adaptively profiling models with task elicitation,” inProceedings of the PREPRINT 12 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 24 985–25 020

    2025

  31. [39]

    Mind2Web 2: Evaluating agentic search with Agent-as-a-Judge,

    B. Gou, Z. Huang, Y . Ning, Y . Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. Jiménez Gutiérrez, Y . Shu, C. H. Song, J. Wu, S. Chen, H. N. Moussa, T. Zhang, J. Xie, Y . Li, T. Xue, Z. Liao, K. Zhang, B. Zheng, Z. Cai, V . Rozgic, M. Ziyadi, H. Sun, and Y . Su, “Mind2Web 2: Evaluat...

    2025

  32. [40]

    One-Eval: An agentic system for automated and traceable LLM evaluation,

    C. Shen, Y . Hou, M. Pan, R. He, Z. H. Wong, M. Qiang, Z. Liu, H. Liang, P. Lai, Z. Sheng, and W. Zhang, “One-Eval: An agentic system for automated and traceable LLM evaluation,”arXiv preprint arXiv:2603.09821, 2026

    2026

  33. [41]

    AgenticEval: Toward agentic and self-evolving safety evaluation of large language models,

    Y . Wang, X. Wang, Y . Yao, X. Li, X. Yang, Y . Teng, X. Ma, and Y . Wang, “AgenticEval: Toward agentic and self-evolving safety evaluation of large language models,” inFindings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 14 789–14 808

    2026

  34. [42]

    Evaluating hallucination in text-to-image diffusion models with scene- graph based question-answering agent,

    Z. Qin, D. Cheng, H. Wang, H. Yi, Y . Shao, Z. Fan, K. Li, and Q. Lao, “Evaluating hallucination in text-to-image diffusion models with scene- graph based question-answering agent,”arXiv preprint arXiv:2412.05722, 2024

    2024 arXiv

  35. [43]

    A unified agentic framework for evaluating conditional image generation,

    J. Wang, X. Yang, L. Wang, Z. Xu, Y . Wang, Y . Wang, W. Luo, K. Zhang, B. Hu, and M. Zhang, “A unified agentic framework for evaluating conditional image generation,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2025

  36. [44]

    EdiVal-Agent: An object-centric framework for automated, fine- grained evaluation of multi-turn editing,

    T. Chen, Y . Zhang, Z. Zhang, P. Yu, S. Wang, Z. Wang, K. Lin, X. Wang, Z. Yang, L. Li, C.-C. Lin, J. Xie, O. Leong, L. Wang, Y . N. Wu, and M. Zhou, “EdiVal-Agent: An object-centric framework for automated, fine- grained evaluation of multi-turn editing,”arXiv preprint arXiv:...

    2026

  37. [45]

    RewardHarness: Self-evolving agentic post-training,

    Y . Zhang, P. Du, B. Li, C. Wei, J. Miao, H. Zhang, S. Cai, Y . Wang, D. Jiang, Y . Zhang, P. Nie, W. Chen, C. Yu, and K. R. Allen, “RewardHarness: Self-evolving agentic post-training,”arXiv preprint arXiv:2605.08703, 2026

    2026 arXiv

  38. [46]

    VideoGen-Eval: Agent-based system for video generation evaluation,

    Y . Yang, K. Fan, S. Sun, H. Li, A. Zeng, F. Han, W. Zhai, W. Liu, Y . Cao, and Z.-J. Zha, “VideoGen-Eval: Agent-based system for video generation evaluation,”arXiv preprint arXiv:2503.23452, 2025

    2025 arXiv

  39. [47]

    VQQA: An agentic approach for video evaluation and quality improvement,

    Y . Song, T. Pfister, and Y . Song, “VQQA: An agentic approach for video evaluation and quality improvement,”arXiv preprint arXiv:2603.12310, 2026

    2026

  40. [48]

    VideoArgus: Agentic rubric-grounded unified evaluation for video generation and editing,

    Z. Zeng, Z. Wang, Y . Yu, H. Hua, and J. Luo, “VideoArgus: Agentic rubric-grounded unified evaluation for video generation and editing,”arXiv preprint arXiv:2608.05485, 2026

    2026 arXiv

  41. [49]

    AgentRewardBench: Evaluating automatic evaluations of web agent trajectories,

    X. H. Lù, A. Kazemnejad, N. Meade, A. Patel, D. Shin, A. Zambrano, K. Sta ´nczak, P. Shaw, C. J. Pal, and S. Reddy, “AgentRewardBench: Evaluating automatic evaluations of web agent trajectories,” in Second Conference on Language Modeling, 2025. [Online]. Available: https://ope...

    2025

  42. [50]

    AJ-Bench: Benchmarking Agent-as-a-Judge for environment-aware evaluation,

    W. Shi, Y . Wang, Y . Zhao, Y . Chen, F. Feng, X. Hao, X. Su, Q. Gu, H. Su, X. Cai, and X. He, “AJ-Bench: Benchmarking Agent-as-a-Judge for environment-aware evaluation,” inFindings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 25 371–25 413

    2026

  43. [51]

    Time to REFLECT: Can we trust LLM judges for evidence-based research agents?

    L. Wang, Y . He, P. Chen, A. Yehudai, Y . Liu, R. Ying, M. Shmueli- Scheuer, and A. Cohan, “Time to REFLECT: Can we trust LLM judges for evidence-based research agents?”arXiv preprint arXiv:2605.19196, 2026

    2026 arXiv

  44. [52]

    Establishing best practices in building rigorous agentic benchmarks,

    Y . Zhu, T. Jin, Y . Pruksachatkun, A. Zhang, S. Liu, S. Cui, S. Kapoor, S. Longpre, K. Meng, R. Weiss, F. Barez, R. Gupta, J. Dhamala, J. Mer- izian, M. Giulianelli, H. Coppock, C. Ududec, A. Kellermann, J. Sekhon, J. Steinhardt, S. Schwettmann, A. Narayanan, M. A. Zaharia, I...

    2025

  45. [53]

    Webarena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Friedet al., “Webarena: A realistic web environment for building autonomous agents,” inAdvances in Neural Information Processing Systems, 2024

    2024

  46. [54]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments,

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Leiet al., “Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” inAdvances in Neural Information Processing Systems, vol. 37, 2024

    2024

  47. [55]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,

    J. Wang, H. Xu, J. Ye, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,” inICLR 2024 Workshop on Large Language Model Agents, 2024

    2024

  48. [56]

    Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web,

    R. Kapoor, Y . P. Butala, M. Russak, J. Y . Koh, K. Kamble, W. Alshikh, and R. Salakhutdinov, “Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web,” in Computer Vision – ECCV 2024, 2024, pp. 161–178

    2024

  49. [57]

    MMInA: Benchmarking multihop multimodal internet agents,

    S. Tian, Z. Zhang, L. Chen, and Z. Liu, “MMInA: Benchmarking multihop multimodal internet agents,” inFindings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 13 682–13 697

    2025

  50. [58]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  51. [59]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,”Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  52. [60]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations, 2023

    2023

  53. [61]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  54. [62]

    Algorithm of thoughts: Enhancing exploration of ideas in large language models,

    B. Sel, A. Al-Tawaha, V . Khattar, R. Jia, and M. Jin, “Algorithm of thoughts: Enhancing exploration of ideas in large language models,” in International Conference on Machine Learning, 2024

    2024

  55. [63]

    Graph of thoughts: Solving elaborate problems with large language models,

    M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyket al., “Graph of thoughts: Solving elaborate problems with large language models,” inProceedings of the AAAI Conference on Artificial Intelligence, ...

    2024

  56. [64]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” inInternational Conference on Learning Representa- tions, 2023

    2023

  57. [65]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  58. [66]

    Api-bank: A comprehensive benchmark for tool-augmented llms,

    M. Li, Y . Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y . Li, “Api-bank: A comprehensive benchmark for tool-augmented llms,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 3102–3116

    2023

  59. [67]

    Toolllm: Facilitating large language models to master 16000+ real-world apis,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qianet al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,” inInternational Conference on Learning Representations, 2024

    2024

  60. [68]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  61. [69]

    Gorilla: Large language model connected with massive apis,

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,” inAdvances in Neural Information Processing Systems, vol. 37, 2024

    2024

  62. [70]

    T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot,

    D. Jiang, Z. Guo, R. Zhang, Z. Zong, H. Li, L. Zhuo, S. Yan, P.-A. Heng, and H. Li, “T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot,”Advances in Neural Information Processing Systems, vol. 38, pp. 39 856–39 890, 2026

    2026

  63. [71]

    VChain: Chain-of-visual-thought for reasoning in video generation,

    Z. Huang, N. Yu, G. Chen, H. Qiu, P. Debevec, and Z. Liu, “VChain: Chain-of-visual-thought for reasoning in video generation,” inAnnual Meeting of the Association for Computational Linguistics (ACL Findings), 2026, 2026

    2026

  64. [72]

    τ 2-Bench: Evaluating conversational agents in a dual-control environment,

    V . Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan, “ τ 2-Bench: Evaluating conversational agents in a dual-control environment,”arXiv preprint arXiv:2506.07982, 2025

    2025 arXiv

  65. [73]

    An illusion of progress? assessing the current state of web agents,

    T. Xue, W. Qi, T. Shi, C. H. Song, B. Gou, D. Song, H. Sun, and Y . Su, “An illusion of progress? assessing the current state of web agents,” in Second Conference on Language Modeling, 2025. [Online]. Available: https://openreview.net/forum?id=6jZi4HSs6o

    2025

  66. [74]

    PaperBench: Evaluating AI’s ability to replicate AI research,

    G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan, “PaperBench: Evaluating AI’s ability to replicate AI research,” inProceedings of the 42nd International Conference on Machi...

    2025

  67. [75]

    Clipscore: A reference-free evaluation metric for image captioning,

    J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y . Choi, “Clipscore: A reference-free evaluation metric for image captioning,” inProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.