pith. sign in

arxiv: 1902.03570 · v1 · pith:KV7NYLNYnew · submitted 2019-02-10 · 💻 cs.AI · cs.CL· cs.CV· cs.LG

EvalAI: Towards Better Evaluation Systems for AI Agents

Deshraj Yadav , Rishabh Jain , Harsh Agrawal , Prithvijit Chattopadhyay , Taranjeet Singh , Akash Jain , Shiv Baran Singh , Stefan Lee
show 1 more author
Dhruv Batra
This is my paper
classification 💻 cs.AI cs.CLcs.CVcs.LG
keywords evalailearningmachineagentsartificialevaluatingintelligencemodels
0
0 comments X
read the original abstract

We introduce EvalAI, an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence algorithms (AI) at scale. EvalAI is built to provide a scalable solution to the research community to fulfill the critical need of evaluating machine learning models and agents acting in an environment against annotations or with a human-in-the-loop. This will help researchers, students, and data scientists to create, collaborate, and participate in AI challenges organized around the globe. By simplifying and standardizing the process of benchmarking these models, EvalAI seeks to lower the barrier to entry for participating in the global scientific effort to push the frontiers of machine learning and artificial intelligence, thereby increasing the rate of measurable progress in this domain.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. nuScenes: A multimodal dataset for autonomous driving

    cs.LG 2019-03 accept novelty 8.0

    nuScenes provides the first public autonomous-driving dataset that includes synchronized 360-degree data from cameras, radars, and lidar together with 3D bounding-box annotations across 1000 scenes.

  2. Whose Good, Whose Place? The Moral Geography of Agentic AI for Social Good

    cs.CY 2026-05 unverdicted novelty 7.0

    Survey of 112 agentic AI for social good papers reveals moral-geographic asymmetry with 73% lacking geographic context (lowest for SDG 16) and only 25% reporting deployments.

  3. MyoChallenge 2025: A New Benchmark for Human Athletic Intelligence

    cs.RO 2026-05 unverdicted novelty 6.0

    MyoChallenge 2025 introduces standardized table tennis and soccer tasks for musculoskeletal models in the MyoSuite simulation framework to benchmark athletic motor control algorithms.

  4. Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agents

    cs.HC 2025-09 unverdicted novelty 5.0

    Industry markets AI agents for orchestration, creation, and insight, but a usability study with 31 participants reveals users face challenges from capability misalignment and lack of meta-cognition in tools like Opera...