By Lindsey (11th Grade)
Since 2025, research teams have released seven benchmarks that test whether AI agents can build working games, not just short pieces of code. Most of them follow the same pipeline, shown below.
how a benchmark tests an AI · 5 steps
The AI agent never sees the hidden tests that grade its work, so it can’t tailor its code to pass them. The only way to score well is to produce a game that behaves correctly when it runs.
The table below compares the seven. Each benchmark uses its own tasks and grading method, so 50% on one is not equivalent to 50% on another. Compare scores within a benchmark, not across them.
| Benchmark | Released | Game tool | Tasks | What the AI must do | How it is graded | Best AI score | Code on GitHub |
|---|---|---|---|---|---|---|---|
| GameLogicBench | Sep 2026 | Godot engine | 72 | Implement game rules such as scoring and movement | Inspects the game’s state frame by frame as it runs | 52.8% | NJU-LINK/GameLogicBench |
| GameASG-Bench | Sep 2026 | Web browser | 47 | Build a complete browser game from a written spec | A scripted player sends keyboard input and checks how the game responds | 55.3% | areal-project/GameASG-Bench |
| GameXpert-Bench | Aug 2026 | Web browser | 214 | Build, debug and improve games over multiple rounds | Automated tests plus human ratings of visuals and game feel | 39.0 on fixing bugs (out of 100) | Not released yet (project page) |
| GameEngineBench | Jul 2026 | Unreal Engine (C++) | 110 | Add features to existing professional game projects | Code must compile; hidden tests then verify the feature | 55.5% | Not public |
| JAMER | Jun 2026 | Godot engine | 300 | Rebuild or complete entire game-jam projects | Runs the result and compares its behavior to the original game | 80.4% on small games, 5.7% on big ones | Not linked yet |
| GameDevBench | Feb 2026 | Godot engine | 333 | Tasks from online tutorials: animation, art, menus, gameplay | Automatic tests inside the game engine | 53.8% | waynchi/gamedevbench |
| V-GameGym | Sep 2025 | Pygame (Python) | 2,219 | Write a small playable game in Python | An AI judge scores the code, screenshots and gameplay video | About 45% | alibaba/SKYLENAGE-GameCodeGym |
Three features these benchmarks share:
- They use real game tools. Three use Godot, a free, open-source game engine, and two run in a web browser. GameEngineBench uses Unreal Engine, which many professional studios build with, and V-GameGym uses Python’s Pygame library.
- A computer does the grading. Most benchmarks run the game automatically and check the result against hidden tests, which makes scores reproducible. Two add other judges: V-GameGym uses an AI model as the grader, and GameXpert-Bench adds human ratings.
- They test more than one step. Newer benchmarks also ask the AI to find and fix bugs in an existing game, which is closer to how real development works.
