AI models can build apps, fix bugs and work independently with code. But how well do they handle real users' requests? A new study, Co-Design Bench, found a striking gap between coding agents' capabilities and the quality of their finished work. Even the best of the ten tested models, GPT-6 Astra, fully completed just 26.4% of tasks, although it met nearly 70% of individual requirements on average.
Researchers from Alibaba and several universities, including Fudan University and the Chinese University of Hong Kong, conducted the study. The paper Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models was published on October 26, 2026; its results attracted technology media attention on November 8.
The main finding: AI can do much of a programmer's work, but bringing it to a fully finished result is considerably harder.
How the coding agents were tested
Unlike many traditional coding tests, Co-Design Bench is based on real user requests. Researchers analyzed 26,314 sessions with coding tools, identified 37,463 unique queries and used them to create 273 test tasks. They favored difficult cases, such as requests that users repeatedly clarified or revised, or where unsuccessful changes were rolled back.
The tasks fell into two groups:
- Multimodal Editing (100 tasks): local changes to app interfaces and content, sometimes using screenshots, video, audio or documents.
- Repository Editing (173 tasks): more complex revisions to existing projects, including interface redesigns, new features and coordinated changes across components.
Experts wrote 2,331 acceptance criteria. They checked whether the result met the user's specific requirements, not simply whether code existed.
For example, a task could require changing a product price based on a document, recalculating dependent totals and keeping data correct on other pages. If the model changed the price but forgot the total, the task was incomplete. Another task involved a video editor: the agent had to respect clip limits, update the timeline and allow new clips after old ones were removed.
These requests go beyond “build a nice website”: the agent must write code and make the system work together.
Results: GPT-6 Astra leads, but finishes only a quarter of tasks
The researchers tested ten models in the same Claude Code agent environment. Each received a starting project and a task and could inspect files, change code and run the app. Each run had a limit of three hours and 120 agent steps.
Models ranked by fully completed tasks
| Model | Fully completed |
|---|---|
| GPT-6 Astra | 26.4% |
| Claude Opus 5 | 18.3% |
| Qwen3.8-Max | 16.5% |
| Kimi K3 | 15.8% |
| Qwen3.8-27B | 15.0% |
| GPT-5.6 Sol | 14.3% |
| GLM-5.3 Flash | 10.6% |
| Claude Opus 4.8 | 10.3% |
| GLM-5.2 | 9.5% |
| Grok 4.6 | 6.6% |
Source: Co-Design Bench, table 5. The figures show the share of tasks meeting every criterion. They are not a universal ranking of coding quality.
Even GPT-6 Astra left at least one requirement unmet in 73.6% of tasks. Claude Opus 5 finished fewer than one in five, and Grok 4.6 just 6.6%. That does not mean the remaining tasks failed completely.
GPT-6 Astra met an average of 69.9% of requirements per task, but satisfied every requirement in only 26.4% of tasks.
Imagine asking an AI to build a store page with ten required features. The agent implements seven and leaves three unfinished. A substantial amount of work is done, but the project is not ready for the customer. The researchers identify this gap between partial completion and finished delivery as a central challenge for today's agents.
More complex tasks leave more room for errors
Performance dropped particularly on coordinated changes to existing apps. GPT-6 Astra fully completed 41% of the more localized multimodal tasks, but only 17.9% of repository-level tasks.
The researchers also found that all ten models performed worse when a request combined changes to an app's structure, appearance, interactivity and content. Average completion of individual criteria was:
- 65.4% for tasks involving one or two types of change;
- 59.1% for tasks involving three or four types.
That is a difference of 6.3 percentage points. The problem is not necessarily writing a particular function; coordinating changes, preserving existing behavior and checking the whole result can be harder.
An agent might add a dark-mode switch but forget to save the choice after a reload, or build a menu that looks right on a computer but fails on a phone. A task can look finished while part of its functionality remains broken.
Why older benchmarks can show almost perfect results
On WebCompass-edit, modern agents can achieve near-perfect scores. For example, Claude Opus 4.8 reached 99.4% average completion of checked functions in an agentic run. In Co-Design Bench, even the leading model met 69.9% of requirements on average.
These figures cannot be compared directly because the tasks and evaluation methods differ. The contrast nevertheless shows how much the choice of benchmark affects the impression of AI capabilities. A standardized edit may be easier to assess than a real request with attachments, dependencies and constraints that are easy to miss. A high benchmark score does not guarantee that an agent can independently finish a complex commercial project.
The agent environment matters too
The researchers separately tested two models in three agent environments: Codex, OpenCode and Qwen Code. For Qwen3.8-27B, the share of requirements met on repository-editing tasks ranged from 56.0% to 74.9%, depending on the environment: almost 19 percentage points for the same model.
The result depends on more than the model itself. Available tools, file handling, command execution and verification all shape its work. In some settings, changing the environment even affected the relative ranking of the two models. Comparing model names alone is therefore not enough; the whole system matters.
Can we trust the results?
Co-Design Bench has limitations. First, tasks were selected with difficulty and previous user problems in mind. It would be wrong to claim that AI fails at 74% of all everyday programming tasks. The study describes a particular set of challenging web-app revisions.
Second, assessment was mostly performed by automated AI judges. One inspected source code and the running app; another challenged positive verdicts to find fake functionality, hard-coded answers and other shortcomings. Two frontend experts also checked 200 model-task cases, but automated assessment can still make mistakes.
Third, agents worked from a fixed request without a later conversation. In real development a user can clarify requirements, point out errors and request another attempt. The comparison also covered specific versions of ten models, not every current system, so its results cannot automatically be extended to other generations.
Does this mean AI cannot replace programmers yet?
The results suggest a change in a programmer's work rather than that AI agents are useless. Models can already perform a substantial part of development, but people still need to inspect the final result, find missed requirements and check how components interact.
A high share of completed requirements does not necessarily translate into the same amount of time saved. Fixing what remains could be a quick adjustment or difficult debugging across much of a project. The authors did not directly measure how long programmers would spend on fixes, so the 73.6% incomplete-task figure does not establish how often manual programming is required.
Additional verification remains necessary. The main lesson: agents can already do much of the work, but still struggle to meet every requirement reliably. Future tools may need better verification, debugging and completion of complex tasks as well as better code generation. Turning a partially finished result into a working product may become a key measure of an AI programmer's quality.