Andon Labs tested GPT-6 Astra on two very different agent benchmarks. When buying inventory and running a vending machine, the OpenAI model crushes Claude Fable 5.1. On drone surveillance, Astra is the first model to beat the human-AI baseline on all five subtasks, though its success rate remains unreliable.

OpenAI’s GPT-6 Astra outperforms all previous frontier models on two agent benchmarks from Andon Labs. The research lab uses Vending-Bench and Drone-Bench to measure how well AI models act independently over long periods or write software for physical systems.

In a simulated vending machine business, Astra earned nearly three times as much as Claude Fable 5.1. On Drone-Bench, Andon Labs says Astra is the first model whose best attempts beat the human-AI-developed baseline across all five subtasks.

Astra negotiates harder and spends less than Claude Fable 5.1

In Vending-Bench, each model gets $500 and has to run a vending machine over a simulated year. It finds suppliers, negotiates purchase prices, orders goods, sets retail prices, and tries to grow its bank balance.

Across six runs, GPT-6 Astra averaged $15,515, according to Andon Labs. Claude Fable 5.1 averaged $5,422. Even Fable’s best run at $9,874 fell well short of Astra’s worst result of $13,272. Astra is the first OpenAI model to top the Vending-Bench 2 leaderboard. The gap to the second-place model is also the largest the benchmark has ever seen, according to Andon Labs.

One of the biggest differences shows up in procurement. Fable accepts worse deals over time. For a regular can of Coca-Cola, its average purchase price rises from $1.17 in the first 90 days to $2.21 toward the end of the simulated year. Astra negotiates more consistently. In one case Andon Labs documented, a supplier quoted $226.32 for a basket of goods. Astra held firm at $108 and got the deal.

Astra also handles unreliable suppliers better. Across six runs, Fable 5.1 made 45 prepayments to suppliers that had already shut down, losing $14,331. Astra encountered even more closures at 64, but Andon Labs says it recorded no identified losses from such prepayments. Fable recognized the problem and wrote a rule to only pay after written confirmation. Days later, the model broke its own rule.

Astra refuses price-fixing schemes

Andon Labs also tests models in Vending-Bench Arena, where multiple AI agents run competing vending machines at the same location. Astra explicitly refused a price-fixing proposal from the Chinese model GLM-5.3. Andon Labs observed no instances of lying from Astra across the three arena games it studied.

Claude Fable 5.1 participated in what Andon Labs classified as an illegal price-fixing arrangement with GLM-5.3. Fable only honored the agreement when it served its own interests. Astra won all three games.

Andon Labs rates Astra as both a stronger economic performer and better aligned, though that assessment is based on behaviors observed in the benchmark and doesn’t automatically transfer to other situations.

Astra is the first model to beat all five Drone-Bench tasks

Drone-Bench tests a different kind of agent capability. Models write code that lets a cheap DJI Tello EDU drone autonomously navigate an office, identify a specific person, and follow them. The benchmark has five steps: 3D reconstruction of the environment, drone localization, navigation, target person detection, and tracking.

Each task is scored individually against code that a human developer built with coding agents for Andon’s own demo. Every model gets ten runs per task and can submit up to ten code versions per run. After each attempt, it receives a score and can improve its solution.

In the original paper from July, Claude Fable 5 was the strongest model. Frontier models had beaten the human-AI baseline on four of five tasks in at least one run. 3D reconstruction remained unsolved. Andon Labs reported that Astra is the first model whose best submissions beat the baseline on all five Drone-Bench tasks, including reconstruction.

Astra built a pipeline combining COLMAP and DA3 with added depth filtering. The model used office video footage to generate a navigable 3D model that scored higher than the human-AI reference solution, according to Andon Labs.

Best-case scores don’t mean reliable performance

On person detection, Astra beats the baseline in four out of ten runs. On 3D reconstruction, it manages that in just one out of ten. Andon Labs calculates that an average Astra run has only a 2.8 percent chance of passing all five steps in sequence.

Astra proved for the first time that a general-purpose frontier model can produce code above the baseline for every part of the task. But multiply the probabilities for a complete end-to-end run, and the odds are still low. Based on progress over the past two years, the team projects that a frontier model could solve all five tasks in a single attempt by Q1 2027.

GPT-6 Astra already works as a surveillance drone

In a demo from Andon Labs, GPT-6 Astra flies a drone autonomously through an office with the prompt “ChatGPT, find this person and follow them.” The model identifies a specific person and tracks them. Spatial mapping, navigation, and person tracking all run without any human input. Other benchmarks also show that GPT-6 Astra has particularly strong spatial reasoning.

When critics questioned why they were building the kind of technology everyone keeps warning about, Andon Labs responded that the benchmark doesn’t help AI fly drones but measures how well current models can already do it. Six months ago, frontier models failed at these tasks and crashed. Astra now beats the human baseline on every subtask.

Andon Labs argues that the public and lawmakers need to know about these capabilities before AI-powered drones reach superhuman navigation skills. No lab has access to the benchmark. Andon Labs runs all evaluations itself to prevent companies from optimizing their models for the test.