Research
ORO AI’s research on simulation environments and benchmarks for AI that makes economic decisions: SMB Bench, ORO Bench, AssistantEval and ShoppingBench.
oroagents.comCrawled Nov 14, 2026
ORO AI’s research on simulation environments and benchmarks for AI that makes economic decisions: SMB Bench, ORO Bench, AssistantEval and ShoppingBench.
AssistantEval gives assistant apps and bare frontier models the same 34 everyday tasks in simulated services. Claude Opus 5.5 passed 80%; the best app, Pally, passed 69%.
CandyBench evaluates the frontier capabilities in candy distribution. Three robots, one candy station, Halloween 2026: about 580 completed handoffs (95% CI 422–738) from 41 hand-annotated clips.
The same shopping agents average 0.71 on practice tasks and 0.18 on hidden ones. ORO Bench compiles new hidden tasks from a frozen 11.1M-product catalog and grades the one order each agent places from the environment’s own record.
ShoppingBench, created by Alibaba Group (Wang et al., 2025), ran as Bittensor SN15’s benchmark. ORO post-trained Qwen3-4B on the subnet’s trajectories; it solves 42.7% of 75 held-out problems, up from 18.0% for the base model.
SMB Bench is an AI agent benchmark: a frontier model runs a simulated wholesale buying desk for a 90-day quarter, scored on profit against a fixed buying rule.
Same model, same 90-day quarter. Switch on one delayed consequence and its profit falls from $9,401 to $1,186. What SMB Bench shows about AI running a business.
Anthropic demonstrated that buyers cannot detect when an AI agent overpays on their behalf.
ORO's trajectory paper on Subnet 15 as a source of training data: distilled Qwen3-4B results on ShoppingBench, plus the released filter, corpus splits and arena mechanics.
Research write-ups and updates from ORO AI, an applied AI research lab.
The company tested six frontier models and three assistant apps on 34 everyday tasks, with Claude Opus 5.5 completing 80% of its runs.
The leaderboard initially included agents that failed to shop, highlighting the need for genuine reasoning in competition.
The ORO Bench competition allows participants to build shopping agents and earn Bittensor emissions on Subnet 15.
ORO builds high-fidelity simulation environments grounded in real data to teach AI the tacit knowledge required to run safe and profitable autonomous businesses.