ORBIT-Q: A new benchmark for evaluating autonomous coding agents in scientific quantum programming
By
[Submitted on 3 Jul 2026]
Summary
ORBIT-Q (Open Research Benchmark for Integrated Tasks in Quantum Computing) is a new dual-axis benchmark designed to evaluate autonomous coding agents in scientific quantum programming. Unlike conventional programming benchmarks, ORBIT-Q tests whether generated code preserves physical fidelity, differentiable workflows, framework-native semantics, and scalable representations. It combines a multi-tier verification pipeline supporting two comparisons: different agent configurations on a fixed quantum framework, and different quantum frameworks on a fixed agent. Evaluations found TensorCircuit-NG (TC) to be the most capable framework, and Codex with GPT-5.5 the strongest agent configuration on TC. However, a significant gap remains between autonomous agents and human expert implementations. The benchmark also evaluates agent-side resource use and artifact-side runtime efficiency.
Source
bskyORBIT-Q: A new benchmark for evaluating autonomous coding agents in scientific quantum programmingarxiv.orgKey quotes
· 5 pulledAutonomous coding agents perform well on many conventional programming tasks, but scientific computing demands a rigorous validation paradigm that extends beyond simple functional test completion
ORBIT-Q contributes a carefully curated suite of complex, research-level quantum workflows that serves as a challenging testbed for modern scientific programming
A significant performance and design gap remains between frontier autonomous agents and human expert reference implementations
ORBIT-Q combines a rigorous multi-tier verification pipeline to support two orthogonal comparisons
TensorCircuit-NG (TC) exhibits the highest capability and performance efficiency among the evaluated quantum software frameworks under agent-driven programming
You might also wanna read
New Benchmark Reveals High Rates of Outcome-Driven Constraint Violations in Autonomous AI Agents
Researchers introduce a new benchmark for evaluating autonomous AI agents' safety, specifically focusing on outcome-driven constraint violat
arxiv.org·4mo ago
The Evolution of AI: From Static Benchmarks to Inference-Time Search for Autonomous Agents
The article explores the shift from traditional AI benchmarking to inference-time search as the future of AI development. It discusses how c
adlrocha.substack.com·6mo ago
Red Queen Gödel Machine: An Evolutionary Framework for Self-Improving AI with Dynamic Evaluation
This paper introduces the Red Queen Gödel Machine (RQGM), an evolutionary framework for recursive self-improvement of AI agents under non-st
arxiv.org·8d agoRed Queen Gödel Machine: An Evolutionary Framework for Self-Improving AI with Dynamic Evaluation
This paper introduces the Red Queen Gödel Machine (RQGM), an evolutionary framework for recursive self-improvement of AI agents under non-st
arxiv.org·8d agoACM Transactions on Quantum Computing (tqc) Vol 7, Issue 2
SkillsBench: A Benchmark for Evaluating AI Agent Skills Across Diverse Tasks
SkillsBench is a new benchmark for evaluating how well AI agent skills work across diverse tasks. The benchmark includes 86 tasks across 11
arxiv.org·4mo ago
ProgramBench: New Benchmark Reveals Language Models Struggle to Build Complete Software Projects From Scratch
This paper introduces ProgramBench, a new benchmark designed to evaluate the ability of language model-based software engineering agents to
arXiv.org·2mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.