PACE: A Framework for Predicting LLM Agent Performance Using Proxy Evaluations
By
Yueqi Song ,
Summary
This paper introduces PACE (Proxy for Agentic Capability Evaluation), a framework that predicts performance on expensive, time-consuming agentic benchmarks (like SWE-Bench and GAIA) by evaluating LLM agents on a small, carefully selected subset of atomic evaluation instances. The authors argue that non-agentic benchmarks testing individual capabilities (reasoning, code generation) are fast and cheap, and investigate whether they can serve as accurate proxies for costly agentic evaluations that can cost thousands of dollars and take days to complete.
Source
Twitter / XPACE: A Framework for Predicting LLM Agent Performance Using Proxy Evaluationshuggingface.coKey quotes
· 3 pulledEvaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure.
A single evaluation can cost thousands of dollars and take days to complete.
In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run.
You might also wanna read
Token Consumption Analysis in LLM-Based Multi-Agent Software Engineering Systems
This paper analyzes token consumption patterns in LLM-based Multi-Agent (LLM-MA) systems applied to software engineering tasks. Using the Ch
arxiv.org·1mo ago
SWE-CI: A Continuous Integration Benchmark for Evaluating LLM Agents' Code Maintenance Capabilities
Researchers propose SWE-CI, a new benchmark for evaluating LLM-powered software engineering agents' ability to maintain codebases over time
arxiv.org·4mo ago
Study Reveals "Constraint Decay" in LLM Agents for Backend Code Generation Under Structural Requirements
This paper presents a systematic study on how LLM agents handle structural constraints in multi-file backend code generation. The authors in
arxiv.org·1mo ago
GetProfile: Open-Source LLM Proxy for AI Agent Memory and User Profiles
GetProfile is an open-source LLM proxy that provides AI agents with persistent memory and structured user understanding capabilities. The to
Product Hunt·6mo ago
Redesigning Data Systems for LLM Agent Workloads: Addressing Agentic Speculation Challenges
This research paper argues that data systems need to be redesigned to better support LLM agents, which are predicted to become the dominant
arxiv.org·9mo ago
Comparing Code-Driven and LLM-Driven Workflows for Internal Agent Development
The article discusses the author's experience building internal agents and compares code-driven versus LLM-driven workflows. While initially
Comments
Sign in to join the conversation.
No comments yet. Be the first.