Research Study: Repository Context Files Reduce Coding Agent Performance and Increase Costs
By
mustaphah
Warm and crisp on the edges. A bagel with a bit of bite.
Summary
This research paper evaluates the effectiveness of repository-level context files (like AGENTS.md) for coding agents. The study examines whether these context files, which are commonly used to tailor coding agents to specific repositories, actually improve task completion performance. The researchers tested coding agents on established SWE-bench tasks and a novel collection of issues from repositories with developer-committed context files. Surprisingly, they found that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference costs by over 20%. The study reveals that context files encourage broader exploration but ultimately make tasks harder due to unnecessary requirements.
Key quotes
· 4 pulledAcross multiple coding agents and LLMs, we find that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference cost by over 20%.
Behaviorally, both LLM-generated and developer-provided context files encourage broader exploration (e.g., more thorough testing and file traversal), and coding agents tend to respect their instructions.
Ultimately, we conclude that unnecessary requirements from context files make tasks harder, and human-written context files should describe only minimal requirements.
Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks.
