LiveBrowseComp reveals LLM search agents rely on memorized knowledge, not genuine web searching

Summary

This paper introduces the concept of Intrinsic Knowledge Dependence (IKD), showing that LLM-based search agents often rely on pre-trained knowledge rather than genuine web searching when answering questions on benchmarks like BrowseComp. Agents answer up to 44.5% of questions without tools and generate over half their search queries from internal hypotheses. To address this, the authors introduce LiveBrowseComp, a new benchmark with 335 human-authored questions based on facts published within 90 days, ensuring answers cannot be derived from model training data. On LiveBrowseComp, all agents score below 2% closed-book accuracy, and search-augmented scores drop 25-40 points compared to BrowseComp, revealing that static benchmarks conflate memory with genuine search capability.