Research Proposal: Measuring LLM Perplexity Scaling Laws Across Codebase Sizes for Safer Software
By
Gwern
Summary
This research proposal explores empirically measuring how coding LLM perplexity scales with codebase size to estimate scaling laws of 'predictability' across programming languages. Using Lean as a test case for formal languages, the author argues that languages with better scaling law exponents will become easier for LLMs to understand, fix, and write, potentially leading to safer and more secure software. The proposal suggests both expensive approaches (training from scratch) and cheap approaches (measuring perplexity over increasingly large context windows) to investigate this relationship.
Source
bskyResearch Proposal: Measuring LLM Perplexity Scaling Laws Across Codebase Sizes for Safer Softwaregwern.netKey quotes
· 3 pulledCodebases, and programming languages, which have better exponents in their scaling laws will eventually become easier for LLMs to understand, fix, and write.
Research idea: empirically measure the scaling of coding LLM perplexity over codebase size to estimate the scaling laws of 'predictability' by programming language or other factors.
We can measure this in contemporary LLMs expensively, by training from scratch and finetuning, or cheaply, by measuring perplexity over increasingly large context windows of source code.
You might also wanna read
Strategies for Scaling LLMs to Larger Codebases Through Guidance and Oversight
The article explores strategies for scaling Large Language Models (LLMs) to work effectively with larger codebases, focusing on the importan
blog.kierangill.xyz·6mo ago
Critical Analysis of LLM Limitations in Software Development
The article critiques the limitations and problems with Large Language Models (LLMs) and generative AI, using a personal experience with a n
deobald.ca·4mo ago
How AI Coding Tools Are Teaching New Lessons About Software Development Principles
The article explores how large language models (LLMs) and AI-driven coding workflows are revealing new insights about software development p
thefuriousopposites.com·3mo ago
Scaling Laws Limit Reliability of Large Language Models, Study Finds
This research paper demonstrates that the scaling laws governing large language models (LLMs) fundamentally limit their ability to improve p
arxiv.org·10mo ago
Testing LLM Effectiveness on Large Codebases: A 10,000+ Line Project Experience
The article challenges the common belief that Large Language Models (LLMs) become ineffective on larger codebases. The author shares their p
medium.com·10mo ago
Reflections on LLMs and Their Impact on Software Development Practices
The author shares personal reflections on the current state of Large Language Models (LLMs) and AI in software development, questioning whet
Comments
Sign in to join the conversation.
No comments yet. Be the first.