The Additive–Transformative Gap: Why LLMs Cannot Safely Maintain Real Software Systems
By
Jh Evans
Master baker tier. Every paragraph earns its place on the tray.
Summary
This article argues that current LLMs and AI agents cannot safely modify or maintain real software systems, despite impressive code-generation demos. It introduces the concept of an "Additive–Transformative Gap," distinguishing between additive tasks (reading, mapping, planning, generating new code) which LLMs can handle, and transformative tasks (modifying, refactoring, debugging existing systems) which require causal reasoning about system behavior—something LLMs lack. The article contends that system-level software maintenance requires understanding causal structures and side effects, not just pattern-matching on training data.
Key quotes
· 3 pulledThe first three tasks are additive; the last three are transformative.
Applying new code is self-contained, additive work; modifying an existing system requires understanding its causal structure.
LLMs can generate code, but they cannot modify or maintain systems because system‑level work requires causal reasoning, not pattern‑matching.
You might also wanna read
Why Treating LLMs as Black-Box Problem Solvers Fails: Lessons from Processing 100 Compliance PDFs
The article discusses the author's experience transforming 100 messy compliance PDFs into structured JSON rules. It critiques the common app
towardsdatascience.com·4d ago
Study finds LLMs persist in treating false claims as true despite explicit warnings
A study on fine-tuning large language models (LLMs) reveals that even after explicit warnings that certain claims are false, the models cont
arstechnica.com·23h ago