Attention Residuals: A Drop-in Replacement for Standard Residual Connections in Transformers

Summary

Attention Residuals (AttnRes) is a novel architectural modification for Transformers that replaces standard residual connections with attention-based mechanisms. Instead of uniform additive accumulation of previous layer outputs, AttnRes enables each layer to selectively attend to and aggregate earlier representations through learned, input-dependent attention over depth. The approach comes in two variants: Full AttnRes where each layer attends over all previous outputs, and Block AttnRes which groups layers into blocks to reduce memory requirements from O(Ld) to O(Nd). This drop-in replacement aims to improve Transformer performance by allowing more sophisticated information flow between layers.