T5Gemma 2: Next-Generation Encoder-Decoder Models with Multi-Modal Capabilities

Summary

T5Gemma 2 is the next generation of encoder-decoder models based on Gemma 3, featuring multi-modal and long-context capabilities. The model introduces architectural improvements including tied word embeddings across encoder and decoder, and merged decoder self- and cross-attention to reduce parameter count. It offers compact pre-trained models at sizes of 270M-270M (~3 billion parameters) and demonstrates strong performance across various benchmarks including natural language understanding, generation, and multi-modal tasks.