ReverseEOL: A simple method to improve text embeddings by reversing input text in decoder-only LLMs
By
[Submitted on 4 Jun 2026]
A weekday bagel. Dependable, satisfying, no fuss.
Summary
This paper introduces ReverseEOL (Reverse prompting with Explicit One-word Limitation), a method to improve training-free text embeddings from decoder-only Large Language Models. The key insight is that causal attention in decoder-only LLMs prevents earlier tokens from accessing future context, creating biased representations. ReverseEOL addresses this by augmenting standard forward embeddings with an additional reversed embedding derived from reversed input text. This reversal exposes each token to previously inaccessible context, providing complementary information. Combining forward and reversed embeddings yields richer final representations. Experiments on STS and MTEB benchmarks show significant improvements across diverse LLM architectures and scales.
Key quotes
· 5 pulledReverseEOL augments the standard forward embedding with an additional reversed embedding derived from the reversed input text.
Since reversing the input exposes each token to context inaccessible in the original order, the resulting reversed embedding effectively provides complementary information to the original one.
Combining the forward and reversed embeddings yields a richer final representation.
Comprehensive experiments on STS and MTEB benchmarks demonstrate that ReverseEOL significantly improves the performance of existing training-free baselines across a broad range of LLMs with diverse architectures and scales.
Extensive ablations and analyses further confirm the necessity of our reversal mechanism.
You might also wanna read
LLM-Deflate: Reversing Model Training to Extract Structured Datasets from Large Language Models
LLM-Deflate is a novel technique that reverses the training process of Large Language Models by systematically extracting structured dataset
scalarlm.com·8mo ago
Zebra-Llama: Efficient Hybrid Language Models Combining SSMs and Attention Layers
Researchers propose Zebra-Llama, a family of hybrid language models (1B, 3B, 8B) that combine State Space Models (SSMs) and Multi-head Laten
arxiv.org·6mo ago
Research Proves Transformer Language Models Are Injective and Invertible
This research paper challenges the conventional view that transformer language models are non-injective due to non-linear components. The au
arxiv.org·7mo ago
ChunkLLM: A Lightweight Framework for Accelerating Large Language Model Inference
ChunkLLM is a lightweight, pluggable framework designed to accelerate large language model inference by addressing computational inefficienc
arxiv.org·7mo ago
Fast-dLLM: Training-Free Acceleration Method for Diffusion Language Models Using KV Cache and Parallel Decoding
Researchers introduce Fast-dLLM, a training-free acceleration method for diffusion-based large language models that addresses their slower i
arxiv.org·7mo ago
Ouro: Looped Language Models That Build Reasoning into Pre-Training Through Latent Space Iteration
Researchers introduce Ouro, a family of pre-trained Looped Language Models (LoopLM) that build reasoning capabilities directly into the pre-
arxiv.org·5mo ago