Mutual Coupling-Aware Channel Estimation for Holographic MIMO Systems: A Vector Factorization Design Paradigm

Holographic multiple-input and multiple-output (HMIMO) systems have emerged as a promising technology for enabling precise wavefront control and achieving fine-grained spatial resolution thanks to their capability to manipulate electromagnetic fields. However, the dense antenna configurations may lead to severe mutual coupling (MC) effects, which significantly complicates channel state information acquisition. To address this challenge, this paper investigates channel estimation for HMIMO systems with explicit consideration of MC effects. Specifically, we first derive a MC model based on multiport theory. It is shown that the Toeplitz structure of MC matrix utilized in conventional MIMO systems does not persist in HMIMO systems. As a result, existing channel estimation algorithms cannot be directly applied to HMIMO systems. To bridge this gap, we propose an approximate unitary diagonalization representation for the MC matrix via plane decomposition, which significantly reduces the number of parameters to be estimated. Moreover, by leveraging the wavenumber-domain sparsity of the HMIMO channel, we formulate the MC-aware channel estimation as a vector factorization (VF) problem within a variational Bayesian inference framework, where Markov chain and pattern-coupling priors are incorporated to capture the statistical characteristics of both the wavenumber-domain channel and the MC coefficients. To effectively resolve the dependencies induced by the structured priors, we further propose a bilinear VF-tailored hybrid message passing (BiVF-HMP) algorithm, which exploits the power of variational inference, belief propagation, and the first-order optimality. Simulation results validate the effectiveness of the proposed BiVF-HMP algorithm and highlight the importance of considering MC effects for channel estimation.