Contents Online

# Communications in Information and Systems

## Volume 10 (2010)

### Number 3

### Variational Bayes and a Problem of Reliable Communication I: Finite Systems

Pages: 155 – 182

DOI: http://dx.doi.org/10.4310/CIS.2010.v10.n3.a2

#### Authors

#### Abstract

This paper is the first in a two-part study of a variational Bayesian method and its application to a problem of reliable communication. The variational method expresses a Bayesian posterior distribution as the unique minimizer of a quantity dubbed *apparent information*. This has the same nature as the *free energy* of statistical mechanics. The minimum apparent information coincides with the *full information* of the observation.

Reliable communication over an error prone channel can be achieved by the use of random block coding, as originally proposed by Shannon. The *primary* Bayesian problem in this context, is that of estimating the transmitted block from observations of the output of the channel. Scaling limits for the various information quantities are derived for this problem; these show that the primary problem undergoes a second-order phase transition, in a very precise sense, at the channel capacity; the code rate is shown to play the role of absolute temperature.

Shannon's *reliability function* is recovered as the scaling limit of the full information of a *secondary* Bayesian problem, in which the channel noise and random code are estimated from the observation of a block decoding error. This secondary problem undergoes a third-order phase transition at a second critical code rate.