Back to the index. Or to the chambers

This article has 9 links. View as Cloud or List.

Continuous Measure Relation Theory Distribution Normal Random Variable Covariance Matrix Random Vector Shannons Theorem Entropy Joint Normal Distribution

1	Measure
1	Continuous
1	Relation Theory
3	Distribution
4	Normal Random Variable
6	Random Vector
8	Covariance Matrix
10	Shannons Theorem Entropy
11	Joint Normal Distribution

Loading ...

Planetmath Browser (2008—2009)

BSD licence | A django site

All articles taken from PlanetMath.org snapshot under CC-BY-SA licence.

→ The original article on PlanetMath.org

Other Formats: LaTeX

New! You can click on formulas to copy the LaTeX source to your clipboard. | Math Videos

Differential Entropy

Let $(X, \mathfrak{B}, \mu)$ be a probability space, and let $f \in L^p(X, \mathfrak{B}, \mu)$ , $\vert\vert f\vert\vert _{p} = 1$ be a function. The differential entropy is defined as

$\displaystyle h(f) \equiv -\int_{X} \vert f\vert^p \log \vert f\vert^p\ d\mu$

(1)

Differential entropy is the continuous version of the Shannon entropy, $H[\mathbf{p}] = -\sum_{i} p_i \log p_i$ . Consider first , the uniform 1-dimensional distribution on . The differential entropy is

$\displaystyle h(u_a) = -\int_{0}^{a} \frac{1}{a} \log \frac{1}{a}\ d\mu = \log a.$

(2)

Next consider probability distributions such as the function

$\displaystyle g = \frac{1}{2 \pi \sigma}e^{-\frac{(t-\mu)^2}{2 \sigma^2}},$

(3)

the 1-dimensional Gaussian. This pdf has differential entropy

$\displaystyle h(g) = -\int_{\mathbb{R}} g \log g\ dt = \frac{1}{2} \log 2 \pi e \sigma^2.$

(4)

For a general -dimensional Gaussian $\mathcal{N}_{n}(\mathbf{\mu},\mathbf{K})$ with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{K}$ , $K_{ij} = \mathrm{cov}(x_i, x_j)$ , we have

$\displaystyle h(\mathcal{N}_{n}(\mathbf{\mu},\mathbf{K})) = \frac{1}{2} \log (2 \pi e)^n \vert\mathbf{K}\vert$

(5)

where $\vert\mathbf{K}\vert = \det{\mathbf{K}}$ .