The shannon entropy of a bit $(p,1-p)$ is $$H(p)=-p\log(p)-(1-p)\log(1-p)$$. This is a well behaved function that uniquley assigns each state (up to permutation of its elements, i.e. $(p,1-p)\rightarrow (1-p,p)$ ) an entropy. Why is it then that i cannot find an inverse function that tells me $p$ when i know the $H(p)$ ? Many thanks!
$\endgroup$ 31 Answer
$\begingroup$For one thing, $H(p)$ is not invertible, since $H(p)=H(1-p)$. Both $H(0)=H(1)=0$, so what would define $H^{-1}(0)$ to be?
In theory this can be resolved by choosing a single branch of $H^{-1}$; you have two choices for each $H^{-1}$, so choose the greater one, for example. But finding the formula is still difficult. This is likely a transcendental equation, one whose solutions can't be expressed in terms of elementary functions. These happen a lot, for example, $\tan x=x$, $xe^x=1$, etc. Just because a function is easy to compute, there is no reason to believe its inverse should be just as easy.
$\endgroup$
