Entropy modulo a prime

Building on work of Kontsevich, we introduce a definition of the entropy of a finite probability distribution in which the ‘probabilities’ are integers modulo a prime $p$. The entropy, too, is an integer $\operatorname{mod} p$. Entropy $\operatorname{mod} p$ is shown to be uniquely characterized by a functional equation identical to the one that characterizes ordinary Shannon entropy. We also establish a sense in which certain real entropies have residues $\operatorname{mod} p$, connecting the concepts of entropy over $\mathbb{R}$ and over $\mathbb{Z} / p \mathbb{Z}$. Finally, entropy $\operatorname{mod} p$ is expressed as a polynomial which is shown to satisfy several identities, linking into work of Cathelineau, Elbaz–Vincent and Gangl on polylogarithms.

Keywords

entropy, $p$-derivation, information loss, modular arithmetic, fundamental equation of information theory, Faddeev’s theorem

2010 Mathematics Subject Classification

Primary 94A17. Secondary 11A07, 11A99, 11T06, 13N15.

Full Text (PDF format)

The author was supported by a Leverhulme Trust Research Fellowship.

Received 28 March 2019

Accepted 27 November 2020

Published 18 June 2021