In information theory, it is well known that for a Markov Chain, \(X_1 \rightarrow X_2 \rightarrow X_3\) , if the number of states in \(X_1\) is k and the number of states in \(X_2\) is m with \(m < k\) , then \(I(X_1;X_3) \le \log m\) . Thus, a bottleneck limits the amount of mutual information between the input to the Markov chain a random variable later than the bottleneck. With this in mind, a theoretical framework for analyzing deep neural networks called the Information Bottleneck has been proposed, where the desire is to have the simplest possible representation that captures all of the relevant information. The simple representation is what produces the information bottleneck.