Unsupervised learning via maximizing mutual information in neural population coding

Shannon's information theory is a widely used tool for many applications including statistical inference, natural language processing, thermal physics, pattern recognition and neuroscience etc. However, it is a big challenge to effectively calculate the mutual information (MI) for these applications. In this paper, we propose an effective asymptotic bounds and approximation for evaluating MI in the context of neural population coding, especially for the case of high-dimensional inputs. Based on that, an unsupervised framework is presented to learn representations, e.g, complete, overcomplete or un-dercomplete bases from the input datasets. Our experiments on MNIST digits image and natural image patches data sets showed robustness and efficiency for extracting salient features compared to existing methods.