Abstract and research overview
Paper record · Original source linked above
Understanding the informative behaviour of deep neural networks is challenged by misused estimators and the complexity of network structure, which leads to inconsistent observations and diversified interpretation. Here we propose the LogDet estimator – a reliable matrix-based entropy estimator that approximates Shannon differential entropy. We construct informative measurements based on LogDet estimator, verify our method with comparable experiments and utilize it to analyse neural network behaviour. Our results demonstrate the LogDet estimator overcomes the drawbacks that emerge from highly diverse and degenerated distribution thus is reliable to estimate entropy in neural networks. The Network analysis results also find a functional distinction between shallow and deeper layers, which can help understand the compression phenomenon in the Information bottleneck theory of neural networks.
Recommended citation: Zhanghao Zhouyin and Ding Liu. “Understanding neural networks with logarithm determinant entropy estimator.” Neurocomputing 657, 131520 (2025).
This page is part of the author's maintained research bibliography. The text may be an author-written overview or an abstract from an earlier manuscript version; consult the linked publisher or preprint for the version of record.
Citation
Zhanghao Zhouyin and Ding Liu. "Understanding neural networks with logarithm determinant entropy estimator." Neurocomputing 657, 131520 (2025).