# Distilling a Neural Network Into a Soft Decision Tree

**Authors:** Nicholas Frosst, Geoffrey Hinton  
**arXiv:** [1711.09784](https://arxiv.org/abs/1711.09784)  
**Date:** 2017-11-27  
**Summary page:** https://www.summarizepaper.com/en/paper/1711.09784v1/

## Key points

- Deep neural networks excel in scenarios with high-dimensional input data, complex input-output relationships, and a large number of labeled training examples.
- One challenge with deep neural networks is the lack of interpretability due to their reliance on distributed hierarchical representations.
- Frosst and Hinton propose distilling knowledge from trained neural networks into soft decision tree models to address the interpretability issue.
- The soft decision tree generated from the neural network generalizes better than traditional decision trees learned directly from training data.
- This approach offers a promising solution to enhance the interpretability of deep neural networks while maintaining high performance in classification tasks.

## Summary

In their paper titled "Distilling a Neural Network Into a Soft Decision Tree," authors Nicholas Frosst and Geoffrey Hinton discuss the effectiveness of deep neural networks in performing classification tasks. These networks excel in scenarios where the input data is high dimensional, the relationship between input and output is complex, and there are a large number of labeled training examples available. However, one challenge with deep neural networks is the difficulty in explaining why a particular classification decision is made on a specific test case. This lack of interpretability stems from the reliance of neural networks on distributed hierarchical representations. To address this issue, Frosst and Hinton propose a novel approach to distill the knowledge acquired by a trained neural network into a soft decision tree model. By expressing the learned knowledge in a model that relies on hierarchical decisions instead of distributed representations, they aim to make it easier to explain individual classification decisions. The soft decision tree generated from the neural network generalizes better than traditional decision trees learned directly from training data. This innovative method presented by Frosst and Hinton offers a promising solution to enhance the interpretability of deep neural networks while maintaining their high performance in classification tasks. The ability to distill complex neural network knowledge into more interpretable models like soft decision trees could have significant implications for various applications in machine learning and artificial intelligence.