# LIT: Block-wise Intermediate Representation Training for Model Compression

**Authors:** Animesh Koratana, Daniel Kang, Peter Bailis, Matei Zaharia  
**arXiv:** [1810.01937](https://arxiv.org/abs/1810.01937)  
**Date:** 2018-10-02  
**Summary page:** https://www.summarizepaper.com/en/paper/1810.01937v1/

## Key points

- Knowledge Distillation (KD) is a technique for reducing computational burden in deep network inference
- KD involves training a smaller student model using the output of a larger teacher model
- Hint training (FitNets) regresses intermediate representations of the student to those of the teacher
- Block-wise Intermediate Representation Training (LIT) extends the use of intermediate representations in deep network compression
- LIT surpasses KD and hint training in terms of performance
- LIT trains a student model with the same width as the teacher but with shallower depth by comparing their intermediate representations
- LIT uses previous block's representation from the teacher as input to current student block during training to avoid unstable representations
- LIT achieves significant reductions in network depth without sacrificing accuracy on CIFAR10 and Amazon Reviews datasets
- LIT outperforms KD and hint training in achieving smaller networks with comparable accuracy levels
- LIT improves accuracy of student models compared to corresponding teacher models on ResNet, ResNeXt, and VDCNN architectures
- LIT is effective in compressing GAN generators that output pixels instead of probabilities

## Summary

In the field of deep learning, Knowledge Distillation (KD) is a widely used technique for reducing the computational burden of deep network inference. It involves training a smaller and faster student model using the output of a larger teacher model. To further enhance this approach, hint training (also known as FitNets) was introduced, which focuses on regressing the intermediate representations of the student model to those of the teacher model. In this research, the authors propose a novel method called Block-wise Intermediate Representation Training (LIT) that extends the utilization of intermediate representations in deep network compression. LIT surpasses both KD and hint training in terms of performance. The technique incorporates two key ideas: 1) LIT trains a student model with the same width as the teacher but with shallower depth by directly comparing their intermediate representations; 2) LIT employs the intermediate representation from the previous block in the teacher model as an input to the current student block during training. This helps to avoid unstable intermediate representations in the student network. The experiments conducted demonstrate that LIT achieves significant reductions in network depth without sacrificing accuracy. For instance, on CIFAR10 dataset, LIT can compress a ResNeXt-110 model to a ResNeXt-20 model (5.5x compression), and on Amazon Reviews dataset, it can compress a VDCNN-29 model to a VDCNN-9 model (3.2x compression), all without any loss in accuracy. Moreover, LIT outperforms KD and hint training when it comes to achieving smaller networks with comparable accuracy levels. Additionally, applying LIT to identical student/teacher architectures leads to improved accuracy of the student models compared to their corresponding teacher models; outperforming another recent technique called Born Again Networks procedure on ResNet, ResNeXt and VDCNN architectures. Furthermore, unlike KD which primarily supports probability-based outputs; LIT proves to be effective in compressing GAN (Generative Adversarial Network) generators that output pixels instead of probabilities. Overall, the proposed LIT technique presents a promising approach for model compression by leveraging intermediate representations; achieving substantial reductions in network depth without sacrificing accuracy; and outperforming existing methods such as KD and hint training.