# MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

**Authors:** Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei  
**arXiv:** [2012.15828](https://arxiv.org/abs/2012.15828)  
**Date:** 2020-12-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2012.15828v1/

## Key points

- MiniLMv2 is a method for compressing pretrained Transformers through self-attention relation distillation.
- It generalizes the deep self-attention distillation technique used in MiniLM and focuses on distilling only the self-attention relations.
- Multi-head self-attention relations are defined as scaled dot-products between query, key, and value vectors within each self-attention module.
- MiniLMv2 allows for more flexibility by not restricting the number of attention heads in the student model.
- The fine-grained self-attention relations effectively utilize the interaction knowledge learned by Transformer models.
- Layer selection strategies for teacher models are explored beyond relying solely on the last layer as in MiniLM.
- Experimental results show that models distilled using MiniLMv2 outperform state-of-the-art methods when using base-size and large-size teachers like BERT and RoBERTa.
- MiniLMv2 provides an effective approach for compressing pretrained Transformers by distilling self-attention relations in natural language processing tasks.

## Summary

The paper presents a method called MiniLMv2 for compressing pretrained Transformers through self-attention relation distillation. The authors generalize the deep self-attention distillation technique used in MiniLM and focus on distilling only the self-attention relations for task-agnostic compression. They define multi-head self-attention relations as scaled dot-products between query, key, and value vectors within each self-attention module. This relational knowledge is then used to train the student model. One advantage of MiniLMv2 is its simplicity and unified principle. Unlike previous methods, there is no restriction on the number of attention heads in the student model, allowing for more flexibility. Additionally, the fine-grained self-attention relations effectively utilize the interaction knowledge learned by Transformer models. The paper also explores layer selection strategies for teacher models, going beyond relying solely on the last layer as in MiniLM. Experimental results demonstrate that models distilled from base-size and large-size teachers (such as BERT and RoBERTa) using MiniLMv2 outperform state-of-the-art methods. Overall, MiniLMv2 provides an effective approach for compressing pretrained Transformers by distilling self-attention relations. Its simplicity, flexibility in attention head numbers, and utilization of fine-grained interaction knowledge make it a promising method for task-agnostic compression in natural language processing tasks.