# BERT-DRE: BERT with Deep Recursive Encoder for Natural Language Sentence Matching

**Authors:** Ehsan Tavan, Ali Rahmati, Maryam Najafi, Saeed Bibak, Zahed Rahmati  
**arXiv:** [2111.02188](https://arxiv.org/abs/2111.02188)  
**Date:** 2021-11-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.02188v2/

## Key points

- The paper presents a deep neural architecture for Natural Language Sentence Matching (NLSM) to identify semantic similarity in input text pairs.
- NLSM models predict a category or scale value for a pair of input texts to indicate their similarity or relationship.
- Two main steps are used: designing a model to obtain proper representation of the text and developing a matching mechanism to extract complex interactions.
- The authors collected a Persian Religious question matching dataset containing 18,000 samples with two questions, appropriate answers, and match or not-match labels for each question pair.
- BERT-DRE model was implemented by adding recursive encoder module to BERT and achieved an F1-score of 90.27% on the test data when trained and evaluated using annotated religious dataset making it the strongest model among those studied.
- Deep neural networks are preferred approach in NLSM field and researchers use convolutional and recurrent neural networks to determine semantic features and relationships among sentences.
- BERT-DRE outperforms its predecessor achieving an accuracy score of 90.29% when employing proposed architecture on same dataset.

## Summary

This paper presents a deep neural architecture for Natural Language Sentence Matching (NLSM) that aims to identify the semantic similarity of input text pairs. The objective of NLSM models is to predict a category or scale value for a pair of input texts, which indicates their similarity or relationship. To achieve this goal, NLSM models generally use two main steps: designing a model to obtain a proper representation of whatever text will be analyzed so that it can extract semantic features from it, and by using the representation obtained from texts, developing a matching mechanism to extract complex interactions. In this study, the authors collected a Persian Religious question matching dataset containing 18,000 samples with two questions, appropriate answers, and match or not-match labels for each question pair. The dataset was used for designing a chatbot by crawling religious questions from religious question-answering websites and annotating two similar questions for each question and generating dissimilar questions automatically. They implemented BERT with Deep Recursive Encoder (BERT-DRE) model by adding recursive encoder module to BERT (Devlin et al., 2019). Three Bi-LSTM layers with residual connection were used to design a recursive encoder and an attention module was used on top of this encoder. To obtain the final vector, a pooling layer consisting of average and maximum pooling was used. The authors evaluated BERT-DRE with related models using the introduced religious and benchmark datasets such as SNLI, FarsTail, MultiNLI, SciTail. It was noted that their BERT-DRE model achieved an F1-score of 90.27% on the test data when trained and evaluated using annotated religious dataset making it the strongest model among those studied. Furthermore, in order to better evaluate the BERT-DRE model's performance on other datasets such as SNLI, MultiNLI, FarsTail, and SciTail they achieved appropriate F1-scores. The authors investigated related models to the field of NLSM and found that deep neural networks are the preferred approach. To determine semantic features and relationships among sentences, researchers use convolutional and recurrent neural networks. The structure of convolutional neural networks (CNNs) make them very capable of extracting local features while recurrent neural networks can extract temporal features by considering natural language texts as a sequence of words. The Long Short Term Memory (LSTM), a variant of RNN’s is capable of extracting long-term dependency in an appropriate way. In conclusion, this paper focuses on improving BERT results in NLSM task by adding deep recursive encoder to BERT so called BERT with Deep Recursive Encoder (BERT-DRE). The authors experiment their model on four benchmarks: SNLI; FarsTail; MultiNLI; SciTail; as well as novel Persian religious questions dataset. Comparisons between BERT-DRE and BERT are conducted showing that in all cases BERT-DRE outperforms its predecessor - achieving an accuracy score 89.70% using only the original algorithm versus 90.29% when employing proposed architecture on same dataset..