# Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond

**Authors:** Paul Owoicho, Ivan Sekulić, Mohammad Aliannejadi, Jeffrey Dalton, Fabio Crestani  
**arXiv:** [2304.13874](https://arxiv.org/abs/2304.13874)  
**Date:** 2023-04-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.13874v3/

## Key points

- Research focuses on assessing user feedback in mixed-initiative conversational search (CS) systems
- Proposed framework called ConvSim uses a user simulator to provide feedback and answer clarifying questions
- Effective utilization of user feedback leads to a 16% increase in retrieval performance
- Increasing the number of feedback rounds results in a 35% relative improvement after three rounds
- Over 30,000 transcripts of system-simulator interactions are released for further research
- Existing methods have shortcomings, but novel adaptations lead to improvements in recall and nDCG@3
- Incorporating answers to clarifying questions improves recall and nDCG@3
- Multiple rounds of simulator-system interactions enhance retrieval effectiveness
- Experimental framework based on simulated user-system interactions addresses challenges in CS systems

## Summary

This research focuses on exploring methods for assessing user feedback in mixed-initiative conversational search (CS) systems. While CS systems have made significant advancements, incorporating user feedback remains a challenge due to the lack of system-user conversational interaction data. To address this, the researchers propose a user simulator-based framework called ConvSim. This simulator is capable of providing feedback to system responses and answering clarifying questions. The experiments conducted on various state-of-the-art passage retrieval and neural re-ranking models show that effective utilization of user feedback can lead to a 16% increase in retrieval performance in terms of nDCG@3. Additionally, consistent improvements are observed as the number of feedback rounds increases, with a 35% relative improvement after three rounds. This highlights the need for specific feedback processing modules and presents opportunities for advancements in CS. To support further research in this area, the researchers release over 30,000 transcripts of system-simulator interactions based on established CS datasets. The study also identifies shortcomings in existing methods and proposes solutions. For example, standard T5 query rewriter struggles with processing feedback, but a novel adaptation of the T5 method leads to improvements in recall and nDCG@3 by 10% and 16%, respectively. Incorporating answers to clarifying questions also yields improvements in recall (18%) and nDCG@3 (12%). Furthermore, it is found that multiple rounds of simulator-system interactions result in further enhancements in retrieval effectiveness. To overcome the lack of research on feedback utilization and appropriate data, the researchers develop an experimental framework based on simulated user-system interactions. This framework allows for evaluating multiple mixed-initiative CS systems and addressing challenges such as contextual query resolution, asking clarifying questions, and incorporating user feedback. Overall, this study contributes to advancing the field of conversational search by demonstrating the benefits of utilizing user feedback and providing a comprehensive experimental framework for future research.