# A Markovian Formalism for Active Querying

**Authors:** Sid Ijju  
**arXiv:** [2306.08001](https://arxiv.org/abs/2306.08001)  
**Date:** 2023-06-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.08001v1/

## Key points

- Active learning algorithms have played a crucial role in recent advancements in artificial intelligence
- The research in this field lacks a comprehensive organizing framework
- The authors propose a Markovian formalism for active learning to capture the active learning process as a whole
- Various aspects of active learning, such as querying, dataset augmentation, and reward updates, can be viewed as transitions between meta-states within a Markovian system
- Two potential directions for further research are suggested: non-deterministic data addition using probability threshold parameter and non-deterministic changes to the learned reward function based on state changes
- Dataset tuning through modifying old examples using new data obtained through querying is proposed
- Many existing methods overlook specifying the reward space within the Markovian setup, which is crucial for developing truly intelligent robots.

## Summary

Active learning algorithms have played a crucial role in recent advancements in artificial intelligence. However, the research in this field lacks a comprehensive organizing framework. In this paper, the authors propose a Markovian formalism for active learning and survey existing literature to demonstrate the effectiveness of their proposed approach. Their formalism adopts a partially observable Markovian system perspective to capture the active learning process as a whole. The authors specifically outline how various aspects of active learning, such as querying, dataset augmentation, and reward updates, can be viewed as transitions between meta-states within a Markovian system. They argue that by considering these aspects as part of a larger system, it becomes possible to better understand and analyze the dynamics of active learning. To provide more context, the authors suggest two potential directions for further research. Firstly, they propose developing methods that add data to the dataset in a non-deterministic manner using a probability threshold parameter. This approach allows for consideration of the value of each query in terms of its contribution to the overall learning process. Secondly, they discuss the possibility of non-deterministically changing the learned reward function based on changes in the state. By representing the reward function as a distribution over possible outcomes instead of just one possibility, it becomes possible to capture uncertainty and variability in the learning algorithm's output. The authors also highlight an area for future exploration related to dataset tuning. Instead of simply adding or not changing examples in the dataset, they suggest modifying old examples using new data obtained through querying. For instance, reducing the labeled reward for queries deemed unfavorable by human experts could lead to more refined and accurate models. Lastly, they mention that many existing methods overlook specifying the reward space within this Markovian setup. They emphasize that explicitly defining and understanding this aspect is crucial for developing truly intelligent robots. In summary, this paper introduces a Markovian formalism for active learning and demonstrates its organizing capability through an analysis of existing literature.