# Learning Factored Representations in a Deep Mixture of Experts

**Authors:** David Eigen, Marc'Aurelio Ranzato, Ilya Sutskever  
**arXiv:** [1312.4314](https://arxiv.org/abs/1312.4314)  
**Date:** 2013-12-16  
**Summary page:** https://www.summarizepaper.com/en/paper/1312.4314v1/

## Key points

- Combines outputs from multiple expert networks specializing in different aspects of input space
- Key is training a "gating" network to assign inputs to specific expert distributions
- Potential for constructing larger and more efficient networks during testing
- Allows for parallelized training
- Stacked model with multiple sets of gating mechanisms and expert networks
- Exponentially increases effective experts by associating inputs with various combinations at different layers
- Maintains manageable model size
- Autonomously develops location-dependent ("where") experts at initial layer and class-specific ("what") experts at subsequent layers on MNIST dataset
- Effectively utilizes distinct combinations of experts on speech monophones dataset
- Showcases ability to learn complex representations based on varying inputs
- Enhances performance and enables nuanced understanding within deep learning architectures

## Summary

In their study titled "Learning Factored Representations in a Deep Mixture of Experts," authors David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever explore the concept of Mixtures of Experts. This approach involves combining outputs from multiple expert networks that specialize in different aspects of the input space. The key to this method is training a "gating" network that assigns each input to a specific distribution over these experts. It has shown potential for constructing larger and more efficient networks during testing while allowing for parallelized training. Building upon this foundation, the researchers introduce a novel extension known as the Deep Mixture of Experts. This stacked model incorporates multiple sets of gating mechanisms and expert networks, exponentially increasing the number of effective experts by associating each input with various combinations at different layers while maintaining a manageable model size. Through experimentation on a randomly translated version of the MNIST dataset, it was observed that the Deep Mixture of Experts autonomously develops location-dependent ("where") experts at the initial layer and class-specific ("what") experts at subsequent layers. Furthermore, when applied to a dataset featuring speech monophones, it became evident that distinct combinations of experts were effectively utilized by the model. This showcases the versatility and adaptability of the Deep Mixture of Experts in learning complex representations based on varying inputs. Overall, this research highlights how leveraging stacked models with multiple sets of gating and expert networks can enhance performance and enable more nuanced understanding within deep learning architectures.