# Neuromorphic Visual Scene Understanding with Resonator Networks

**Authors:** Alpha Renner, Lazar Supic, Andreea Danielescu, Giacomo Indiveri, Bruno A. Olshausen, Yulia Sandamirskaya, Friedrich T. Sommer, E. Paxon Frady  
**arXiv:** [2208.12880](https://arxiv.org/abs/2208.12880)  
**Date:** 2022-08-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2208.12880v3/

## Key points

- Understanding a visual scene and inferring object identities and poses is a challenge
- Authors propose a neuromorphic solution using an efficient factorization network
- Vector Symbolic Architectures (VSA) framework uses complex-valued vectors for image models
- Hierarchical Resonator Networks (HRN) address non-commutative nature of translation and rotation in scenes
- Multi-compartment spiking phasor neuron model implements complex-valued resonator networks on neuromorphic hardware
- Synthetic scenes demonstrate successful inference of objects and poses
- Approach can be applied to real-world scenarios in machine vision and robotics

## Summary

Understanding a visual scene and accurately inferring the identities and poses of individual objects within it remains an ongoing challenge. In this paper, the authors propose a novel neuromorphic solution that leverages an efficient factorization network based on three key concepts. Firstly, they introduce a computational framework called Vector Symbolic Architectures (VSA) that utilizes complex-valued vectors. This framework employs vector binding operations to generate image models capable of representing geometric transformations. By treating binding as the equivariant operation for these transformations, scenes can be described as sums of vector products. Secondly, the authors design Hierarchical Resonator Networks (HRN) to address the non-commutative nature of translation and rotation in visual scenes when both are used together. The HRN enables the definition of a partitioned architecture where vector binding is equivariant for horizontal and vertical translation within one partition, and for rotation and scaling within another partition. Lastly, they develop a multi-compartment spiking phasor neuron model specifically designed to implement complex-valued resonator networks on neuromorphic hardware. This model allows for efficient mapping of the resonator network onto low-power neuromorphic hardware. To demonstrate their approach, synthetic scenes composed of simple 2D shapes undergoing rigid geometric transformations and color changes are used to show that their proposed method successfully infers objects and their poses in these scenes. Additionally, a companion paper showcases how this approach can be applied to real-world scenarios in machine vision and robotics. Overall, this research presents an innovative solution for visual scene understanding by combining Vector Symbolic Architectures with Hierarchical Resonator Networks and implementing them on neuromorphic hardware. The proposed method shows promise in accurately identifying objects and determining their poses in both synthetic and real-world environments.