Modality-Agnostic Variational Compression of Implicit Neural Representations

In their paper titled "Modality-Agnostic Variational Compression of Implicit Neural Representations," authors Jonathan Richard Schwarz, Jihoon Tack, Yee Whye Teh, Jaeho Lee, and Jinwoo Shin introduce a novel neural compression algorithm that is modality-agnostic and based on a functional perspective of data. This algorithm is parameterized as an Implicit Neural Representation (INR) and addresses the gap between latent coding and sparsity by generating compact latent representations that are non-linearly mapped to a soft gating mechanism. By allowing for the specialization of a shared INR network to individual data items through subnetwork selection, the authors achieve improved performance in optimizing the rate/distortion trade-off in a modality-agnostic space using neural compression. The proposed approach, known as Variational Compression of Implicit Neural Representations (VC-INR), demonstrates superior performance compared to previous quantization schemes used for other INR techniques. Through experiments across various modalities such as images, climate data, 3D shapes and scenes, audio, and video, the authors showcase the effectiveness of VC-INR without relying on modality-specific biases. Notably, VC-INR surpasses well-known codecs like JPEG 2000, MP3, and AVC/HEVC in their respective modalities. Overall,this research presents a significant advancement in neural compression techniques by leveraging INRs and variational methods to achieve state-of-the-art results across diverse data types without the need for modality-specific adjustments.

Introduction: Neural compression is a rapidly growing field that aims to reduce the size of neural networks while maintaining their performance. This has become increasingly important as the use of deep learning models in various applications continues to grow, leading to larger and more complex networks. However, these large models require significant computational resources and storage space, making them difficult to deploy on resource-constrained devices such as mobile phones or IoT devices.

In their paper titled "Modality-Agnostic Variational Compression of Implicit Neural Representations," authors Jonathan Richard Schwarz, Jihoon Tack, Yee Whye Teh, Jaeho Lee, and Jinwoo Shin introduce a novel approach for neural compression that is modality-agnostic and based on a functional perspective of data. This new algorithm leverages implicit neural representations (INRs) and variational methods to achieve state-of-the-art results across diverse data types without the need for modality-specific adjustments.

Background: The traditional approach to neural compression involves quantizing weights or activations in a trained network. However, this method often leads to suboptimal results due to the gap between latent coding and sparsity. To address this issue, INRs have been proposed as an alternative representation for data that can be compressed efficiently without sacrificing performance.

INRs are parameterized by a shared network architecture that maps input data onto latent codes through non-linear transformations. These codes are then used as inputs into another network known as the decoder which reconstructs the original input from the latent code representation. This framework allows for efficient encoding and decoding of data while also providing flexibility in choosing different architectures for different modalities.

Methodology: The proposed approach in this paper, known as Variational Compression of Implicit Neural Representations (VC-INR), builds upon previous work on INRs by introducing a soft gating mechanism that enables specialization of a shared INR network to individual data items through subnetwork selection. This means that instead of having a single INR network for all data, VC-INR allows for the creation of multiple specialized subnetworks that can better represent specific types of data.

To achieve this, the authors introduce a variational compression framework that optimizes the rate/distortion trade-off in a modality-agnostic space using neural compression. This is done by incorporating an additional loss term into the training process that encourages sparsity in the latent codes while also minimizing reconstruction error. The resulting compressed representation is then used to reconstruct the original input through a decoder network.

Results: The authors evaluate their proposed approach on various modalities including images, climate data, 3D shapes and scenes, audio, and video. They compare VC-INR with other quantization schemes used for INRs and demonstrate its superior performance across all modalities without relying on modality-specific biases.

Notably, VC-INR outperforms well-known codecs like JPEG 2000, MP3, and AVC/HEVC in their respective modalities. For example, when compressing images from ImageNet dataset at different bitrates (0.1 - 2 bits per pixel), VC-INR achieves higher PSNR (Peak Signal-to-Noise Ratio) values compared to JPEG 2000. Similarly, when compressing audio from MusicNet dataset at different bitrates (16 - 256 kbps), VC-INR outperforms MP3 in terms of both PSNR and MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) scores.

Conclusion: In conclusion, "Modality-Agnostic Variational Compression of Implicit Neural Representations" presents a significant advancement in neural compression techniques by leveraging INRs and variational methods to achieve state-of-the-art results across diverse data types without relying on modality-specific adjustments. The proposed approach not only improves upon previous INR-based methods but also surpasses traditional codecs used for specific modalities. This research has the potential to greatly impact the field of neural compression and enable more efficient deployment of deep learning models in various applications.