Search Results for « Wasserstein Auto-Encoder (WAE) »

We propose the Wasserstein Auto-Encoder (WAE)---a new algorithm for building a generative model of the data distribution. WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational Auto-Encoder (VAE). This regularizer encourages the encoded training distribution to match the prior. We compare our algorithm with several other techniques and show that it is a generalization of adversarial auto-encoders (AAE). Our experiments show that WAE shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality, as measured by the FID score.

We study the role of latent space dimensionality in Wasserstein auto-encoders (WAEs). Through experimentation on synthetic and real datasets, we argue that random encoders should be preferred over deterministic encoders. We highlight the potential of WAEs for representation learning with promising results on a benchmark disentanglement task.

In recent years, the use of deep learning approaches in healthcare has yielded promising results in a variety of fields, most notably in the detection of adverse drug reactions (ADRs) and drug recommendations. This paper promises a breakthrough in this field by using Wasserstein autoencoders (WAEs) for personalized medicine recommendation and ADR detection. WAEs' capacity to manage complex data distributions and develop meaningful latent representations makes them ideal for modeling heterogeneous healthcare data. This study intends to improvise the precision and efficiency of drug recommendation systems while also improving patient safety by combining WAEs and early ADR detection strategies. Previous research has used social media data for pharmacovigilance, drug repositioning, and other machine learning algorithms to detect ADRs. However, our proposed methodology offers a novel perspective by combining Wasserstein autoencoders with ADR detection methods, outperforming existing approaches. Preliminary results show that the proposed methodology surpasses current methodologies, with much greater accuracy in ADR identification and medicine recommendation. In particular, the proposed model achieves an ADR detection accuracy of 96.04%, which is 15% higher than the most sophisticated techniques, with considerable improvements in precision, recall, and accuracy metrics. In conclusion, our study seeks to develop customized medicine in healthcare, perhaps leading to dramatically improved patient outcomes and safety.

We are concerned with a worst-case scenario in model generalization, in the sense that a model aims to perform well on many unseen domains while there is only one single domain available for training. We propose a new method named adversarial domain augmentation to solve this Out-of-Distribution (OOD) generalization problem. The key idea is to leverage adversarial training to create "fictitious" yet "challenging" populations, from which a model can learn to generalize with theoretical guarantees. To facilitate fast and desirable domain augmentation, we cast the model training in a meta-learning scheme and use a Wasserstein Auto-Encoder (WAE) to relax the widely used worst-case constraint. Detailed theoretical analysis is provided to testify our formulation, while extensive experiments on multiple benchmark datasets indicate its superior performance in tackling single domain generalization.

The maximum mean discrepancy (MMD) has found numerous applications in statistics and machine learning, among which is its use as a penalty in the Wasserstein auto‐encoder (WAE). In this paper, we compute closed‐form expressions for estimating the Gaussian kernel‐based MMD between a given distribution and the standard multivariate normal distribution. This formula reveals a connection to the Baringhaus–Henze–Epps–Pulley (BHEP) statistic of the Henze–Zirkler test and provides further insights about the MMD. We introduce the standardized version of MMD as a penalty for the WAE training objective, allowing for a better interpretability of MMD values and more compatibility across different hyperparameter settings. Next, we propose code normalization–using batch normalization at the code layer–which has the benefits of making the kernel width selection easier, reducing the training effort, and preventing outliers in the aggregate code distribution. Our experiments on synthetic and real data show that the analytic formulation improves over the commonly used stochastic approximation of the MMD and demonstrate that code normalization provides significant benefits when training WAEs.

Non-adversarial generative models such as variational auto-encoder (VAE), Wasserstein auto-encoders with maximum mean discrepancy (WAE-MMD), sliced-Wasserstein auto-encoder (SWAE) are relatively easy to train and have less mode collapse compared to Wasserstein auto-encoder with generative adversarial network (WAE-GAN). However, they are not very accurate in approximating the target distribution in the latent space because they don't have a discriminator to detect the minor difference between real and fake. To this end, we develop a novel non-adversarial framework called Tessellated Wasserstein Auto-encoders (TWAE) to tessellate the support of the target distribution into a given number of regions by the centroidal Voronoi tessellation (CVT) technique and design batches of data according to the tessellation instead of random shuffling for accurate computation of discrepancy. Theoretically, we demonstrate that the error of estimate to the discrepancy decreases when the numbers of samples $n$ and regions $m$ of the tessellation become larger with rates of $\mathcal{O}(\frac{1}{\sqrt{n}})$ and $\mathcal{O}(\frac{1}{\sqrt{m}})$, respectively. Given fixed $n$ and $m$, a necessary condition for the upper bound of measurement error to be minimized is that the tessellation is the one determined by CVT. TWAE is very flexible to different non-adversarial metrics and can substantially enhance their generative performance in terms of Fréchet inception distance (FID) compared to VAE, WAE-MMD, SWAE. Moreover, numerical results indeed demonstrate that TWAE is competitive to the adversarial model WAE-GAN, demonstrating its powerful generative ability.

In the field of image generation, especially for auto-encoder models, how to extract better features and obtain better quality reconstruction samples by modifying network structure and training algorithms has always been the focus of attention. For example, Variational Auto-Encoder (VAE), which is a very popular auto-encoder model, is a theoretically rigorously derived image generation model. For commonly used auto-encoders, such as VAE, Regularized Auto-Encoder (RAE), and Wasserstein Auto-Encoder (WAE), the mean square error (MSE) is all used as the loss function of the reconstructed process, which makes the blur problem of the reconstruction samples unavoidable. Especially for larger-sized images, the blur phenomenon is more obvious. To solve this problem, Perceptual Loss Function is used in some cases. Although it can improve the image quality to a certain extent, the amount of calculation is large and the image quality improvement in auto-encoder is also relatively limited. For this reason, we try to propose a new loss function, Wavelet loss function, to better generate and reconstruct images. Wavelet transform is applied to the reconstructed image loss function of the auto-encoder, and the frequency characteristics of the decomposed image are used to constrain it. We conducted comparative experiments on two larger-size image datasets (FaceSrub, COIL20) and a small-size image dataset (Fashion_MNIST), and proved the effectiveness of the wavelet loss function. At the same time, we propose a new image quality index: wavelet high-frequency signal-to-noise ratio (WHF-SNR), which can better measure the quality of the reconstructed image of the auto-encoder.

This article presents a computational framework for the Wasserstein auto-encoding of merge trees (MT-WAE), a novel extension of the classical auto-encoder neural network architecture to the Wasserstein metric space of merge trees. In contrast to traditional auto-encoders which operate on vectorized data, our formulation explicitly manipulates merge trees on their associated metric space at each layer of the network, resulting in superior accuracy and interpretability. Our novel neural network approach can be interpreted as a non-linear generalization of previous linear attempts (Pont et al. 2023) at merge tree encoding. It also trivially extends to persistence diagrams. Extensive experiments on public ensembles demonstrate the efficiency of our algorithms, with MT-WAE computations in the orders of minutes on average. We show the utility of our contributions in two applications adapted from previous work on merge tree encoding (Pont et al. 2023). First, we apply MT-WAE to merge tree compression, by concisely representing them with their coordinates in the final layer of our auto-encoder. Second, we document an application to dimensionality reduction, by exploiting the latent space of our auto-encoder, for the visual analysis of ensemble data. We illustrate the versatility of our framework by introducing two penalty terms, to help preserve in the latent space both the Wasserstein distances between merge trees, as well as their clusters. In both applications, quantitative experiments assess the relevance of our framework. Finally, we provide a C++ implementation that can be used for reproducibility.

Learning diverse and high-performance behaviors from a limited set of demonstrations is a grand challenge. Traditional imitation learning methods usually fail in this task because most of them are designed to learn one specific behavior even with multiple demonstrations. Therefore, novel techniques for quality diversity imitation learning, which bridges the quality diversity optimization and imitation learning methods, are needed to solve the above challenge. This work introduces Wasserstein Quality Diversity Imitation Learning (WQDIL), which 1) improves the stability of imitation learning in the quality diversity setting with latent adversarial training based on a Wasserstein Auto-Encoder (WAE), and 2) mitigates a behavior-overfitting issue using a measure-conditioned reward function with a single-step archive exploration bonus. Empirically, our method significantly outperforms state-of-the-art IL methods, achieving near-expert or beyond-expert QD performance on the challenging continuous control tasks derived from MuJoCo environments.

To address the challenges in learning deep generative models (e.g.,the blurriness of variational auto-encoder and the instability of training generative adversarial networks, we propose a novel deep generative model, named Wasserstein-Wasserstein auto-encoders (WWAE). We formulate WWAE as minimization of the penalized optimal transport between the target distribution and the generated distribution. By noticing that both the prior $P_Z$ and the aggregated posterior $Q_Z$ of the latent code Z can be well captured by Gaussians, the proposed WWAE utilizes the closed-form of the squared Wasserstein-2 distance for two Gaussians in the optimization process. As a result, WWAE does not suffer from the sampling burden and it is computationally efficient by leveraging the reparameterization trick. Numerical results evaluated on multiple benchmark datasets including MNIST, fashion- MNIST and CelebA show that WWAE learns better latent structures than VAEs and generates samples of better visual quality and higher FID scores than VAEs and GANs.