# TransformerG2G: Adaptive time-stepping for learning temporal graph embeddings using transformers

**Authors:** Alan John Varghese, Aniruddha Bora, Mengjia Xu, George Em Karniadakis  
**arXiv:** [2307.02588](https://arxiv.org/abs/2307.02588)  
**Date:** 2023-07-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.02588v1/

## Key points

- Graph embedding is a popular technique for analyzing complex graph-structured data
- Existing methods focus on static graphs and do not consider temporal dynamics
- The proposed model, TransformerG2G, incorporates long-range dependencies from the historical context of a dynamic graph
- Transformer encoder is used to learn intermediate node representations from current state and previous context of the graph
- Two projection layers generate lower-dimensional multivariate Gaussian distributions as latent embeddings for each node at each timestamp
- TransformerG2G outperforms conventional multi-step methods and prior work (DynG2G) in terms of link prediction accuracy and computational efficiency, particularly for high degrees of novelty
- Time-dependent attention weights reveal automatic adaptive time stepping enabled by the transformer
- Attention weights uncover temporal dependencies, identify influential elements, and provide insights into complex interactions within the graph structure
- Strong correlation found between attention weights and node degree at different stages of graph topology evolution
- TransformerG2G model with uncertainty quantification improves upon existing methods for dynamic graph embedding by capturing long-term dependencies and achieving better performance in link prediction tasks

## Summary

Graph embedding has become a popular technique for analyzing complex graph-structured data in various domains. However, most existing methods focus on static graphs and do not consider the temporal dynamics of evolving graphs. In this paper, we propose a graph embedding model called TransformerG2G that incorporates long-range dependencies from the historical context of a dynamic graph. We use the advanced transformer encoder to learn intermediate node representations from both the current state and previous context of the graph. Additionally, we employ two projection layers to generate lower-dimensional multivariate Gaussian distributions as latent embeddings for each node at each timestamp. We evaluate our model on diverse benchmarks with varying levels of "novelty" measured by TEA plots. Our experiments show that TransformerG2G outperforms conventional multi-step methods and our prior work (DynG2G) in terms of link prediction accuracy and computational efficiency, particularly for high degrees of novelty. Furthermore, we analyze the learned time-dependent attention weights across multiple graph snapshots, which reveal an automatic adaptive time stepping enabled by the transformer. By examining these attention weights, we can uncover temporal dependencies, identify influential elements, and gain insights into the complex interactions within the graph structure. For example, we find a strong correlation between attention weights and node degree at different stages of graph topology evolution. In summary, our proposed TransformerG2G model with uncertainty quantification improves upon existing methods for dynamic graph embedding by effectively capturing long-term dependencies and achieving better performance in link prediction tasks. The learned time-dependent attention weights provide valuable insights into temporal dynamics and interactions within evolving graphs.