# Improving Contextual Congruence Across Modalities for Effective Multimodal Marketing using Knowledge-infused Learning

**Authors:** Trilok Padhi, Ugur Kursuncu, Yaman Kumar, Valerie L. Shalin, Lane Peterson Fronczek  
**arXiv:** [2402.03607](https://arxiv.org/abs/2402.03607)  
**Date:** 2024-02-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.03607v1/

## Key points

- Study focuses on predicting success of multimodal marketing campaigns on crowdfunding platforms
- Integration of common sense knowledge into Visual Language Models (VLMs)
- Dataset includes pairs of images and text with binary labels for campaign success
- Goal is to determine likelihood of campaign reaching funding goal within specific timeline
- Framework employs modular and flexible text and image encoders
- Pretrained BERT, RoBERTa for text and ViT, ResNet for image encoders fine-tuned using bidirectional transformers
- Knowledge retrieval involves generating text captions for images using multimodal LVMs, with BLIP outperforming other models
- Clustering analysis shows impact of external knowledge on semantic relationships between modalities
- t-SNE visualizations demonstrate denser clusters with closer centroids when external knowledge included, reducing semantic distance between modalities
- Semantic similarity between text and image modalities increases by approximately 9.9% with external knowledge inclusion
- Research aims to improve prediction accuracy and advance marketing theory through early detection of persuasive multi-modal campaigns on crowdfunding platforms

## Summary

This study addresses the challenge of predicting the success of multimodal marketing campaigns on crowdfunding platforms by integrating explicit common sense knowledge into large Visual Language Models (VLMs). The dataset consists of pairs of images and text with binary labels indicating campaign success. The goal is to determine the likelihood of a campaign reaching its funding goal within a specified timeline. To enhance VLM performance, a modular and flexible framework for text and image encoders is employed. Pretrained text (e.g., BERT, RoBERTa) and image encoders (e.g., ViT, ResNet) are jointly fine-tuned using bidirectional transformers. Vision encoders such as ResNet-152 and Vision Transformers are experimented with, producing output vectors for each image. Knowledge retrieval involves generating text captions for images using multimodal LVMs, with BLIP performing better than other models. Clustering analysis is conducted over text and image captions to demonstrate how external knowledge impacts semantic relationships between modalities. t-SNE visualizations show that including external knowledge results in denser clusters with closer centroids, indicating reduced semantic distance between modalities. Semantic similarity between text and image modalities increases by approximately 9.9% when external knowledge is included. This research not only aims to improve prediction accuracy but also contributes to advancing marketing theory through early detection of persuasive multi-modal campaigns and assessment of marketing strategies on crowdfunding platforms.