Can GPT-4V Detect Misleading Visualizations?
Jason Huang Alexander - University of Massachusetts Amherst, Amherst, United States
Priyal H Nanda - University of Masssachusetts Amherst, Amherst, United States
Kai-Cheng Yang - Northeastern University, Boston, United States
Ali Sarvghad - University of Massachusetts Amherst, Amherst, United States
Room: To Be Announced
Abstract
The proliferation of misleading visualizations online, particularly during critical events like public health crises and elections, poses a significant risk of misinformation. This work investigates the capability of GPT-4V to detect misleading visualizations. Utilizing a dataset of tweet-visualization pairs with various visual misleaders, we tested GPT-4V under four experimental conditions: naive zero-shot, naive few-shot, guided zero-shot, and guided few-shot. Our results demonstrate that GPT-4V can detect misleading visualizations with moderate accuracy without prior training (naive zero-shot) and that performance considerably improves by providing the model with the definitions of misleaders (guided zero-shot). However, combining definitions with examples of misleaders (guided few-shot) did not yield further improvements. This study underscores the feasibility of using large vision-language models such as GTP-4V to combat misinformation and emphasizes the importance of optimizing prompt engineering to enhance detection accuracy.