An Exploratory Pilot Study of the Effectiveness of XAI Visual Explanations for Novice Users

Abstract

Most visual analytics tools for Explainable AI (XAI) are designed for domain experts, leaving a substantial gap in accessibility for broader populations. We present an exploratory study evaluating how non-expert users interact with popular interpretability techniques: SHAP (SHapley Additive exPlanations), influence functions, and ``what-if'' analysis. Utilizing a highly engaging sports statistics dataset (NFL Hall of Fame prediction), we extract four core user design goals through initial interviews. We then develop and evaluate an interactive dashboard system. Our exploratory study suggests that novice users prefer tabular data views and low-friction, interactive ``what-if'' slider adjustments over complex, aggregated visualizations like Beeswarm plots. Furthermore, we identify critical barriers to trust when model logic conflicts with user intuition, offering concrete design implications for future non-expert XAI interfaces.