Abstract:
This paper reports on a between-subject, comparative online study of three
information visualization demonstrators that each displayed the same dataset
by way of an identical scatterplot technique, yet were different in style in
terms of visual and interactive embellishment. We validated stylistic
adherence and integrity through a separate experiment in which a small cohort
of participants assigned our three demonstrators to predefined groups of
stylistic examples, after which they described the styles with their own
words. From the online study, we discovered significant differences in how
participants execute specific interaction operations, and the types of
insights that followed from them. However, in spite of significant
differences in apparent usability, enjoyability and usefulness between the
style demonstrators, no variation was found on the self-reported depth,
expert-rated depth, confidence or difficulty of the resulting insights. Three
different methods of insight analysis have been applied, revealing how style
impacts the creation of insights, ranging from higher-level pattern seeking
to a more reflective and interpretative engagement with content, which is
what underlies the patterns. As this study only forms the first step in
determining how the impact of style in information visualization could be
best evaluated, we propose several guidelines and tips on how to gather,
compare and categorize insights through an online evaluation study,
particularly in terms of analyzing the concise, yet wide variety of insights
and observations in a trustworthy and reproducable manner.