IEEE VIS 2024 Content: Position paper: Proposing the use of an “Advocatus Diaboli” as a pragmatic approach to improve transparency in qualitative data analysis and reporting

Position paper: Proposing the use of an “Advocatus Diaboli” as a pragmatic approach to improve transparency in qualitative data analysis and reporting



Judith Friedl-Knirsch - University of Applied Sciences Upper Austria, Hagenberg, Austria

 Room: Bayshore I

2024-10-14T12:30:00ZGMT-0600Change your timezone on the schedule page
2024-10-14T12:30:00Z

Exemplar figure, described by caption below — A sketch of the Advocatus Diaboli process for qualitative data analysis. First, the primary researcher analyses the collected data. Then a secondary researcher assumes the position of an Advocatus Diaboli and attempts to disprove the findings of the primary researcher based on the collected data. Finally, both researchers discuss the findings of the Advocatus Diaboli and adapt the results if necessary.

Full Video

Abstract

Qualitative data analysis is widely adopted for user evaluation, not only in the Visualisation community but also related communities, such as Human-Computer Interaction and Augmented and Virtual Reality. However, the data analysis process is often not clearly described and the results are often simply listed in the form of interesting quotes from or summaries of quotes that were uttered by study participants. This position paper proposes an early concept for the use of a researcher as an “Advocatus Diaboli”, or devil’s advocate, to try to disprove the results of the data analysis by looking for quotes that contradict the findings or leading questions and task designs. Whatever this devil’s advocate finds can then be used to reiterate on the findings and the analysis process to form more suitable theories. On the other hand, researchers are enabled to clarify why they did not include this in their theory. This process could increase transparency in the qualitative data analysis process and increase trust in these findings, while being mindful of the necessary resources.