Cost-Benefit Modeling of Interactive Visual Machine Learning Systems

Authors

Yixuan Wang (Arizona State Unversity), Rolando Garcia (Arizona State University), Alex Endert (Georgia Institute of Technology), Ross Maciejewski (Arizona State University)

Presentation

Session: Talk to my agent
Time: Tuesday, Nov 10, 10:00 – 10:12 (US/Eastern) · session 10:00 – 11:30
Location: Hall Essex center

Keywords

Visual analytics for machine learning, interactive visual machine learning, cost-beneﬁt analysis

Abstract

Machine learning systems often benefit from human involvement, but human-in-the-loop support is not free: teams may need to build specialized interfaces, analysts must spend time interacting with them, and some interactions trigger costly retraining or re-optimization. The central design question is therefore not simply whether interaction helps, but when it is worth the added effort relative to other interventions, such as improving the data, revising the model or training specification, or relying on automated workflows alone. In this paper, we build upon Van Wijk's foundational economic model of visualization, extend it by explicitly incorporating machine learning components, and present a cost-benefit framework for reasoning about these choices in interactive visual machine learning systems. We identify measurable quantities that govern practical trade-offs, including analyst time, interaction frequency, training latency, model improvement, downstream operational savings, and failure-prevention value. We organize these quantities into workflow-specific models for three common intervention targets: data iteration, model or specification iteration, and evaluating and understanding model behavior. We then instantiate the framework on representative systems to show how it supports break-even analysis, latency bounds, and stopping criteria for interactive analysis. The contribution is a practical decision framework for reasoning about when interactive visual tooling is likely to be justified, which intervention can provide value, and where human effort can be allocated more effectively across IVML workflows.

For Practitioners

This paper is highly relevant for machine learning engineers, data scientists, HCI researchers, and visual analytics designers. In practice, these teams can use the framework to figure out if adding a "human-in-the-loop" feature is actually worth the trouble. Building interactive tools takes time, and retraining models is computationally expensive, so this paper gives them a structured way to weigh those costs against the actual benefits. They can also use the workflow models to decide exactly where human feedback makes the most economic sense—whether that's cleaning up training data, steering the model's logic, or simply evaluating the outputs. Finally, the break-even and marginal efficiency analyses give teams a concrete way to set "stopping criteria." This helps them figure out exactly when to cut off interactive analysis, ensuring that analysts don't waste time and compute power chasing diminishing returns.