IEEE VIS 2024 Content: Fine-Tuned Large Language Model for Visualization System: A Study on Self-Regulated Learning in Education

Fine-Tuned Large Language Model for Visualization System: A Study on Self-Regulated Learning in Education



Lin Gao - Fudan University, Shanghai, China

 Jing Lu - Fudan University, ShangHai, China

 Zekai Shao - Fudan University, Shanghai, China

 Ziyue Lin - Fudan University, Shanghai, China

 Shengbin Yue - Fudan unversity, ShangHai, China

 Chiokit Ieong - Fudan University, Shanghai, China

 Yi Sun - Fudan University, Shanghai, China

 Rory Zauner - University of Vienna, Vienna, Austria

 Zhongyu Wei - Fudan University, Shanghai, China

 Siming Chen - Fudan University, Shanghai, China

 Room: Bayshore V

2024-10-18T12:54:00ZGMT-0600Change your timezone on the schedule page
2024-10-18T12:54:00Z

Exemplar figure, described by caption below — In applying workflow to Self-Regulated Learning (SRL) in education, we outline the process in three phases. Phase 1 involves establishing a fundamental understanding of the SRL task (A1) and collecting data on artificial intelligence (A2). The design requirements (B) align with the design requirements. Phase 2 details the SRL pipeline sub-tasks and visualizations (C1), leading to the creation of fine-tuning data (C2). In phase 3, we enhance the fine-tuning effects and visualization interactions by integrating user feedback within the visualization system.

Fast forward

Full Video

Keywords

Fine-tuned large language model, visualization system, self-regulated learning, intelligent tutorial system

Abstract

Large Language Models (LLMs) have shown great potential in intelligent visualization systems, especially for domain-specific applications. Integrating LLMs into visualization systems presents challenges, and we categorize these challenges into three alignments: domain problems with LLMs, visualization with LLMs, and interaction with LLMs. To achieve these alignments, we propose a framework and outline a workflow to guide the application of fine-tuned LLMs to enhance visual interactions for domain-specific tasks. These alignment challenges are critical in education because of the need for an intelligent visualization system to support beginners' self-regulated learning. Therefore, we apply the framework to education and introduce Tailor-Mind, an interactive visualization system designed to facilitate self-regulated learning for artificial intelligence beginners. Drawing on insights from a preliminary study, we identify self-regulated learning tasks and fine-tuning objectives to guide visualization design and tuning data construction. Our focus on aligning visualization with fine-tuned LLM makes Tailor-Mind more like a personalized tutor. Tailor-Mind also supports interactive recommendations to help beginners better achieve their learning goals. Model performance evaluations and user studies confirm that Tailor-Mind improves the self-regulated learning experience, effectively validating the proposed framework.