Cooperative Design Optimization through Natural Language Interaction

UIST2025
Ryogo Niwa*1,2Shigeo Yoshida1Yuki Koyama3,4Yoshitaka Ushiku1
1OMRON SINIC X Corporation2University of Tsukuba3National Institute of Advanced Industrial Science and Technology (AIST)4University of Tokyo* work done as an intern at OMRON SINIC X.
papervideo

TL;DR In this work, we propose the concept of cooperative design optimization through natural language. To realize this, we present a novel technique to integrate an LLM with the Bayesian Optimization (BO)-based optimization procedure, enabling designers to intervene through natural language and understand the system’s intentions behind its suggestions.

Overview

Designing successful interactions requires identifying optimal design parameters. To do so, designers often conduct iterative user testing and exploratory trial-and-error. This involves balancing multiple objectives in a high-dimensional space, making the process time-consuming and cognitively demanding. System-led optimization methods, such as those based on Bayesian optimization, can determine for designers which parameters to test next. However, they offer limited opportunities for designers to intervene in the optimization process, negatively impacting the designer’s experience. We propose a design optimization framework that enables natural language interactions between designers and the optimization system, facilitating cooperative design optimization. This is achieved by integrating system-led optimization methods with Large Language Models (LLMs), allowing designers to intervene in the optimization process and better understand the system's reasoning. Experimental results show that our method provides higher user agency than a system-led method and shows promising optimization performance compared to manual design. It also matches the performance of an existing cooperative method with lower cognitive load.

Video

Method

We integrate an LLM into the BO process to realize cooperative design optimization through natural language interaction. In each iteration, the system selects a single point (i.e., a set of parameter values) to be evaluated next with the help of an LLM from multiple candidates generated by a batch BO technique. In this way, we can integrate the designer’s request into the optimization process.

In each step, our system first samples 𝑞 candidates (𝑞 = 8 in this figure) using a technique called BO. Then, the LLM receives a prompt consisting of the task information, the designer’s request, the previously evaluated parameter-performance pairs, and the predicted performance of the candidates. Finally, it chooses the parameter set that best meets the designer’s request and provides its reason for that choice in natural language.

Results

User Study 1: Comparing Levels of Control

In User Study 1, we compared three conditions in a design optimization task: Designer-led (manual design), System-led (BO-led), and Cooperative (Natural Language), our proposed method. Eighteen participants with prior UI/UX experience experienced all three conditions. The results show that the Cooperative (Natural Language) condition preserves a higher sense of user agency compared to the BO-led condition, and shows promising optimization performance compared to manual design. These findings indicate that our proposed method can balance user agency and optimization performance.

Left figure (User agency): A box plot showing the distribution of agency scores for the three design optimization methods (Designer-led, BO-led, and Cooperative) in Study 1. The BO-led approach has the lowest median score, while both Designer-led and Cooperative exhibit similarly higher medians that cluster in the positive range. Designer-led and Cooperative each differ significantly from BO-led. Right figure (Optimization performance): A box plot comparing the relative hypervolume for three design optimization approaches: Designer-led, BO-led, and Cooperative in Study 1. Overall, Designer-led yields the lowest values, Cooperative the next highest, and BO-led the highest. The medians follow the same pattern, with Designer-led < Cooperative < BO-led. In the Cooperative condition, lower outliers are visible and shown as individual points. There is a statistically significant difference between the BO-led approach and both the Designer-led and Cooperative approaches.

User Study 2: Comparing Cooperation Approaches

In User Study 2, we compared two cooperative methods: Cooperative (Explicit Constraint), which required designers to specify constraints in the search space through a GUI, and Cooperative (Natural Language), our proposed method that allowed designers to flexibly make requests in natural language and receive explanations from the system. Twelve participants with prior UI/UX experience took part, and all of them experienced both conditions in a within-participant design. The results show that Cooperative (Natural Language) reduces NASA-TLX weighted scores and cognitive load, while achieving optimization performance comparable to Cooperative (Explicit Constraint).

Left figure (NASA-TLX scores): A box plot comparing the NASA-TLX (task load) scores in Study 2 for two conditions: Cooperative (Explicit Constraint) and Cooperative (Natural Language). The natural language condition shows significantly lower overall scores, including a lower median, and exhibits a narrower distribution of scores. Right figure (Optimization performance): A box plot comparing the relative hypervolume for two interface conditions in Study 2: Cooperative (Explicit Constraint) and Cooperative (Natural Language). The figure contrasts the efficiency of design optimization under these two approaches. While their medians are comparable, the Cooperative (Natural Language) condition exhibits a slightly higher overall distribution. The Cooperative (Explicit Constraint) condition has a somewhat lower median.

Acknowledgments

This work was supported by JST Moonshot R&D Program Grant Number JPMJMS2236.

Contact

contact@sinicx.com
Shigeo Yoshida

Citation

# arXiv version Link: https://arxiv.org/abs/2508.16077 BibTeX: (to appear)
# UIST 2025 version BibTeX: (to appear)
© 2024 OMRON SINIC X Corporation, all rights reserved.