publications

2025

1. Towards Unobtrusive Physical AI: Augmenting Everyday Objects with Intelligence and Robotic Movement for Proactive Assistance

   Violet Yinuo Han, Jesse T. Gonzalez, Christina Yang, Zhiruo Wang, Scott E. Hudson, and Alexandra Ion

   ACM UIST’25, Busan, Korea, Sep 2025

   
   

   Abstract DOI PDF Video Code

   Users constantly interact with physical, most often passive, objects. Consider if familiar objects instead proactively assisted users, e.g., a stapler moving across the table to help users organize documents, or a knife moving away to prevent injury as the user is inatten- tively about to lean against the countertop. In this paper, we build on the qualities of tangible interaction and focus on recognizing user needs in everyday tasks to enable ubiquitous yet unobtrusive tangible interaction. To achieve this, we introduce an architecture that leverages large language models (LLMs) to perceive users’ environment and activities, perform spatial-temporal reasoning, and generate object actions aligned with inferred user intentions and object properties. We demonstrate the system’s utility provid- ing proactive assistance with multiple objects and in various daily scenarios. To evaluate our system components, we compare our system-generated output for user goal estimation and object action recommendation with human-annotated baselines, with results indicating good agreement.

   What if everyday objects can become proactive robotic agents that offer us help when we need it?

2. Transforming Everyday Objects into Dynamic Interfaces using Smart Flat-Foldable Structures

   Violet Yinuo Han, Amber Yinglei Chen, Mason Zadan, Jesse T. Gonzalez, Dinesh K. Patel, Wendy Fangyu Yu, Carmel Majidi, and Alexandra Ion

   ACM UIST’25, Busan, Korea, Sep 2025

   
   

   Abstract DOI PDF Video

   Dynamic physical interfaces are often dedicated devices designed to adapt their physical properties to user needs. In this paper, we present an actuation system that allows users to transform their existing objects into dynamic physical user interfaces. We design our actuation system to integrate as a self-contained locomotion layer into existing objects that are small-scale, i.e., hand-size rather than furniture-size. We envision that such objects can act as col- laborators: as a studio assistant in a painter’s palette, as tutors in a student’s ruler, or as caretakers for plants evading direct sunlight. The key idea is to decompose the actuation into (1) energy input and (2) steering to achieve a flat form factor. The energy input is provided by simple vibration. We implement steering through differential friction controlled by flat-foldable compliant structures that can be activated electrically. We study the mechanism and its performance, and show its application scenarios enabling dynamic interactions with objects.

   What if there is a "motion layer" that can replace objects’ bottom surfaces, allowing them to move without changing their form and functionalities?

3. A Dynamic Bayesian Network Based Framework for Multimodal Context-Aware Interactions

   Violet Yinuo Han, Tianyi Wang, Hyunsung Cho, Kashyap Todi, Ajoy Savio Fernandes, Andre Levi, Zheng Zhang, Tovi Grossman, Alexandra Ion, and Tanya Jonker

   ACM IUI’25, Cagliari, Italy, Mar 2025

   
   

   Abstract DOI PDF Video

   Multimodal context-aware interactions integrate multiple sensory inputs, such as gaze, gestures, speech, and environmental signals, to provide adaptive support across diverse user contexts. Building such systems is challenging due to the complexity of sensor fusion, real-time decision-making, and managing uncertainties from noisy inputs. To address these challenges, we propose a hybrid approach combining a dynamic Bayesian network (DBN) with a large language model (LLM). The DBN offers a probabilistic framework for modeling variables, relationships, and temporal dependencies, enabling robust, real-time inference of user intent, while the LLM incorporates world knowledge for contextual reasoning. We demonstrate our approach with a tri-level DBN implementation for tangible interactions, integrating gaze and hand actions to infer user intent in real time. A user evaluation with 10 participants in an everyday office scenario showed that our system can accurately and efficiently infer user intentions, achieving 0.83 per frame accuracy, even in complex environments. These results validate the effectiveness of the DBN+LLM framework for multimodal context-aware interactions.

   How can a system efficiently infer user intents from multimodal inputs in real time and generalize to unseen contexts?

2024

1. 

   Robotic Metamaterials: A Modular System for Hands-On Configuration of Ad-Hoc Dynamic Applications

   Zhitong Cui, Shuhong Wang, Violet Yinuo Han, Tucker Rae-Grant, Willa Yunqi Yang, Alan Zhu, Scott E. Hudson, and Alexandra Ion

   ACM CHI’24, Honolulu, HI, May 2024

   
   

   Abstract DOI PDF Video

   We propose augmenting initially passive structures built from simple repeated cells, with novel active units to enable dynamic, shape-changing, and robotic applications. Inspired by metamaterials that can employ mechanisms, we build a framework that allows users to configure cells of this passive structure to allow it to perform complex tasks. A key benefit is that our structures can be repeatedly (re)configured by users inserting our configuration units to turn the passive material into, e.g., locomotion robots, integrated motion platforms, or interactive interfaces, as we demonstrate in this paper.

   What if users can author dynamic applications with modular robotic metamaterials?

2023

1. 

   BlendMR: A Computational Method to Create Ambient Mixed Reality Interfaces

    Best Paper Award

   Violet Yinuo Han, Hyunsung Cho, Kiyosu Maeda, Alexandra Ion, and David Lindlbauer

   ACM ISS’23, Pittsburgh, PA, Nov 2023

   
   

   Abstract DOI PDF Video Talk Code

   Mixed Reality (MR) systems display content freely in space, and present nearly arbitrary amounts of information, enabling ubiquitous access to digital information. This approach, however, introduces clutter and distraction if too much virtual content is shown. We present BlendMR, an optimization-based MR system that blends virtual content onto the physical objects in users’ environments for ambient information display. Our approach takes existing 2D applications and meshes of physical objects as input. It analyses the geometry of the physical objects and identifies regions that are suitable hosts for virtual elements. Using a novel integer programming formulation, our approach then optimally maps selected contents of the 2D applications onto the object, optimizing for factors such as importance and hierarchy of information, viewing angle, and geometric distortion. We evaluate BlendMR by comparing it to a 2D window baseline. Study results show that BlendMR decreases clutter and distraction, and is preferred by users. We demonstrate the applicability of BlendMR in a series of results and usage scenarios.

   What if future always-on MR interfaces can blend into our existing physical objects for ambient information display?

2. 

   Parametric Haptics: Versatile Geometry-Based Tactile Feedback Devices

   Violet Yinuo Han, Abena Boadi-Agyemang, Yuyu Lin, David Lindlbauer, and Alexandra Ion

   ACM UIST’23, San Francisco, CA, Nov 2023

   
   

   Abstract DOI PDF Video Talk

   Haptic feedback is important for immersive, assistive, or multimodal interfaces, but engineering devices that generalize across applications is notoriously difficult. To address the issue of versatility, we propose Parametric Haptics, geometry-based tactile feedback devices that are customizable to render a variety of tactile sensations. To achieve this, we integrate the actuation mechanism with the tactor geometry into passive 3D printable patches, which are then connected to a generic wearable actuation interface consisting of micro gear motors. The key benefit of our approach is that the 3D-printed patches are modular, can consist of varying numbers and shapes of tactors, and that the tactors can be grouped and moved by our actuation geometry over large areas of the skin. The patches are soft, thin, conformable, and easy to customize to different use cases, thus potentially enabling a large design space of diverse tactile sensations. In our user study, we investigate the mapping between geometry parameters of our haptic patches and users’ tactile perceptions. Results indicate a good agreement between our parameters and the reported sensations, showing initial evidence that our haptic patches can produce a wide range of sensations for diverse use scenarios. We demonstrate the utility of our approach with wearable prototypes in immersive Virtual Reality (VR) scenarios, embedded into wearable objects such as glasses, and as wearable navigation and notification interfaces. We support designing such patches with a design tool in Rhino.

   How can haptic experiences be customized, digitalized, shared, and easily integratable into diverse applications and even objects?

3. 

   Permeable Thermistor Temperature Sensors Based on Porous Melamine Foam

   Hugo Souza Oliveira, Niloofar Saeedzadeh Khaanghah, Violet Yinuo Han, Alejandro Carrasco-Pena, Alexandra Ion, Michael Haller, Giuseppe Cantarella, and Niko Münzenrieder

   IEEE Sensors Letters, Nov 2023

   
   

   Abstract DOI

   Flexible sensors and electronics have gained much attention in recent years. They are especially interesting due to their abilities to conform to static and dynamic surfaces while keeping their functionality. These characteristics make them relevant for a wide range of applications, from health care and fitness monitoring to soft robotics. In this work, we go beyond simple mechanical flexibility and present a lightweight and permeable flexible sensor utilizing melamine foam as a substrate. The foam is coated with metallic copper (Cu) and semiconductive Indium-Gallium-Zinc-Oxide (InGaZnO) to form a thermistor-type temperature sensor. The sensor showed a very stable response when cycling the temperature between 25 °C and 51 °C, exhibiting a maximum sensitivity of −01.6%∘C−1 , a permeability of 366.6gm−2h−1 at 24 °C, and a maximum resistance variation of −2.9%RH−1 when varying the relative humidity from 40% to 70%. The device also remained fully functional even after being bent to a radius of 5 mm.

   Lightweight permeable temperature sensors built on melamine foam for flexible wearable applications.