People have their own habits; from how they organize their workspace, over which mug they reach for in the morning, to how they prepare for cooking. Such preferences are context-dependent, dynamic, and often difficult to articulate. As embodied AI agents, such as robots, enter personal spaces, effective interaction requires understanding these implicit and evolving preferences; otherwise, such systems risk being more hindering than helpful. We propose an architecture that learns user preferences from real-world behavioral observation without requiring explicit instruction or feedback. We introduce a factor-graph-based user model that captures preferences across spatial arrangement, tool selection, and workflow, and continuously adapts as tasks and contexts change. The model expands with new observations, enabling open-ended, in situ learning of user-specific preference patterns. Our evaluation shows accurate, calibrated predictions across domains and robust adaptation over time. We demonstrate personalized robot assistance scenarios illustrating seamless and context-aware human–agent interaction.

Users constantly interact with physical, most often passive, objects. Consider if familiar objects instead proactively assisted users, e.g., a stapler moving across the table to help users organize documents, or a knife moving away to prevent injury as the user is inatten- tively about to lean against the countertop. In this paper, we build on the qualities of tangible interaction and focus on recognizing user needs in everyday tasks to enable ubiquitous yet unobtrusive tangible interaction. To achieve this, we introduce an architecture that leverages large language models (LLMs) to perceive users’ environment and activities, perform spatial-temporal reasoning, and generate object actions aligned with inferred user intentions and object properties. We demonstrate the system’s utility provid- ing proactive assistance with multiple objects and in various daily scenarios. To evaluate our system components, we compare our system-generated output for user goal estimation and object action recommendation with human-annotated baselines, with results indicating good agreement.

Dynamic physical interfaces are often dedicated devices designed to adapt their physical properties to user needs. In this paper, we present an actuation system that allows users to transform their existing objects into dynamic physical user interfaces. We design our actuation system to integrate as a self-contained locomotion layer into existing objects that are small-scale, i.e., hand-size rather than furniture-size. We envision that such objects can act as col- laborators: as a studio assistant in a painter’s palette, as tutors in a student’s ruler, or as caretakers for plants evading direct sunlight. The key idea is to decompose the actuation into (1) energy input and (2) steering to achieve a flat form factor. The energy input is provided by simple vibration. We implement steering through differential friction controlled by flat-foldable compliant structures that can be activated electrically. We study the mechanism and its performance, and show its application scenarios enabling dynamic interactions with objects.

Multimodal context-aware interactions integrate multiple sensory inputs, such as gaze, gestures, speech, and environmental signals, to provide adaptive support across diverse user contexts. Building such systems is challenging due to the complexity of sensor fusion, real-time decision-making, and managing uncertainties from noisy inputs. To address these challenges, we propose a hybrid approach combining a dynamic Bayesian network (DBN) with a large language model (LLM). The DBN offers a probabilistic framework for modeling variables, relationships, and temporal dependencies, enabling robust, real-time inference of user intent, while the LLM incorporates world knowledge for contextual reasoning. We demonstrate our approach with a tri-level DBN implementation for tangible interactions, integrating gaze and hand actions to infer user intent in real time. A user evaluation with 10 participants in an everyday office scenario showed that our system can accurately and efficiently infer user intentions, achieving 0.83 per frame accuracy, even in complex environments. These results validate the effectiveness of the DBN+LLM framework for multimodal context-aware interactions.

Mixed Reality (MR) systems display content freely in space, and present nearly arbitrary amounts of information, enabling ubiquitous access to digital information. This approach, however, introduces clutter and distraction if too much virtual content is shown. We present BlendMR, an optimization-based MR system that blends virtual content onto the physical objects in users’ environments for ambient information display. Our approach takes existing 2D applications and meshes of physical objects as input. It analyses the geometry of the physical objects and identifies regions that are suitable hosts for virtual elements. Using a novel integer programming formulation, our approach then optimally maps selected contents of the 2D applications onto the object, optimizing for factors such as importance and hierarchy of information, viewing angle, and geometric distortion. We evaluate BlendMR by comparing it to a 2D window baseline. Study results show that BlendMR decreases clutter and distraction, and is preferred by users. We demonstrate the applicability of BlendMR in a series of results and usage scenarios.

Haptic feedback is important for immersive, assistive, or multimodal interfaces, but engineering devices that generalize across applications is notoriously difficult. To address the issue of versatility, we propose Parametric Haptics, geometry-based tactile feedback devices that are customizable to render a variety of tactile sensations. To achieve this, we integrate the actuation mechanism with the tactor geometry into passive 3D printable patches, which are then connected to a generic wearable actuation interface consisting of micro gear motors. The key benefit of our approach is that the 3D-printed patches are modular, can consist of varying numbers and shapes of tactors, and that the tactors can be grouped and moved by our actuation geometry over large areas of the skin. The patches are soft, thin, conformable, and easy to customize to different use cases, thus potentially enabling a large design space of diverse tactile sensations. In our user study, we investigate the mapping between geometry parameters of our haptic patches and users’ tactile perceptions. Results indicate a good agreement between our parameters and the reported sensations, showing initial evidence that our haptic patches can produce a wide range of sensations for diverse use scenarios. We demonstrate the utility of our approach with wearable prototypes in immersive Virtual Reality (VR) scenarios, embedded into wearable objects such as glasses, and as wearable navigation and notification interfaces. We support designing such patches with a design tool in Rhino.