Ongoing Research

Personalized interactions in the IOT Space

Ongoing research focuses on the relationships between people and the smart spaces made possible by the Internet of Things (IoT). While existing deployments of the IoT are often controlled by enterprises (e.g., building owners, municipalities, etc.), the increasing pervasiveness of user-facing smart devices opens the possibility of a more personalizable world. These devices range from lights, locks, thermostats, and smoke detectors to window shades, refrigerators, printers, stop lights, and cameras. We start with the assumption that such devices are embedded in everyday environments. They can be fixed (e.g., installed on the walls of a room), moveable (e.g., attached to furniture), or mobile (e.g., carried by a person or vehicle). We ask a simple yet fundamental question: what does an individual expect from such an environment? In the most basic terms, the answer is itself simple: one expects natural, responsive interactions with the available functionality. Since the system is assumed to be "smart," one might also expect the quality of the experience to improve over time.

This reserch is in collaboration with Christine Julien at UT Austin and involves students at both institutions.

Mobile Computing Laboratory (MobiLab)

MobiLab, located next to my office, is an experimental playground for testing out ideas about interacting with the IoT space. The latest addition to the lab is a set of smart glasses. Apps are being developed to allow users to control devices by simply reading a QR codes that redirects the smart glasses to uploading a custom interface for the device the user is looking at.

• Algorithms for controlling spatial characteristics

In the smart spaces made possible by recent advances in IoT (Internet of Things) technology, users control the space by interfacing with individual devices reachable via a wireless network. Some of the interactions are narrowly focused on features of a particular device (turning on the TV) while others indirectly control properties of the space (a thermostat affects the ambient temperature). This investigaion is are primarily concerned with interactions that affect the user’s perception of the space, as opposed to invoking particular device features. Thus, we introduce the concept of spatial characteristics (e.g., illumination, security) to refer to properties of the surrounding space that are of interest to the user, and which are generally controlled by an aggregate set of devices. This leads us to explore the notion of spatial characteristics from a pragmatic perspective, provide an appropriate formal specification for it, and examine the range of algorithmic solutions required to enable user control over spatial characteristics without explicit reference to devices in the space.

• Conflict prediction in Smart Spaces

In a smart space influenced by multiple parties, conflicts can arise when competing users try to control the same devices in different ways. Such conflicts usually require user negotiation to resolve and thus lower people’s satisfaction and trust in the smart system. Finding a conflict is the first step to resolving it, and the timing when a conflict is identified impacts the options for resolution. Most existing approaches identify conflicts only at the time they occur, which offers little help to the users in resolving the conflicts, especially without them having to compromise. A better solution is to predict potential conflicts in advance so that the users can coordinate themselves to avoid conflict situations beforehand. Our approach proposes a novel context-aware conflict prediction framework that addresses the research gaps identified in existing literature. We mine habit patterns from the user’s previous interactions with smart devices in the various environments they occupy. These habits serve as inputs to our conflict prediction algorithm which takes the habits of pairs of users and outputs context situations in which those users have the potential to conflict.

• Middleware for Mediating Interactions in Smart IoT Spaces

The Internet of Things (IoT) is a major technological development likely to have a profound effect on all aspects of society. Among other things, it promises smooth and personalized interactions between people and the spaces they inhabit and visit. Unfortunately, we are not yet at the point of interacting with smart spaces per se; rather, we simply interact with collections of devices having different interfaces, offered by different manufacturers, living in different administrative domains, and using different apps. Our research promises to take us a step closer to achieving the personalized interaction modalities the IoT technology is capable of offering. The starting point is to reimagine the smart space as being defined by spatial characteristics (e.g., illumination, security, temperature, etc.) with most devices receding from the user’s explicit awareness. Users can specify their needs from the environment in terms of the abstract characteristics. Key to accomplishing this is the introduction of the concept of the Space Broker, a software agent that manages available devices so as to meet user requirements expressed in terms of spatial characteristics.

• Device control through Augmented reality

Increasing numbers of smart devices have enriched the possibilities of the IoT. These advances also impose significant cognitive overhead on the end-users; to interact with IoT devices, a user must be knowledgeable about the devices, the applications, the network, etc. ArcIoT is a system that enables intuitive interaction with IoT devices by leveraging Augmented Reality (AR) and a novel relative indoor localization service. In ArcIoT, the user points a smartphone camera towards a device to control, chooses the device by a tap on the screen, and interacts with the displayed interface. Continuously and transparently, ArcIoT maintains a map of devices in the environment, which allows it to quickly and reliably control them. Compared to existing solutions, ArcIoT does not need predefined markers to recognize a device or require any change to physical infrastructure. ArcIoT is also robust to changes in the locations of the IoT devices in the environment. We successfully deployed ArcIoT in different home environments and evaluated its performance to demonstrate that ArcIoT helps users control devices in everyday environments accurately, responsively and intuitively.

• Programming Abstractions for Personalizing Interactions

Advances in sensing and networking along with ubiquitous Internet connectivity have paved the way for today’s massive Internet of Thing (IoT) market. Despite the vast potential of connecting to myriad devices across homes, office buildings, and public spaces, there is still a large need to unify the scattered protocols, hubs, and cloud services while personalizing enduser experiences. Enabling personalized IoT experiences requires an expressive and flexible middleware that enables simplified development of applications that address diverse individual needs and seamlessly cross multiple vendors and administrative domains. Warble is a middleware for such personalized IoT applications; Warble encapsulates device and protocol complexities, represents interaction with IoT devices as flexible programming abstractions, and enables applications to learn from their prior interactions in the IoT on behalf of their users.

• Seamless Context-Aware Automation

Advances in mobile computing capabilities and an increasing number of Internet of Things (IoT) devices have enriched the possibilities of the IoT but have also increased the cognitive load required of IoT users. Existing context-aware systems provide various levels of automation in the IoT. Many of these systems adaptively take decisions on how to provide services based on assumptions made a priori. The approaches are difficult to personalize to an individual’s dynamic environment, and thus today’s smart IoT spaces often demand complex and specialized interactions with the user in order to provide tailored services. rIoT isa framework for seamless and personalized automation of human-device interaction in the IoT. rIoT leverages existing technologies to operate across heterogeneous devices and networks to provide a one-stop solution for device interaction in the IoT. rIoT exploits similarities between contexts and employs a decision-tree like method to adaptively capture a user’s preferences from a small number of interactions with the IoT space.