Mid-air Gestural Interaction

In collaboration with Dr. Daniel Vogel, we explored the design of whole body gestures with low arm fatigue. As part of this research we introduced the idea of using soft-constraints (i.e. something that physically penalizes or encourages certain types of movement) in elicitation studies to minimize participant biases. Gesture elicitation studies have become a widely used tool to inform the design of gesture sets. In a gestural elicitation study, participants are shown a system action (the “referent”) and asked to propose a gesture to trigger it (the “symbol”). An advantage of these studies is they should not be confined to current technologies—but rather the focus is on user desires, not limitations of recognition technology or previous design decisions.

Despite these benefits of elicitation studies, Morris et al. argue legacy bias is a potential problem with elicitation studies. This is when prior experience with interfaces and technologies make it hard to uncover new gestures for an emerging medium. Researchers have shown that participants proposed touch gestures resembling legacy mouse input [16]. For whole-body gestures (performed by moving any part of the body without a device), we extended the definition of legacy bias beyond direct experience. We include secondhand knowledge of current body tracking capabilities and gestures portrayed in popular culture such as gaming system advertisements or science fiction films. With this extended definition of legacy bias, we argued elicited whole-body gestures may have more to do with what is easy to track with current sensing technologies or more expressive for the sake of cinematic performance. For example, we found that for map panning, participants often mimic a gesture popularized by the Microsoft Kinect and the film Minority Report: they raise their arm to chest level and move it left and right. Frequent use of large arm gestures like this can cause “gorilla arm” fatigue. Alternative gestures using other body parts or subtler arm movements may be more appropriate. We believe legacy bias is compounded by what we call performance bias, which occurs when the artificial, time-limited study setting biases participants against considering long-term performance. It is difficult to get participants to consider aspects such as fatigue when their primary goal is to complete the study quickly and receive remuneration.

To overcome these issues, we proposed and used a soft constraint (wrist weights to penalize large arm movements) to reduce legacy and performance biases in gesture elicitation studies. Results from our study demonstrated that this soft constraint can counteract legacy and performance bias and thus, elicit more alternative gestures with lower arm fatigue. This work appeared at CHI ‘15.

Lastly, we are currently exploring how multimodal interactions including gestures, speech and facial expressions can be used to enable two-way communication between users and computers. This work is part of collaboration with Drs. Bruce Draper and Ross Beveridge who are experts in computer vision. This work is currently being funded by DARPA’s Communicating with Computers program. This $2.1 million award started July 2015 and is expected to run until 2020.

Exploring the Use of Gesture in Collaborative Tasks
Isaac Wang, Pradyumna Narayana, Dhruva Patil, Gururaj Mulay, Rahul Bangar, Bruce Draper, Ross Beveridge, and Jaime Ruiz. 2017. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 2990-2997. DOI: https://doi.org/10.1145/3027063.3053239