Wearable Interaction

Hui-Shyong Yeo and Aaron Quigley


Personal and intimate mobile and wearable devices such as head-mounted displays, smartwatches, smartphones and tablets are seeing increasing use. Within SACHI we are interested in wearable devices, wearable displays, wearable interaction in mobile settings, with second-screen scenarios, or as part of coordinated multi-display environments from the desktop, second-screen to gigapixel display walls.

Wearable devices

The screen size of a smartwatch provides limited space to enable expressive multi-touch input, resulting in a markedly difficult and limited experience. In SACHI we are exploring ways enhance and extend the wearable experience. One of our projects is called WatchMI which stands for Watch Movement Input without requiring additional or hardware modification of the smartwatch. WatchMI enhances touch interaction on a smartwatch to support continuous pressure touch, twist, pan gestures and their combinations. Along with studying these primary WatchMI interactions we have also demonstrated a wide range of applications which WatchMI interactions support including, map navigation, an alarm clock, a music player, pan gesture recognition, text entry, file explorer and controlling remote devices or a game character. Our software analyzes, in real-time, the data from a built-in Inertial Measurement Unit (IMU) in order to determine with great accuracy and different levels of granularity. On this page you can find links to WatchMI videos, the WatchMI paper presented at MobileHCI 2016 and some of the media coverage of this work. This work has been undertaken with colleagues in KAIST, South Korea.

Following WatchMI is Sidetap & Slingshot Gestures on Unmodified Smartwaches. We present a technique for detecting gestures on the edge of an unmodified smartwatch. We demonstrate two exemplary gestures, i) Sidetap – tapping on any side and ii) Slingshot – pressing on the edge and then releasing quickly. Our technique is lightweight, as it relies on measuring the data from the internal Inertial measurement unit (IMU) only. With these two gestures, we expand the input expressiveness of a smartwatch, allowing users to use intuitive gestures with natural tactile feedback, e.g., for the rapid navigation of a long list of items with a tap, or act as shortcut commands to launch applications. It can also allow for eyes-free interaction or subtle interaction where visual attention is not available.

Coordinated, on (ie. wearable) and around, body devices

Displays on and around the body such as smartwatches, head-mounted displays or tablets enables users to interact on the go. However, diverging input and output fidelities of these devices can lead to interaction seams that inhibit efficient mobile interaction when users employ multiple devices at once. Our work on MultiFi demonstrates Multi-Fidelity Interaction with Displays On and Around the Body and was presented at CHI 2015. MultiFI combines the strengths of multiple displays (smart watch and head mounted display or mobile device and head mounted displays).

The display coordination in MultiFi presents three powerful concepts with (1) body-aligned mode, the devices share a common information space, which is spatially registered to the user’s body, (2) device-aligned mode, where the information space is spatially registered to the touchscreen device and moves with it and (3) side-by-side mode, interaction is redirected from one device to the other without requiring a spatial relationship among devices. Intuitively, in device-aligned mode, content which is displayed is glued to a single reference display eg. a high res map shown on a smart-watch, with a low res head mounted display of the “rest of the map” shown around the prephiery of the smart-watch. When the watch moves, the content of the head mounted display follows! On this page you can find links to MultiFi videos/talks, the MutliFI paper presented at CHI 2015 and some follow up journal and workshop papers. This work has been undertaken with colleagues in the University of Passau, Germay and Graz University of Technology, Graz, Austria.

In ItchyNose, we explored subtle and discreet on body interaction – detecting finger movements on the nose by using EOG sensors embedded in the frame of a pair of eyeglasses. Eyeglasses wearers can use their fingers to exert different types of movement on the nose, such as flicking, pushing or rubbing. These subtle gestures can be used to control a wearable computer without calling attention to the user in public.


2016: Our 3 Min overview video of WatchMI

Sidetap & Slingshot Gestures on Unmodified Smartwatches

2015: Our 30 teaser video of MultiFI

ItchyNose video

2015: Watch the full MultiFi talk below from CHI

Recent Publications



More Links

  • https://verdictforu.com/2016/08/25/watchmi-intends-to-change-how-you-use-smartwatches/
  • http://phoneradar.com/watchmi-technology-change-smart-watch/
  • http://androidcommunity.com/new-smartwatch-technology-developed-under-experiment-for-usability-20160824/
  • http://www.theinquirer.net/inquirer/news/2467851/boffins-develop-smartwatch-ui-that-could-turn-wearables-into-gaming-controllers
  • http://www.v3.co.uk/v3-uk/news/2467832/researchers-show-off-pressure-touch-and-movement-control-tech-for-wearables
  • https://www.wareable.com/smartwatches/watchmi-smartwatch-gesture-controls-video-android-wear-3094
  • http://www.tuttoandroid.net/android/android-wear-watchmi-press-touch-video-403206/
  • http://www.wearabletechnology-news.com/news/2016/aug/16/research-examines-different-pressure-and-inputs-greater-smartwatch-functionality/
  • http://www.smartwatch.me/t/watchmi-system-lets-users-access-smartwatch-functions-with-motions/2386
  • http://www.smartwatchpro.ru/watchmi/http://www.undergraduateawards.com/st-andrews-researchers-create-software-making-smartwatches-easier-use/
  • http://wearables4u.info/watchmi-wants-to-bring-new-gesture-controls-to-existing-smartwatches.html
  • http://telegraf.com.ua/tehnologii/2761851-blagodarya-sisteme-watchmi-rasshiryatsya-vozmozhnosti-umnyih-chasov.html
  • http://vistanews.ru/computers/74577-sistema-watchmi-rasshirit-vozmozhnosti-umnyh-chasov.html
  • http://www.pianetacellulare.it/Articoli/Produttori/42746_WatchMI-Software-per-Smartwatch-che-moltiplica-le-possibili.php
  • http://www.pianetacellulare.it/Articoli/Produttori/42746_WatchMI-Software-per-Smartwatch-che-moltiplica-le-possibili.php
  • http://www.smartwatchandroid.it/news/229/smartwatch_android_-_il_meglio_della_settimana_21_08_16.htm
  • http://www.dianxi.cc/keji/shouji/134750.html
  • http://www.gulfeyes.net/technology/565582.html
  • http://smartwatchviet.net/threads/watchmi-cong-nghe-moi-cho-phep-tao-them-cu-chi-tren-smartwatch-khong-can-phan-cung-bo-sung.41/
  • http://androidhungers.com/latest-technology-news/android-wearable-devicehttp://smartchimps.com/smart-news/university-makes-smartwatches-easier-to-use/
  • http://www.startlr.com/researchers-have-developed-a-new-way-to-interact-with-smart-clock/
  • http://electronics360.globalspec.com/article/7155/computer-scientists-are-making-smartwatches-easier-to-use
  • http://www.financialexpress.com/industry/tech/new-methods-make-smartwatches-easier-to-use/347949/
  • https://www.smartwatch.de/news/watchmi-ruestet-smartwatches-ueber-die-software-auf/
  • https://techxplore.com/news/2016-08-methods-smartwatches-easier.html
  • http://www.dailypioneer.com/vivacity/watches-get-smarter.html
  • http://wireless.electronicspecifier.com/wearables/making-smartwatches-easier-to-use
  • http://www.xxiiw.com/showinfo-5-3088-0.html