SpeCam: Sensing Surface Color and Material with the Front-Facing Camera of a Mobile Device

Today, mobile computing readily affords us the opportunity to determine our approximate location and to use this information to customize our interactions.
However, fine-grained location information within an environment often relies on new mobile hardware or sensing infrastructures.
SpeCam is a lightweight surface color and material sensing approach for mobile devices which only uses the front-facing camera and the display as a multi-spectral light source. We leverage the natural use of mobile devices (placing it facedown) to detect the material underneath and therefore infer the location or placement of the device. SpeCam can then be used to support discreet micro-interactions to avoid the numerous distractions that users daily face with today’s mobile devices.
Being able to determine the location of a device with high precision offers several unexplored opportunities of interaction: for example, a user could transfer information by placing a phone on a computer, or trigger specific applications on the device by placing it on a predetermined area of their desk and other furniture.
Full video:

This work will be presented at the prestigious 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, ACM MobileHCI 2017 on Wednesday September 6th in Vienna, Austria. The paper “SpeCam: Sensing Surface Color and Material with the Front-Facing Camera of a Mobile Device” has received an honourable mention award at MobileHCI, one of only 5 such papers in the entire program.
Publication:
Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, David Harris-Birtill, and Aaron Quigley. 2017. SpeCam: sensing surface color and material with the front-facing camera of a mobile device. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’17). ACM, New York, NY, USA, Article 25, 9 pages. DOI: https://doi.org/10.1145/3098279.3098541
PDF