Fits Like a Glove: Rapid and Reliable Hand Shape Personalization

David Joseph Tan (1,2), Thomas Cashman (1), Jonathan Taylor (3), Andrew Fitzgibbon (1), Daniel Tarlow (1), Sameh Khamis (3), Shahram Izadi (3), Jamie Shotton (1)

 (1) Microsoft Research; (2) TU Munich; (3) perceptiveIO
Note:  This work was conducted at Microsoft Research

View the paper here.  |  View on Microsoft Research

We present a fast, practical method for personalizing a hand shape basis to an individual user's detailed hand shape using only a small set of depth images. To achieve this, we minimize an energy based on a sum of render-and-compare cost functions called the golden energy. However, this energy is only piecewise continuous, due to pixels crossing occlusion boundaries, and is therefore not obviously amenable to efficient gradient-based optimization. A key insight is that the energy is the combination of a smooth low-frequency function with a high-frequency, low-amplitude, piecewise continuous function. A central finite difference approximation with a suitable step size can therefore jump over the discontinuities to obtain a good approximation to the energy's low-frequency behavior, allowing efficient gradient-based optimization. Experimental results quantitatively demonstrate for the first time that detailed personalized models improve the accuracy of hand tracking and achieve competitive results in both tracking and model registration.