HoloDesk is a novel interactive system combining an optical see through display and Kinect camera to create the illusion that users are directly interacting with 3D graphics. A virtual image of a 3D scene is rendered through a half silvered mir- ror and spatially aligned with the real-world for the viewer. Users easily reach into an interaction volume displaying the virtual image. This allows the user to literally get their hands into the virtual display. A novel real-time algorithm for rep- resenting hands and other physical objects, which are sensed by the Kinect inside this volume, allows physically realistic interaction between real and virtual 3D objects.