Making Things See: A Book for O'Reilly about the Kinect

31 May, 2011

It's been quiet around here for awhile, but not for me out in the real world. Since my last post here, I've finished my master's thesis and graduated from ITP. Over the next couple of weeks I plan to catch up on documenting many of the projects from the last few months of the school year so keep watch here for more posts soon.

My time at ITP was an amazing two years that I can't believe is already over. I spent so long looking forward to the program and planning for it that, in the end, it seemed to go by in a whirlwind. I met some of the most brilliant, creative, and kind people I've ever come across and I hope to be working and collaborating with many of them for years to come.

So, what comes next? I have two announcements to share about my future plans.

First off, I'm excited to announce that I've been contracted by O'Reilly to write a book about the Microsoft Kinect. The book is tentatively titled Making Things See: Computer Vision with the Microsoft Kinect, Processing, and Arduino. My goal is to introduce users to working with the Kinect's depth camera and skeleton tracking abilities in their own projects and also to put those abilities in the wider context of the fields of gestural interfaces, 3D scanning, computer animation, and robotics. It is my belief that the Kinect is just the first in a generation of new computer vision technologies that are going to enable a vast number of programmers and creative technologists to build all kinds of amazing new applications and hacks. The key to building these applications will not simply be grokking the particular workings of the Kinect and its associated libraries as they happen to exist right now, but gaining a richer understanding of the wider areas of study that these new abilities unlock. Gaining the ability to access depth images and user skeleton data unlocks a door that opens into a vast territory that researchers and computer scientists have been exploring for some time now. My hope is to translate some of what those researchers have learned into a map that students, creative coders, and other newcomers like myself can use to make really cool stuff in this area.

Further, since the particular details of working with the Kinect are rapidly evolving with developments in the associated open source and commercial communities — and since the Kinect is unlikely to remain the only widely available depth camera for long — providing some broader context is the only way to ensure that the book stays relevant beyond the current micro-moment in the technological development cycle.

Making Things See is going to use the Processing creative coding environment for all of its examples. It will be targeted at people with some familiarity with Processing or equivalent programming experience (i.e. people who've read Casey Reas and Ben Fry's Getting Started with Processing or Dan Shiffman's excellent Learning Processing or something similar). I'll start with the absolute basics of accessing the Kinect and work my way up to showing you how to build relatively sophisticated projects that take advantage of rich concepts from the application areas such as user-mimicking robotic armss, gestural interfaces that automate the occupational therapy exercise process, digitally-fabricated prints from 3D scans, and motion capture puppets that render beautiful 3D animations.

The book is tentatively scheduled to come out in physical form this holiday season with a digital Early Release this summer as soon as I have the first major section completed. If you have any requests for areas you'd like to see covered or any other constructive comments, I'd love to hear about them in the comments. You can also follow me on twitter: @atduskgreg for updates and announcements as things progress.

I'm also proud to announce that I've been selected as one of ITP's Resident Researchers for the 2011-2012 school year. This means that I'll be helping teach students as well as working on my own projects with support from the school. I'm especially excited to continue the transformation of ITP's web curriculum I helped kick off this year. It looks like ITP is going to be using Ruby and Sinatra in nearly all of its web programming classes come the fall so I'll be doing all I can to use my experience in this area to support those efforts. I'll be working closely with Clay Shirky who heads up the web applications effort in the department as well as fellow resident Rune Madsen. In addition to the transition to Ruby we'll be working to make it ever easier to build web-connected physical devices with the Arduino. With all the excitement around the Android ADK and the use of the iPhone headphone jack to connect arbitrary hardware it's going to be a very exciting year for integrating physical computing projects with mobile devices and, hence, for networked objects in general. Clay, Rune, and I will be working with Tom Igoe and the rest of the ITP faculty to make it as easy-as-possible for students to dive into this area.

I'm sad my time as a student at ITP has come to an end, but I'm incredibly excited about these (and other still developing) prospects.