360 Degree Interactive Camera with HTML5 Video, Canvas, and Processing.js

29 September, 2010

A couple of days ago, I posted about a project I'm working on with some fellow students to create a system for making 360 degree documentation of circuits. At that point we had our demo working in Open Frameworks, while our eventual goal was to implement it on the web and also add annotations as overlays.

We've made progress! Today Rune got Processing.js working on top of a video element. His demo uses Processing.js to draw on a canvas element that is absolute positioned on top of a video tag. This was proof that the basic structure would work: using javascript to overlay annotations on top of a moving piece of video.

This afternoon, I took the next step: using javascript to control the playback of the video. In HTML5, the video element has a pretty straightforward API, allowing you to play, pause, and position the currentFrame, everything one would need to recreate our rotation-via-scrubbing effect. The basic idea is to move to a frame in the video's full duration that corresponds proportionally to the position of the mouse horizontally over the width of the video on the screen: move the mouse left the video "rewinds", move it right it "fast forwards".

At first I encountered some frustrating problems due to the fact that when the video first renders it is not ready to be manipulated. While you can read attributes like currentFrame and duration and call functions like play() and pause(), you can not set attributes until the video's readyState attribute is set to 4. This was rather hidden in the documentation and was hard to figure out as code I would construct in Firebug would work successfully there and would then fail when run within the page's script tag.

Once I understood this problem and worked around it, implementing the scrubbing as described was pretty straightforward. Play with the demo (movie may take some time to load; be patient). Read the code on GitHub. And keep an eye out for further advancements coming soon.