It looks like the world of Kinect hacks is about to get a bit more interesting.

While many of the Kinect-based projects we see use one or two units, this 3D telepresence system developed by UNC Chapel Hill student [Andrew Maimone] under the guidance of [Henry Fuchs] has them all beat.

The setup uses up to four Kinect sensors in a single endpoint, capturing images from various angles before they are processed using GPU-accelerated filters. The video captured by the cameras is processed in a series of steps, filling holes and adjusting colors to create a mesh image. Once the video streams have been processed, they are overlaid with one another to form a complete 3D image.

The result is an awesome real-time 3D rendering of the subject and surrounding room that reminds us of this papercraft costume. The 3D video can be viewed at a remote station which uses a Kinect sensor to track your eye movements, altering the video feed’s perspective accordingly. The telepresence system also offers the ability to add in non-existent objects, making it a great tool for remote technology demonstrations and the like.

Check out the video below to see a thorough walkthrough of this 3D telepresence system.

The way developers are hacking Microsoft’s Kinect motion controller to lead it into all sorts of exciting new directions continues to impress. While some are using it as a musical instrument controller, others have taken it into the world of dance.

This impressive video was sent through to us from Singapore. Edwin Toh, a developer at Kinetic Singapore, expains that it was created as a marketing stunt for drinks brand F&N. “Dancers are transformed into colourful bubbles which react and replicate the dancers’ moves,” he says. “As dancers groove to the music, sounds will be triggered. Depending on which screen region the dancers move to and trigger, different pitches will be produced, leading to a unique mish-mash of creative sounds and movements.”

Beyond the fact that brands are now jumping on the Kinect hack bandwagon, what’s exciting about this video is that it looks just like a pop video but rendered in real time. If developers are doing this mere months after Kinect’s initial release, what will they be doing in a year from now?

It’s hard to assemble an April Fool’s Joke involving technology these days, because actual inventions keep proving stranger than fiction. When Google created a prank involving gestures for controlling email, it was only a matter of time before someone whipped up a prototype that actually did the job.

The Moog Music company, therefore, may be asking for trouble. Their highly-entertaining polyphonic Theremin is spot-on parody, down to the “Stairway to Heaven” solo. And part of the geekier joke for Theremin players is the knowledge that the technology behind this instrument makes what they’re describing safely impossible.

But what’s impossible with conventional Theremin technology could be very possible with computer vision – even the goofy gestures in Moog’s faked video. Artist, inventor, and musician Tim Thompson has been at the bleeding edge of new music instruments for some time. It wouldn’t be overstatement to say Tim was using multi-touch before multi-touch was cool. When I shared a booth with him at Maker Faire a few years ago, he had with him FingerWorks hardware, a now-discontinued tactile, multi-touch pad, and was using it to play visuals live. In a pattern too often repeated in technology, the independent niche tool was snapped up by a larger player. In this case, that larger player was Apple – and, apparently backed at least in part by FingerWorks’ know-how and patents, Apple made history.

In a new project filmed by the superb Modulate This!, Tim works instead with touch-less control, using the Kinect to track multiple areas of expression. (Tim is using the free environment Cinder, which joins tools like Processing and OpenFrameworks as well-liked options for Kinect hackers. In this case, the Kinect support itself comes from libfreenect, the open-source drivers for Mac, Windows, and Linux.)

What he’s built, in other words, is a true polyphonic Theremin – able to play more than one line and employ more than a monophonic gesture, all without touch. The joke may be on Moog.

Read the full story on Modulate This, Mark Mosher’s all-original repository for great writing on music making.
An Exclusive First Look at Tim Thompson’s Kinect-Based Instrument: MultiMultiTouchTouch
(Thanks to Tim and Roger Linn for sending this my way!)

Part of the value of trying extreme ideas is to demonstrate not only advantages, but disadvantages. And I still find some reason to express healthy skepticism. The similarity to the Theremin isn’t accidental in the Kinect experiments. These projects also inherit the Theremin’s weaknesses. A lack of tactile feedback means it’s difficult to orient pitch or achieve precise control, without the resistance a physical object provides. Reliance on gestural control also opens the opportunity for accidental input and calibration challenges. (The Kinect fares better than the Theremin, but it’s not immune to similar problems, if for different reasons.) Taking a page from the Theremin, Tim’s physical frame makes a big difference – while it doesn’t provide tactile resistance, it at least creates a point of reference in physical space.

The Kinect also adds a new problem the Theremin didn’t face: latency. All of this means if you still like knobs, keys, strings, or even physical multi-touch (which can in certain variations provide excellent tactile feedback via deformable meshes), you needn’t worry. Your revolution may not be Kinect-ified.

But if there were one perfect design for musical instruments, we’d all play just one instrument. Instead, the history of instrument design across the world is an evolutionary explosion of different tradeoffs, different playing styles, and resulting different musical idioms. Any joke can become an instrument, just as any instrument – to someone – can seem like a joke. And that means if you’re looking for something new, you might just celebrate every day as if it’s April Fool’s Day. No kidding.

Updated: Tim offers some comments. He says what other musicians experimenting with Kinect have told me – that while it has certain restrictions as a solo instrumental controller, there’s tremendous potential for multi-user scenarios like installations. And that is itself significant (back to the question of choosing tradeoffs in order to accomplish goals). Tim writes:

Folks whose goal is to replace conventional instruments are sure to be disappointed, as you describe. You could add more detail on other goals:

Goal: using it for art installations at events like Burning Man, creating new and “casual” instruments which are unusual yet inviting and easy to play. Matt Bell ran an experiment related to that goal:

Goal: creating controllers which have a much larger visual appeal to an audience, who deserve performers more interesting to look at than someone hunched over buttons and sliders. That’s the reason why musicians like Mark Mosher are interested, in the same way he’s interested in the Percussa Audiocubes, for their visual appeal in performances.

Goal: provide an instrument that dancers can use in performances. I’ll be exploring this in the fall, with a choreographer friend.

Good food for thought; feel free to discuss more in comments.

As an entry to the Eyebeam DataVizChallenge to visualize your taxes, Budget Climb by Frankie Cheung, Zach Schwartz, and Fred Truman places you in a virtual environment where you can climb and stand on top of tax data.

Created using openframeworks, the Microsoft Kinect and OpenNI Budget Climb is a physically interactive data environment where we can explore 26 years of federal spending - giving us a unique perspective on how our government spends our money. In order to explore the data we must exert physical effort, revealing how the budget is distributed in a novel and tangible way.

Maybe not the best way to learn about the data, but I like the idea of data exploration as a game. See it in action below. It reminds me of Irad Lee's Spamology.

[Budget Climb | Thanks, Fred]

Today we have a special treat, three projects combining the “fastest selling consumer electronics device”, Kinect, and the “fastest selling indie java game that once kept us from sleeping for an entire weekend”, Minecraft!

[Sean Oczkowski] writes in to tell us about his efforts to play Minecraft with Kinect using no more than the OpenKinect Java wrapper on Ubuntu.  The code was written in about 4 days with some help from Wikipedia.  Using  histograms to locate the player in the field of view, the script calculates the center mass of the body and defines interactions for the limb occupying that quadrant of the screen. [Sean] does an excellent job of running through the whole process as well as the decisions made along the way. The whole thing is a bit like running in place, and we can’t imagine the flailing that will occur during the inevitable creeper encounter.

Next we have  [Wade McGillis] with his award winning Minecraft Kinect Controller. [Wade] provides source code and executables at his site. This version of control uses skeletal tracking data to sense the user’s gestures. This still involves holding your hands out like a zombie but it is a bit more versatile as one can pass their arms in front of their own body.

Finally [Nathan Viniconis] has been doing some very interesting work using the Kinect to import giant three dimensional models into the game world. [Nathan] then goes the extra mile and animates the figures! Check out the video below for the really impressive results. We here at Hackaday feel that this is the most appropriate use of this technology, and may begin building gigantic statues of ourselves on public servers.

Check out the the tricrafta (minefecta?) of videos after the jump!