It’s rare that I opine on my blog, but since we’ve recently been able to talk openly about our involvement with Microsoft Surface, I’ve been feeling like it’s worth discussing this much-maligned creature. People seem to treat it a bit like Michael Jackson — a reclusive enigma, rarely seen in the wild, impossible to capture on film, difficult to understand. But there seem to be no shortage of initiatives aiming to bring the “Surface experience” into the hands of the many, not just the privileged few.

Now, we’re fairly platform agnostic over at Stimulant — we’ve had a chance to work with the most homebrew of systems all the way up to some of the most polished. Surface is just one of many tools in our arsenal, but I feel like it’s a bit misunderstood. Sure you can build your own multitouch system for the price of a Wii remote, a projector and a laptop. But that’s about where the similarities end. Even higher end multitouch systems seem to either ignore or miss some of the finer points of Surface, and that’s what I’d like to discuss.

Surface > Multitouch

Yes, Surface senses a multiple touches. But it’s also inherently multi-user (given its flat orientation) and most interestingly, it features object recognition. The last one there seems to be one of the most overlooked features of Surface. Beyond blob detection, Surface can recognize a finger and determine in which direction that finger is pointing. Most infrared-based multi-touch systems are happy to get a blob at all, and call it a day. Beyond that, the cameras housed inside (with help from some smart software) can can recognize unique shapes, as well as read data from “domino” tags — small 3/4″ stickers with a series of dots on them that encode a short identifier. These tags could theoretically be printed or engraved on physical items as well, enabling an easy way to interface with said objects. This thinking “beyond the blob” really sets Surface apart in its capabilities — at least with regard to other offerings out there today. It adds a level of richness to the user experience (when leveraged properly, of course) that I’ve yet to see equaled elsewhere.

Easy to build

There are a smattering of open source frameworks showing up that enable multitouch hacking, from players such as NUIGroup and Ideo, and I applaud them all. But none of them feature a rich and full SDK that comes with well-thought out controls — it not only makes rapid prototyping easy, it enforces evolving best practices for user interaction. Plus, you’ve got the enitre .NET 3.5 platform at your fingertips. Admittedly, there’s a very sharp learning curve for folks who haven’t dabbled in Windows Presentation Foundation. But we’ve seen first-hand how much effort is going in to testing, developing and documenting this SDK. Think of what the iPhone SDK did to enable and “standardize” user experiences for 3rd party application development on the iPhone. It’s like that, only for a big-ass table.

Also worth mentioning that Surface can also be programmed against using XNA, which, though lacking any sort of UI controls or safety nets, opens Surface hacking up to a world of game developers already familiar with the framework.

I’m hoping that we will see these open source SDKs evolve into something as polished as the iPhone or Surface SDK, but until then, it seems like there’s something to be said for the cathedral over the bazaar.

Built to Last

The world of “installation computing” is a mixed bag. On one hand, you’ve got complete control over the hardware, so there’s no worrying about tweaking your app for the odd edge case. On the other hand, when something breaks, you can guess who’s getting a phone call. So given an opportunity to design and install and application that needs to persist in a public space, if you’re smart, you’re going to go with the burliest, most bulletproof, battle-tested option. The homebrew and grassroots options out there are amazingly functional, but whose pager goes off when a mac mini melts down or someone kicks the box and the webcamera gets out of alignment? If you’re playing with Surface, Microsoft is offering a solid level of on-site support for its units. That’s enough to let me sleep soundly at night.

Not to mention that Surface units are built to support the weight of a full human, withstand spilled drinks and survive rough play. The whole thing is built like a tank, and it’s been through multiple design revs so while you may laugh at that form factor, you’ll thank Microsoft for having put these things through years of the lather/rinse/repeat cycle to get it where it is today. Sure, it’s not perfect, but I’d have a hard time deploying a Cubit in a public space in a remote city where someone wasn’t there to tend to it night and day. These DIY pieces are fantastic for development and exploration, but I’m not so sure about permanent installation. It’s a frustrating conundrum — you can build an inexpensive system to play with, but where do you deploy your work? Ah, life on the bleeding edge.

Now, don’t get me wrong, I’m a complete supporter of every single effort out there exploring these new interaction paradigms; heck, we’re building our company around just these kinds of hardware solutions. The more people that start working in the arena, the faster the technology makes its way into the mainstream, and the more familiar these solutions get, the lower the barrier for designing solutions that can be enjoyed and used successfully by a larger audience. I merely wanted to highlight some of the characteristics of Surface that seem to go unmentioned in most discussions, all of which are why we’re particularly excited about this platform. I expect that we’ll be hacking together several franken-systems over time because we like to play, too. But when showtime hits, it’s nice to know there’s a piece of hardware out there that has our back.

We recently acquired some high-quality media documenting our latest project (thank to our friends over at 24g), which lends itself to a much more compelling post. We were engaged by Obscura Digital to create a multi-touch UI for General Motors for the Greenbuild Conference in Chicago. Similar in concept to the HP Interactive Canvas, Obscura engineered an all new rig that measured 18 feet x 5.5 feet, with 3 independent interaction areas, each driven by a separate CPU and projector.

(click image for full-size shot)

The rig was fully self-contained, and had its own custom-crafted “travel case”. Just box it up and roll it on to a semi. Obscura also significantly improved the touch tracking this time around, with a combination of improved software, setup optimizations, and a new cocktail for the film on front. Like butter!

The app leveraged our existing WPF-based Multitouch framework, and featured playful drag-and-drop interactions, large “swiping” motions to navigate through content and some mild particle physics. We’ve come a long way in our understanding of the ergonomic design of large-format multitouch apps (low-dpi + big screen + close proximity = severe UX challenges), and are starting to fold some of these learnings back in to the framework.

Each “silo” highlighted a different aspect of GM’s efforts in exploring alternative fuels and greening their vehicles and production processes. A picture is worth a thousand words, but a video does a much better job of storytelling.

This video is from the Greenbuild Conference, the other shots are from the 2007 Electric Vehicle Show in Anaheim, CA ,where the wall surfaced for a repeat performance.

Mike and I were discussing ways to translate multiple-finger motions into simple matrix transformations for multi-touch over dinner; a busy afternoon later and he’s produced a great example that makes it look easy. This certainly won’t appear new to followers of multi-touch interaction design, but it’s the first time I’ve seen the code both released and explained in detail. I’ll be porting this to our WPF framework over the next few weeks and will be sure to share my results on the big screen. Thanks, Mike!

Here are some more videos of the HP Interactive Canvas (from Obscura Digital via YouTube):

Still waiting to get a hold of the video with Martha Stewart…

I’m finally free to talk about what I’ve been working on for the past month! I worked very closely with Obscura Digital in San Francisco to build what is, to our knowledge, the world’s largest multi-touch, multi-user touchscreen. It was installed for the D5 conference in Carlsbad this past week. The installation was commissioned by HP (via Goodby Silverstein Partners) and tied in to their “The Computer is Personal Again” campaign — a large “interactive canvas” that allowed anyone to “personalize” the display with their touch.

The final surface weighed in at 16 feet by 7.5 feet, and sat roughly 2.5 feet off the floor. I was responsible for implementing the UI, which was built in WPF. The app allowed users to browse through 5 years of D conference videos, photos, articles and quotes, in addition to a live music sequencer and real-time stock quotes. I’m still waiting for some high quality videos to come through, but in the meantime, you can get an idea of the interaction model and scale of the piece from these short phonecam vids:

Get the Flash Player to see this player.

Get the Flash Player to see this player.

The biggest challenge on the WPF side was developing a multi-touch input framework from scratch. All of the familiar “Mouse*” events were useless to me; I had to extract point data from a UDP stream coming across the network from the machine which was translating touch in to x,y coordinates. I’m actually very pleased with the solution, it proved to be very performant and easy to work with. At the end of the day, I was surprised at how much could actually be accomplished in this model, and it only served to get me more excited about multi-touch surface computing. Perhaps this warrants some future posts on working with multi-touch, as it certainly seems to be the flavor of the month right now.

Highlights of the conference included showing Walt Mossberg how to interact with it (his response: “Very cool.” which, by the way, is my first Walt Mossberg review, and I’m ecstatic that it was positive ), in addition to Martha Stewart (yes, that Martha Stewart). Mossberg also mentioned the project while onstage with Bill Gates and Steve Jobs, in the same breath as the iPhone and the Microsoft Surface initiative, which are both real, shipping products (mention at 5:50). You can’t get better exposure than that.

Oh yeah, the entire project was completed, soup to nuts, in 4 weeks. That includes building hardware from scratch (which to me is just mindblowing), and only adds to my respect for the creative and technical geniuses at Obscura Digital. Expect some better videos and more posts on this topic soon, but for now, I’m happy to just catch my breath!