Doing That Crazy Hand Jive: Gesture And The Future Of User Experience

The technology industry is going retro — moving away from remote controls, mice and joysticks to something that arrives without batteries, wires or a user manual.

It’s called a hand.

In the coming months, the likes of Microsoft, Hitachi and major PC makers will begin selling devices that will allow people to flip channels on the TV or move documents on a computer monitor with simple hand gestures. The technology, one of the most significant changes to human-device interfaces since the mouse appeared next to computers in the early 1980s, was being shown in private sessions during the immense Consumer Electronics Show here last week. Past attempts at similar technology have proved clunky and disappointing. In contrast, the latest crop of gesture-powered devices arrives with a refreshing surprise: they actually work.

[…]

Just as Microsoft’s gaming system hits the market, so should TVs
from Hitachi in Japan that will let people turn on their screens, scan
through channels and change the volume on their sets with simple hand
motions. Laptops and other computers should also arrive later this year
with built-in cameras that can pick up similar gestures. Such
technology could make today’s touch-screen tools obsolete as people use
gestures to control, for instance, the playback or fast-forward of a
DVD.

To bring these gesture functions to life, device makers
needed to conquer what amounts to one of computer science’s grand
challenges. Electronics had to see the world around them in fine detail
through tiny digital cameras. Such a task meant giving a TV, for
example, a way to identify people sitting on a couch and to recognize a
certain hand wave as a command and not a scratching of the nose.

Little
things like the sun, room lights and people’s annoying habit of doing
the unexpected stood as just some of the obstacles companies had to
overcome.

GestureTek, with offices in Silicon Valley and
Ottawa, has spent a quarter-century trying to perfect its technology
and has enjoyed some success. It helps TV weather people, museums and
hotels create huge interactive displays.

This past work,
however, has relied on limited, standard cameras that perceive the
world in two dimensions. The major breakthrough with the latest gesture
technology comes through the use of cameras that see the world in three
dimensions, adding that crucial layer of depth perception that helps a
computer or TV recognize when someone tilts their hand forward or nods
their head.

via Ashee Vance, www.nytimes.com

This advance is one of several that will form the basis of an entirely new experience for computing.

Gestural UI, or ‘hand jive’ as I call it, once deployed as a built in aspect of future computers, like touchpads and mouses are today, will set the stage for a rethink about user experience.

First we will see hand jive as a way to manipulate the gears of now-tradition windowed UIs: pulling down a menu in an app, moving windows around, dragging a file to the trash.

In the future, we’ll have real Minority Report stuff, without the enormous touch screens: we’ll also see the emergence of augmented reality goggles — Terminator goggles — where we can toggle back and forth between 100% computer screen sorts of display to 100% augmented reality. And the goggles — as an integrated part of the computing device — will be watching our hands for commands, and watching the world for reality to augment.

The combination of these trends will make computing primarily mobile: we’ll have an iPhone sized device we carry all the time, which will be a phone and a PC. We will be free of LCD screens — in general — courtesy of our goggles, and free of keyboards, courtesy of hand jive. A keyboard can be imaged on any flat surface by the goggles, and we can type without a physical keyboard because the gestural system is watching our fingers in 3D. And of course, a lot of things could be done without typing, especially once kids start using sign language and voice to communicate with computers. (I say kids because that’s who start first.)

The other parts of this tectonic shift in UX will include the end of the document-centric folder/file/desktop metaphor, where information in managed in documents, based on old school filing cabinets. I believe that innovation like the Litl OS, which has shifted to a TV-influenced UI of channels, treating information as something that flows and not something static, sitting in a document.

And of course, the social web will be the foundation of future computing, as opposed to a document-centric world in which people are an afterthought.

How To Hand Jive

via Ehow