Not A Wall, A World: The Future Of User Experience

I loved Minority Report’s gestural interface, as a scifi representation of what we think a police state might use to watch us, given the ability to move through a nearly infinite amount of data — and time — searching for clues.

Apparently, that interface is not just the stuff of Hollywood, as it appears that John Underkoffler, the guy that mocked up that experience for the movie, has been off actually building the system he literally is hand waving into existence.

MG Siegler thinks this represents the future of computing. I disagree, but first, MG’s thoughts:

- MG Siegler, The iPad Is Step 1 In The Future Of Computing. This Is Step 2 (Or 3).

While we may not have been at this year’s TED conference, apparently, Oblong was. And apparently, it wowed the crowd. And it should have. If you’ve seen the movie Minority Report, you’ve seen the system they’re building.

No, really. The co-founder of Oblong, John Underkoffler, is the man who came up with the gesture-based interface used in the Steven Spielberg movie. And now he’s building it in real life.

The demo I saw a couple years ago was stunning, but it was still just a video. Apparently, at TED, the audience got to see it in action. NYT’s Bits blog detailed some of it in a post yesterday. For those not at TED, Oblong has also made a few demo videos in the past, which I’ll embed below. Again, this is Minority Report.

Oblong’s coming out party couldn’t come at a better time. Following the unveiling of Apple’s iPad, there has been a lot of talk about the future of computing at a fundamental level. That is to say, after decades of dominance by the keyboard and mouse, we’re finally talking about other, more natural, methods of input. The iPad is one step to a multi-touch gesture system (as is this 10/GUI awesome demo), but this Oblong system is the next step beyond that.

I don’t believe that huge displays based on petabytes of information — like Cruise was surfing — is likely to be the prototypical user experience for normal people in the near term. In some narrowly defined industries — military, cinematography — such displays may be temporarily of interest. But the future of user experience is a logical extension of what we have been seeing in consumer electronics: a continued movement to small, mobile, and personal.

Yesterday, I posted a Nokia video that I think is much more true to life. I reproduce it here, again, in the form of the complete video, and an image pulled out.

complete video

one screenshot from the video

The Nokia example is based on a few assumptions:

Augmented reality glasses will become the standard user display — Instead of huge displays, hung on walls with giant panels, people will wear augmented reality glasses. These will display on the inside of the glass images that provide access to various sorts of information.

Displays will become less complex than today’s file/folder/desktop jumble, and interaction will be based on simple eye movements and gestures — User interaction will rely on eye tracking and gestural interfaces to represent selection, expansion, playing video or audio, and the like. In this example, the woman looks at the name of an artist in a playlist long enough and the environment interprets that as a selection. At some points in the demo she flicks her hand to represent clicking or scrolling. Note she doesn’t wear gloves or special hand gear: the glasses have cameras that watch her hands. They don’t show her doing it but either a generalized sign language could be used for more complex communications — more than selecting an emoticon, like she does — or a virtual keyboard could be displayed, and ‘keystrokes’ recorded, again, by the glasses observing our hands.

I don’t think that the grand gestural, ‘orchestra conductor’ sort of scenario that we saw in Minority Report will be the norm, although in specialized contexts — like gaming, war fighting, and brain surgery — those sorts of advanced gestural languages might be developed.

Social interaction with others will be the primary modality of all future operating environments, and other activities will principally be constructed to help filter and aggregate social channels — This is not well-represented in either the Nokia video or Minority Report. In the Nokia example, the woman is mostly dabbling with relatively conventional streams and stores — weather and news, and riffling through a music library — while occasionally being pinged by an overly attentive boyfriend. Imagine a more rich scenario of a marketing executive racing through the streets of New York, communicating with four colleagues in an open semi-public sort of way, with integrated information streams of plans, designs, and marketing campaign mockups. And at the same time receiving local augmented reality information about the streets she is passing through, like GPS coordinates, a map showing her destination and where her four colleagues are, offers from the food truck she passes, and a global stream of socialized news and information from her network of friends, fans, and connections.

***

This can be condensed to the shorthand: not a wall, a world.

The steampunk idea that we will continue to have displays like today’s TV screens or PC monitors is dubious. I would give up mine in a heartbeat. More important, there is a world out there, and amplifying what we are already looking at — like the street we are walking on — with relevant information — like where the bus stop is, or what kind of food that restaurant serves — is so obviously helpful it doesn’t really need to be motivated.

We will continue to have personal and mobile computing experiences in the near future, because mostly we work and play on personal devices. Yes, there is the occasional meeting in a face to face setting where currently we use large displays, but this will be replaced by shared augmented reality: a presentation, for example, could be controlled by one person (or more) and viewed by a larger group. But this wouldn’t be projected on the wall, necessarily. Instead, it would be shared via each attendee’s glasses. We might be looking at a blank wall, or we might be walking through a virtual representation of a building being designed, or a product being assembled.

Amplifying the social through this sort of user experience would be phenomenal. Wandering around at a business meeting, a party, or a conference, and seeing salient information about the people you are looking at — where they work, when you last talked, the names of their loved ones, their pet peeves, whether they follow you and know of your work — would be an immense help. And would potentially change the nature of our social contract in startling ways. This is what I am expecting to appear, and very soon. Not 2045.

Why Closed Works: Moving Past Steampunk Thinking About The Future Of Computing

I have been arguing for some time (see Tim O’Reilly on Web 3.0) that the metaphors of computing user experience (or UX, as we tend to say these days) are holding us back from new ways of structuring our interaction through computers.

First of all, way back when we were debugging fire with oscilloscopes, and inadvertantly invented personal computers, no one considered that the primary dynamic of computing would be social. We invented the web to chat among ourselves, to write for each other, and share photos, and argue. We invented a web structured in such a way that all its roads lead back to us.

But, when we were cooking up Unix, DOS, Windows, and Mac OS, we thought people would be ‘working’. And working meant fooling around with documents (files), since that what we thought was the logical atom of information: files.

But it’s not.

In fact, the layout of information on my hard drive bears little resemblance to what I do all day, since the overwhelming amount of my time is taken up in communication with others, buried in email, Twitter, blogs, and various other coordinative tools.

Sometimes I am forced to know where a bit of information is embedded in some document — against my will — and to the extent that is possible, I try not to know such things.

More importantly, I am not alone: more people all the time are using their computers as social interaction devices, and files, folders, and the desktop are just a launch pad for that.

I believe that we are seeing the first glimpses of a break with the past, when we will not care if and where information lives in files on some physical level, either in the cloud or on a local hard drive.

If I am sharing a video with a bunch of friends does it matter if it is located in one place on the web, or distributed in twenty servers and streamed on demand, or sitting on my local hard drive and streamed to them? I don’t care.

But the thing that is blocking us from moving forward, to a better user experience centered on social interaction and not physical data, are the existing metaphors of OS’s. Since we are living in a world of general purpose computers running Unix, Mac OS, and Windows — and we need to have them interoperate — we seem stuck in the 90’s.

The only way forward is to build a new user experience on top of the physical hardware and software that form a platform for it, and conceal its nasty details from us.

This is one aspect of the genius of the iPhone and iPad generation of devices: we don’t need to know about the files and folders. We don’t need a desktop with data bundles lying in piles.

Instead, we have a suite of applications that serve up myriad useful metaphors. Photo apps deal with photos, not files. Expense report apps deal with expenses. Twitter apps reveal the twitter stream. And so on.

To have a break with the past, and to make the past a platform, we have to push it under and not pretend that all its constructs are desirable. We need to push files, folders and the notion of a desktop under the surface of a better user experience, and keep it under. Let a new generation of user experience shield us from that drudgery and detail.

This break with the past is made faster and less difficult if the new system is closed: if we can’t escape its premises, and start rooting around in the basement, moving files and putzing around. Apps have to be limited in what they can do at the physical level, because the whole edifice can come down if apps can punch a whole in the foundations.

I expect that Apple won’t be the only one to devise new OS’s that head in this direction, and perhaps Chrome OS or a new Mac OS kernel might become the predominant infrastructure for a presentation layers like iPad and iPhone. They will become more powerful, faster, and will blossom in new directions, like augmented reality.

I know there are a lot of techies out there that idealize a hypothetical future that’s a kind of steampunk mixture: amazing new functionality — like full augmented reality, digital currencies, and crowdsourced sousveillance — but somehow all strapped to the back of operating environments they are comfortable with, like Linux and the desktop-oriented user experiences of the ’90s. Sorry, folks. No steam-powered rockets to Mars in this episode.

To move ahead, something has to give. To get more abstract, the foundation has to be more closed. And that’s one of the reasons that the iPad/iPhone devices represent a breakthrough.