Messiness At Scale

I stumbled onto a hilarious but unenlightening Twitter flame war instigated by Dave Winer — the Godfather of RSS — in response to MG Siegler’s ‘RSS is dead’ wisecrack.

At the risk of putting my fingers in the sausage machine, let me add a touch of nuance:

• RSS has declined in use, as web heads shift their source of ‘things to read’ away from RSS readers — like Google Reader — to tools like Twitter and Flipboard.
• The role of RSS in web infrastructure is being threatened by non-RSS based architectures, like Flipboard’s. That product ignores RSS and fetches through the URL to get directly at images, text, and other content.

Winer is ideologically opposed to closed, proprietary approaches like that of Twitter (or, by extension, of Flipboard):

Dave Winer, What I mean by “the open web”

Anyway, here’s what I meant by “open web.”

I meant not in a corporate blogging silo.

If I put stuff in Twitter, the only way to get it out is through a heavily regulated and always-changing API. It will change a lot in the coming months and years. It will certainly narrow more than it expands. I feel very confident in predicting this, because I understand where Twitter is going.

If you put stuff in Facebook, it’s even more silo’d than it is in Twitter.

However, if you put stuff in WordPress, even on wordpress.com, you have full fluidity. You are not silo’d. You can get data in and out using widely-supported APIs that are implemented by Drupal, Movable Type, TypePad, etc etc. At least there’s some compatibility. And in a pinch you could probably move your content to a static website and have it be useful.

If you write in static HTML and RSS, you’re very portable, there will be no lock-in at all.

So to the extent you’re locked in, that’s the extent you are not on the open web. The perfectly open web has zero lock-in. The silos are totally locked-in and therefore not on the open web.

Winer’s complaints are about control of our content: that we should be able to easily manage what we write. It’s a political argument. 

But his points fly in the face of innovation, where a Twitter or Quora or Facebook create very different — and not solitary — models of open social discourse, which need to be managed in ways that are different from old school blogging. It’s not every man for himself, anymore. Time is a shared resource on today’s web: our time is not our own, anymore. And that’s largely good.

I liken this problem to the trade offs inherent in living in large cities versus towns or the country. There’s more noise, bigger crowds, and longer lines at the DMV: more things that we can’t control, or where our control is restricted, relative to folks living in bucolic Des Moines.

Only in cities we get superlinear scaling, as Geoffrey West and his colleagues have shown:

Jonah Lehrer, A Physicist Turns the City Into an Equation

When a superlinear equation is graphed, it looks like the start of a roller coaster, climbing into the sky. The steep slope emerges from the positive feedback loop of urban life — a growing city makes everyone in that city more productive, which encourages more people to move to the city, and so on. According to West, these superlinear patterns demonstrate why cities are one of the single most important inventions in human history. They are the idea, he says, that enabled our economic potential and unleashed our ingenuity. “When we started living in cities, we did something that had never happened before in the history of life,” West says. “We broke away from the equations of biology, all of which are sublinear. Every other creature gets slower as it gets bigger. That’s why the elephant plods along. But in cities, the opposite happens. As cities get bigger, everything starts accelerating. There is no equivalent for this in nature. It would be like finding an elephant that’s proportionally faster than a mouse.

I maintain that Twitter, Facebook, and other ‘closed’ systems are really something else: they are dense and complex social systems, more like modern cities than Web 1.0 publishing platforms. And, like cities, there is more going on, less being controlled by specifications like RSS, and the food is better, the music is better, and there is more dangerous sex taking place.

Brian Eno uses the term ‘scenius’ to define the quality of the great cities, their ability to foster deep shared understanding and purpose for large networks of people. This collective intellect arises from messiness at scale, not carefully mediated and clearly defined standards. 

Said differently, the best food comes from cities with the highest number of health code violations, and the best art is produced where the largest number of building code infractions are found.

So, if you are looking for clean bathrooms and no traffic jams, stay in Iowa. But it is in cities — dense, loud, unplanned, messy — where the breakthroughs emerge.

Getting back to the specific case, here, let’s look at Flipboard. Flipboard rejects the use of neat-and-tidy RSS, and reaches through the URLs it finds in Twitter to directly paw the text, images, and links placed into articles and posts, and then it chooses what to display based on a proprietary algorithm inside the guts of the app, not based on the publisher’s RSS specification. 

Flipboard, Twitter, and other dense, complex social tools create a messier world, one that has superlinear scale. The tradeoff between complete ‘openness’ (or individual control of information and its experience) and superlinear social density is one I am willing to make. And so are all the users of these tools, or should I say, residents of these cities?

West Coast v East Coast Tech Scene Mudslinging

Antonio at Adgrok collates a grab bag of reasons as to why NYC isn’t as good a place for (tech) start-ups, but forgets a few keypoints: NYC is the world center for finance, entertainment, art, and media circles, while the Bay Area is the center of its own circle. Silicon Valley was originally the outgrowth of government funding for Standford, and later for the development of the semiconductor industry. The rest just happened because the VCs and established companies were there. So, once VCs and established tech companies exist in NYC — which we are seeing — we can expect similar results.

I lived in SF halftime for five years and I was struck by how insular the tech scene is. Perhaps my disaffection was personal, or I am simply more of an east coast guy. But I think New York is the place where the most social scenes intersect, here in America. It has what Brian Eno calls scenius, the communal form of genius, in abundance. True, SF has its own scenius as well, but it is narrower, more tech-obsessed, and less international than NYC’s.

The fact that Antonio doesn’t give that a value higher than low-cost housing is a sign of some of what is wrong in the Bay Area these days.

And trying to diss NYC’s food offerings is laughable. But I agree about Katz’s Deli making a great pastrami.

related articles

• Hotbed (stoweboyd.com)
• Scenius (stoweboyd.com)

Social Scenes: The Invisible Calculus Of Culture

Today, in New York, I heard Clay Shirky talk two times — midday at the Betaworks monthly Brown Bag Lunch, and this evening at the New York Tech Meetup — on the same topic. He is extrapolating in very interesting ways from the research of social scientists Nicholas Christakis and James Fowler on the social dimension buried in the data of the Framingham Heart Study.

In a nutshell, it turns out that the activities of the ‘third neighborhood’ influence you in ways you may be completely unaware of.  These are people that you do not know, but are (dis)connected to you by two removes: the friends of your friends’ friends. Christakis and Fowler found that obesity, smoking, and many other medical factors strongly correlated with the prevalence of corresponding activities in these large social scenes:

- Clive Thompson, Are Your Friends Making You Fat?

Christakis knew about the Framingham Heart Study and arranged a visit to the town to learn more. The study seemed promising: he knew it had been underway for more than 50 years and had followed more than 15,000 people, spanning three generations, so in theory, at least, it could offer a crucial moving picture. But how to track social connections? During his visit, Christakis asked one of the coordinators of the study how she and her colleagues were able to stay in contact with so many people for so long. What happened if a family moved away? The woman reached under her desk and pulled out a green sheet. It was a form that staff members used to collect information from every participant each time they came in to be examined — and it asked them to list all their family and at least one of their friends. “They asked you, ‘Who is your spouse, who are your children, who are your parents, who are your siblings, where do they live, who is your doctor, where do you work, where do you live, who is a close friend who would know where to find you in four years if we can’t find you?” Christakis said. “And they were writing all this stuff down.” He felt a jolt of excitement: he and Fowler could use these thousands of green forms to manually reconstruct the social ties of Framingham — who knew whom, going back decades.

Over the next few years, Christakis and Fowler managed a team that painstakingly sifted through the records. When they were done, they had a map of how 5,124 subjects were connected, tracing a web of 53,228 ties between friends and family and work colleagues. Next they analyzed the data, beginning with tracking patterns of how and when Framingham residents became obese. Soon they had created an animated diagram of the entire social network, with each resident represented on their computer screens as a dot that grew bigger or smaller as he or she gained or lost weight over 32 years, from 1971 to 2003. When they ran the animation, they could see that obesity broke out in clusters. People weren’t just getting fatter randomly. Groups of people would become obese together, while other groupings would remain slender or even lose weight.

And the social effect appeared to be quite powerful. When a Framingham resident became obese, his or her friends were 57 percent more likely to become obese, too. Even more astonishing to Christakis and Fowler was the fact that the effect didn’t stop there. In fact, it appeared to skip links. A Framingham resident was roughly 20 percent more likely to become obese if the friend of a friend became obese — even if the connecting friend didn’t put on a single pound. Indeed, a person’s risk of obesity went up about 10 percent even if a friend of a friend of a friend gained weight.

“People are connected, and so their health is connected,” Christakis and Fowler concluded when they summarized their findings in a July 2007 article in The New England Journal of Medicine, the first time the prestigious journal published a study of how social networks affect health. Or as Christakis and Fowler put it in “Connected,” their coming book on their findings: “You may not know him personally, but your friend’s husband’s co-worker can make you fat. And your sister’s friend’s boyfriend can make you thin.

Obesity was only the beginning. Over the next year, the sociologist and the political scientist continued to analyze the Framingham data, finding more and more examples of contagious behavior. Smoking, they discovered, also appeared to spread socially — in fact, a friend taking up smoking increased your chance of lighting up by 36 percent, and if you had a three-degrees-removed friend who started smoking, you were 11 percent more likely to do the same. Drinking spread socially, as did happiness and even loneliness. And in each case one’s individual influence stretched out three degrees before it faded out. They termed this the “three degrees of influence” rule about human behavior: We are tied not just to those around us, but to others in a web that stretches farther than we know.

This research brings to mind the obsrvation of Blaise Pascal, “The heart has its reasons that the mind knows not.” It appears that negative behaviors like overeating and smoking are in some hidden way transmitted through our social networks, even when we are not in contact with those others who are influencing us. Likewise, it turns out that happiness is spread in a similarly diffuse and oblique fashion.

Getting back to Clay: he wonders what this means for the way that modern social tools work, like Twitter, for example.

In social tools, we are each the center of our own universe, and we are connected to our friends (who are each the center of their own universes, too). We are aware that our friends have friends we don’t know (or are aware of in the most insubstantial of ways). And these friends of friends likewise have friends, which are unknown to us as well.

But despite their anonymity and distance from us, they are influencing us, as Christakis and Fowler showed. But out tools, like Twitter, don’t allow us to deal with this mass of people — which is likely to be on the order of a million people, plus or minus — in any way at all. It is not addressible, or searchable, or filterable. I can’t find out what TV my social scene is watching, or what music they like, or how they voted in the last election.

Shirky points out that it is easy to find out what my friends are doing, or what the world as a whole is doing, but what the world is doing is fairly ‘bland’ as he puts it. The world’s combined interests lead to the dropping out of all the odd and eclectic, and you are left with Lady Gaga and Obama. BIg surprise.

But my social scene — the group that actually influences my thinking, moods, and buying behavior — in completely untapped and untappable by out tools today.

However, its clear that it could be tapped: just as in the Framingham Heart Study. It’s possible (and not even very technically challening) to create the swirling, dynamic, and ever-changing opinions and activities of your one million closest ‘friends’, only a few hundred that you know well and perhaps a few thousand that you ‘know of’. We are all surrounded by ‘dark matter’, the next ring in the social cosmology out past those you know and the friends of those you know: a million people exerting an invisible influence on those that influence those that influence you. If that group is down on smoking, you will be getting social cues to not smoke. If they are crazy about Korean food, you will be served kim chi at dinner parties. If they are into country and western music, you will find yourself shopping for cowboy boots with your cousin.

Shirky clearly states that he doesn’t know where this will lead, even if he is right. But I think that it is obvious that we would like to explicit see and measure the influences in our unverse (each in their own overlapping universes), so on a personal level this may be a tremendous adjunct to the filtering, amplifying, and serendipity that we all want social tools to help us with. And perhaps just as much as a possible driver of technical experimentation in this sector, companies would like to know how influence is channeled and how it impacts individuals. The underbelly of this is exactly that: that marketers would like to tap into this social juju, and influence us through social ties that we can’t even touch directly.

But it is always the brightest light that casts the darkest shadow.

It comes as no surprise that there is value — and power — in identifying the wellspring of our desires and the foundation of our apsirations. Social scenes may turn out to be the crux of this transitive and reflexive influence that we exchange in ten thousand ways, every day. If it turns out that our place in the world — our position in an invisble sphere of one million almost friends of ours — defines strongly who we are, what we love, and who we hate, would we be surprised? Not me.

This may be just the face of tribalism, proven through scientific observation. I am choosing to use the term ‘social scene’ though, because tribalism has so many connotations and associations that could take us off the track. Also, it was Brian Eno that coined the term ‘scenius’ to represent the positive side of a social scene:

via Kevin Kelly, quoting Brian Eno

Scenius stands for the intelligence and the intuition of a whole cultural scene. It is the communal form of the concept of the genius.

We are a result of the accumulated sum of influences that are being tallied behind our backs, and behind the backs of all those that we know. Apparently, we are impacted by a hidden calculus in which we are the integral of the specturm of influences on all those we hold dear.

The Bantu people have a saying “Through people we become human,” and ever aspect of our identity and psychology is shaped by the cultural milieu in which we are part. I have said for years, “I am made greater by the sum of my connections, and so are my connections,’ alluding in a recursive way to these hidden network dynamics. And of course we want better tools to bring these indistinct and indirect forces into high relief. Clay is right about that.