Reaching The Limits: Twitter-in-the-Large and Subtags
Two scare pieces have appeared at around the same moment, both ignited by the spike in Twitter usage that accompanies SxSW.
The basic premise is that we are reaching — or have already reached — the limits of what I will call Twitter-in-the-Small, an era when Twitter was a community of a few hundred thousand bitheads. As the Twittersphere grows to include millions — and not too longer after, hundreds of millions — very different practices and technologies may be required for the service to scale into Twitter-in-the-Large.
Daniel Terdiman touches on the limitations of today’s #hashtags as a means of indicating the topic of tweets:
[At SXSW, attendees confront Twitter saturation | Webware - CNET.
[…]
At SXSW, the standard is for everyone to include the tag “#sxsw” in their tweets. For example, on Friday, I was looking for sources for a different story and tweeted, “If you are launching an iPhone app at #sxsw, or know someone who is, please let me know. Thanks!”
That’s a great convention because it allows anyone wanting to know what’s going on to search Twitter for posts using any search term important to them. That has proven useful for people wanting to find out what’s going on after earthquakes, the Mumbai terrorist attacks, the Hudson River airplane crash, and many other events. At SXSW in 2007 and 2008, this was a big part of how people navigated their experiences.
At a conference with scores of panels and seemingly just as many parties, being able to determine what’s worthwhile is crucial for people trying to get the most out of their time here.
This year because of the conference’s impressive growth and Twitter’s broader mainstream appeal, it has become almost impossible to find the same value as in the past. I did a search for the “#sxsw” tag on Saturday afternoon and found that there had been 392 tweets with the term in just the previous 10 minutes. That number mushroomed to more than 1,500 in the previous hour.
There were nearly 400 tweets using the #sxsw tag in just 10 minutes during the SXSW conference on Saturday afternoon.
While those numbers demonstrate that people here are without question using Twitter like never before, it also means that it’s never been harder to find what you’re looking for amid the flood of posts about the panels, barbecue, Web celebrity spottings, and deep thoughts about social media.
This has forced people accustomed to relying on simple Twitter searches to get creative to find the nuggets they need.
So… the limits here are a function of the simple model of one #hashtag to denote an entire swirling mess like SxSW.
Many conferences in the past have created more structured #hastags to indicate different tracks or sessions, such as the Web 2.0 Expo. Those events often use a base for the #hashtags, like #w2e, and then append other letters to indicate tracks and sessions, like #w2em2, which was a session called “Marketing 2.0”.
The problem around this approach is that the #hashtags are top-down: the conference organizers have to create them and promote them.
Alternatively, conventions could arise spontaneously to solve the #hashtag overload problem. Recall that #hashtags themselves are a more or less bottom-up convention, that Chris Messina suggested, leading to people like us using #hashtags before any specific tools existed to support them.
One solution is to use combinations of tags — like “#sxsw #party” or “#w2e #marketing” — in combination with smarter tools and search. This is already happening in part.
An alternative is to build more structure into the tags. We could start using some delimiter that explicitly makes tags richer, like ‘.’ or ‘/’. For example, “#sxsw.party” or “#w2e.marketing”.
This approach, which I call subtags, has the benefit of being more concise, and more obviously subcategorizing the tweet as related to SxSW parties or marketing sessions at Web 2.0 Expo. The problem is that Twitter search deals with these delimiters in strange ways. For example, using the ‘.’ works for the search:
#hashtag.sub - Twitter Search, originally uploaded by Stowe Boyd.
But in the search result, the #hashtag link is truncated to “#hashtag” dropping off the suffix standing for the subcategory. Of course the fix to this minor glitch is simple: the Twitter search guys, and anyone else dealing with #hashtags, simply have to tweak their code to do the right thing for subtags. And the right thing is half implemented: the base tag points to the “#hashtag” search. The fix is to make the “.sub” suffix point to the search for “#hashtag.sub”. And tweak to find all SxSW subtags in the search for “#sxsw”, which also doesn’t work at the moment.
Nova Spivack writes a similarly panicked piece on Twitter-in-the-Large, where he suggests that very different filtering regimes will be necessary since the arrivistes will be spewing all sorts of crap into the Twittersphere, and we will drown before we find tweets of interest:
[via Can Twitter Survive What is About to Happen to It?
The Solution: New Ways to Filter Twitter
The solution to this is filtering. But filtering capabilities are weak at best in existing Twitter apps. And even if app developers start adding them, there are limitations built into Twitter’s messaging system that make it difficult to do sophisticated filtering.
Number of Followers as a Filter. One way to filter would be to use social filtering to infer the value of content. For example, content by people with more followers might have a higher reputation score. But let’s face it, there are people on Twitter who are acquiring followers using all sorts of tricky techniques — like using auto-follow or simply following everyone they can find in the hopes that they will be followed back. Or offering money or prizes to followers — a recent trend. The number of followers someone has does not necessarily reflect reputation.
Re-Tweeting Activity as a Filter. A better measure of reputation might be how many times someone is re-tweeted. RT’s definitely indicate whether someone is adding value to the network. That is worth considering.
Social Network Analysis as a Filter. One might also analyze the social graph to build filters. For example, by looking at who is followed by who. Something similar to Google PageRank might even be possible in Twitter. You could figure out that for certain topics, certain people are more central than others, by analyzing how many other people who tweet about those topics are following them. Ok good. Nobody can patent this now.
Some of these concepts are fairly shopworn at this point. I think the number of followers argument has been hashed and rehashed, with the conventional wisdom being that the number of followers is only a indicator of something, since the system is too easily gamed. But it has the benefit of being simple to measure, and it is proudly emblazoned on all our Twitter pages while none of the more sophisticated possibilities are.
Retweeting is an interesting metric, which is perhaps correlated with influence, but the jury is out on that, since it is tied to number of followers. So it could be that ‘Twisdom’ is some measure based on Retweets/Followers: the ability to jump on things that other people want to retweet. But I am still unsure on that.
The social network analysis concept has been around for eons, just not directly applied in Twitter: this is the idea of karma, where a social reputation is built through an analysis of who you are connected to, who responds to your messages, and so on.
Most importantly, we learned during the big surge into blogging a few years back that the newbies are not very discerning about people’s historic reputations, and the laws of large numbers lead to large numbers becoming the only stat that seems to resonate with those of less subtle discernment. Therefore, I predict that the only metrics that matter will be the ones that Twitter, Inc. offers up as part of our profiles, which is Following and Followers. Since Following is totally in the hands of the individual, the only central metric that represents popularity and maybe value is Followers. So, despite the learned psychological dissection of the topic by all of us, Followers will be the thing that matters.
As a result, when media sources want to talk to someone knowledgeable about Twitter, they will ping someone with 117K followers, and not someone with 7800. The cycle perpetuates itself, as more people learn about that popular twitterer, they gather more followers: the inexorable power laws at work.
Spivak suggests that new sorts of metadata might help the overload problem:
Metadata for Filtering. But we are going to need more than inferred filtering I believe. We are going to need ways to filter Twitter messages by sender, type of content, size, publisher, trust, popularity, content rating, MIME type, etc. This is going to require metadata in Twitter, ultimately.
Broadly speaking there are two main ways that metadata could be added to Twitter:
- Metadata Added Outside Twitter. Twitter messages could simply be URLs that point to further resources that in turn carry the actual body and metadata of each message. Thus a message might just be a single URL. Clicking that URL would yield a web page with the content and then XML or RDF metadata about the message. If this were to happen, Twitter messages would be simply URLs created and sent by outside client software — and they would require outside software (special Twitter clients) to unpack and read them.
- Metadata Added Inside Twitter. Another solution would be for Twitter to extend their message schema so every Twitter message has two parts, a 140 char body and a metadata section with a certain amount of space as well. This would be great. It would be a good move for the people at Twitter to jump the gun by enabling this sooner rather than later. It will help them protect their control over their own franchise.
I bet that neither of these solutions will happen as part of the actual Twitter stack. However, all manner of external companies will arise that would like to provide better lenses to make sense of the Twittersphere as it explodes, like the expanding galaxy. Those advances we will have to wait for, but we can start using subtags today.
