A speech at ETech, April, 2003; published July 1, 2003 on the “Networks, Economics, and Culture” mailing list.

This is a lightly edited version of the keynote I gave on Social Software at the O’Reilly Emerging Technology conference in Santa Clara on April 24, 2003
Good morning, everybody. I want to talk this morning about social software …there’s a surprise. I want to talk about a pattern I’ve seen over and over again in social software that supports large and long-lived groups. And that pattern is the pattern described in the title of this talk: “A Group Is Its Own Worst Enemy.”

In particular, I want to talk about what I now think is one of the core challenges for designing large-scale social software. Let me offer a definition of social software, because it’s a term that’s still fairly amorphous. My definition is fairly simple: It’s software that supports group interaction. I also want to emphasize, although that’s a fairly simple definition, how radical that pattern is. The Internet supports lots of communications patterns, principally point-to-point and two-way, one-to-many outbound, and many-to-many two-way.

Prior to the Internet, we had lots of patterns that supported point-to-point two-way. We had telephones, we had the telegraph. We were familiar with technological mediation of those kinds of conversations. Prior to the Internet, we had lots of patterns that supported one-way outbound. I could put something on television or the radio, I could publish a newspaper. We had the printing press. So although the Internet does good things for those patterns, they’re patterns we knew from before.

Prior to the Internet, the last technology that had any real effect on the way people sat down and talked together was the table. There was no technological mediation for group conversations. The closest we got was the conference call, which never really worked right — “Hello? Do I push this button now? Oh, shoot, I just hung up.” It’s not easy to set up a conference call, but it’s very easy to email five of your friends and say “Hey, where are we going for pizza?” So ridiculously easy group forming is really news.

We’ve had social software for 40 years at most, dated from the Plato BBS system, and we’ve only had 10 years or so of widespread availability, so we’re just finding out what works. We’re still learning how to make these kinds of things.

Now, software that supports group interaction is a fundamentally unsatisfying definition in many ways, because it doesn’t point to a specific class of technology. If you look at email, it obviously supports social patterns, but it can also support a broadcast pattern. If I’m a spammer, I’m going to mail things out to a million people, but they’re not going to be talking to one another, and I’m not going to be talking to them — spam is email, but it isn’t social. If I’m mailing you, and you’re mailing me back, we’re having point-to-point and two-way conversation, but not one that creates group dynamics.

So email doesn’t necessarily support social patterns, group patterns, although it can. Ditto a weblog. If I’m Glenn Reynolds, and I’m publishing something with Comments Off and reaching a million users a month, that’s really broadcast. It’s interesting that I can do it as a single individual, but the pattern is closer to MSNBC than it is to a conversation. If it’s a cluster of half a dozen LiveJournal users, on the other hand, talking about their lives with one another, that’s social. So, again, weblogs are not necessarily social, although they can support social patterns.

Nevertheless, I think that definition is the right one, because it recognizes the fundamentally social nature of the problem. Groups are a run-time effect. You cannot specify in advance what the group will do, and so you can’t substantiate in software everything you expect to have happen.

Now, there’s a large body of literature saying “We built this software, a group came and used it, and they began to exhibit behaviors that surprised us enormously, so we’ve gone and documented these behaviors.” Over and over and over again this pattern comes up. (I hear Stewart [Brand, of the WELL] laughing.) The WELL is one of those places where this pattern came up over and over again.

This talk is in three parts. The best explanation I have found for the kinds of things that happen when groups of humans interact is psychological research that predates the Internet, so the first part is going to be about W.R. Bion’s research, which I will talk about in a moment, research that I believe explains how and why a group is its own worst enemy.

The second part is: Why now? What’s going on now that makes this worth thinking about? I think we’re seeing a revolution in social software in the current environment that’s really interesting. 

And third, I want to identify some things, about half a dozen things, in fact, that I think are core to any software that supports larger, long-lived groups. 

Part One: How is a group its own worst enemy?

So, Part One. The best explanation I have found for the ways in which this pattern establishes itself, the group is its own worst enemy, comes from a book by W.R. Bion called “Experiences in Groups,” written in the middle of the last century.

Bion was a psychologist who was doing group therapy with groups of neurotics. (Drawing parallels between that and the Internet is left as an exercise for the reader.) The thing that Bion discovered was that the neurotics in his care were, as a group, conspiring to defeat therapy. 

There was no overt communication or coordination. But he could see that whenever he would try to do anything that was meant to have an effect, the group would somehow quash it. And he was driving himself crazy, in the colloquial sense of the term, trying to figure out whether or not he should be looking at the situation as: Are these individuals taking action on their own? Or is this a coordinated group?

He could never resolve the question, and so he decided that the unresolvability of the question was the answer. To the question: Do you view groups of people as aggregations of individuals or as a cohesive group, his answer was: “Hopelessly committed to both.”

He said that humans are fundamentally individual, and also fundamentally social. Every one of us has a kind of rational decision-making mind where we can assess what’s going on and make decisions and act on them. And we are all also able to enter viscerally into emotional bonds with other groups of people that transcend the intellectual aspects of the individual. 

In fact, Bion was so convinced that this was the right answer that the image he put on the front cover of his book was a Necker cube, one of those cubes that you can look at and make resolve in one of two ways, but you can never see both views of the cube at the same time. So groups can be analyzed both as collections of individuals and having this kind of emotive group experience. 

Now, it’s pretty easy to see how groups of people who have formal memberships, groups that have been labeled and named like “I am a member of such-and-such a guild in a massively multi-player online role-playing game,” it’s easy to see how you would have some kind of group cohesion there. But Bion’s thesis is that this effect is much, much deeper, and kicks in much, much sooner than many of us expect. So I want to illustrate this with a story, and to illustrate the illustration, I’ll use a story from your life. Because even if I don’t know you, I know what I’m about to describe has happened to you. 

You are at a party, and you get bored. You say “This isn’t doing it for me anymore. I’d rather be someplace else. I’d rather be home asleep. The people I wanted to talk to aren’t here.” Whatever. The party fails to meet some threshold of interest. And then a really remarkable thing happens: You don’t leave. You make a decision “I don’t like this.” If you were in a bookstore and you said “I’m done,” you’d walk out. If you were in a coffee shop and said “This is boring,” you’d walk out.

You’re sitting at a party, you decide “I don’t like this; I don’t want to be here.” And then you don’t leave. That kind of social stickiness is what Bion is talking about. 

And then, another really remarkable thing happens. Twenty minutes later, one person stands up and gets their coat, and what happens? Suddenly everyone is getting their coats on, all at the same time. Which means that everyone had decided that the party was not for them, and no one had done anything about it, until finally this triggering event let the air out of the group, and everyone kind of felt okay about leaving.

This effect is so steady it’s sometimes called the paradox of groups. It’s obvious that there are no groups without members. But what’s less obvious is that there are no members without a group. Because what would you be a member of? 

So there’s this very complicated moment of a group coming together, where enough individuals, for whatever reason, sort of agree that something worthwhile is happening, and the decision they make at that moment is: This is good and must be protected. And at that moment, even if it’s subconscious, you start getting group effects. And the effects that we’ve seen come up over and over and over again in online communities.

Now, Bion decided that what he was watching with the neurotics was the group defending itself against his attempts to make the group do what they said they were supposed to do. The group was convened to get better, this group of people was in therapy to get better. But they were defeating that. And he said, there are some very specific patterns that they’re entering into to defeat the ostensible purpose of the group meeting together. And he detailed three patterns.

The first is sex talk, what he called, in his mid-century prose, “A group met for pairing off.” And what that means is, the group conceives of its purpose as the hosting of flirtatious or salacious talk or emotions passing between pairs of members.

You go on IRC and you scan the channel list, and you say “Oh, I know what that group is about, because I see the channel label.” And you go into the group, you will also almost invariably find that it’s about sex talk as well. Not necessarily overt. But that is always in scope in human conversations, according to Bion. That is one basic pattern that groups can always devolve into, away from the sophisticated purpose and towards one of these basic purposes.

The second basic pattern that Bion detailed: The identification and vilification of external enemies. This is a very common pattern. Anyone who was around the Open Source movement in the mid-Nineties could see this all the time. If you cared about Linux on the desktop, there was a big list of jobs to do. But you could always instead get a conversation going about Microsoft and Bill Gates. And people would start bleeding from their ears, they would get so mad.

If you want to make it better, there’s a list of things to do. It’s Open Source, right? Just fix it. “No, no, Microsoft and Bill Gates grrrrr …”, the froth would start coming out. The external enemy — nothing causes a group to galvanize like an external enemy.

So even if someone isn’t really your enemy, identifying them as an enemy can cause a pleasant sense of group cohesion. And groups often gravitate towards members who are the most paranoid and make them leaders, because those are the people who are best at identifying external enemies.

The third pattern Bion identified: Religious veneration. The nomination and worship of a religious icon or a set of religious tenets. The religious pattern is, essentially, we have nominated something that’s beyond critique. You can see this pattern on the Internet any day you like. Go onto a Tolkein newsgroup or discussion forum, and try saying “You know, The Two Towers is a little dull. I mean loooong. We didn’t need that much description about the forest, because it’s pretty much the same forest all the way.”

Try having that discussion. On the door of the group it will say: “This is for discussing the works of Tolkein.” Go in and try and have that discussion. 

Now, in some places people say “Yes, but it needed to, because it had to convey the sense of lassitude,” or whatever. But in most places you’ll simply be flamed to high heaven, because you’re interfering with the religious text.

So these are human patterns that have shown up on the Internet, not because of the software, but because it’s being used by humans. Bion has identified this possibility of groups sandbagging their sophisticated goals with these basic urges. And what he finally came to, in analyzing this tension, is that group structure is necessary. Robert’s Rules of Order are necessary. Constitutions are necessary. Norms, rituals, laws, the whole list of ways that we say, out of the universe of possible behaviors, we’re going to draw a relatively small circle around the acceptable ones.

He said the group structure is necessary to defend the group from itself. Group structure exists to keep a group on target, on track, on message, on charter, whatever. To keep a group focused on its own sophisticated goals and to keep a group from sliding into these basic patterns. Group structure defends the group from the action of its own members. 

In the Seventies — this is a pattern that’s shown up on the network over and over again — in the Seventies, a BBS called Communitree launched, one of the very early dial-up BBSes. This was launched when people didn’t own computers, institutions owned computers.

Communitree was founded on the principles of open access and free dialogue. “Communitree” — the name just says “California in the Seventies.” And the notion was, effectively, throw off structure and new and beautiful patterns will arise.

And, indeed, as anyone who has put discussion software into groups that were previously disconnected has seen, that does happen. Incredible things happen. The early days of Echo, the early days of usenet, the early days of Lucasfilms Habitat, over and over again, you see all this incredible upwelling of people who suddenly are connected in ways they weren’t before.

And then, as time sets in, difficulties emerge. In this case, one of the difficulties was occasioned by the fact that one of the institutions that got hold of some modems was a high school. And who, in 1978, was hanging out in the room with the computer and the modems in it, but the boys of that high school. And the boys weren’t terribly interested in sophisticated adult conversation. They were interested in fart jokes. They were interested in salacious talk. They were interested in running amok and posting four-letter words and nyah-nyah-nyah, all over the bulletin board.

And the adults who had set up Communitree were horrified, and overrun by these students. The place that was founded on open access had too much open access, too much openness. They couldn’t defend themselves against their own users. The place that was founded on free speech had too much freedom. They had no way of saying “No, that’s not the kind of free speech we meant.”

But that was a requirement. In order to defend themselves against being overrun, that was something that they needed to have that they didn’t have, and as a result, they simply shut the site down.

Now you could ask whether or not the founders’ inability to defend themselves from this onslaught, from being overrun, was a technical or a social problem. Did the software not allow the problem to be solved? Or was it the social configuration of the group that founded it, where they simply couldn’t stomach the idea of adding censorship to protect their system. But in a way, it doesn’t matter, because technical and social issues are deeply intertwined. There’s no way to completely separate them.

What matters is, a group designed this and then was unable, in the context they’d set up, partly a technical and partly a social context, to save it from this attack from within. And attack from within is what matters. Communitree wasn’t shut down by people trying to crash or syn-flood the server. It was shut down by people logging in and posting, which is what the system was designed to allow. The technological pattern of normal use and attack were identical at the machine level, so there was no way to specify technologically what should and shouldn’t happen. Some of the users wanted the system to continue to exist and to provide a forum for discussion. And other of the users, the high school boys, either didn’t care or were actively inimical. And the system provided no way for the former group to defend itself from the latter.

Now, this story has been written many times. It’s actually frustrating to see how many times it’s been written. You’d hope that at some point that someone would write it down, and they often do, but what then doesn’t happen is other people don’t read it.

The most charitable description of this repeated pattern is “learning from experience.” But learning from experience is the worst possible way to learn something. Learning from experience is one up from remembering. That’s not great. The best way to learn something is when someone else figures it out and tells you: “Don’t go in that swamp. There are alligators in there.” 

Learning from experience about the alligators is lousy, compared to learning from reading, say. There hasn’t been, unfortunately, in this arena, a lot of learning from reading. And so, lessons from Lucasfilms’ Habitat, written in 1990, reads a lot like Rose Stone’s description of Communitree from 1978.

This pattern has happened over and over and over again. Someone built the system, they assumed certain user behaviors. The users came on and exhibited different behaviors. And the people running the system discovered to their horror that the technological and social issues could not in fact be decoupled.

There’s a great document called “LambdaMOO Takes a New Direction,” which is about the wizards of LambdaMOO, Pavel Curtis’s Xerox PARC experiment in building a MUD world. And one day the wizards of LambdaMOO announced “We’ve gotten this system up and running, and all these interesting social effects are happening. Henceforth we wizards will only be involved in technological issues. We’re not going to get involved in any of that social stuff.”

And then, I think about 18 months later — I don’t remember the exact gap of time — they come back. The wizards come back, extremely cranky. And they say: “What we have learned from you whining users is that we can’t do what we said we would do. We cannot separate the technological aspects from the social aspects of running a virtual world.

“So we’re back, and we’re taking wizardly fiat back, and we’re going to do things to run the system. We are effectively setting ourselves up as a government, because this place needs a government, because without us, the place was falling apart.”

People who work on social software are closer in spirit to economists and political scientists than they are to people making compilers. They both look like programming, but when you’re dealing with groups of people as one of your run-time phenomena, that is an incredibly different practice. In the political realm, we would call these kinds of crises a constitutional crisis. It’s what happens when the tension between the individual and the group, and the rights and responsibilities of individuals and groups, gets so serious that something has to be done.

And the worst crisis is the first crisis, because it’s not just “We need to have some rules.” It’s also “We need to have some rules for making some rules.” And this is what we see over and over again in large and long-lived social software systems. Constitutions are a necessary component of large, long-lived, heterogenous groups. 

Geoff Cohen has a great observation about this. He said “The likelihood that any unmoderated group will eventually get into a flame-war about whether or not to have a moderator approaches one as time increases.” As a group commits to its existence as a group, and begins to think that the group is good or important, the chance that they will begin to call for additional structure, in order to defend themselves from themselves, gets very, very high.

Part Two: Why now? 

If these things I’m saying have happened so often before, have been happening and been documented and we’ve got psychological literature that predates the Internet, what’s going on now that makes this important?

I can’t tell you precisely why, but observationally there is a revolution in social software going on. The number of people writing tools to support or enhance group collaboration or communication is astonishing.

The web turned us all into size queens for six or eight years there. It was loosely coupled, it was stateless, it scaled like crazy, and everything became about How big can you get? “How many users does Yahoo have? How many customers does Amazon have? How many readers does MSNBC have?” And the answer could be “Really a lot!” But it could only be really a lot if you didn’t require MSNBC to be answering those readers, and you didn’t require those readers to be talking to one another.

The downside of going for size and scale above all else is that the dense, interconnected pattern that drives group conversation and collaboration isn’t supportable at any large scale. Less is different — small groups of people can engage in kinds of interaction that large groups can’t. And so we blew past that interesting scale of small groups. Larger than a dozen, smaller than a few hundred, where people can actually have these conversational forms that can’t be supported when you’re talking about tens of thousands or millions of users, at least in a single group.

We’ve had things like mailing lists and BBSes for a long time, and more recently we’ve had IM, we’ve had these various patterns. And now, all of a sudden, these things are popping up. We’ve gotten weblogs and wikis, and I think, even more importantly, we’re getting platform stuff. We’re getting RSS. We’re getting shared Flash objects. We’re getting ways to quickly build on top of some infrastructure we can take for granted, that lets us try new things very rapidly.

I was talking to Stewart Butterfield about the chat application they’re trying here. I said “Hey, how’s that going?” He said: “Well, we only had the idea for it two weeks ago. So this is the launch.” When you can go from “Hey, I’ve got an idea” to “Let’s launch this in front of a few hundred serious geeks and see how it works,” that suggests that there’s a platform there that is letting people do some really interesting things really quickly. It’s not that you couldn’t have built a similar application a couple of years ago, but the cost would have been much higher. And when you lower costs, interesting new kinds of things happen.

So the first answer to Why Now? is simply “Because it’s time.” I can’t tell you why it took as long for weblogs to happen as it did, except to say it had absolutely nothing to do with technology. We had every bit of technology we needed to do weblogs the day Mosaic launched the first forms-capable browser. Every single piece of it was right there. Instead, we got Geocities. Why did we get Geocities and not weblogs? We didn’t know what we were doing. 

One was a bad idea, the other turns out to be a really good idea. It took a long time to figure out that people talking to one another, instead of simply uploading badly-scanned photos of their cats, would be a useful pattern. 

We got the weblog pattern in around ’96 with Drudge. We got weblog platforms starting in ’98. The thing really was taking off in 2000. By last year, everyone realized: Omigod, this thing is going mainstream, and it’s going to change everything. 

The vertigo moment for me was when Phil Gyford launched the Pepys weblog, Samuel Pepys’ diaries of the 1660’s turned into a weblog form, with a new post every day from Pepys’ diary. What that said to me was: Phil was asserting, and I now believe, that weblogs will be around for at least 10 years, because that’s how long Pepys kept a diary. And that was this moment of projecting into the future: This is now infrastructure we can take for granted.

Why was there an eight-year gap between a forms-capable browser and the Pepys diaries? I don’t know. It just takes a while for people to get used to these ideas.

So, first of all, this is a revolution in part because it is a revolution. We’ve internalized the ideas and people are now working with them. Second, the things that people are now building are web-native.

When you got social software on the web in the mid-Nineties, a lot of it was: “This is the Giant Lotus Dreadnought, now with New Lightweight Web Interface!” It never felt like the web. It felt like this hulking thing with a little, you know, “Here’s some icons. Don’t look behind the curtain.”

A weblog is web-native. It’s the web all the way in. A wiki is a web-native way of hosting collaboration. It’s lightweight, it’s loosely coupled, it’s easy to extend, it’s easy to break down. And it’s not just the surface, like oh, you can just do things in a form. It assumes http is transport. It assumes markup in the coding. RSS is a web-native way of doing syndication. So we’re taking all of these tools and we’re extending them in a way that lets us build new things really quickly.

Third, in David Weinberger’s felicitous phrase, we can now start to have a Small Pieces Loosely Joined pattern. It’s really worthwhile to look into what Joi Ito is doing with the Emergent Democracy movement, even if you’re not interested in the themes of emerging democracy. This started because a conversation was going on, and Ito said “I am frustrated. I’m sitting here in Japan, and I know all of these people are having these conversations in real-time with one another. I want to have a group conversation, too. I’ll start a conference call.

“But since conference calls are so lousy on their own, I’m going to bring up a chat window at the same time.” And then, in the first meeting, I think it was Pete Kaminski said “Well, I’ve also opened up a wiki, and here’s the URL.” And he posted it in the chat window. And people can start annotating things. People can start adding bookmarks; here are the lists.

So, suddenly you’ve got this meeting, which is going on in three separate modes at the same time, two in real-time and one annotated. So you can have the conference call going on, and you know how conference calls are. Either one or two people dominate it, or everyone’s like “Oh, can I — no, but –“, everyone interrupting and cutting each other off.

It’s very difficult to coordinate a conference call, because people can’t see one another, which makes it hard to manage the interrupt logic. In Joi’s conference call, the interrupt logic got moved to the chat room. People would type “Hand,” and the moderator of the conference call will then type “You’re speaking next,” in the chat. So the conference call flowed incredibly smoothly.

Meanwhile, in the chat, people are annotating what people are saying. “Oh, that reminds me of So-and-so’s work.” Or “You should look at this URL…you should look at that ISBN number.” In a conference call, to read out a URL, you have to spell it out — “No, no, no, it’s w w w dot net dash…” In a chat window, you get it and you can click on it right there. You can say, in the conference call or the chat: “Go over to the wiki and look at this.”

This is a broadband conference call, but it isn’t a giant thing. It’s just three little pieces of software laid next to each other and held together with a little bit of social glue. This is an incredibly powerful pattern. It’s different from: Let’s take the Lotus juggernaut and add a web front-end.

And finally, and this is the thing that I think is the real freakout, is ubiquity. The web has been growing for a long, long time. And so some people had web access, and then lots of people had web access, and then most people had web access.

But something different is happening now. In many situations, all people have access to the network. And “all” is a different kind of amount than “most.” “All” lets you start taking things for granted.

Now, the Internet isn’t everywhere in the world. It isn’t even everywhere in the developed world. But for some groups of people — students, people in high-tech offices, knowledge workers — everyone they work with is online. Everyone they’re friends with is online. Everyone in their family is online.

And this pattern of ubiquity lets you start taking this for granted. Bill Joy once said “My method is to look at something that seems like a good idea and assume it’s true.” We’re starting to see software that simply assumes that all offline groups will have an online component, no matter what.

It is now possible for every grouping, from a Girl Scout troop on up, to have an online component, and for it to be lightweight and easy to manage. And that’s a different kind of thing than the old pattern of “online community.” I have this image of two hula hoops, the old two-hula hoop world, where my real life is over here, and my online life is over there, and there wasn’t much overlap between them. If the hula hoops are swung together, and everyone who’s offline is also online, at least from my point of view, that’s a different kind of pattern.

There’s a second kind of ubiquity, which is the kind we’re enjoying here thanks to Wifi. If you assume whenever a group of people are gathered together, that they can be both face to face and online at the same time, you can start to do different kinds of things. I now don’t run a meeting without either having a chat room or a wiki up and running. Three weeks ago I ran a meeting for the Library of Congress. We had a wiki, set up by Socialtext, to capture a large and very dense amount of technical information on long-term digital preservation.

The people who organized the meeting had never used a wiki before, and now the Library of Congress is talking as if they always had a wiki for their meetings, and are assuming it’s going to be at the next meeting as well — the wiki went from novel to normal in a couple of days.

It really quickly becomes an assumption that a group can do things like “Oh, I took my PowerPoint slides, I showed them, and then I dumped them into the wiki. So now you can get at them.” It becomes a sort of shared repository for group memory. This is new. These kinds of ubiquity, both everyone is online, and everyone who’s in a room can be online together at the same time, can lead to new patterns.

Part Three: What can we take for granted?

If these assumptions are right, one that a group is its own worst enemy, and two, we’re seeing this explosion of social software, what should we do? Is there anything we can say with any certainty about building social software, at least for large and long-lived groups? 

I think there is. A little over 10 years ago, I quit my day job, because Usenet was so interesting, I thought: This is really going to be big. And I actually wrote a book about net culture at the time: Usenet, the Well, Echo, IRC and so forth. It launched in April of ’95, just as that world was being washed away by the web. But it was my original interest, so I’ve been looking at this problem in one way or another for 10 years, and I’ve been looking at it pretty hard for the a year and a half or so.

So there’s this question “What is required to make a large, long-lived online group successful?” and I think I can now answer with some confidence: “It depends.” I’m hoping to flesh that answer out a little bit in the next ten years.

But I can at least say some of the things it depends on. The Calvinists had a doctrine of natural grace and supernatural grace. Natural grace was “You have to do all the right things in the world to get to heaven…” and supernatural grace was “…and God has to anoint you.” And you never knew if you had supernatural grace or not. This was their way of getting around the fact that the Book of Revelations put an upper limit on the number of people who were going to heaven.

Social software is like that. You can find the same piece of code running in many, many environments. And sometimes it works and sometimes it doesn’t. So there is something supernatural about groups being a run-time experience. 

The normal experience of social software is failure. If you go into Yahoo groups and you map out the subscriptions, it is, unsurprisingly, a power law. There’s a small number of highly populated groups, a moderate number of moderately populated groups, and this long, flat tail of failure. And the failure is inevitably more than 50% of the total mailing lists in any category. So it’s not like a cake recipe. There’s nothing you can do to make it come out right every time.

There are, however, I think, about half a dozen things that are broadly true of all the groups I’ve looked at and all the online constitutions I’ve read for software that supports large and long-lived groups. And I’d break that list in half. I’d say, if you are going to create a piece of social software designed to support large groups, you have to accept three things, and design for four things.

Three Things to Accept

1.) Of the things you have to accept, the first is that you cannot completely separate technical and social issues. There are two attractive patterns. One says, we’ll handle technology over `here, we’ll do social issues there. We’ll have separate mailing lists with separate discussion groups, or we’ll have one track here and one track there. This doesn’t work. It’s never been stated more clearly than in the pair of documents called “LambdaMOO Takes a New Direction.” I can do no better than to point you to those documents.

But recently we’ve had this experience where there was a social software discussion list, and someone said “I know, let’s set up a second mailing list for technical issues.” And no one moved from the first list, because no one could fork the conversation between social and technical issues, because the conversation can’t be forked.

The other pattern that’s very, very attractive — anybody who looks at this stuff has the same epiphany, which is: “Omigod, this software is determining what people do!” And that is true, up to a point. But you cannot completely program social issues either. So you can’t separate the two things, and you also can’t specify all social issues in technology. The group is going to assert its rights somehow, and you’re going to get this mix of social and technological effects.

So the group is real. It will exhibit emergent effects. It can’t be ignored, and it can’t be programmed, which means you have an ongoing issue. And the best pattern, or at least the pattern that’s worked the most often, is to put into the hands of the group itself the responsibility for defining what value is, and defending that value, rather than trying to ascribe those things in the software upfront.

2.) The second thing you have to accept: Members are different than users. A pattern will arise in which there is some group of users that cares more than average about the integrity and success of the group as a whole. And that becomes your core group, Art Kleiner’s phrase for “the group within the group that matters most.” 

The core group on Communitree was undifferentiated from the group of random users that came in. They were separate in their own minds, because they knew what they wanted to do, but they couldn’t defend themselves against the other users. But in all successful online communities that I’ve looked at, a core group arises that cares about and gardens effectively. Gardens the environment, to keep it growing, to keep it healthy.

Now, the software does not always allow the core group to express itself, which is why I say you have to accept this. Because if the software doesn’t allow the core group to express itself, it will invent new ways of doing so. 

On alt.folklore.urban , the discussion group about urban folklore on Usenet, there was a group of people who hung out there and got to be friends. And they came to care about the existence of AFU, to the point where, because Usenet made no distinction between members in good standing and drive-by users, they set up a mailing list called The Old Hats. The mailing list was for meta-discussion, discussion about AFU, so they could coordinate efforts formally if they were going to troll someone or flame someone or ignore someone, on the mailing list.Addendum, July 2, 2003: A longtime a.f.u participant says that the Old Hat list was created to allow the Silicon Valley-dwelling members to plan a barbecue, so that they could add a face-to-face dimension to their virtual interaction. The use of the list as a backstage area for discussing the public newsgroup arose after the fact.

Then, as Usenet kept growing, many newcomers came along and seemed to like the environment, because it was well-run. In order to defend themselves from the scaling issues that come from of adding a lot of new members to the Old Hats list, they said “We’re starting a second list, called the Young Hats.”

So they created this three-tier system, not dissimilar to the tiers of anonymous cowards, logged-in users, and people with high karma on Slashdot. But because Usenet didn’t let them do it in the software, they brought in other pieces of software, these mailing lists, that they needed to build the structure. So you don’t get the program users, the members in good standing will find one another and be recognized to one another.

3.) The third thing you need to accept: The core group has rights that trump individual rights in some situations. This pulls against the libertarian view that’s quite common on the network, and it absolutely pulls against the one person/one vote notion. But you can see examples of how bad an idea voting is when citizenship is the same as ability to log in. 

In the early Nineties, a proposal went out to create a Usenet news group for discussing Tibetan culture, called soc.culture.tibet. And it was voted down, in large part because a number of Chinese students who had Internet access voted it down, on the logic that Tibet wasn’t a country; it was a region of China. And in their view, since Tibet wasn’t a country, there oughtn’t be any place to discuss its culture, because that was oxymoronic. 

Now, everyone could see that this was the wrong answer. The people who wanted a place to discuss Tibetan culture should have it. That was the core group. But because the one person/one vote model on Usenet said “Anyone who’s on Usenet gets to vote on any group,” sufficiently contentious groups could simply be voted away. 

Imagine today if, in the United States, Internet users had to be polled before any anti-war group could be created. Or French users had to be polled before any pro-war group could be created. The people who want to have those discussions are the people who matter. And absolute citizenship, with the idea that if you can log in, you are a citizen, is a harmful pattern, because it is the tyranny of the majority. 

So the core group needs ways to defend itself — both in getting started and because of the effects I talked about earlier — the core group needs to defend itself so that it can stay on its sophisticated goals and away from its basic instincts. 

The Wikipedia has a similar system today, with a volunteer fire department, a group of people who care to an unusual degree about the success of the Wikipedia. And they have enough leverage, because of the way wikis work, they can always roll back graffiti and so forth, that that thing has stayed up despite repeated attacks. So leveraging the core group is a really powerful system.

Now, when I say these are three things you have to accept, I mean you have to accept them. Because if you don’t accept them upfront, they’ll happen to you anyway. And then you’ll end up writing one of those documents that says “Oh, we launched this and we tried it, and then the users came along and did all these weird things. And now we’re documenting it so future ages won’t make this mistake.” Even though you didn’t read the thing that was written in 1978.

All groups of any integrity have a constitution. The constitution is always partly formal and partly informal. At the very least, the formal part is what’s substantiated in code — “the software works this way.” 

The informal part is the sense of “how we do it around here.” And no matter how is substantiated in code or written in charter, whatever, there will always be an informal part as well. You can’t separate the two.

Four Things to Design For

1.) If you were going to build a piece of social software to support large and long-lived groups, what would you design for? The first thing you would design for is handles the user can invest in. 

Now, I say “handles,” because I don’t want to say “identity,” because identity has suddenly become one of those ideas where, when you pull on the little thread you want, this big bag of stuff comes along with it. Identity is such a hot-button issue now, but for the lightweight stuff required for social software, its really just a handle that matters. 

It’s pretty widely understood that anonymity doesn’t work well in group settings, because “who said what when” is the minimum requirement for having a conversation. What’s less well understood is that weak pseudonymity doesn’t work well, either. Because I need to associate who’s saying something to me now with previous conversations. 

The world’s best reputation management system is right here, in the brain. And actually, it’s right here, in the back, in the emotional part of the brain. Almost all the work being done on reputation systems today is either trivial or useless or both, because reputations aren’t linearizable, and they’re not portable. 

There are people who cheat on their spouse but not at cards, and vice versa, and both and neither. Reputation is not necessarily portable from one situation to another, and it’s not easily expressed. 

eBay has done us all an enormous disservice, because eBay works in non-iterated atomic transactions, which are the opposite of social situations. eBay’s reputation system works incredibly well, because it starts with a linearizable transaction — “How much money for how many Smurfs?” — and turns that into a metric that’s equally linear. 

That doesn’t work well in social situations. If you want a good reputation system, just let me remember who you are. And if you do me a favor, I’ll remember it. And I won’t store it in the front of my brain, I’ll store it here, in the back. I’ll just get a good feeling next time I get email from you; I won’t even remember why. And if you do me a disservice and I get email from you, my temples will start to throb, and I won’t even remember why. If you give users a way of remembering one another, reputation will happen, and that requires nothing more than simple and somewhat persistent handles. 

Users have to be able to identify themselves and there has to be a penalty for switching handles. The penalty for switching doesn’t have to be total. But if I change my handle on the system, I have to lose some kind of reputation or some kind of context. This keeps the system functioning.

Now, this pulls against the sense that we’ve had since the early psychological writings about the Internet. “Oh, on the Internet we’re all going to be changing identities and genders like we change our socks.” 

And you see things like the Kaycee Nicole story, where a woman in Kansas pretended to be a high school student, and then because the invented high school student’s friends got so emotionally involved, she then tried to kill the Kaycee Nicole persona off. “Oh, she’s got cancer and she’s dying and it’s all very tragic.” And of course, everyone wanted to fly to meet her. So then she sort of panicked and vanished. And a bunch of places on the Internet, particularly the MetaFilter community, rose up to find out what was going on, and uncovered the hoax. It was sort of a distributed detective movement.

Now a number of people point to this and say “See, I told you about that identity thing!” But the Kaycee Nicole story is this: changing your identity is really weird. And when the community understands that you’ve been doing it and you’re faking, that is seen as a huge and violent transgression. And they will expend an astonishing amount of energy to find you and punish you. So identity is much less slippery than the early literature would lead us to believe. 

2.) Second, you have to design a way for there to be members in good standing. Have to design some way in which good works get recognized. The minimal way is, posts appear with identity. You can do more sophisticated things like having formal karma or “member since.” 

I’m on the fence about whether or not this is a design or accepting. Because in a way I think members in good standing will rise. But more and more of the systems I’m seeing launching these days are having some kind of additional accretion so you can tell how much involvement members have with the system. 

There’s an interesting pattern I’m seeing among the music-sharing group that operates between Tokyo and Hong Kong. They operate on a mailing list, which they set up for themselves. But when they’re trading music, what they’re doing is, they’re FedExing one another 180-gig hard-drives. So you’re getting .wav files and not MP3s, and you’re getting them in bulk. 

Now, you can imagine that such a system might be a target for organizations that would frown on this activity. So when you join that group, your user name is appended with the user name of the person who is your sponsor. You can’t get in without your name being linked to someone else. You can see immediately the reputational effects going on there, just from linking two handles. 

So in that system, you become a member in good standing when your sponsor link goes away and you’re there on your own report. If, on the other hand, you defect, not only are you booted, but your sponsor is booted. There are lots and lots of lightweight ways to accept and work with the idea of member in good standing. 

3.) Three, you need barriers to participation. This is one of the things that killed Usenet. You have to have some cost to either join or participate, if not at the lowest level, then at higher levels. There needs to be some kind of segmentation of capabilities. 

Now, the segmentation can be total — you’re in or you’re out, as with the music group I just listed. Or it can be partial — anyone can read Slashdot, anonymous cowards can post, non-anonymous cowards can post with a higher rating. But to moderate, you really have to have been around for a while. 

It has to be hard to do at least some things on the system for some users, or the core group will not have the tools that they need to defend themselves. 

Now, this pulls against the cardinal virtue of ease of use. But ease of use is wrong. Ease of use is the wrong way to look at the situation, because you’ve got the Necker cube flipped in the wrong direction. The user of social software is the group, not the individual.

I think we’ve all been to meetings where everyone had a really good time, we’re all talking to one another and telling jokes and laughing, and it was a great meeting, except we got nothing done. Everyone was amusing themselves so much that the group’s goal was defeated by the individual interventions. 

The user of social software is the group, and ease of use should be for the group. If the ease of use is only calculated from the user’s point of view, it will be difficult to defend the group from the group is its own worst enemy style attacks from within. 

4.) And, finally, you have to find a way to spare the group from scale. Scale alone kills conversations, because conversations require dense two-way conversations. In conversational contexts, Metcalfe’s law is a drag. The fact that the amount of two-way connections you have to support goes up with the square of the users means that the density of conversation falls off very fast as the system scales even a little bit. You have to have some way to let users hang onto the less is more pattern, in order to keep associated with one another. 

This is an inverse value to scale question. Think about your Rolodex. A thousand contacts, maybe 150 people you can call friends, 30 people you can call close friends, two or three people you’d donate a kidney to. The value is inverse to the size of the group. And you have to find some way to protect the group within the context of those effects. 

Sometimes you can do soft forking. Live Journal does the best soft forking of any software I’ve ever seen, where the concepts of “you” and “your group” are pretty much intertwingled. The average size of a Live Journal group is about a dozen people. And the median size is around five. 

But each user is a little bit connected to other such clusters, through their friends, and so while the clusters are real, they’re not completely bounded — there’s a soft overlap which means that though most users participate in small groups, most of the half-million LiveJournal users are connected to one another through some short chain. 

IRC channels and mailing lists are self-moderating with scale, because as the signal to noise ratio gets worse, people start to drop off, until it gets better, so people join, and so it gets worse. You get these sort of oscillating patterns. But it’s self-correcting.

And then my favorite pattern is from MetaFilter, which is: When we start seeing effects of scale, we shut off the new user page. “Someone mentions us in the press and how great we are? Bye!” That’s a way of raising the bar, that’s creating a threshold of participation. And anyone who bookmarks that page and says “You know, I really want to be in there; maybe I’ll go back later,” that’s the kind of user MeFi wants to have. 

You have to find some way to protect your own users from scale. This doesn’t mean the scale of the whole system can’t grow. But you can’t try to make the system large by taking individual conversations and blowing them up like a balloon; human interaction, many to many interaction, doesn’t blow up like a balloon. It either dissipates, or turns into broadcast, or collapses. So plan for dealing with scale in advance, because it’s going to happen anyway.

Conclusion

Now, those four things are of course necessary but not sufficient conditions. I propose them more as a platform for building the interesting differences off. There are lots and lots and lots of other effects that make different bits of software interesting enough that you would want to keep more than one kind of pattern around. But those are commonalities I’m seeing across a range of social software for large and long-lived groups. 

In addition, you can do all sorts of things with explicit clustering, whether it’s guilds in massively multi-player games, or communities on Live Journal or what have you. You can do things with conversational artifacts, where the group participation leaves behind some record. The Wikipedia right now, the group collaborated online encyclopedia is the most interesting conversational artifact I know of, where product is a result of process. Rather than “We’re specifically going to get together and create this presentation” it’s just “What’s left is a record of what we said.” 

There are all these things, and of course they differ platform to platform. But there is this, I believe, common core of things that will happen whether you plan for them or not, and things you should plan for, that I think are invariant across large communal software. 

Writing social software is hard. And, as I said, the act of writing social software is more like the work of an economist or a political scientist. And the act of hosting social software, the relationship of someone who hosts it is more like a relationship of landlords to tenants than owners to boxes in a warehouse. 

The people using your software, even if you own it and pay for it, have rights and will behave as if they have rights. And if you abrogate those rights, you’ll hear about it very quickly.

That’s part of the problem that the John Hegel theory of community — community leads to content, which leads to commerce — never worked. Because lo and behold, no matter who came onto the Clairol chat boards, they sometimes wanted to talk about things that weren’t Clairol products. 

“But we paid for this! This is the Clairol site!” Doesn’t matter. The users are there for one another. They may be there on hardware and software paid for by you, but the users are there for one another. 

The patterns here, I am suggesting, both the things to accept and the things to design for, are givens. Assume these as a kind of social platform, and then you can start going out and building on top of that the interesting stuff that I think is going to be the real result of this period of experimentation with social software. 

Thank you very much.

First published June 3, 2003 on the “Networks, Economics, and Culture” mailing list.

Yesterday, the FCC adjusted the restrictions on media ownership, allowing newspapers to own TV stations, and raising the ownership limitations on broadcast TV networks by 10%, to 45% from 35%. It’s not clear whether the effects of the ruling will be catastrophic or relatively unimportant, and there are smart people on both sides of that question. It is also unclear what effect the internet had on the FCC’s ruling, or what role it will play now.

What is clear, however, is a lesson from the weblog world: inequality is a natural component of media. For people arguing about an ideal media landscape, the tradeoffs are clear: Diverse. Free. Equal. Pick two.

The Developing Debate

The debate about media and audience size used to be focussed on the low total number of outlets, mainly because there were only three national television networks. Now that more than 80% of the country gets their television from cable and satellite, the concern is concentration. In this view, there may be diverse voices available on the hundred or more TV channels the average viewer gets, but the value of that diversity is undone by the fact that large media firms enjoy the lion’s share of the audience’s cumulative attention.

A core assumption in this debate is that if media were free of manipulation, the audience would be more equally distributed, so the concentration of a large number of viewers by a small number of outlets is itself evidence of impermissible control. In this view, government intervention is required simply to restore the balance we would expect in an unmanipulated system.

For most of the 20th century, we had no way of testing this proposition. The media we had were so heavily regulated and the outlets so scarce that we had no other scenarios to examine, and the growth of cable in the last 20 years involved local monopoly of the wire into the home, so it didn’t provide a clean test of an alternative.

Weblogs As Media Experiment

In the last few years, however, we have had a clean test, and it’s weblogs. Weblogs are the freest media the world has ever known. Within the universe of internet users, the costs of setting up a weblog are minor, and perhaps more importantly, require no financial investment, only time, thus greatly weakening the “freedom of the press for those who can afford one” effect. Furthermore, there is no Weblog Central — you do not need to incorporate your weblog, you do not need to register your weblog, you do not need to clear your posts with anyone. Weblogs are the best attempt we’ve seen to date of making freedom of speech and freedom of the press the same freedom, in Mike Godwin’s famous phrase.

And in this free, decentralized, diverse, and popular medium we find astonishing inequality, inequality so extreme it makes the distribution of television ratings look positively egalitarian. In fact, a review of any of the weblog tracking initiatives such as Technorati or the blogging ecosystem project shows thousand-fold imbalances between the most popular and average weblogs. These inequalities often fall into what’s known as a power law distribution, a curve where a tiny number of sites account for a majority of the in-bound links, while the vast majority of sites have a very small number of such links. (Although the correlation with links and traffic is not perfect, it is a strong proxy for audience size.)

The reasons for this are complex (I addressed some of them in Power Laws, Weblogs, and Inequality), but from the point of view of analyzing the FCC ruling, the lesson of weblog popularity is clear: inequality can arise in systems where users are free to make choices among a large set of options, even in the absence of central control or manipulation. Inequality is not a priori evidence of manipulation, in other words; it can also be a side effect of large systems governed by popular choice.

In the aftermath of the FCC ruling, and given what we have learned from the development of weblogs, the debate on media concentration can now be sharpened to a single question: if inequality is a fact of life, even in diverse and free systems, what should our reaction be?

‘Pick Two’ Yields Three Positions

There are three coherent positions in this debate: The first is advocacy of free and equal media, which requires strong upper limits on overall diversity. This was roughly the situation of the US broadcast television industry from 1950-1980. Any viewer was free to watch shows from any network, but having only three national networks kept any one of them from becoming dominant. (Funnily enough, Gunsmoke, the most popular television show in history, enjoyed a 45% audience share, the same upper limit now proposed by the FCC for overall audience size.)

Though this position is logically coherent, the unprecedented explosion of media choice makes it untenable in practice. Strong limits on the number of media outlets accessible by any given member of the public now exist in only two places: broadcast radio and newspapers, not coincidently, the two media least affected by new technologies of distribution. 

The second coherent position is advocacy of diverse and equal media, which requires constraints on freedom. This view is the media equivalent of redistributive taxation, where an imbalance in audience size is seen as being so corrosive of democratic values that steps must be taken to limit the upper reach of popular media outlets, and to subsidize in some way less popular ones. In practice, this position is advocacy of diverse and less unequal media. This is the position taken by the FCC, who yesterday altered regulations rather than removing them. (There can obviously be strong disagreement within this group about the kind and degree of regulations.)

People who hold this view believe that regulation is preferable to inequality, and will advocate governmental intervention in any market where the scarcity in the number of channels constrains number of outlets the locals have access to (again, radio and newspapers are the media with the most extreme current constraints.)

More problematic for people who hold this view are unequal but unconstrained media such as weblogs. As weblogs grow in importance, we can expect at least some members of the “diverse and equal” camp to advocate regulation of weblogs, on the grounds that the imbalance between Glenn Reynolds of InstaPundit.com and J. Random Blogger is no different than the imbalance between Clear Channel and WFMU. This fight will pit those who advocate government intervention only where there is scarcity (whether regulatory or real) vs. those who advocate regulation wherever there is inequality, even if it arises naturally and in an unconstrained system.

The third coherent position is advocacy of diverse and free media, which requires abandonment of equality as a goal. For this camp, the removal of regulation is desirable in and of itself, whatever the outcome. Given the evidence that diverse and free systems migrate to unequal distributions, the fact of inequality is a necessarily acceptable outcome to this group. However, in truly diverse systems, with millions of choices rather than hundreds, the imbalance between popular and average media outlets is tempered by the imbalance between the most popular outlets and the size of the system as a whole. As popular as Glenn Reynolds may be, InstaPundit is no Gunsmoke; no one weblog is going to reach 45% of the audience. In large diverse systems, freedom increases the inequality between outlets, but the overall size and growth weakens the effects of concentration.

This view is the least tested in practice. While the “diverse and equal” camp is advocating regulation and therefore an articulation of the status quo, people who believe that our goals should be diversity and freedom and damn the consequences haven’t had much effect on the traditional media landscape to date, so we have very little evidence on the practical effect of their proposals. The most obvious goal for this group is radical expansion of media choice in all dimensions, and a subsequent dropping of all mandated restrictions. For this view to come to pass, restrictions on internet broadcast of radio and TV should be dropped, web radio stations must live in the same copyright regime broadcast stations do, much more unlicensed spectrum must be made available, and so on.

And this the big risk. Though the FCC’s ruling is portrayed as deregulation, it is nothing of the sort. It is simply different regulation, and it adjusts percentages within a system of scarcity, rather than undoing the scarcity itself. It remains to be seen if the people supporting the FCC’s current action are willing to go all the way to the weblogization of everything, but this is what will be required to get to the benefits of the free and diverse scenario. In the absence of regulation, the only defense against monopolization is to create a world where, no matter how many media outlets a single company can buy, more can appear tomorrow. The alternative — reduction of regulation without radical expansion — is potentially the worst of both worlds.

The one incoherent view is the belief that a free and diverse media will naturally tend towards equality. The development of weblogs in their first five years demonstrates that is not always true, and gives us reason to suspect it may never be true. Equality can only be guaranteed by limiting either diversity or freedom.

The best thing that could come from the lesson of weblog popularity would be an abandoning of the idea that there will ever be an unconstrained but egalitarian media utopia, a realization ideally followed by a more pragmatic discussion between the “diverse and free” and “diverse and equal” camps.

First published May 20, 2003 on the “Networks, Economics, and Culture” mailing list.

Grid Computing is, according to the Grid Information Centre a way to “…enable the sharing, selection, and aggregation of a wide variety of geographically distributed computational resources.” It is, in other words, an attempt to make Sun’s famous pronouncement “The Network Is The Computer” an even more workable proposition. (It is also an instantiation of several of the patterns of decentralization that used to travel together under the name peer-to-peer.)

Despite the potential generality of the Grid, most of the public pronouncements are focusing on the use of Grids for supercomputing. IBM defines it more narrowly: Grid Computing is “… applying resources from many computers in a network-at the same time-to a single problem” , and the MIT Technology Review equated Grid technology with supercomputing on tap when it named Grids one of “Ten Technologies That Will Change the World.”

This view is wrong. Supercomputing on tap won’t live up to to this change-the-world billing, because computation isn’t a terribly important part of what people do with computers. This is a lesson we learned with PCs, and it looks like we will be relearning it with Grids.

The Misnomer of the Personal Computer

Though most computational power lives on the world’s hundreds of millions of PCs, most PCs are not used for computation most of the time. There are two reasons for this, both of which are bad news for predictions of a supercomputing revolution. The first is simply that most people are not sitting at their computer for most hours of the day. The second is because even when users are at their computers, they are not tackling computationally hard problems, and especially not ones that require batch processing — submit question today, get answer tomorrow (or next week.) Indeed, whenever users encounter anything that feels even marginally like batch processing — a spreadsheet that takes seconds to sort, a Photoshop file that takes a minute to render — they begin hankering for a new PC, because they care about peak performance, not total number of cycles available over time. The only time the average PC performs any challenging calculations is rendering the visual for The Sims or WarCraft.

Therein lies the conundrum of the Grid-as-supercomputer: the oversupply of cycles the Grid relies on exists because of a lack of demand. PCs are used as many things — file cabinets and communications terminals and typewriters and photo albums and jukeboxes — before they are used as literal computers. If most users had batch applications they were willing to wait for even as long as overnight, the first place they would look for spare cycles would be on their own machines, not on some remote distributed supercomputer. Simply running their own PC round the clock would offer a 10x to 20x improvement, using hardware they already own.

If users needed Grid-like power, the Grid itself wouldn’t work, because the unused cycles the Grid is going to aggregate wouldn’t exist. Of all the patterns supported by decentralization, from file-sharing to real-time collaboration to supercomputing, supercomputing is the least general.

The Parallel with Push 

There is a parallel between Grids and Push technology, that glamorous flameout of the mid-90s. The idea behind Push, exemplified by the data-displaying screensaver Pointcast, was that because users suffered from limited bandwidth and periodic disconnection (e.g. laptops on airplanes), they would sign up to have data pushed to them, which they could then experience at their leisure. This, we were told, would create a revolution in the way people use the internet. (This notion reached its apotheosis in a Wired magazine cover story, “Push!”, whose subtitle read “Kiss your browser goodbye: The radical future of media beyond the Web”.)

As it turned out, user’s response to poor connectivity was to agitate for better connectivity, because like CPUs, users want bandwidth that provides good peak performance, even if that means most of it gets “wasted.” Shortly after the Wired cover, it was PointCast we kissed goodbye.

Push’s collapse was made all the more spectacular because of its name. The label Push seemed to suggest a sweeping new pattern of great importance. Had the technology been given a duller but more descriptive name, like “forward caching,” it would have generated much less interest in the beginning, but might also not have been so prematurely consigned to the list of failed technologies.

Forward caching is in fact a key part of some applications. In particular, companies building decentralized groupware like Groove , Kubi Software, and Shinkuro, all of whom use forward caching of shared files to overcome the difficulties caused by limited bandwidth and partially disconnected nodes, just the issues Push was supposed to address. By pushing the name Push, the Pointcast’s of the world made it harder to see that though forward caching was not universally important, it was still valuable in some areas.

Distributed Batch Processing

So it is with Grids. The evocative name suggests that computation is so critical that we must have a global infrastructure to provide all those cycles we’ll be needing next time our boss asks us to model an earthquake, or we have to help our parents crack a cryptographic key. The broadness of the term masks the specialised nature of the technology, which should probably be called “distributed batch processing.”

Like forward caching, distributed batch processing is useful in a handful of areas. The SETI@Home project runs on distributed batch processing, as does the distributed.net cryptographic key-breaking tool. The sequencing of the SARS virus happened using distributed batch processing. Distributed batch processing could be useful in fields like game theory, where scenarios could be exhaustively tested on the cheap, or animated film, where small studios or even individuals could afford acces to Pixar-like render farms.

Distributed batch processing is real progress for people who need supercomputing power, but having supercomputing on tap doesn’t make you a researcher anymore than having surfboard wax on tap would make you a surfer. Indeed, to the consternation of chip manufacturers (and the delight of researchers who want cheap cycles), people don’t even have much real use for the computational power on the machines they buy today.

History has not been kind to business predictions based on an undersupply of cycles, and the business case for selling access to supercomputing on tap is grim. Assuming that a $750 machine with a 2 gigahertz chip can be used for 3 years, commodity compute time now costs roughly a penny a gigahertz/hour. If Grid access costs more than a penny a ghz/hr, building a dedicated supercomputer starts to be an economical proposition, relative to buying cycles from a Grid. (And of course Moore’s Law sees to it that these economics get more adverse every year.)

Most of the for-profit work on supercomputing Grids will be in helping businesses harness their employees’ PCs so that the CFO can close the books quickly — cheap, one-shot contracts, in other words, that mostly displace money from the purchase of new servers. The cost savings for the average business will be nice of course, but saving money by deferring server purchases is hardly a revolution.

People Matter More Than Machines

We have historically overestimated the value of connecting machines to one another, and underestimated the value of connecting people, and by emphasizing supercomputing on tap, the proponents of Grids are making that classic mistake anew. During the last great age of batch processing, the ARPAnet’s designers imagined that the nascent network would be useful as a way of providing researchers access to batch processing at remote locations. This was wrong, for two reasons: first, it turned out researchers were far more interested in getting their own institutions to buy computers they could use locally than in using remote batch processing, and Moore’s Law made that possible as time passed. Next, once email was ported to the network, it became a far more important part of the ARPAnet backbone than batch processing was. Then as now, access to computing power mattered less to the average network user than access to one another.

Though Sun was incredibly prescient in declaring “The Network is the Computer” at a time when PCs didn’t even ship with built-in modems, the phrase is false in some important ways — a network is a different kind of thing than a computer. As long ago as 1968, J.R. Licklider predicted that computers would one day be more important as devices of communication than of computation, a prediction that came true when email overtook the spreadsheet as the core application driving PC purchases.

What was true of the individual PC is true of the network as well — changes in computational power are nice, but changes in communications power are profound. As we learned with Push, an intriguing name is no substitute for general usefulness. Networks are most important as ways of linking unevenly distributed resources — I know something you don’t know; you have something I don’t have — and Grid technology will achieve general importance to the degree that it supports those kinds of patterns. The network applications that let us communicate and share in heterogeneous environments, from email to Kazaa, are far more important uses of the network than making all the underlying computers behave as a single supercomputer.

First published March 28, 2003 on the “Networks, Economics, and Culture” mailing list. 

“The future always comes too fast and in the wrong order.” — Alvin Toffler

For most of the past year, on many US airlines, those phones inserted into the middle seat have borne a label reading “Service Disconnected.” Those labels tell a simple story — people don’t like to make $40 phone calls. They tell a more complicated one as well, about the economics of connectivity and about two competing visions for access to our various networks. One of these visions is the one everyone wants — ubiquitous and convenient — and the other vision is the one we get — spotty and cobbled together. 

Call the first network “perma-net,” a world where connectivity is like air, where anyone can send or receive data anytime anywhere. Call the second network “nearly-net”, an archipelago of connectivity in an ocean of disconnection. Everyone wants permanet — the providers want to provide it, the customers want to use it, and every few years, someone announces that they are going to build some version of it. The lesson of in-flight phones is that nearlynet is better aligned with the technological, economic, and social forces that help networks actually get built. The most illustrative failure of permanet is the airphone. The most spectacular was Iridium. The most expensive will be 3G. 

“I’m (Not) Calling From 35,000 Feet”

The airphone business model was obvious — the business traveler needs to stay in contact with the home office, with the next meeting, with the potential customer. When 5 hours of the day disappears on a flight, value is lost, and business customers, the airlines reasoned, would pay a premium to recapture that value.

The airlines knew, of course, that the required investment would make in-flight calls expensive at first, but they had two forces on their side. The first was a captive audience — when a plane was in the air, they had a monopoly on communication with the outside world. The second was that, as use increased, they would pay off the initial investment, and could start lowering the cost of making a call, further increasing use.

What they hadn’t factored in was the zone of connectivity between the runway and the gate, where potential airphone users were physically captive, but where their cell phones still worked. The time spent between the gate and the runway can account for a fifth of even long domestic flights, and since that is when flight delays tend to appear, it is a disproportionately valuable time in which to make calls.

This was their first miscalculation. The other was that they didn’t know that competitive pressures in the cell phone market would drive the price of cellular service down so fast that the airphone would become more expensive, in relative terms, after it launched. 

The negative feedback loop created by this pair of miscalculations marginalized the airphone business. Since price displaces usage, every increase in the availability on cell phones or reduction in the cost of a cellular call meant that some potential users of the airphone would opt out. As users opted out, the projected revenues shrank. This in turn postponed the date at which the original investment in the airphone system could be paid back. The delay in paying back the investment delayed the date at which the cost of a call could be reduced, making the airphone an even less attractive offer as the number of cell phones increased and prices shrank still further.

66 Tears

This is the general pattern of the defeat of permanet by nearlynet. In the context of any given system, permanet is the pattern that makes communication ubiquitous. For a plane ride, the airphone is permanet, always available but always expensive, while the cell phone is nearlynet, only intermittently connected but cheap and under the user’s control. 

The characteristics of the permanet scenario — big upfront investment by few enough companies that they get something like monopoly pricing power — is usually justified by the assumption that users will accept nothing less than total connectivity, and will pay a significant premium to get it. This may be true in scenarios where there is no alternative, but in scenarios where users can displace even some use from high- to low-priced communications tools, they will.

This marginal displacement matters because a permanet network doesn’t have to be unused to fail. It simply has to be underused enough to be unprofitable. Builders of large networks typically overestimate the degree to which high cost deflects use, and underestimate the number of alternatives users have in the ways they communicate. And in the really long haul, the inability to pay off the initial investment in a timely fashion stifles later investment in upgrading the network.

This was the pattern of Iridium, Motorola’s famously disastrous network of 66 satellites that would allow the owner of an Iridium phone to make a phone call from literally anywhere in the world. This was permanet on a global scale. Building and launching the satellites cost billions of dollars, the handsets cost hundreds, the service cost dollars a minute, all so the busy executive could make a call from the veldt.

Unfortunately, busy executives don’t work in the veldt. They work in Pasedena, or Manchester, or Caracas. This is the SUV pattern — most SUV ads feature empty mountain roads but most actual SUVs are stuck in traffic. Iridium was a bet on a single phone that could be used anywhere, but its high cost eroded any reason to use an Iridium phone in most of the perfectly prosaic places phone calls actually get made.

3G: Going, Going, Gone

The biggest and most expensive permanet effort right now is wireless data services, principally 3G, the so-called 3rd generation wireless service, and GPRS, the General Packet Radio Service (though the two services are frequently lumped together under the 3G label.) 3G data services provide always on connections and much higher data rates to mobile devices than the widely deployed GSM networks do, and the wireless carriers have spent tens of billions worldwide to own and operate such services. Because 3G requires licensed spectrum, the artificial scarcity created by treating the airwaves like physical property guarantees limited competition among 3G providers. 

The idea here is that users want to be able to access data any time anywhere. This is of course true in the abstract, but there are two caveats: the first is that they do not want it at any cost, and the second and more worrying one is that if they won’t use 3G in environments where they have other ways of connecting more cheaply.

The nearlynet to 3G’s permanet is Wifi (and, to a lesser extent, flat-rate priced services like email on the Blackberry.) 3G partisans will tell you that there is no competition between 3G and Wifi, because the services do different things, but of course that is exactly the problem. If they did the same thing, the costs and use patterns would also be similar. It’s precisely the ways in which Wifi differs from 3G that makes it so damaging. 

The 3G model is based on two permanetish assumptions — one, that users have an unlimited demand for data while traveling, and two, that once they get used to using data on their phone, they will use it everywhere. Both assumptions are wrong.

First, users don’t have an unlimited demand for data while traveling, just as they didn’t have an unlimited demand for talking on the phone while flying. While the mobile industry has been telling us for years that internet-accessible cellphones will soon outnumber PCs, they fail to note that for internet use, measured in either hours or megabytes, the PC dwarfs the phone as a tool. Furthermore, in the cases where users do demonstrate high demand for mobile data services by getting 3G cards for their laptops, the network operators have been forced to raise their prices, the opposite of the strategy that would drive use. Charging more for laptop use makes 3G worse relative to Wifi, whose prices are constantly falling (access points and Wifi cards are now both around $60.)

The second problem is that 3G services don’t just have the wrong prices, they have the wrong kind of prices — metered — while Wifi is flat-rate. Metered data gives the user an incentive to wait out the cab ride or commute and save their data intensive applications for home or office, where sending or receiving large files creates no additional cost. The more data intensive a users needs are, the greater the price advantage of Wifi, and the greater their incentive to buy Wifi equipment. At current prices, a user can buy a Wifi access point for the cost of receiving a few PDF files over a 3G network, and the access point, once paid for, will allow for unlimited use at much higher speeds. 

The Vicious Circle 

In airline terms, 3G is like the airphone, an expensive bet that users in transit, captive to their 3G provider, will be happy to pay a premium for data communications. Wifi is like the cell phone, only useful at either end of travel, but providing better connectivity at a fraction of the price. This matches the negative feedback loop of the airphone — the cheaper Wifi gets, both in real dollars and in comparison to 3G, the greater the displacement away from 3G, the longer it will take to pay back the hardware investment (and, in countries that auctioned 3G licenses, the stupefying purchase price), and the later the day the operators can lower their prices.

More worryingly for the operators, the hardware manufacturers are only now starting to toy with Wifi in mobile devices. While the picture phone is a huge success as a data capture device, the most common use is “Take picture. Show friends. Delete.” Only a fraction of the photos that are taken are sent over 3G now, and if the device manufacturers start making either digital cameras or picture phones with Wifi, the willingness to save a picture for free upload later will increase. 

Not all permanets end in total failure, of course. Unlike Iridium, 3G is seeing some use, and that use will grow. The displacement of use to cheaper means of connecting, however, means that 3G will not grow as fast as predicted, raising the risk of being too little used to be profitable.

Partial Results from Partial Implementation

In any given situation, the builders of permanet and nearlynet both intend to give the customers what they want, but since what customers want is good cheap service, it is usually impossible to get there right away. Permanet and nearlynet are alternate strategies for evolving over time.

The permanet strategy is to start with a service that is good but expensive, and to make it cheaper. The nearlynet strategy is to start with a service that is lousy but cheap, and to make it better. The permanet strategy assumes that quality is the key driver of a new service, and permanet has the advantage of being good at every iteration. Nearlynet assumes that cheapness is the essential characteristic, and that users will forgo quality for a sufficient break in price.

What the permanet people have going for them is that good vs. lousy is not a hard choice to make, and if things stayed that way, permanet would win every time. What they have going against them, however, is incentive. The operator of a cheap but lousy service has more incentive to improve quality than the operator of a good but expensive service does to cut prices. And incremental improvements to quality can produce disproportionate returns on investment when a cheap but lousy service becomes cheap but adequate. The good enough is the enemy of the good, giving an edge over time to systems that produce partial results when partially implemented. 

Permanet is as Permanet Does

The reason the nearlynet strategy is so effective is that coverage over cost is often an exponential curve — as the coverage you want rises, the cost rises far faster. It’s easier to connect homes and offices than roads and streets, easier to connect cities than suburbs, suburbs than rural areas, and so forth. Thus permanet as a technological condition is tough to get to, since it involves biting off a whole problem at once. Permanet as a personal condition, however, is a different story. From the user’s point of view, a kind of permanet exists when they can get to the internet whenever they like.

For many people in the laptop tribe, permanet is almost a reality now, with home and office wired, and any hotel or conference they attend Wifi- or ethernet-enabled, at speeds that far outstrip 3G. And since these are the people who reliably adopt new technology first, their ability to send a spreadsheet or receive a web page faster and at no incremental cost erodes the early use the 3G operators imagined building their data services on. 

In fact, for many business people who are the logical customers for 3G data services, there is only one environment where there is significant long-term disconnection from the network: on an airplane. As with the airphone itself, the sky may be a connection-poor environment for some time to come, not because it isn’t possible to connect it, but because the environment on the plane isn’t nearly nearlynet enough, which is to say it is not amenable to inexpensive and partial solutions. The lesson of nearlynet is that connectivity is rarely an all or nothing proposition, much as would-be monopolists might like it to be. Instead, small improvements in connectivity can generally be accomplished at much less cost than large improvements, and so we continue growing towards permanet one nearlynet at a time.

First published February 8, 2003 on the “Networks, Economics, and Culture” mailing list.

Version 1.1: Changed 02/10/03 to point to the updated “Blogging Ecosystem” project, and to Jason Kottke’s work using Technorati.com data. Added addendum pointing to David Sifry’s “Technorati Interesting Newcomers” list, which is in part a response to this article.

A persistent theme among people writing about the social aspects of weblogging is to note (and usually lament) the rise of an A-list, a small set of webloggers who account for a majority of the traffic in the weblog world. This complaint follows a common pattern we’ve seen with MUDs, BBSes, and online communities like Echo and the WELL. A new social system starts, and seems delightfully free of the elitism and cliquishness of the existing systems. Then, as the new system grows, problems of scale set in. Not everyone can participate in every conversation. Not everyone gets to be heard. Some core group seems more connected than the rest of us, and so on.

Prior to recent theoretical work on social networks, the usual explanations invoked individual behaviors: some members of the community had sold out, the spirit of the early days was being diluted by the newcomers, et cetera. We now know that these explanations are wrong, or at least beside the point. What matters is this: Diversity plus freedom of choice creates inequality, and the greater the diversity, the more extreme the inequality.

In systems where many people are free to choose between many options, a small subset of the whole will get a disproportionate amount of traffic (or attention, or income), even if no members of the system actively work towards such an outcome. This has nothing to do with moral weakness, selling out, or any other psychological explanation. The very act of choosing, spread widely enough and freely enough, creates a power law distribution.

A Predictable Imbalance #

Power law distributions, the shape that has spawned a number of catch-phrases like the 80/20 Rule and the Winner-Take-All Society, are finally being understood clearly enough to be useful. For much of the last century, investigators have been finding power law distributions in human systems. The economist Vilfredo Pareto observed that wealth follows a “predictable imbalance”, with 20% of the population holding 80% of the wealth. The linguist George Zipf observed that word frequency falls in a power law pattern, with a small number of high frequency words (I, of, the), a moderate number of common words (book, cat cup), and a huge number of low frequency words (peripatetic, hypognathous). Jacob Nielsen observed power law distributions in web site page views, and so on.

We are all so used to bell curve distributions that power law distributions can seem odd. The shape of Figure #1, several hundred blogs ranked by number of inbound links, is roughly a power law distribution. Of the 433 listed blogs, the top two sites accounted for fully 5% of the inbound links between them. (They were InstaPundit and Andrew Sullivan, unsurprisingly.) The top dozen (less than 3% of the total) accounted for 20% of the inbound links, and the top 50 blogs (not quite 12%) accounted for 50% of such links.

Figure #1: 433 weblogs arranged in rank order by number of inbound links.  The data is drawn from N.Z Bear’s 2002 work on the blogosphere ecosystem. The current version of this project can now be found at http://www.myelin.co.nz/ecosystem/.

The inbound link data is just an example: power law distributions are ubiquitous. Yahoo Groups mailing lists ranked by subscribers is a power law distribution. (Figure #2) LiveJournal users ranked by friends is a power law. (Figure #3) Jason Kottke has graphed the power law distribution of Technorati link data. The traffic to this article will be a power law, with a tiny percentage of the sites sending most of the traffic. If you run a website with more than a couple dozen pages, pick any time period where the traffic amounted to at least 1000 page views, and you will find that both the page views themselves and the traffic from the referring sites will follow power laws.

Figure #2: All mailing lists in the Yahoo Groups Television category, ranked by number of subscribers (Data from September 2002.) Figure #3: LiveJournal users ranked by number of friends listed. (Data from March 2002)

Rank Hath Its Privileges #

The basic shape is simple – in any system sorted by rank, the value for the Nth position will be 1/N. For whatever is being ranked — income, links, traffic — the value of second place will be half that of first place, and tenth place will be one-tenth of first place. (There are other, more complex formulae that make the slope more or less extreme, but they all relate to this curve.) We’ve seen this shape in many systems. What’ve we’ve been lacking, until recently, is a theory to go with these observed patterns.

Now, thanks to a series of breakthroughs in network theory by researchers like Albert-Laszlo Barabasi, Duncan Watts, and Bernardo Huberman among others, breakthroughs being described in books like Linked, Six Degrees, and The Laws of the Web, we know that power law distributions tend to arise in social systems where many people express their preferences among many options. We also know that as the number of options rise, the curve becomes more extreme. This is a counter-intuitive finding – most of us would expect a rising number of choices to flatten the curve, but in fact, increasing the size of the system increases the gap between the #1 spot and the median spot.

A second counter-intuitive aspect of power laws is that most elements in a power law system are below average, because the curve is so heavily weighted towards the top performers. In Figure #1, the average number of inbound links (cumulative links divided by the number of blogs) is 31. The first blog below 31 links is 142nd on the list, meaning two-thirds of the listed blogs have a below average number of inbound links. We are so used to the evenness of the bell curve, where the median position has the average value, that the idea of two-thirds of a population being below average sounds strange. (The actual median, 217th of 433, has only 15 inbound links.) 

Freedom of Choice Makes Stars Inevitable #

To see how freedom of choice could create such unequal distributions, consider a hypothetical population of a thousand people, each picking their 10 favorite blogs. One way to model such a system is simply to assume that each person has an equal chance of liking each blog. This distribution would be basically flat – most blogs will have the same number of people listing it as a favorite. A few blogs will be more popular than average and a few less, of course, but that will be statistical noise. The bulk of the blogs will be of average popularity, and the highs and lows will not be too far different from this average. In this model, neither the quality of the writing nor other people’s choices have any effect; there are no shared tastes, no preferred genres, no effects from marketing or recommendations from friends.

But people’s choices do affect one another. If we assume that any blog chosen by one user is more likely, by even a fractional amount, to be chosen by another user, the system changes dramatically. Alice, the first user, chooses her blogs unaffected by anyone else, but Bob has a slightly higher chance of liking Alice’s blogs than the others. When Bob is done, any blog that both he and Alice like has a higher chance of being picked by Carmen, and so on, with a small number of blogs becoming increasingly likely to be chosen in the future because they were chosen in the past.

Think of this positive feedback as a preference premium. The system assumes that later users come into an environment shaped by earlier users; the thousand-and-first user will not be selecting blogs at random, but will rather be affected, even if unconsciously, by the preference premiums built up in the system previously.

Note that this model is absolutely mute as to why one blog might be preferred over another. Perhaps some writing is simply better than average (a preference for quality), perhaps people want the recommendations of others (a preference for marketing), perhaps there is value in reading the same blogs as your friends (a preference for “solidarity goods”, things best enjoyed by a group). It could be all three, or some other effect entirely, and it could be different for different readers and different writers. What matters is that any tendency towards agreement in diverse and free systems, however small and for whatever reason, can create power law distributions.

Because it arises naturally, changing this distribution would mean forcing hundreds of thousands of bloggers to link to certain blogs and to de-link others, which would require both global oversight and the application of force. Reversing the star system would mean destroying the village in order to save it.

Inequality and Fairness #

Given the ubiquity of power law distributions, asking whether there is inequality in the weblog world (or indeed almost any social system) is the wrong question, since the answer will always be yes. The question to ask is “Is the inequality fair?” Four things suggest that the current inequality is mostly fair.

The first, of course, is the freedom in the weblog world in general. It costs nothing to launch a weblog, and there is no vetting process, so the threshold for having a weblog is only infinitesimally larger than the threshold for getting online in the first place.

The second is that blogging is a daily activity. As beloved as Josh Marshall (TalkingPointsMemo.com) or Mark Pilgrim (DiveIntoMark.org) are, they would disappear if they stopped writing, or even cut back significantly. Blogs are not a good place to rest on your laurels.

Third, the stars exist not because of some cliquish preference for one another, but because of the preference of hundreds of others pointing to them. Their popularity is a result of the kind of distributed approval it would be hard to fake.

Finally, there is no real A-list, because there is no discontinuity. Though explanations of power laws (including the ones here) often focus on numbers like “12% of blogs account for 50% of the links”, these are arbitrary markers. The largest step function in a power law is between the #1 and #2 positions, by definition. There is no A-list that is qualitatively different from their nearest neighbors, so any line separating more and less trafficked blogs is arbitrary.

The Median Cannot Hold #

However, though the inequality is mostly fair now, the system is still young. Once a power law distribution exists, it can take on a certain amount of homeostasis, the tendency of a system to retain its form even against external pressures. Is the weblog world such a system? Are there people who are as talented or deserving as the current stars, but who are not getting anything like the traffic? Doubtless. Will this problem get worse in the future? Yes.

Though there are more new bloggers and more new readers every day, most of the new readers are adding to the traffic of the top few blogs, while most new blogs are getting below average traffic, a gap that will grow as the weblog world does. It’s not impossible to launch a good new blog and become widely read, but it’s harder than it was last year, and it will be harder still next year. At some point (probably one we’ve already passed), weblog technology will be seen as a platform for so many forms of publishing, filtering, aggregation, and syndication that blogging will stop referring to any particularly coherent activity. The term ‘blog’ will fall into the middle distance, as ‘home page’ and ‘portal’ have, words that used to mean some concrete thing, but which were stretched by use past the point of meaning. This will happen when head and tail of the power law distribution become so different that we can’t think of J. Random Blogger and Glenn Reynolds of Instapundit as doing the same thing.

At the head will be webloggers who join the mainstream media (a phrase which seems to mean “media we’ve gotten used to.”) The transformation here is simple – as a blogger’s audience grows large, more people read her work than she can possibly read, she can’t link to everyone who wants her attention, and she can’t answer all her incoming mail or follow up to the comments on her site. The result of these pressures is that she becomes a broadcast outlet, distributing material without participating in conversations about it.

Meanwhile, the long tail of weblogs with few readers will become conversational. In a world where most bloggers get below average traffic, audience size can’t be the only metric for success. LiveJournal had this figured out years ago, by assuming that people would be writing for their friends, rather than some impersonal audience. Publishing an essay and having 3 random people read it is a recipe for disappointment, but publishing an account of your Saturday night and having your 3 closest friends read it feels like a conversation, especially if they follow up with their own accounts. LiveJournal has an edge on most other blogging platforms because it can keep far better track of friend and group relationships, but the rise of general blog tools like Trackback may enable this conversational mode for most blogs.

In between blogs-as-mainstream-media and blogs-as-dinner-conversation will be Blogging Classic, blogs published by one or a few people, for a moderately-sized audience, with whom the authors have a relatively engaged relationship. Because of the continuing growth of the weblog world, more blogs in the future will follow this pattern than today. However, these blogs will be in the minority for both traffic (dwarfed by the mainstream media blogs) and overall number of blogs (outnumbered by the conversational blogs.)

Inequality occurs in large and unconstrained social systems for the same reasons stop-and-go traffic occurs on busy roads, not because it is anyone’s goal, but because it is a reliable property that emerges from the normal functioning of the system. The relatively egalitarian distribution of readers in the early years had nothing to do with the nature of weblogs or webloggers. There just weren’t enough blogs to have really unequal distributions. Now there are.

Addendum: #David Sifry, creator of the Technorati.com, has created the Technorati Interesting Newcomers List, in part spurred by this article. The list is designed to flag people with low overall link numbers, but who have done something to merit a sharp increase in links, as a way of making the system more dynamic.

First published January 21, 2003 on the ‘Networks, Economics, and Culture’ mailing list.

The first and last thirds of the music industry have been reconfigured by digital tools. The functions in the middle have not.

Thanks to software like ProTools and CakeWalk, the production of music is heavily digital. Thanks to Napster and its heirs like Gnutella and Kazaa, the reproduction and distribution of music is also digital. As usual, this digitization has taken an enormous amount of power formerly reserved for professionals and delivered it to amateurs. But the middle part — deciding what new music should be available — is still analog and still professionally controlled.

The most important departments at a record label are Artists & Repertoire, and Marketing. A&R’s job is to find new talent, and Marketing’s job is to publicize it. These are both genuinely hard tasks, and unlike production or distribution, there is no serious competition for those functions outside the labels themselves. Prior to its demise, Napster began publicizing itself as a way to find new music, but this was a fig leaf, since users had to know the name of a song or artist in advance. Napster did little to place new music in an existing context, and the current file-sharing networks don’t do much better. In strong contrast to writing and photos, almost all the music available on the internet is there because it was chosen by professionals.

Aggregate Judgments

The curious thing about this state of affairs is that in other domains, we now use amateur input for finding and publicizing. The last 5 years have seen the launch of Google, Blogdex, Kuro5in, Slashdot, and many other collaborative filtering sites that transform the simple judgments of a few participants into aggregate recommendations of remarkably high quality.

This is all part of the Big Flip in publishing generally, where the old notion of “filter, then publish” is giving way to “publish, then filter.” There is no need for Slashdot’s or Kuro5hin’s owners to sort the good posts from the bad in advance, no need for Blogdex or Daypop to pressure people not to post drivel, because lightweight filters applied after the fact work better at large scale than paying editors to enforce minimum quality in advance. A side-effect of the Big Flip is that the division between amateur and professional turns into a spectrum, giving us a world where unpaid writers are discussed side-by-side with New York Times columnists.

The music industry is largely untouched by the Big Flip. The industry harvests the aggregate taste of music lovers and sells it back to us as popularity, without offering anyone the chance to be heard without their approval. The industry’s judgment, not ours, still determines the entire domain in which any collaborative filtering will subsequently operate. A working “publish, then filter” system that used our collective judgment to sort new music before it gets played on the radio or sold at the record store would be a revolution.

Core Assumptions

Several attempts at such a thing have been launched, but most are languishing, because they are constructed as extensions of the current way of producing music, not alternatives to it. A working collaborative filter would have to make three assumptions. 

First, it would have to support the users’ interests. Most new music is bad, and the users know it. Sites that sell themselves as places for bands to find audiences are analogous to paid placement on search engines — more marketing vehicle than real filter. FarmFreshMusic, for example lists its goals as “1. To help artists get signed with a record label. 2. To help record labels find great artists efficiently. 3. To help music lovers find the best music on the Internet.” Note who comes third. 

Second, life is too short to listen to stuff you hate. A working system would have to err more on the side of false negatives (not offering you music you might like) rather than false positives (offering you music you might not like). With false negatives as the default, adventurous users could expand their preferences at will, while the mass of listeners would get the Google version — not a long list of every possible match, but rather a short list of high relevance, no matter what has been left out. 

Finally, the system would have to use lightweight rating methods. The surprise in collaborative filtering is how few people need to be consulted, and how simple their judgments need to be. Each Slashdot comment is moderated up or down only a handful of times, by only a tiny fraction of its readers. The Blogdex Top 50 links are sometimes pointed to by as few as half a dozen weblogs, and the measure of interest is entirely implicit in the choice to link. Despite the almost trivial nature of the input, these systems are remarkably effective, given the mass of mediocrity they are sorting through. 

A working filter for music would similarly involve a small number of people (SMS voting at clubs, periodic “jury selection” of editors a la Slashdot, HotOrNot-style user uploads), and would pass the highest ranked recommendations on to progressively larger pools of judgment, which would add increasing degrees of refinement about both quality and classification. 

Such a system won’t undo inequalities in popularity, of course, because inequality appears whenever a large group expresses their preferences among many options. Few weblogs have many readers while many have few readers, but there is no professional “weblog industry” manipulating popularity. However, putting the filter for music directly in the hands of listeners could reflect our own aggregate judgments back to us more quickly, iteratively, and with less distortion than the system we have today.

Business Models and Love

Why would musicians voluntarily put new music into such a system? 

Money is one answer, of course. Several sorts of businesses profit from music without needing the artificial scarcity of physical media or DRM-protected files. Clubs and concert halls sell music as experience rather than as ownable object, and might welcome a system that identified and marketed artists for free. Webcasting radio stations are currently forced to pay the music industry per listener without extracting fees from the listeners themselves. They might be willing to pay artists for music unencumbered by per-listener fees. Both of these solutions (and other ones, like listener-supported radio) would offer at least some artists some revenues, even if their music were freely available elsewhere. 

The more general answer, however, is replacement of greed with love, in Kevin Kelly’s felicitous construction. The internet has lowered the threshold of publishing to the point where you no longer need help or permission to distribute your work. What has happened with writing may be possible with music. Like writers, most musicians who work for fame and fortune get neither, but unlike writers, the internet has not offered wide distribution to people making music for the love of the thing. A system that offered musicians a chance at finding an audience outside the professional system would appeal to at least some of them. 

Music Is Different

There are obvious differences here, of course, as music is unlike writing in several important ways. Writing tools are free or cheap, while analog and digital instruments can be expensive, and writing can be done solo, while music-making is usually done by a group, making coordination much more complex. Furthermore, bad music is far more painful to listen to than bad writing is to read, so the difference between amateur and professional music may be far more extreme. 

But for all those limits, change may yet come. Unlike an article or essay, people will listen to a song they like over and over again, meaning that even a small amount of high-quality music that found its way from artist to public without passing through an A&R department could create a significant change. This would not upend the professional music industry so much as alter its ecosystem, in the same way newspapers now publish in an environment filled with amateur writing. 

Indeed, the world’s A&R departments would be among the most avid users of any collaborative filter that really worked. The change would not herald the death of A&R, but rather a reconfiguration of the dynamic. A world where the musicians already had an audience when they were approached by professional publishers would be considerably different from the system we have today, where musicians must get the attention of the world’s A&R departments to get an audience in the first place. 

Digital changes in music have given us amateur production and distribution, but left intact professional control of fame. It used to be hard to record music, but no longer. It used to be hard to reproduce and distribute music, but no longer. It is still hard to find and publicize good new music. We have created a number of tools that make filtering and publicizing both easy and effective in other domains. The application of those tools to new music could change the musical landscape.

First published January 7, 2003 on the ‘Networks, Economics, and Culture’ mailing list. 

To understand what’s going to happen to the telephone companies this year thanks to WiFi (otherwise known as 802.11b) and Voice over IP (VoIP) you only need to know one story: ZapMail.

The story goes like this. In 1984, flush from the success of their overnight delivery business, Federal Express announced a new service called ZapMail, which guaranteed document delivery in 2 hours. They built this service not by replacing their planes with rockets, but with fax machines.

This was CEO Fred Smith’s next big idea after the original delivery business. Putting a fax machine in every FedEx office would radically reconfigure the center of their network, thus slashing costs: toner would replace jet fuel, bike messenger’s hourly rates would replace pilot’s salaries, and so on. With a much less expensive network, FedEx could attract customers with a discount on regular delivery rates, but with the dramatically lower costs, profit margins would be huge compared to actually moving packages point to point. Lower prices, higher margins, and to top it all off, the customer would get their documents in 2 hours instead of 24. What’s not to love?

Abject failure was not to love, as it turned out. Two years and hundreds of millions of dollars later, FedEx pulled the plug on ZapMail, allowing it to vanish without a trace. And the story of ZapMail’s collapse holds a crucial lesson for the telephone companies today.

The Customer is the Competitor

ZapMail had three fatal weaknesses.

First of all, Federal Express didn’t get that faxing was a product, not a service. FedEx understood that faxing would be cheaper than physical delivery. What they missed, however, was that their customers understood this too. The important business decision wasn’t when to pay for individual faxes, as the ZapMail model assumed, but rather when to buy a fax machine. The service was enabled by the device, and the business opportunity was in selling the devices.

Second, because FedEx thought of faxing as a service, it failed to understand how the fax network would be built. FedEx was correct in assuming it would take hundreds of millions of dollars to create a useful network. (It has taken billions, in fact, over the last two decades.) However, instead of the single massive build out FedEx undertook, the network was constructed by individual customers buying one fax machine at a time. The capital expenditure was indeed huge, but it was paid for in tiny chunks, at the edges of the network.

Finally, because it misunderstood how the fax network would be built, FedEx misunderstood who its competition was. Seeing itself in the delivery business, it thought it had only UPS and DHL to worry about. What FedEx didn’t see was that its customers were its competition. ZapMail offered two hour delivery for slightly reduced prices, charged each time a message was sent. A business with a fax machine, on the other hand, could send and receive an unlimited number of messages almost instantaneously and at little cost, for a one-time hardware fee of a few hundred dollars.

There was simply no competition. ZapMail looked good next to FedEx’s physical delivery option, but compared to the advantages enjoyed by the owners of fax machines, it was laughable. If the phone network offered cheap service, it was better to buy a device to tap directly into that than to allow FedEx to overcharge for an interface to that network that created no additional value. The competitive force that killed ZapMail was the common sense of its putative users.

ZapPhone

The business Fred Smith imagined being in — build a network that’s cheap to run but charge customers as it if were expensive — is the business the telephone companies are in today. They are selling us a kind of ZapPhone service, where they’ve digitized their entire network up to the last mile, but are still charging the high and confusing rates established when the network was analog.

The original design of the circuit-switched telephone network required the customers to lease a real circuit of copper wire for the duration of their call. Those days are long over, as copper wires have been largely replaced by fiber optic cable. Every long distance phone call and virtually every local call is now digitized for at least some part of its journey.

As FedEx was about faxes, the telephone companies are in deep denial about the change from analog to digital. A particularly clueless report written for the telephone companies offers this choice bit of advice:

Telcos gain billions in service fees from […] services like Call Forwarding and Call Waiting […]. Hence, capex programs that shift a telco, say, from TDM to IP, as in a softswitch approach that might have less capital intensity, must absolutely preserve the revenue stream. [ http://www.proberesearch.com/alerts/refocusing.htm]

You don’t need to know telephone company jargon to see that this is the ZapMail strategy. 

Step #1: Scrap the existing network, which relies on pricey hardware switches and voice-specific protocols like Time Division Multiplexing (TDM). 
Step #2: Replace it with a network that runs on inexpensive software switches and Internet Protocol (IP). This new network will cost less to build and be much cheaper to run. 
Step #3: “Preserve the revenue stream” by continuing to charge the prices from the old, expensive network.

This will not work, because the customers don’t need to wait for the telephone companies to offer services based on IP. The customers already have access to an IP network — it’s called the internet. And like the fax machine, they are going to buy devices that enable the services they want on top of this network, without additional involvement by the telephone companies.

Two cheap consumer devices loom large on this front, devices that create enormous value for the owners while generating little revenue for the phone companies. The first is WiFi access points, which allow the effortless sharing of broadband connections, and the second is VoIP converters, which provide the ability to route phone calls over the internet from a regular phone.

WiFi — Wireless local networks

In classic ZapMail fashion, the telephone companies misunderstand the WiFi business. WiFi is a product, not a service, and they assume their competition is limited to other service companies. There are now half a dozen companies selling wireless access points; at the low end, Linksys sells a hundred dollar device for the home that connects to DSL or cable modems, provides wireless access, and has a built-in ethernet hub to boot. The industry has visions of the “2nd phone line” effect coming to data networking, where multi-computer households will have multiple accounts, but if customers can share a high-speed connection among several devices with a single product, the service business will never materialize.

The wireless ISPs are likely to fare no better. Most people do their computing at home or at work, and deploying WiFi to those two areas will cost at worst a couple hundred bucks, assuming no one to split the cost with. There may be a small business in wiring “third places” — coffee shops, hotels, and meeting rooms — but that will be a marginal business at best. WiFi is the new fax machine, a huge value for consumers that generates little new revenue for the phone companies. And, like the fax network, the WiFi extension to the internet will cost hundreds of millions of dollars, but it will not be built by a few companies with deep pockets. It will be built by millions of individual customers, a hundred dollars at a time.

VoIP — Phone calls at internet prices

Voice over IP is another area where a service is becoming a product. Cisco now manufactures an analog telephone adapter (ATA) with a phone jack in the front and an ethernet jack in the back. The box couldn’t be simpler, and does exactly what you’d expect a box with a phone jack in the front and an ethernet jack in the back to do. The big advantage is that unlike the earlier generation of VoIP products — “Now you can use your computer as a phone!” — the ATA lets you use your phone as a phone, allowing new competitors to offer voice service over any high-speed internet connection.

Vonage.com, for example, is giving away ATAs and offering phone service for $40 a month. Unlike the complex billing structures of the existing telephone companies, Vonage prices the phone like an ISP subscription. A Vonage customer can make an unlimited number of unlimited-length domestic long distance calls for their forty bucks, with call waiting, call forwarding, call transfer, web-accessible voicemail and caller ID thrown in free. Vonage can do this because, like the telephone companies, they are offering voice as an application on a digital network, but unlike the phone companies, they are not committed to charging the old prices by pretending that they are running an analog network.

Voice quality is just one feature among many

True to form, the telephone companies also misunderstand the threat from VoIP (though here it is in part because people have been predicting VoIPs rise since 1996.) The core of the misunderstanding is the MP3 mistake: believing that users care about audio quality above all else. Audiophiles confidently predicted that MP3s would be no big deal, because the sound quality was less than perfect. Listeners, however, turned out to be interested in a mix of things, including accessibility, convenience, and price. The average music lover was willing, even eager, to give up driving to the mall to buy high quality but expensive CDs, once Napster made it possible to download lower quality but free music.

Phone calls are like that. Voice over IP doesn’t sound as good as a regular phone call, and everyone knows it. But like music, people don’t want the best voice quality they can get no matter what the cost, they want a minimum threshold of quality, after which they will choose phone service based on an overall mix of features. And now that VoIP has reached that minimum quality, VoIP offers one feature the phone companies can’t touch: price.

The service fees charged by the average telephone company (call waiting, caller ID, dial-tone and number portability fees, etc) add enough to the cost of a phone that a two-line household that moved only its second line to VoIP could save $40 a month before making their first actual phone call. By simply paying for the costs of the related services, a VoIP customer can get all their domestic phone calls thrown in as a freebie. 

As with ZapMail, the principal threat to the telephone companies’ ability to shrink costs but not revenues is their customers’ common sense. Given the choice, an increasing number of customers will simply bypass the phone company and buy the hardware necessary to acquire the service on their own.

And hardware symbiosis will further magnify the threat of WiFi and VoIP. The hardest part of setting up VoIP is simply getting a network hub in place. Once a hub is installed, adding an analog telephone adapter is literally a three-plug set-up: power, network, phone. Meanwhile, one of the side-effects of installing WiFi is getting a hub with open ethernet ports. The synergy is obvious: Installing WiFi? You’ve done most of the work towards adding VoIP. Want VoIP? Since you need to add a hub, why not get a WiFi-enabled hub? (There are obvious opportunities here for bundling, and later for integration — a single box with WiFi, Ethernet ports, and phone jacks for VoIP.)

The economic logic of customer owned networks

According to Metcalfe’s Law, the value of an internet connection rises with the number of users on the network. However, the phone companies do not get to raise their prices in return for that increase in value. This is a matter of considerable frustration to them.

The economic logic of the market suggests that capital should be invested by whoever captures the value of the investment. The telephone companies are using that argument to suggest that they should either be given monopoly pricing power over the last mile, or that they should be allowed to vertically integrate content with conduit. Either strategy would allow them to raise prices by locking out the competition, thus restoring their coercive power over the customer and helping them extract new revenues from their internet subscribers.

However, a second possibility has appeared. If the economics of internet connectivity lets the user rather than the network operator capture the residual value of the network, the economics likewise suggest that the user should be the builder and owner of the network infrastructure.

The creation of the fax network was the first time this happened, but it won’t be the last. WiFi hubs and VoIP adapters allow the users to build out the edges of the network without needing to ask the phone companies for either help or permission. Thanks to the move from analog to digital networks, the telephone companies’ most significant competition is now their customers, because if the customer can buy a simple device that makes wireless connectivity or IP phone calls possible, then anything the phone companies offer by way of competition is nothing more than the latest version of ZapMail.

First published on O’Reilly’s OpenP2P on January 7, 2003.

A persistent criticism of open source software is that it is more about copying existing features than creating new ones. While this criticism is overblown, the literature of open source is clearer on debugging than on design. This note concerns an attempt to apply debugging techniques to feature requests and concludes by describing Ben Hammersley’s attempt to create such a system, implemented as an RSS feed.

A key observation in Eric Raymond’s The Cathedral and the Bazaar is: “Given enough eyeballs, all bugs are shallow.” Raymond suggests that Brook’s Law–“Adding more programmers to a late software project makes it later”–doesn’t apply here, because debugging is less collaborative than most other software development. He quotes Linus on the nature of debugging: “Somebody finds the problem, and somebody else understands it. And I’ll go on record as saying that finding it is the bigger challenge.”

Finding a bug doesn’t mean simply pointing out broken behavior, though. Finding a bug means both locating it and describing it clearly. The difference between “It doesn’t work” and “Whenever I resize the login window, the Submit button disappears” is the difference between useless mail and useful feedback. Both the description and the fix are vital, and the description precedes the fix.

Enter the LazyWeb

There is evidence that this two-step process applies to features as well, in a pattern Matt Jones has dubbed the LazyWeb. The original formulation was “If you wait long enough, someone will write/build/design what you were thinking about.” But it is coming to mean “I describe a feature I think should exist in hopes that someone else will code it.” Like debugging, the success of the LazyWeb is related at least in part to the quality of the descriptions. A feature, schema, or application described in enough detail can give the right developer (usually someone thinking about the same problem) a clear idea of how to code it quickly. 

Examples of the LazyWeb in action are Stephen Johnson’s URL catcher as built by Andre Torrez, and Ben Trott’s “More Like This From Others” feature after Ben Hammersley’s initial characterization.

LazyWeb seems to work for at least three reasons:

1. Developers have blind spotsBecause a developer knows a piece of software in a far more detailed way than their users, they can have blind spots. A good LazyWeb description provides a developer with a alternate perspective on the problem. And sometimes the triviality of the coding involved keeps developers from understanding how valuable a feature would be to its users, as with Yoz Grahame’s “get, search and replace, display” script for revising the text of web pages. Sometimes four lines of code can make all the difference.
2. Developers have social itchesThe canonical motivation for open source developers is that they want to “scratch an itch.” In this view, most open source software is written with the developer as the primary user, with any additional use seen as a valuable but secondary side-effect.Sometimes, though, the itch a developer has is social: they want to write software other people will adopt. In this case, the advantage of the LazyWeb is not just that a new application or feature is described clearly, but that it is guaranteed to have at least one grateful user. Furthermore, LazyWeb etiquette involves publicizing any solution that does arise, meaning that the developer gets free public attention, even if only to a select group. If writing software that gets used in the wild is a motivation, acting on a LazyWeb description is in many ways a karmically optimal move.
3. Many eyes make features shallowThis is really a meta-advantage. The above advantages would apply even to a conversation between a single describer and a single developer, if they were the right people. Expanding the conversation to include more describers and developers increases the possibility that at least one such pairing will occur.

Transaction Costs and Coordination Costs

The transaction costs of the LazyWeb are extremely low. Someone describes; someone else codes. The describer can write sketchily or in great detail. No developers are required to read the description; and those who do read it can ignore or modify the proposed design. The interface between the parties is lightweight, one-way, and optional.

However, the coordination costs of the LazyWeb as a whole are very high, and they will grow as more people try it. More people can describe features than write software, just as more people can characterize bugs than fix them. Unlike debugging, however, a LazyWeb description does not necessarily have a target application or a target group of developers. This creates significant interface problems, since maximal LazyWeb awareness would have every developer reading every description, an obvious impossibility. (Shades of Brook’s Law.)

This would be true even if the LazyWeb were confined to skilled programmers. The ability of system architects, say, to describe new visual layout tools, or graphics programmers to characterize their filesharing needs, ensures that there will always be more capable describers than suitable developers.

Thus the LazyWeb is currently limited to those environments that maximize the likelihood that a developer with a social itch and a good grasp of the problem space will happen to read a particular LazyWeb description. In practice, this means that successful LazyWeb requests work best when posted on a few blogs read by many developers. Far from being a “More describers than developers” scenario, in other words, the current LazyWeb has many fewer describers than developers, with the developers fragmented across several sites.

Sounds Like a Job for RSS

One common answer to this kind of problem is to launch a portal for all LazyWeb requests. (There have been earlier experiments in this domain, like http://www.halfbakery.com, http://www.shouldexist.org, andhttp://www.creativitypool.com, and Magnetbox has launched a Lazyweb-specific site.) These sites are meant to be brokers between describers and developers.

However, nearly a decade of experimentation with single-purpose portals shows that most of them fail. As an alternative to making a LazyWeb portal, creating an RSS feed of LazyWeb descriptions has several potential advantages, including letting anyone anywhere add to the feed, letting sites that serve developers present the feed in an existing context, and letting the developers themselves fold, spindle, and mutilate the feed in any way they choose.

Ben Hammersley has designed a version of a LazyWeb feed. It has three moving parts. 

The first part is the collection of descriptions. Hammersley assumes that a growing number of people will be writing LazyWeb descriptions, and that most of these descriptions will be posted to blogs.

The second part is aggregation. He has created a trackback address, http://blog.mediacooperative.com/mt-tb.cgi/1080, for LazyWeb posts. Blog posts that point to this address are aggregated and presented at http://www.benhammersley.com/lazyweb/.

The third part is the RSS feed itself, athttp://www.benhammersley.com/lazyweb/index.rdf, which is simply the XML version of http://www.benhammersley.com/lazyweb/. However, because it is a feed, third parties can subscribe to it, filter it, present it as a sidebar on their own sites, and so on.

It’s easy to see new features that could be added to this system. A LazyWeb item in RDF has only four elements, set by the Trackback spec — title, link, description, and date. Thus almost all the onus on filtering the feed is on the subscriber, not the producer. An RDF format with optional but recommended tags (type: feature, schema, application, etc; domain: chat, blog, email, etc.) might allow for higher-quality syndication, but would be hard with the current version of Trackback. Alternatively, community consensus about how to use title tags to characterize feature requests could help.

And not everyone with a LazyWeb idea runs a Trackback-enabled weblog, so having some way for those people to register their ideas could be useful. Hooks for automated translation could make the feed more useful to developers working in languages other than English, and so on.

But for all the possible new features, this is a good start, having achieved a kind of bootstrap phase analogous to compiler development. The original work came out of a LazyWeb characterization made during a larger conversation Jones, Hammersley, and I have been having about social software, and some of the early LazyWeb requests are themselves feature descriptions for the system.

Will It Work?

Will it work? Who knows. Like any experiment, it could die from inactivity. It could also be swamped by a flood of low-quality submissions. It may be that the membrane that a weblog forms around its readers is better for matching describers and developers than an open feed would be. And Paul Hammond has suggested that “Any attempt to invoke the LazyWeb directly will cause the whole thing to stop working.” 

It’s worth trying, though, because the potential win is so large. If the benefits open source development offers for fixing bugs can be applied to creating features as well, it could confer a huge advantage on the development of Mob Software.

Stefano Mazzocchi of the Cocoon project has said “Anyway, it’s a design pattern: ‘good ideas and bad code build communities, the other three combinations do not.” This is extremely hard to understand, it’s probably the most counter-intuitive thing about open source dynamics.” If Mazzocchi is right, then a high-quality stream of feature requests could be a powerful tool for building communities of developers and users, as well as providing a significant advantage to open over closed source development.

The closed source shops could subscribe to such a feed as well, of course, but their advantage on the feature front isn’t speed, it’s secrecy. If a small group of closed source developers is working on a feature list that only they know, they will often ship it first. But for good ideas in the public domain, the open and closed development teams will have the same starting gun. And releasing early and often is where open development has always excelled.

It’s too early to tell if LazyWeb is just the flavor of the month or points to something profound about the way ideas can spread. And it’s much too early to know if an RSS feed is the right way to spread LazyWeb ideas to the developers best able to take advantage of them. But it’s not too early to know that it’s worth trying.

First published on the ‘Networks, Economics, and Culture mailing list.

For decades, the privacy debate has centered on questions about databases and database interoperability: How much information about you exists in the world’ databases? How easily is it retrieved? How easily is it compared or combined with other information?

Databases have two key weaknesses that affect this debate. The first is that they deal badly with ambiguity, and generally have to issue a unique number, sometimes called a primary key, to every entity they store information on. The US Social Security number is a primary key that points to you, the 6-letter Passenger Name Record is a primary key that points to a particular airline booking, and so on. This leads to the second weakness: since each database maintains its own set of primary keys, creating interoperability between different databases is difficult and expensive, and generally requires significant advance coordination.

Privacy advocates have relied on these weaknesses in creating legal encumbrances to issuing and sharing primary keys. They believe, rightly, that widely shared primary keys pose a danger to privacy. (The recent case of Princeton using its high school applicants’ Social Security numbers to log in to the Yale admittance database highlights these dangers.) The current worst-case scenario is a single universal database in which all records — federal, state, and local, public and private — would be unified with a single set of primary keys.

New technology brings new challenges however, and in the database world the new challenge is not a single unified database, but rather decentralized interoperability, interoperability brought about by a single universally used ID. The ID is DNA. The interoperability comes from the curious and unique advantages DNA has as a primary key. And the effect will put privacy advocates in a position analogous to that of the RIAA, forcing them to switch from fighting the creation of a single central database to fighting a decentralized and interoperable system of peer-to-peer information storage.

DNA Markers

While much of the privacy debate around DNA focuses on the ethics of predicting mental and physical fitness for job categories and insurance premiums, this is too narrow and too long-range a view. We don’t even know yet how many genes there are in the human genome, so our ability to make really sophisticated medical predictions based on a person’s genome is still some way off. However, long before that day arrives, DNA will provide a cheap way to link a database record with a particular person, in a way that is much harder to change or forge than anything we’ve ever seen.

Everyone has a biological primary key embedded in every cell of their body in the form of DNA, and everyone has characteristic zones of DNA that can be easily read and compared. These zones serve as markers, and they differ enough from individual to individual that with fewer than a dozen of them, a person can be positively identified out of the entire world’s population.

DNA-as-marker, in other words, is a nearly perfect primary key, as close as we can get to being unambiguous and unforgeable. If every person has a primary key that points to their physical being, then the debate about who gets to issue such a key are over, because the keys are issued every time someone is born, and re-issued every time a new cell is created. And if the keys already exist, then the technological argument is not about creating new keys, but about reading existing ones.

The race is on among several biotech firms to be able to sequence a person’s entire genome for $1000. The $1 DNA ID will be a side effect of this price drop, and it’s coming soon. When the price of reading DNA markers drops below a dollar, it will be almost impossible to control who has access to reading a person’s DNA.

There are few if any legal precedents that would prevent collection of this data, at least in the US. There are several large populations that do not enjoy constitutional protections of privacy, such as the armed services, prisoners, and children. Furthermore, most of the controls on private databases rely on the silo approach, where an organization can collect an almost unlimited amount of information about you, provided they abide by the relatively lax rules that govern sharing that information.

Even these weak protections have been enough, however, to prevent the creation of a unified database, because the contents of two databases cannot be easily merged without some shared primary key, and shared primary keys require advance coordination. And it is here, in the area of interoperability, that DNA markers will have the greatest effect on privacy.

You’re the Same You Everywhere

Right now, things like alternate name spellings or alternate addresses make positive matching difficult across databases. Its hard to tell if Eric with the Wyoming driver’s license and Shawn with the Florida arrest record are the same person, unless there is other information to tie them together. If two rows of two different databases are tied to the same DNA ID, however, they point to the same person, no matter what other material is contained in the databases, and no matter how it is organized or labeled.

No more trying to figure out if Mr. Shuler and Mr. Schuller are the same person, no more wondering if two John Smiths are different people, no more trying to guess the gender of J. Lee. Identity collapses to the body, in a way that is far more effective than fingerprints, and far more easily compared across multiple databases than more heuristic measures like retinal scans.

In this model, the single universal database never gets created, not because privacy advocates prevent it, but because it is no longer needed. If primary keys are issued by nature, rather than by each database acting alone, then there is no more need for central databases or advance coordination, because the contents of any two DNA-holding databases can be merged on demand in something close to real time.

Unlike the creation of a vast central database, even a virtual one, the change here can come about piecemeal, with only a few DNA-holding databases. A car dealer, say, could simply submit a DNA marker to a person’s bank asking for a simple yes-or-no match before issuing a title. In the same way the mid-90s ID requirements for US domestic travel benefited the airlines because it kept people from transferring unused tickets to friends of family, we can expect businesses to like the way DNA ties transactions to a single customer identity.

The privacy debate tends to be conducted as a religious one, with the absolutists making the most noise. However, for a large number of people, privacy is a relative rather than an absolute good. The use of DNA as an ID will spread in part because people want it to, in the form of credit cards that cannot be used in other hands or cars that cannot be driven by other drivers. Likewise, demands that DNA IDs be derived from populations who do not enjoy constitutional protections, whether felons or children, will be hard to deflect as the cost of reading an individual’s DNA falls dramatically, and as the public sees the effective use of DNA in things like rape and paternity cases.

Peer-to-Peer Collation of Data

In the same way Kazaa has obviated the need for central storage or coordination for the world’s music, the use of DNA as an ID technology makes radically decentralized data integration possible. With the primary key problem solved, interoperability will arise as a side effect, neither mandated nor coordinated centrally. Fighting this will require different tactics, not least because it is a rear-guard action. The keys and the readers both exist, and the price and general availability of the technology all point to ubiquity and vanishingly low cost within a decade.

This is a different kind of fight over privacy. As the RIAA has discovered, fighting the growth of a decentralized and latent capability is much harder than fighting organizations that rely on central planning and significant resources, because there is no longer any one place to focus the efforts, and no longer any small list of organizations who can be targeted for preventive action. In a world where database interoperability moves from a difficult and costly goal to one that arises as a byproduct of the system, the important question for privacy advocates is how they will handle the change.