First published on O’Reilly’s OpenP2P, 01/18/2001

Many companies in the P2P space are trying to figure out how to deploy P2P resources most effectively in a dynamic system. This problem is particularly acute for the companies selling distributed computation, such as Popular Power and United Devices, or the companies trying to build a general P2P framework, such as ROKU and Globus.

The first successful implementations of distributed computation like SETI@Home and distributed.net relied on non-financial incentives: The participants donated their cycles because they felt good about the project, or because they enjoyed “collaborative competition,” such as distributed.net’s team rankings for its encryption-cracking contests.

Ego gratification is a powerful tool, but it is a finicky one. Someone happy to donate their time to chess problems may not want to donate their cycles to designing bioengineered food. The problem that companies who rely on distributed resources face is how to get people to give time to commercial projects. 

The general solution to this problem seems to be “if you give people an incentive to do something, they will respond, so find the right incentive.” If ego gratification is an effective way to get people to donate resources, the trick is in finding other kinds of incentives to replace ego gratification for commercial services.

The obvious all-purpose incentive is money (or at least some sort of currency), and several systems are being designed with the idea that paying users is the best way convince them to provide resources.

This solution may not work well, however, for things like cycles and disk space, because there are two kinds of resources that can be deployed in a P2P system, and they have very different characteristics — a difference that can best be illustrated by The Parable of the Umbrellas and Taxis.

Umbrellas and taxis

Anyone who’s spent any time in New York City knows that when it begins to rain, two things happen immediately: It becomes easier to buy an umbrella and it becomes harder to hail a cab. As soon as the first few drops fall, people appear on the street selling cheap umbrellas, while a lucky few pedestrians occupy all the available cabs.

Why does an increase in demand produce opposite effects on supply — more available umbrellas and fewer available taxis? The answer is the nature of the resources themselves. Umbrellas are small and inexpensive to store, so it’s easy to take them out when it’s raining and put them back when the rain stops. Additional umbrellas can be deployed in response to demand.

Taxis, on the other hand, are large and expensive to store. In addition, taxis have all sorts of up-front costs: registration for a yellow cab or car service, license for the driver, local regulations, the cost of an automobile. These up-front costs can be high or low, but whatever they are, they set some maximum number of cabs available in the city on any given day. And if it starts raining, too bad: Additional taxis cannot be deployed in response to peak demand. Every city has a total number of cabs which represents a compromise between the number of potential riders in sun vs. rain, or 4 a.m. vs. 4 p.m., or April vs. August.

Not all P2P resources are created equal

Some of the resources in a P2P system are umbrellas. Some are taxis.

Umbrella resources are what make things like file-sharing systems so powerful. If you decide at 3:17 a.m. that you must listen to Golden Earring’s “Radar Love” or you will surely die, it’s Napster to the rescue: Your demand produces extra supply.

If, however, you decide that you must have a faster chip, you’re out of luck. For that, you have to make a trip to the store. You could use someone else’s cycles, of course, but you can’t increase the total number of cycles in your system: Your demand does not produce extra supply.

This has ramifications for getting users to provide additional resources to a P2P system. If United Devices wants to use more cycles than you have, you can’t instantly produce more chip. Ditto bandwidth and disk space. You have what you have, and you can’t deploy any more than that in the short term.

Thresholds and gaps

Since the increased demand during a rainstorm doesn’t create more taxis on the spot, the way to have more taxis when it rains is to have more taxis overall — to change the overall capacity of the system. One could make it cheaper to buy a taxi, raise the rates drivers could charge, or relax the legal restrictions on the business, and any of these things would increase capacity.

What doesn’t increase taxi capacity is momentarily increased demand, at least not in well-off economies. It’s easy to see how more cab service could become available — drivers of ordinary cars could simply offer their services to damp pedestrians in times of increased demand. Why don’t they do this?

The answer is that the pedestrians aren’t willing to pay enough to make it worth the drivers’ time. There are many sorts of obstacles here — from the time it would take to haggle, to the laws regulating such a thing — but all these obstacles add up to a higher cost than the pedestrian is willing to pay. A passenger would pay more to get in a cab when it’s raining than when it’s dry, but not enough more to change the behavior of private drivers.

One particular effect here deserves mention: Unlike cab drivers, whose vehicles are deployed for use by others, drivers of private cars are presumably driving those cars for some reason other than picking up random passengers. Part of the threshold that keeps them from picking up riders for a fee is that they are unwilling to give up their own use of their cars for the period the passenger wants to use it.

Variable use equals variable value

With that in mind, consider the case of PC users in a distributed computing system like Popular Power. Assume the average computer chassis costs $1,000. Assume also that the average user replaces their computer every two and a half years. This means $1,000 buys roughly 20,000 hours of device use prior to replacement.

Now imagine you owned such a machine and were using it to play Quake, but Popular Power wanted to use it to design flu vaccines. To compensate you for an hour of your computing time, Popular Power should be willing to offer you a nickel, which is to say 1/20,000th of $1,000, the cost of your device for that hour.

Would you be willing to give up an hour of playing Quake (or working on a spreadsheet, or chatting with your friends) for a nickel? No. And yet, once the cost crosses the nickel threshold, Popular Power is spending enough, pro-rata, to buy their own box. 

The secret to projects like Popular Power, in other words, is to use the gap between the permanence of your computer and the variability of your use of it. Of the 20,000 hours of so you will own your computer, between 1,000 and 5,000 of those hours are likely to be highly valuable to you — too valuable for Popular Power to be able to get you to give them up for any amount of money they are willing to pay. 

Successful P2P programs recognize the implicit difference between the times you want to use your PC and the times you don’t. Napster gives users total control of when they want to be logged in, and allows the owner of any PC to unilaterally terminate any uploads that annoy them. Popular Power runs in the background, only actively using cycles when the user is not.

Incentives that match value

Cycles, disk space, and bandwidth are like taxis: They are resources that get provisioned up front and are used variably from then on. The way to get more such resources within a P2P system is to change the up-front costs — not the ongoing costs — since it is the up-front costs that determine the ceiling on available resources.

There are several sorts of up-front incentives that could raise this ceiling: 

• PeoplePC could take $100 off the price of your computer in return for your spare cycles. 
• AudioGalaxy could pay for the delta between 384- and 768-Kbps DSL in return for running the AudioGalaxy client in the background. 
• United Devices could pay your electricity bill in return for leaving your machine on 24/7.

Those kinds of incentives match the way resources are currently deployed, and those incentives will have far more effect on the resources available to P2P systems than simply telling you that there’s momentary demand for more resources than you have.

Futures markets and big winners

Because these resources need to be provisioned in large chunks when the machines are bought, and because users don’t want to spend their time and effort putting spot prices on unused cycles, the markets that form around P2P resources are not likely to be real-time micromarkets but futures macromarkets. By providing up-front incentives, or ongoing incentives that don’t need to be re-priced (a donation to a charity, offsetting the cost of electricity or bandwidth), companies that have access to distributed computing resources are likely to be able to create and maintain vast pools of untapped computing power. 

As long as end users aren’t required to give up their use of the PC during the 1,000 to 5,000 hours they need it, they will prefer seeing the remaining 15,000 hours used in a simple way they approve of, rather than spending the time working out the bidding between companies paying pennies an hour at best. Evenness of reward and lowered mental transaction costs are a big incentive to adopt a “set it and forget it”attitude.

Indeed, the price fluctuations and market are likely to be at the other end of the scale, on a futures market for vast amounts of bandwidth. If a company wants 100,000 hours of computing time on the day they close their quarterly books, they may be willing to pay more for that than simply getting any 100,000 hours spread over three months. 

This suggests a futures market dealing in massive quantities of computing power, a market participated in by a small group of companies that can guarantee delivery of certain minimums on certain dates, or within certain time periods. The price of a bushel of wheat is a commodity, a price not set or affected by individual wheat producers (without operating a cartel), so the price fluctuation is set by the largest consumers. No one goes onto the futures market to buy guaranteed future delivery of a single barrel of oil or bushel of wheat — the price is set by buying and selling in bulk.

Far from being an atomized many-to-many market of buyers, aggregators, and sellers, distributed computing will likely be a “butterfly” market with many providers of spare cycles, many consumers of cycles, and a very few aggregators, all of whom are pursuing network effects and massive economies of scale. 

The likeliest winners are the companies or consortia that have the most open framework and the most installed clients, because once one or a few leaders emerge, they will be in a better position to create such a futures market than hundreds of also-ran competitors. (Of particular note here is Microsoft, who has access to more desktops than everyone else put together. A real P2P framework, run by Microsoft, could become the market leader in selling aggregated computing power.)

Many people in the P2P world are talking about general P2P frameworks for sharing any and all computing resources, and this language makes it seem like all resources are equally fungible. In fact, the vendors of umbrellas are operating under conditions very different from the operators of taxis. The companies aggregating and reselling the resources that are allocated up front and in big chunks, will likely face brutal competition between an ever-smaller group of ever-larger players. The economics of this part of the business so favor economies of scale that within 12 months, even as the rest of the P2P infrastructure is developing, the winners in the world of distributed computation will be anointed.

First published on O’Reilly OpenP2P, 12/19/2000.

Micropayments are back, at least in theory, thanks to P2P. Micropayments are an idea with a long history and a disputed definition – as the W3C micropayment working group puts it, ” … there is no clear definition of a ‘Web micropayment’ that encompasses all systems,” but in its broadest definition, the word micropayment refers to “low-value electronic financial transactions.”

P2P creates two problems that micropayments seem ideally suited to solve. The first is the need to reward creators of text, graphics, music or video without the overhead of publishing middlemen or the necessity to charge high prices. The success of music-sharing systems such as Napster and Audiogalaxy, and the growth of more general platforms for file sharing such as Gnutella, Freenet and AIMster, make this problem urgent.

The other, more general P2P problem micropayments seem to solve is the need for efficient markets. Proponents believe that micropayments are ideal not just for paying artists and musicians, but for providers of any resource – spare cycles, spare disk space, and so on. Accordingly, micropayments are a necessary precondition for the efficient use of distributed resources.

Jakob Nielsen, in his essay The Case for Micropayments writes, “I predict that most sites that are not financed through traditional product sales will move to micropayments in less than two years,” and Nicholas Negroponte makes an even shorter-term prediction: “You’re going to see within the next year an extraordinary movement on the Web of systems for micropayment … .” He goes on to predict micropayment revenues in the tens or hundreds of billions of dollars.

Alas for micropayments, both of these predictions were made in 1998. (In 1999, Nielsen reiterated his position, saying, “I now finally believe that the first wave of micropayment services will hit in 2000.”) And here it is, the end of 2000. Not only did we not get the flying cars, we didn’t get micropayments either. What happened?

Micropayments: An Idea Whose Time Has Gone

Micropayment systems have not failed because of poor implementation; they have failed because they are a bad idea. Furthermore, since their weakness is systemic, they will continue to fail in the future.

Proponents of micropayments often argue that the real world demonstrates user acceptance: Micropayments are used in a number of household utilities such as electricity, gas, and most germanely telecom services like long distance.

These arguments run aground on the historical record. There have been a number of attempts to implement micropayments, and they have not caught on in even in a modest fashion – a partial list of floundering or failed systems includes FirstVirtual, Cybercoin, Millicent, Digicash, Internet Dollar, Pay2See, MicroMint and Cybercent. If there was going to be broad user support, we would have seen some glimmer of it by now.

Furthermore, businesses like the gas company and the phone company that use micropayments offline share one characteristic: They are all monopolies or cartels. In situations where there is real competition, providers are usually forced to drop “pay as you go” schemes in response to user preference, because if they don’t, anyone who can offer flat-rate pricing becomes the market leader. (See sidebar: “Simplicity in pricing.”)

Simplicity in pricing The historical record for user preferences in telecom has been particularly clear. In Andrew Odlyzko’s seminal work, The history of communications and its implications for the Internet, he puts it this way: “There are repeating patterns in the histories of communication technologies, including ordinary mail, the telegraph, the telephone, and the Internet. In particular, the typical story for each service is that quality rises, prices decrease, and usage increases to produce increased total revenues. At the same time, prices become simpler. “The historical analogies of this paper suggest that the Internet will evolve in a similar way, towards simplicity. The schemes that aim to provide differentiated service levels and sophisticated pricing schemes are unlikely to be widely adopted.”

Why have micropayments failed? There’s a short answer and a long one. The short answer captures micropayment’s fatal weakness; the long one just provides additional detail. 

The Short Answer for Why Micropayments Fail

Users hate them.

The Long Answer for Why Micropayments Fail

Why does it matter that users hate micropayments? Because users are the ones with the money, and micropayments do not take user preferences into account.

In particular, users want predictable and simple pricing. Micropayments, meanwhile, waste the users’ mental effort in order to conserve cheap resources, by creating many tiny, unpredictable transactions. Micropayments thus create in the mind of the user both anxiety and confusion, characteristics that users have not heretofore been known to actively seek out.

Anxiety and the Double-Standard of Decision Making

Many people working on micropayments emphasize the need for simplicity in the implementation. Indeed, the W3C is working on a micropayment system embedded within a link itself, an attempt to make the decision to purchase almost literally a no-brainer.

Embedding the micropayment into the link would seem to take the intrusiveness of the micropayment to an absolute minimum, but in fact it creates a double-standard. A transaction can’t be worth so much as to require a decision but worth so little that that decision is automatic. There is a certain amount of anxiety involved in any decision to buy, no matter how small, and it derives not from the interface used or the time required, but from the very act of deciding.

Micropayments, like all payments, require a comparison: “Is this much of X worth that much of Y?” There is a minimum mental transaction cost created by this fact that cannot be optimized away, because the only transaction a user will be willing to approve with no thought will be one that costs them nothing, which is no transaction at all. 

Thus the anxiety of buying is a permanent feature of micropayment systems, since economic decisions are made on the margin – not, “Is a drink worth a dollar?” but, “Is the next drink worth the next dollar?” Anything that requires the user to approve a transaction creates this anxiety, no matter what the mechanism for deciding or paying is. 

The desired state for micropayments – “Get the user to authorize payment without creating any overhead” – can thus never be achieved, because the anxiety of decision making creates overhead. No matter how simple the interface is, there will always be transactions too small to be worth the hassle.

Confusion and the Double-Standard of Value

Even accepting the anxiety of deciding as a permanent feature of commerce, micropayments would still seem to have an advantage over larger payments, since the cost of the transaction is so low. Who could haggle over a penny’s worth of content? After all, people routinely leave extra pennies in a jar by the cashier. Surely amounts this small makes valuing a micropayment transaction effortless?

Here again micropayments create a double-standard. One cannot tell users that they need to place a monetary value on something while also suggesting that the fee charged is functionally zero. This creates confusion – if the message to the user is that paying a penny for something makes it effectively free, then why isn’t it actually free? Alternatively, if the user is being forced to assent to a debit, how can they behave as if they are not spending money? 

Beneath a certain price, goods or services become harder to value, not easier, because the X for Y comparison becomes more confusing, not less. Users have no trouble deciding whether a $1 newspaper is worthwhile – did it interest you, did it keep you from getting bored, did reading it let you sound up to date – but how could you decide whether each part of the newspaper is worth a penny?

Was each of 100 individual stories in the newspaper worth a penny, even though you didn’t read all of them? Was each of the 25 stories you read worth 4 cents apiece? If you read a story halfway through, was it worth half what a full story was worth? And so on.

When you disaggregate a newspaper, it becomes harder to value, not easier. By accepting that different people will find different things interesting, and by rolling all of those things together, a newspaper achieves what micropayments cannot: clarity in pricing. 

The very micro-ness of micropayments makes them confusing. At the very least, users will be persistently puzzled over the conflicting messages of “This is worth so much you have to decide whether to buy it or not” and “This is worth so little that it has virtually no cost to you.”

User Preferences

Micropayment advocates mistakenly believe that efficient allocation of resources is the purpose of markets. Efficiency is a byproduct of market systems, not their goal. The reasons markets work are not because users have embraced efficiency but because markets are the best place to allow users to maximize their preferences, and very often their preferences are not for conservation of cheap resources.

Imagine you are moving and need to buy cardboard boxes. Now you could go and measure the height, width, and depth of every object in your house – every book, every fork, every shoe – and then create 3D models of how these objects could be most densely packed into cardboard boxes, and only then buy the actual boxes. This would allow you to use the minimum number of boxes.

But you don’t care about cardboard boxes, you care about moving, so spending time and effort to calculate the exact number of boxes conserves boxes but wastes time. Furthermore, you know that having one box too many is not nearly as bad as having one box too few, so you will be willing to guess how many boxes you will need, and then pad the number.

For low-cost items, in other words, you are willing to overpay for cheap resources, in order to have a system that maximizes other, more important, preferences. Micropayment systems, by contrast, typically treat cheap resources (content, cycles, disk) as precious commodities, while treating the user’s time as if were so abundant as to be free.

Micropayments Are Just Payments

Neither the difficulties posed by mental transaction costs nor the the historical record of user demand for simple, predictable pricing offers much hope for micropayments. In fact, as happened with earlier experiments attempting to replace cash with “smart cards,” a new form of financial infrastructure turned out to be unnecessary when the existing infrastructure proved flexible enough to be modified. Smart cards as cash replacements failed because the existing credit card infrastructure was extended to include both debit cards and ubiquitous card-reading terminals.

So it is with micropayments. The closest thing we have to functioning micropayment systems, Qpass and Paypal, are simply new interfaces to the existing credit card infrastructure. These services do not lower mental transaction costs nor do they make it any easier for a user to value a penny’s worth of anything – they simply make it possible for users to spend their money once they’ve decided to.

Micropayment systems are simply payment systems, and the size and frequency of the average purchase will be set by the user’s willingness to spend, not by special infrastructure or interfaces. There is no magic bullet – only payment systems that work within user expectations can succeed, and users will not tolerate many tiny payments.

Old Solutions

This still leaves the problems that micropayments were meant to solve. How to balance users’ strong preference for simple pricing with the enormous number of cheap, but not free, things available on the Net?

Micropayment advocates often act as if this is a problem particular to the Internet, but the real world abounds with items of vanishingly small value: a single stick of gum, a single newspaper article, a single day’s rent. There are three principal solutions to this problem offline – aggregation, subscription, and subsidy – that are used individually or in combination. It is these same solutions – and not micropayments – that are likely to prevail online as well. 

Aggregation

Aggregation follows the newspaper example earlier – gather together a large number of low-value things, and bundle them into a single higher-value transaction.

Call this the “Disneyland” pricing model – entrance to the park costs money, and all the rides are free. Likewise, the newspaper has a single cost, that, once paid, gives the user free access to all the stories.

Aggregation also smoothes out the differences in preferences. Imagine a newspaper sold in three separate sections – news, business, and sports. Now imagine that Curly would pay a nickel to get the news section, a dime for business, and a dime for sports; Moe would pay a dime each for news and business but only a nickel for sports; and Larry would pay a dime, a nickel, and a dime. 

If the newspaper charges a nickel a section, each man will buy all three sections, for 15 cents. If it prices each section at a dime, each man will opt out of one section, paying a total of 20 cents. If the newspaper aggregates all three sections together, however, Curly, Moe and Larry will all agree to pay 25 cents for the whole, even though they value the parts differently.

Aggregation thus not only lowers the mental transaction costs associated with micropayments by bundling several purchase decisions together, it creates economic efficiencies unavailable in a world where each resource is priced separately. 

Subscription

A subscription is a way of bundling diverse materials together over a set period, in return for a set fee from the user. As the newspaper example demonstrates, aggregation and subscription can work together for the same bundle of assets. 

Subscription is more than just aggregation in time. Money’s value is variable – $100 today is better than $100 a month from now. Furthermore, producers value predictability no less than consumers, so producers are often willing to trade lower subscription prices in return for lump sum payments and more predictable revenue stream.

Long-term incentivesGame theory fans will recognize subscription arrangements as an Iterated Prisoner’s Dilemma, where the producer’s incentive to ship substandard product or the consumer’s to take resources without paying is dampened by the repetition of delivery and payment.

Subscription also serves as a reputation management system. Because producer and consumer are more known to one another in a subscription arrangement than in one-off purchases, and because the consumer expects steady production from the producer, while the producer hopes for renewed subscriptions from the consumer, both sides have an incentive to live up to their part of the bargain, as a way of creating long-term value. (See sidebar: “Long-term incentives”.)

Subsidy

Subsidy is by far the most common form of pricing for the resources micropayments were meant to target. Subsidy is simply getting someone other than the audience to offset costs. Again, the newspaper example shows that subsidy can exist alongside aggregation and subscription, since the advertisers subsidize most, and in some cases all, of a newspaper’s costs. Advertising subsidy is the normal form of revenue for most Web sites offering content.

The biggest source of subsidy on the Net overall, however, is from the the users themselves. The weblog movement, where users generate daily logs of their thoughts and interests, is typically user subsidized – both the time and the resources needed to generate and distribute the content are donated by the user as a labor of love. 

Indeed, even as the micropayment movement imagines a world where charging for resources becomes easy enough to spawn a new class of professionals, what seems to be happening is that the resources are becoming cheap enough to allow amateurs to easily subsidize their own work.

Against users’ distaste for micropayments, the tools of aggregation, subscription and subsidy will be the principle tools for bridging the gap between atomized resources and demand for simple, predictable pricing.

Playing by the Users’ Rules

Micropayment proponents have long suggested that micropayments will work because it would be great if they did. A functioning micropayment system would solve several thorny financial problems all at once. Unfortunately, the barriers to micropayments are not problems of technology and interface, but user approval. The advantage of micropayment systems to people receiving micropayments is clear, but the value to users whose money and time is involved isn’t.

Because of transactional inefficiencies, user resistance, and the increasing flexibility of the existing financial framework, micropayments will never become a general class of network application. Anyone setting out to build systems that reward resource providers will have to create payment systems that provides users the kind of financial experience they demand – simple, predictable and easily valued. Only solutions that play by these rules will succeed.

First published on O’Reilly’s OpenP2P, 12/15/2000.

After writing on the issue of freeriding, and particularly on why it isn’t the general problem for P2P that people think it is, a possibly neater explanation of the same issue occurred to me. 

I now think that most people working on the freeriding problem are assuming that P2P systems are “Pareto Optimal,” when they actually aren’t. 

Named after the work of Italian economist and sociologist Vilfredo Pareto (1848-1923), Pareto Optimal refers to a situation in which you can’t make anybody better off without making someone else worse off. An analogy is an oversold plane, where there are 110 ticket holders for 100 seats. You could make any of the 10 standby passengers better off, but only by making one of the current passengers worse off. Note that this says nothing about the overall fairness of the system, or even overall efficiency. It simply describes systems where there is equilibrium.

Since free markets are so good at producing competitive equilibrium, free markets abound with Pareto Optimal situations. Furthermore, we are used to needing market incentives to adjust things fairly in Pareto Optimal situations. If you were told you had to get off the plane to make room for a standby passanger, you would think that was unfair, but if you were offered and accepted a travel voucher as recompense, you would not.

I think that we are in fact so used to Pareto Optimal situations that we see them even where they don’t exist. Much of the writing about freeriding assumes that P2P systems are all Pareto Optimal, so that, logically, if someone takes from you but does not give back, they have gained and you have lost.

This is plainly not true. If I leave Napster on over the weekend, I typically get more downloads than I have total songs stored, at a marginal cost of zero. Those users are better off. I am not worse off. The situation is not Pareto Optimal.

Consider the parallel of email. Like mp3s, an email is a file — it takes up disk space and bandwidth. If I send you email, who is the producer and who the consumer in that situation? Should you be charged for the time I spent writing? Should I be charged for the disk space my email takes up on your hard drive? Are lurkers on this list freeriders if they don’t post? Or are the posters freeriders because we are taking up the lurkers’ resources without paying them for the bandwidth and disk our posts are using, to say nothing of their collective time?

Markets are both cause and effect of Pareto Optimal situations. One of the bets made by the “market systems” people is that a market for resources can either move a system to an equilibrium use of resources, or else that the system will move to an equilibrium use of resources of its own accord, at which point a market will be necessary to allocate resources efficiently. 

On the other hand, we are already well-accustomed to situations on the Net where the resources produced and consumed are not in fact subject to real-time market pricing (like the production and consumption of the files that make up email), and it is apparent to everyone who uses those systems that real-time pricing (aka micropayments) would not make the system more efficient overall (though it would cut down on spam, which is the non-Pareto Optimal system’s version of freeriding).

My view is that P2P file-sharing is (or can be, depending on its architecture) in the same position, where some users can be better off without making other users worse off.

First published on Biz2, 10/00.

Premature definition is a danger for any movement. Once a definitive
label is applied to a new phenomenon, it invariably begins shaping —
and possibly distorting — people’s views. So it is with the current
revolution, where Napster, SETI@Home, and their cousins now seem to be
part of a larger and more coherent change in the nature of the
internet. There have been many attempts to describe this change in a
phrase — decentralization, distributed computing — but the label
that seems to have stuck is peer-to-peer. And now that peer-to-peer is
the name of the game, the rush is on to apply this definition both as
a litmus test and as a marketing tool.

This is leading to silliness of the predictable sort — businesses
that have nothing in common with Napster, Gnutella, or Freeserve are
nevertheless re-inventing themselves as “peer-to-peer” companies,
applying the term like a fresh coat of paint over a tired business
model. Meanwhile, newly vigilant interpreters of the revolution are
now suggesting that Napster itself is not “truly peer-to-peer”,
because it relies on a centralized server to host its song list.

It seems obvious, but bears repeating: definitions are only useful as
tools for sharpening one’s perception of reality. If Napster isn’t
peer-to-peer, then “peer-to-peer” is a bad description of what’s
happening. Napster is the killer app for this revolution, and defining
it out of the club after the fact is like saying “Sure it might work
in practice, but it will never fly in theory.”

No matter what you call it, what is happening is this: PCs, and in
particular their latent computing power, are for the first time being
integrated directly into the fabric of the internet.

PCs are the dark matter of the internet. Like the barely detectable
stuff that makes up most of the mass of the universe, PCs are
connected to the internet by the hundreds of millions but have very
little discernable effect on the whole, because they are largely
unused as anything other than dumb clients (and expensive dumb
clients to boot.) From the point of view of most of the internet
industry, a PC is nothing more than a life-support system for a
browser and a place to store cookies.

PCs have been restricted to this expensive-but-dumb client mode for
many historical reasons — slow CPUs, small disks, flakey OSs, slow
and intermittant connections, no permanent IP addresses — but with
the steady growth in hardware quality, connectivity, and user base,
the PCs at the edges of the network now represent an astonishing and
untapped pool of computing power.

At a conservative estimate, the world’s net-connected PCs host an
aggregate 10 billion Mhz of processing power and 10 thousand terabytes
of storage. And this calculation assumes 100 million PCs among the
net’s 300 million users, with an average chip speed of 100 Mhz and an
average 100 Mb hard drive. And these numbers continue to climb —
today, sub-$2K PCs have an order of magnitude more processing power
and two orders of magnitude more storage than this assumed average.

This is the fuel powering the current revolution — the latent
capabilities of PC hardware made newly accessible represent a huge,
untapped resource. No matter how it gets labelled (and peer-to-peer
seems likely to stick), the thing that software like the Gnutella file
sharing system and the Popular Power distributed computing network
have in common is an ability to harness this dark matter, the otherwise
underused hardware at the edges of the net.

Note though that this isn’t just “Return of the PC”, because in these
new models, PCs aren’t just personal computers, they’re promiscious
computers, hosting data the rest of the world has access to, a la
Napster, and sometimes even hosting calculations that are of no use to
the PC’s owner at all, like Popular Powers influenza virus
simulations. Furthermore, the PCs themselves are being disaggregated
— Popular Power will take as much CPU time as it can get but needs
practically no storage, while Gnutella needs vast amounts of disk
space but almost no CPU time. And neither kind of business
particularly needs the operating system — since the important
connection is often with the network rather than the local user, Intel
and Seagate matter more to the peer-to-peer companies than do
Microsoft or Apple.

Its early days yet for this architectural shift, and the danger of the
peer-to-peer label is that it may actually obscure the real
engineering changes afoot. With improvements in hardware, connectivity
and sheer numbers still mounting rapidly, anyone who can figure how to
light up the internet’s dark matter gains access to a large and
growing pool of computing resources, even if some of the functions are
centralized (again, like Napster or Popular Power.)

Its still too soon to see who the major players will be, but don’t
place any bets on people or companies reflexively using the
peer-to-peer label. Bet instead on the people figuring out how to
leverage the underused PC hardware, because the actual engineering
challenges in taking advantage of the world’s PCs matters more — and
will create more value — than merely taking on the theoretical
challenges of peer-to-peer architecture.

Napster has always been a revolution within the commercial music business, not against it, and yesterday’s deal between BMG and Napster demonstrates that at least one of the 5 major labels understands that. The press release was short on details, but the rough outlines of the deal has Bertelsmann dropping its lawsuit and instead working with Napster to create subscription-based access to its entire music
catalog online. Despite a year of legal action by the major labels, and despite the revolutionary fervor of some of Napster’s users, Napster’s success has more to do with the economics of digital music than with copyright law, and the BMG deal is merely a recognition of those economic realities.

Until Napster, the industry had an astonishingly successful run in producing digital music while preventing digital copying from taking place on a wide scale, managing to sideline DAT, Minidisc, and recordable CDs for years. Every time any of the major labels announced an online initiative, it was always based around digital rights
management schemes like SDMI, designed to make the experience of buying and playing digital files at least as inconvenient as physical albums and tapes.

In this environment, Napster was a cold shower. Napster demonstrated how easily and cheaply music could be distributed by people who did not have a vested interest in preserving inefficiency. This in turn reduced the industry to calling music lovers ‘pirates’ (even though Napster users weren’t in it for the money, surely the definition of piracy), or trying to ‘educate’ us about about why we should be happy to pay as much for downloaded files as for a CD (because it was costing them so much to make downloaded music inconvenient.)

As long as the labels kept whining, Napster looked revolutionary, but once BMG finally faced the economic realities of online distribution and flat rate pricing, the obvious partner for the new era was Napster. That era began in earnest yesterday, and the people in for the real surprise are not the music executives, who are after all
adept at reading popular sentiment, and who stand to make more money from the recurring revenues of a subscription model. The real surprise is coming for those users who convinced themselves that Napster’s growth had anything to do with anti-authoritarian zeal.

Despite the rants of a few artists and techno-anarchists who believed that Napster users were willing to go to the ramparts for the cause, large scale civil disobedience against things like like Prohibition or the 55 mph speed limit has usually been about relaxing restrictions, not repealing them. You can still make gin for free in your bathub, but nobody does it anymore, because the legal liquor industry now sells high-quality gin at a reasonable price, with restrictions that society can live with.

Likewise, the BMG deal points to a future where you can subscribe to legal music from Napster for an attractive price, music which, as a bonus, won’t skip, end early, or be misindexed. Faced with the choice between shelling out five bucks a month for high quality legal access or mastering gnutella, many music lovers will simply plump for the subscription. This will in turn reduce the number of copyright violators, making it easier for the industry to go after them, which will drive still more people to legal subscriptions, and so on.

For a moment there, as Napster’s usage went through the roof while the music industry spread insane propaganda about the impending collapse of all professional music making, one could imagine that the collective will of 30 million people looking for free Britney Spears songs constituted some sort of grass roots uprising against The Man. As the BMG deal reverberates through the industry, though, it will become apparent that those Napster users were really just agitating for better prices. In unleashing these economic effects, Napster has almost single-handedly dragged the music industry into the internet age. Now the industry is repaying the favor by dragging Napster into the mainstream of the music business.

First published on Biz2, 10/00

The Domain Name System is coming apart at the seams. DNS, the protocol which maps IP addresses like 206.107.251.22 to domain names like FindDentist.com, is showing its age after almost 20 years. It has proved unable to adapt to dynamic internet addresses, to the number of new services being offered, and particularly to the needs of end users, who are increasingly using their PCs to serve files, host
software, and even search for extra-terrestrial intelligence. As these PCs become a vital part of the internet infrastructure, they need real addresses just as surely as yahoo.com does. This is something the DNS system can’t offer them, but the competitors to DNS can.

The original DNS system was invented, back in the early 80s, for distinctly machine-centric world. Internet-connected computers were rare, occupying a few well-understood niches in academic and government labs. This was a world of permanence: any given computer would always have one and only one IP address, and any given IP address would have one and only one domain name. Neat and tidy and static.

Then along came 1994, the Year of the Web, when the demand for connecting PCs directly to the internet grew so quickly that the IP namespace — the total number of addresses — was too small to meet the demand. In response, the ISPs began doling out temporary IP addresses on an as-needed basis, which kept PCs out of the domain name system: no permanent IP, no domain name. This wasn’t a problem in the mid-90s — PCs were so bad, and modem connections so intermittent, that no one really thought of giving PCs their own domain names.

Over the last 5 years, though, cheap PC hardware has gotten quite good, operating systems have gotten distinctively less flaky, and connectivity via LAN, DSL and cable have given us acceptable connections. Against the background of these remarkable improvements, the DNS system got no better at all — anyone with a PC was still a
second-class citizen with no address, and it was Napster, ICQ, and their cousins, not the managers of the DNS system, who stepped into this breech.

These companies, realizing that interesting services could be run off of PCs if only they had real addresses, simply ignored DNS and replaced the machine-centric model with a protocol-centric one. Protocol-centric addressing creates a parallel namespace for each piece of software, and the mapping of ICQ or Napster usernames to temporary IP addresses is not handled by the net’s DNS servers but by
privately owned servers dedicated to each protocol — the ICQ server matches ICQ names to the users’ current IP address, and so on. As a side-effect of handling dynamic IP addresses, these protocols are also able to handle internet address changes in real time, while current DNS system can take several days to fully log a change.

In Napster’s case, protocol-centric addressing merely turns Napster into customized ftp for music files. The real action is in software like ICQ, which not only uses protocol-centric addressing schemes, but where the address points to a person, not a machine. When I log into ICQ, I’m me, no matter what machine I’m at, and no matter what IP address is presently assigned to that machine. This completely decouples what humans care about — can I find my friends and talk with them online — with how the machines go about it — route message A to IP address X.

This is analgous to the change in telephony brought about by mobile phones. In the same way a phone number is no longer tied to a particular location but is now mapped to the physical location of the phone’s owner, an ICQ address is mapped to me, not to a machine, no matter where I am.

This does not mean that the DNS system is going away, any more than landlines went away with the invention of mobile telephony. It does mean that DNS is no longer the only game in town. The rush is now on, with instant messaging protocols, single sign-on and wallet applications, and the explosion in peer-to-peer businesses, to create
and manage protocol-centric addresses, because these are essentially privately owned, centrally managed, instantly updated alternatives to DNS.

This also does not mean that this change is entirely to the good. While it is always refreshing to see people innovate their way around a bottleneck, sometimes bottlenecks are valuable. While ICQ and Napster came to their addressing schemes honestly, any number of people have noticed how valuable it is to own a namespace, and many business plans making the rounds are just me-too copies of Napster or
ICQ, which will make an already growing list of kinds of addresses — phone, fax, email, url, ICQ, … — explode into meaninglessness.

Protocol-centric namespaces will also force the browser into lesser
importance, as users return to the days they namaged multiple pieces
of internet software, or it will mean that addresses like
icq://12345678 or napster://green_day_fan will have to be added to the
browsers repetoire of recognized URLs. Expect the rise of
‘meta-address’ servers as well, which offer to manage a user’s
addresses for all of these competing protocols, and even to translate
from one kind of address to another. (These meta-address servers will,
of course, need their own addressses as well.)

Its not clear what is going to happen to internet addressing, but it is clear that its going to get a lot more complicated before it gets simpler. Fortunately, both the underlying IP addressing system and the design of URLs can handle this explosion of new protocols and addresses, but that familiar DNS bit in the middle (which really put the dot in dot com) will never recover the central position it has occupied in the last 2 decades, and that means that a critical piece of internet infrastructure is now up for grabs.

---

Thanks to Dan Gilmor of the San Jose Mercury News for pointing out to me the important relationship between peer-to-peer networking and DNS.