Category / Essays

by Clay Shirky

Pretend vs. Real Economy

The difference between the stock market and the economy.

First published in FEED, 06/99.

The Internet happened to Merrill Lynch last week, and it cost them a couple billion dollars — when Merrill announced its plans to open an online brokerage after years of deriding the idea, its stock price promptly fell by a tenth, wiping out $2 billion in its market capitalization. The internet’s been happening like that to a lot of companies lately — Barnes and Noble’s internet stock is well below its recent launch price, Barry Diller’s company had to drop its Lycos acquisition because of damage to the stock prices of both companies, and both Borders and Compaq dumped their CEOs after it became clear that they were losing internet market share. In all of these cases, those involved learned the hard way that the internet is a destroyer of net value for traditional businesses because the internet economy is fundamentally at odds with the market for internet stocks.

The internet that the stock market has been so in love with (call it the “Pretend Internet” for short) is all upside — it enables companies to cut costs and compete without respect to geography. The internet that affects the way existing goods and services are sold, on the other hand (call it the “Real Internet”), forces companies to cut profit margins, and exposes them to competitors without respect to geography. On the Pretend Internet, new products will pave the way for enormous profitability arising from unspecified revenue streams. Meanwhile, on the Real Internet, prices have fallen and they can’t get up. There is a rift here, and its fault line appears wherever offline companies like Merrill tie their stock to their internet offerings. Merrill currently pockets a hundred bucks every time it executes a trade, and when investors see that Merrill online is only charging $30 a trade, they see a serious loss of revenue. When they go on to notice that $30 is something like three times the going rate for an internet stock trade, they see more than loss of revenue, they see loss of value. When a company can cut its prices 70% and still be three times as expensive as its competitors, something has to give. Usually that is the company’s stock price.

The internet is the locus of the future economy, and its effect is the wholesale transfer of information and choice (read: power and leverage) from producer to consumer. Producers (and the stock market) prefer one-of-a-kind businesses who can force their customers to accept continual price increases for the same products. Consumers, on the other hand, prefer commodity businesses where prices start low and keep falling. On the internet, consumers have the upper hand, and as a result, anybody who profited from offline inefficiencies — it used to be hard work to distribute new information to thousands of people every day, for example — are going to see much of their revenue destroyed with no immediate replacement in sight.

This is not to say that the internet produces no new value — on the contrary, it produces enormous value every day. Its just that most of the value is concentrated in the hands of the consumer. Every time someone uses the net to shop on price (cars, plane tickets, computers, stock trades) the money they didn’t spend is now available for other things. The economy grows even as profit margins shrink. In the end, this is what Merrill’s missing market cap tells us — the internet is now a necessity, but there’s no way to use the internet without embracing consumer power, and any business which profits from inefficiency is going to find this embrace more constricting than comforting. The effects of easy price comparison and global reach are going to wring inefficiency (read: profits) out of the economy like a damp dishrag, and as the market comes to terms with this equation between consumer power and lower profit margins, $2 billion of missing value is going to seem like a drop in the bucket.

by Clay Shirky

Who Are You Paying When You’re Paying Attention?

Converting attention to cash.

First published in ACM, 06/99.

Two columns ago, in “Help, the Price of Information Has Fallen and It Can’t Get Up”, I argued that traditional pricing models for informational goods (goods that can theoretically be transmitted as pure data – plane tickets, stock quotes, classified ads) fall apart on the net because so much of what’s actually being paid for when this data is distributed is not the content itself, but its packaging, storage and transportation. This content is distributed either as physical packages, like books or newspapers, or on closed (pronounced ‘expensive’) networks, like Lexis/Nexis or stock tickers, and its cost reflects both these production and distribution expenses and the scaricty that is created when only a few companies can afford to produce and distribute said content. 

The net destroys both those effects, first by removing the need for either printing and distributing physical objects (online newspapers, e-tickets and electronic greeting ‘cards’ are all effortless to distribute relative to their physical counterparts) and by removing many of the barriers to distribution (only a company with access to a printing press can sell classified ads offline, but on the network all it takes is a well-trafficked site), so that many more companies can compete with one another. 

The net effect of all this, pun intended, is to remove the ability to charge direct user fees for many kinds of online content which people are willing to shell out for offline. This does not mean that the net is valueless, however, or that users can’t be asked to pay for content delivered over the Internet. In fact, most users willingly pay for content now. The only hitch is that what they’re paying isn’t money. They’re paying attention. 

THE CURRENCY EXCHANGE MODEL

Much of the current debate surrounding charging user fees on the Internet assumes that content made available over the network follows (or should follow) the model used in the print world – ask users to pay directly for some physical object which contains the content. In some cases, the whole cost of the object is borne by the users, as with books, and in other cases users are simply subsidizing the part of the cost not paid for by advertisements, as with newspapers and magazines. There is, however, another model, one more in line with the things the net does well, where the user pays no direct fees but the providers of the content still get paid – the television model. 

TV networks are like those currency exchange booths for tourists. People pay attention to the TV, and the networks collect this attention and convert it into money at agreed upon rates by supplying it in bulk to their advertisers, generally by calculating the cost to the advertiser of reaching a thousand viewers. The user exchanges their attention in return for the content, and the TV networks exchange this attention for income. These exchange rates rise and fall just like currency markets, based on the perceived value of audience attention and the amount of available cash from the advertiser. 

This model, which generates income by making content widely available over open networks without charging user fees, is usually called ‘ad-supported content’, and it is currently very much in disfavor on the Internet. I believe however, that not only can ad-supported content work on the Internet, I believe it can’t not work. It’s success is guaranteed by the net’s very makeup – the net is simply too good at gathering communities of interest, too good at freely distributing content, and too lousy at keeping anything locked inside subscription networks, for it to fail. Like TV, the net is better at getting people to pay attention than anything else. 

OK SHERLOCK, SO IF THE IDEA OF MAKING MONEY ON THE INTERNET BY CONVERTING ATTENTION INTO INCOME IS SO BRILLIANT, HOW COME EVERYONE ELSE THINKS YOU’RE WRONG?

Its a question of scale and time horizons. 

One of the reasons for the skepticism about applying the TV model to the Internet is the enormous gulf between the two media. This is reflected in both the relative sizes of their audiences and the incomes of those businesses – TV is the quintessential mass medium, commanding tens of millions more viewers than the net does. TV dwarfs the net in both popularity and income. 

Skeptics eyeing the new media landscape often ask “The Internet is fine as a toy, but when will it be like TV?” By this they generally mean ‘When will the net have a TV-style audience with TV-style profits?’ The question “When will the net be like TV” is easy to answer – ‘Never’. The more interesting question is when will TV be like the net, and the answer is “Sooner than you think”. 

A BRIEF DIGRESSION INTO THE BAD OLD DAYS OF NETWORK TV

Many people have written about the differences between the net and television, usually focussing on the difference between broadcast models and packet switched models like multicasting and narrowcasting, but these analyses, while important, overlook one of the principal differences between the two media. The thing that turned TV into the behemoth we know today isn’t broadcast technology but scarcity. 

From the mid-1950s to the mid-1980s, the US national TV networks operated at an artificially high profit. Because the airwaves were deemed a public good, their use was heavily regulated by the FCC, and as a consequence only three companies got to play on a national level. With this FCC-managed scarcity in place, the law of supply and demand worked in the TV networks favor in ways that most industries can only dream of – they had their own private government created and regulated cartel. 

It is difficult to overstate the effect this had on the medium. With just three players, a TV show of merely average popularity would get a third of the available audience, so all the networks were locked in a decades long three-way race for the attention of the ‘average’ viewer. Any business which can get the attention of a third of its 100 million+ strong audience by producing a run-of-the-mill product, while being freed for any other sort of competition by the government, has a license to print money and a barrel of free ink. 

SO WHAT HAPPENED?

Cable happened – the TV universe has been slowly fracturing for the last 20 years or so, with the last 5 years seeing especially sharp movement. With a growing competition from cable (and later sattelite, microwave, a 4th US TV network, and most recently the Internet draining TV watching time), the ability of television to command a vast audience with average work has suffered badly. The two most popular US shows of this year each struggle to get the attention of a fifth of the possible viewers, called a “20 share’ in TV parlance, where 20 is the percentage of the possible audience tuning in. 

The TV networks used to cancel shows with a 20 share, and now that’s the best they can hope for from their most popular shows, and its only going to get worse. As you might imagine, this has played hell with the attention-to-cash conversion machine. When the goal was a creating multiplicity of shows for the ‘average’ viewer, pure volume was good, but in the days of the Wind-Chill Channel and the Abe Vigoda Channel, the networks have had to turn to audience segmentation, to not just counting numbers but counting numbers of women, or teenagers, or Californians, or gardeners, who are watching certain programs. 

The TV world has gone from three channels of pure mass entertainment to tens or even hundreds of interest-specific channels, with attention being converted to cash based not solely on the total number people they attract, but also on how many people with specific interests, or needs, or characteristics. 

Starting to sound a bit like the Web, isn’t it? 

THE TV PEOPLE ARE GOING TO RUE THE DAY THEY EVER HEARD THE WORD ‘DIGITAL’

All this is bad enough from the TV networks point of view, but its a mere scherzo compared to the coming effects of digitality. As I said in an earlier column, apropos CD-ROMs, “Information has been decoupled from objects. Forever.”, and this is starting to be true of information and any form of delivery. A TV is like a book in that it is both the mechanism of distribution and of display – the receiver, decoder and screen travel together. Once television becomes digital, this is over, as any digital content can be delivered over any digital medium. “I just saw an amazing thing on 60 Minutes – here, I’ll mail it to you”, “Whats the URL for ER again?”, “Once everybody’s here, we’ll start streaming Titanic”. Digital Baywatch plus frame relay is the end of ‘appointment TV’. 

Now I am not saying that the net will surpass TV in size of audience or income anytime soon, or that the net’s structure, as is, is suitable for TV content, as is. I am saying that the net’s method of turning attention into income, by letting audience members select what they’re intersted in and when, where and how to view it, is superior to TV’s, and I am saying that as the net’s bandwidth and quality of service increases and television digitizes, many of the advantages TV had move over to the network. 

The Internet is a massive medium, but it is not a mass medium, and this gives it an edge as the scarcity that TV has lived on begins to seriously erode. The fluidity with which the net apportions content to those interested in it without wasting the time of those not interested in it makes it much more suited in the long run for competing for attention in the increasingly fractured environment for television programming, or for any content delivery for that matter. 

In fact, most of the experiments with TV in the last decade – high-definition digital content, interactive shopping and gaming, community organization, and the evergreen ‘video on demand’ – are all things that can be better accomplished by a high bandwidth packet switched network than by traditional TV broadcast signals. 

In the same way that ATT held onto over 2/3rds of its long-distance market for a decade after the breakup only to see it quickly fall below 50% in the last three years, the big 3 TV networks have been coasting on that same kind of intertia. Only recently, prodded by cable and the net, is the sense memory of scarcity starting to fade, and in its place is arising a welter of competition for attention, one that the Internet is poised to profit from enormously. A publicly accessible two-way network that can accomodate both push and pull and can transmit digital content with little regard to protocol has a lot of advantages over TV as an ‘attention to income’ converter, and in the next few years those advantages will make themselves felt. I’m not going to bet on when overall net income surpasses overall TV income, but in an arena where paying attention is the coin of the realm, the net has a natural edge, and I feel confident in predicting that revenues from content will continue to double annually on the net for the forseeable future, while network TV will begin to stagnate, caught flat in a future that looks more like the Internet than it does like network TV.

by Clay Shirky

An Open Letter to Jakob Nielsen

An argument against design standards on the web.

First published on ACM, 06/99.

[For those not subscribing to CACM, Jakob Nielsen and I have come down on opposite sides of a usability debate. Jakob believes that the prevalence of bad design on the Web is an indication that the current method of designing Web sites is not working and should be replaced or augmented with a single set of design conventions. I believe that the Web is an adaptive system and that the prevalence of bad design is how evolving systems work.

Jakob’s ideas are laid out in “User Interface Directions for the Web”, CACM, January 1999.

My ideas are laid out in “View Source… Lessons from the Web’s Massively Parallel Development”, networker, December 1998, and http://www.shirky.com/writings/view-source

Further dialogue between the two of us is in the Letters section of the March 1999 CACM.]

Jakob,

I read your response to my CACM letter with great interest, and while I still disagree, I think I better understand the disagreement, an will try to set out my side of the argument in this letter. Let me preface all of this by noting what we agree on: the Web is host to some hideous dreck; things would be better for users if Web designers made usability more of a priority; and there are some basics of interface usability that one violates at one’s peril.

Where we disagree, however, is on both attitude and method – for you, every Web site is a piece of software first and foremost, and therefore in need of a uniform set of UI conventions, while for me, a Web site’s function is something only determined by its designers and users – function is as function does. I think it presumptous to force a third party into that equation, no matter how much more “efficient” that would make things.

You despair of any systemic fix for poor design and so want some sort of enforcement mechanism for these external standards. I believe that the Web is an adaptive system, and that what you deride as “Digital Darwinism” is what I would call a “Market for Quality”. Most importantly, I believe that a market for quality is in fact the correct solution for creating steady improvements in the Web’s usability.

Let me quickly address the least interesting objection to your idea: it is unworkable. Your plan requires both centralization and force of a sort it is impossible to acheive on the Internet. You say

“…to ensure interaction consistency across all sites it will be necessary to promote a single set of design conventions.”

and

“…the main problem lies in getting Web sites to actually obey any usability rules.”

but you never address who you are proposing to put in the driver’s seat – “it will be necessary” for whom? “[T]he main problem” is a problem for whom? Not for me – I am relieved that there is no authority who can make web site designers “obey” anything other httpd header validity. That strikes me as the Web’s saving grace. With the Web poised to go from 4 million sites to 100 million in the next few years, as you note in your article, the idea of enforcing usability rules will never get past the “thought experiment” stage.

However, as you are not merely a man of action but also a theorist, I want to address why I think enforced conformity to usability standards is wrong, even in theory. My objections break out into three rough categories: creating a market for usability is better than central standards for reasons of efficency, innovation, and morality.

EFFICENCY

In your letter, you say “Why go all the way to shipping products only to have to throw away 99% of the work?” I assume that you meant this as a rhetorical question – after all, how could anybody be stupid enough to suggest that a 1% solution is good enough? The Nielsen Solution – redesign for everybody not presently complying with a “single set of design conventions” – takes care of 100% of the problem, while the Shirky Solution, let’s call it evolutionary progress for the top 1% of sites, well what could that possibly get you?

1% gets you a surprising amount, actually, if it’s properly arranged.

If only the top 1% most trafficked Web sites make usability a priority, those sites would nevertheless account for 70% of all Web traffic. You will recognize this as your own conclusion, of course, as you have suggested on UseIt (http://www.useit.com/alertbox/9704b.html) that Web site traffic is roughly a Zipf distribution, where the thousandth most popular site only sees 1/1000th the traffic of the most popular site. This in turn means that a very small percentage of the Web gets the lion’s share of the traffic. If you are right, then you do not need good design on 70% of the web sites to cover 70% of user traffic, you only need good design on the top 1% of web sites to reach 70% of the traffic.

By ignoring the mass of low traffic sites and instead concentrating on making the popular sites more usable and the usable sites more popular, a market for quality is a more efficient way of improving the Web than trying to raise quality across the board without regard to user interest.

INNOVATION

A market for usability is also better for fostering innovation. As I argue in “View Source…”, good tools let designers do stupid things. This saves overhead on the design of the tools, since they only need to concern themselves with structural validity, and can avoid building in complex heuristics of quality or style. In HTML’s case, if it renders, it’s right, even if it’s blinking yellow text on a leopard-skin background. (This is roughly the equivalent of letting the reference C compiler settle arguments about syntax – if it compiles, it’s correct.)

Consider the use of HTML headers and tables as layout tools. When these practices appeared, in 1994 and 1995 respectively, they infuriated partisans of the SGML ‘descriptive language’ camp who insisted that HTML documents should contain only semantic descriptions and remain absolutely mute about layout. This in turn led to white-hot flamefests about how HTML ‘should’ and ‘shouldn’t’ be used.

It seems obvious from the hindsight of 1999, but it is worth repeating: Everyone who argued that HTML shouldn’t be used as a layout language was wrong. The narrowly correct answer, that SGML was designed as a semantic language, lost out to the need of designers to work visually, and they were able to override partisan notions of correctness to get there. The wrong answer from a standards point of view was nevertheless the right thing to do.

Enforcing any set of rules limits the universe of possibilities, no matter how well intentioned or universal those rules seem. Rules which raise the average quality by limiting the worst excesses risk ruling out the most innovative experiments as well by insisting on a set of givens. Letting the market separate good from bad leaves the door open to these innovations.

MORALITY

This is the most serious objection to your suggestion that standards of usability should be enforced. A web site is an implicit contract between two and only two parties – designer and user. No one – not you, not Don Norman, not anyone, has any right to enter into that contract without being invited in, no matter how valuable you think your contribution might be.

IN PRAISE OF EVOLVEABLE SYSTEMS, REDUX

I believe that the Web is already a market for quality – switching costs are low, word of mouth effects are both large and swift, and redesign is relatively painless compared to most software interfaces. If I design a usable site, I will get more repeat business than if I don’t. If my competitor launches a more usable site, it’s only a click
away. No one who has seen the development of Barnes and Noble and Amazon or Travelocity and Expedia can doubt that competition helps keep sites focussed on improving usability. Nevertheless, as I am a man of action and not just a theorist, I am going to suggest a practical way to improve the workings of this market for usability –
lets call it usable.lycos.com.

The way to allocate resources efficently in a market with many sellers (sites) and many buyers (users) is competition, not standards. Other things being equal, users will prefer a more usable site over its less usable competition. Meanwhile, site owners prefer more traffic to less, and more repeat visits to fewer. Imagine a search engine that weighted usability in its rankings, where users knew that a good way to find a usable site was by checking the “Weight Results by Usability” box and owners knew that a site could rise in the list by offering a good user experience. In this environment, the premium for good UI would align the interests of buyers and sellers around increasing quality. There is no Commisar of Web Design here, no International Bureau of Markup Standards, just an implicit and ongoing compact between users and designers that improvement will be rewarded.

The same effect could be created in other ways – a Nielsen/Norman “Seal of Approval”, a “Usability” category at the various Web awards ceremonies, a “Usable Web Ring”. As anyone who has seen “Hamster Dance” or an emailed list of office jokes can tell you, the net is the most efficent medium the world has ever known for turning user preference into widespread awareness. Improving the market for quality simply harnesses that effect.

Web environments like usable.lycos.com, with all parties maximizing preferences, will be more efficent and less innovation-dampening than the centralized control which would be necessary to enforce a single set of standards. Furthermore, the virtues of such a decentralized system mirrors the virtues of the Internet itself rather than fighting them. I once did a usability analysis on a commerce site which had fairly ugly graphics but a good UI nevertheless. When I queried the site’s owner about his design process, he said “I didn’t know anything when I started out, so I just put up a site with an email link on every page, and my customers would mail me suggestions.”

The Web is a marvelous thing, as is. There is a dream dreamed by engineers and designers everywhere that they will someday be put in charge, and that their rigorous vision for the world will finally overcome the mediocrity around them once and for all. Resist this idea – the world does not work that way, and the dream of centralized control is only pleasant for the dreamer. The Internet’s ability to be adapted slowly, imperfectly, and in many conflicting directions all at once is precisely what makes it so powerful, and the Web has taken those advantages and opened them up to people who don’t know source code from bar code by creating a simple interface design language.

The obvious short term effect of this has been the creation of an ocean of bad design, but the long term effects will be different – over time bad sites die and good sites get better, so while those short-term advantages seem tempting, we would do well to remember that there is rarely any profit in betting against the power of the marketplace in the long haul.

by Clay Shirky

Free PC Business Models

The Economics of the “Free PC” movement.

3/5/1999

“Get a free PC if you fit our demographic profile!” “Get a free PC if you subscribe
to our ISP for 3 years!” “Get a free PC if you spend $100 a month in our online store!”

Suddenly free PCs are everywhere — three offers for free PCs in the last month alone, and more on the way. Is this a gimmick or a con game? Has the hardware industry finally decided to emulate online businesses by having negative revenues? More importantly, is this the step that will lead to universal network access? When PCs and network access are free, what’s to stop everyone from participating in the digital revolution? Could it be that the long-lamented gap between the info-haves and the info-have-nots is soon to be eliminated by these ambitious free PC schemes?

The three offers rolled out last month — from the start-up free-pc.com, the ISP
empire.net, and the online mall shopss.com respectively — are actually the opening salvo of the biggest change in the computer industry since the introduction of the PC in the late 70’s. The free PC marks the end of the transition from PCs as standalone boxes to PCs as network-connected computers, and from PC as product to PC as service. But however revolutionary the underlying business model may be, it is unlikely to change the internet from a limited to a mass medium, at least for the coming years. To understand why, though, you need to understand the underlying economics of the free PC.

We’ve gotten used to cheaper and cheaper computers over the years, but this is different. Free isn’t even the same kind of thing as cheap – a $399 computer is cheaper than a $999 computer, but a free computer isn’t just cheaper still, it’s a different kind of computer altogether. $999 computers and $399 computers are both products; a free computer is a service.

To see why this transition from computer as product to computer as service is a necessary condition for a free PC, consider the forces making the free PC possible in the first place. The first is cost, of course: PC prices have been falling for years, and the recent fall over the last three years from $999 to $399 has been especially precipitous.

A $399 PC is still not free however, and no matter how far prices fall, they will never
fall to zero, so the second part of the equation is cost savings. A $399 PC is cheap
enough that companies can give them away and write off the cost, if they can make back the money elsewhere. A free PC is really a loan on expected future revenues, revenues derived from customer attention of one sort or another.

ABC’s TV slogan, “Hello? It’s free” isn’t quite right. You may not be paying ABC money
when you watch “Nightline,” but you’re still paying attention, and they sell that
attention to their advertisers. TV, radio and a variety of other media have figured out
a way to distribute free programming and profit from consumers’ attention. The low price of computers and their growing use as media outlets and not just as fancy typewriters now brings them into this realm as well. Companies can absorb the cost of giving away hardware if they can profit from getting you to pay attention.

To see how a company might do this, consider these three bits of marketing jargon –
“Customer Acquisition Cost,” “Customer Retention,” and “Lifetime Value of a Customer.” Each of these is a possible source for making back that $399, and in fact, each of the current free PC uses one or more of these sources to underwrite their offering.

Customer Acquisition Cost is simply the marketing budget divided by the number of new customers. If I spend a million dollars in advertising and get 2000 new customers, then my Customer Acquisition Cost is $500 per customer. If, by giving away a $399 PC I can cut my other customer acquisition costs to $100, I come out a dollar ahead per customer, even though I’ve given away hardware to get there. This is the model free-pc.com is following, where they are acquiring an audience cheaply as a first step towards selling that audience’s attention to their advertisers.

Customer Retention is a way of expressing the difficulty of keeping a customer once you get them. Telephone companies, for example, are especially vulnerable to this — customers switch long distance services all the time. If I can give away a $399 PC and keep a customer longer, I make the money back by increasing customer retention. (This is the rationale behind the “free cell phone” as well.) The New England ISP empire.net is following this model — in return for your computer, they have you as a customer for 3 years — guaranteed retention.

Lifetime Value of a Customer is a way of thinking about repeat business — if the average customer at a grocery store spends a hundred dollars a month and lives in the neighborhood for 5 years, then the lifetime value of the average customer is $6000. If a company can give away a $399 PC in order to raise the recipient’s lifetime value, they can absorb the initial cost and make it back over time. This is the model shopss.com is following, where you get a free iMac in return for agreeing to spend $100 a month at their online store for three years.

It’s worth pointing out that all three models depend on one critical element, without
which none of them would work: the Internet. Every one of these plans starts with the assumption that an Internet connection for these PCs is a basic feature, not merely an option. This is is the final step, the thing that turns the computer from product to service. Without the Internet, there is no way to stay connected to your users. Every business model for the free PC treats it as an up-front loan that the user will pay back over time. This requires that a computer be a connected media outlet, not an isolated box, and the Internet is the thing that connects computers together.

A few years ago, engineers at Sun and Oracle began talking about creating an NC — a
“network computer” — but while they’ve been talking, the Internet has been quietly
turning all PCs into NCs, without needing any change to the hardware at all. Everyone giving away free PCs (and there will be more, many more, following this model over the next few months) has recognized that the real PC revolution is not in hardware but use. PCs’ main function has slowly but decisively passed from standalone computing to network connectivity over the 20 years of its life. We’ve already seen that transformation alter the business models of the software world — the free PC movement gives us our first glimpse of how it might transform hardware as well.

If personal computers are now primarily devices of connection rather than computation, this raises the larger question of access. There is a technotopian dream of universal access floating around, where the Internet connects every man, woman and child together in what John Perry Barlow memorably called “the re-wiring of human conciousness.” For years now, old-style progressives have been arguing that we need public-sector investment to make the digital revolution more democratic, while free-market devotees have contended that Moore’s law alone would take care of widening the wired world. Is the free PC the development that will finally lead us into this promised land of universal connectivity?

No.

There’s no such thing as a free lunch. The free PC is not a free product, but a short-term loan in return for your use of a service. The free PC, in all its incarnations, is really a “marketing supported PC” (a more accurate albeit less catchy description). Its cost is supported by your value as a consumer of marketing, or of products bought online. The inverse is also true: people who do not have enough value as consumers won’t get free PCs. (Imagine answering one of those ads for a free cell phone, and telling the phone company that you wanted them to send you the phone itself, but you didn’t want to subscribe to their service.)

For the whole short history of the commercial Internet (1991-present), it has been an
inverted medium, with more representation among higher income brackets. This means that for a marketing supported medium, the most valuable clients for a free PC are, paradoxically, people who already own a PC. Indeed, free-pc.com makes this assumption when they ask the prospective recipient of a free PC how many computers they already own. Access to the Internet is still growing rapidly, but it is growing in stages — first it was populated by computer scientists, then academics, then “early adopters,” and so on. The free PC doesn’t change the Internet’s tendency to expand in phases, it’s just the next phase. The free PC will extend the reach of the Internet, but it will start by reaching the people who are the closest to getting there already, and only expand beyond that audience over time.

The free PC isn’t charity, but it also isn’t a temporary marketing gimmick. It is simply
the meeting of two long-term trends — falling hardware costs and the rising value of
connecting a customer to the Internet. In retrospect, it was inevitable that at some
point those two figures would cross — that the Internet would become valuable enough, and PCs cheap enough, that the the PC could become like a cell phone, a dangle to get people to subscribe to a service. The number of people who can afford to connect to the Internet will continue to rise, not instantly but inexorably, as the value of the Internet itself rises, and the cost of hardware continues to fall. 1999 will be remembered as the year in which this shift from computer as product to computer as service began.

by Clay Shirky

The Interest Horizons and the Limits of Software Love

The economic limits of Open Source.

Commercial software has its limits, and it runs into them regularly. Any commercial developer has a “resource horizon” – some upper boundary of money or programmer time which limits how much can be done on any given project. Projects which lie over this horizon either can’t be accomplished (very few companies could write a new operating system from scratch), or, once started, can’t be completed because of their size or scope (as with the Denver Airport baggage handling system).

Open Source software has no such resource horizon – programmers creating Open Source software do it because they’re interested in it, not because they’re paid to do it. Furthermore, since there’s no requirement to keep the source code secret, a much larger of number of programmers can work on any given project, and they don’t have to work for the same company, or even live in the same country.

However, the lack of this resource horizon does not mean that Open Source software has no limits, it just means that the limits aren’t financial. Commercial companies make software for money, so money is the limiting factor. Open Source developers make software for the love of the thing, so love becomes the limiting factor as well. Unloved software can’t be built using Open Source methods.

Open Source software trades the resource horizon for an interest horizon, the horizon over which a problem simply isn’t interesting enough to attract a group of developers who burn to spend their Friday nights wrestling with, say, meter-reading software for a natural gas pipeline. Consider, from the seminal description of the Open Source process contained in “The Cathedral and the Bazaar” , how many of the principles assume that programmers find problems, rather than vice-versa:

Every good work of software starts by scratching a developer’s personal itch.
…
If you have the right attitude, interesting problems will find you.
…
Often, the most striking and innovative solutions come from realizing that your concept of the problem was wrong.
…
To solve an interesting problem, start by finding a problem that is interesting to you.
…
Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

This is geek nirvana, where interest and innovation have more weight than profit or productivity. Because the articulation of these principles is so new (though the way of working is at least as old as the Internet itself), and because the acheivements of Linux, Apache et al. are so unprecedented, Open Source currently has an aura of unstoppability, a sense that “Maybe Open Source will take over everything!”

It won’t. Wherever Open Source is a good way of working, it is more effective than closed source, for the same reason that open markets are more effective than central planning. The very principles of personal interest outlined above, however, indicate where Open Source isn’t a good way of working in the first place.

Consider the converse of those principles: what if you have a problem that doesn’t scratch some core developer’s personal itch?

“I stayed up all weekend rewriting this awesome time-sheet processing system!” or where innovative solutions aren’t an option in an existing system? “We could make the package much simpler if we just bill them every year instead of every month.”

or where you can’t have a large enough group of beta testers?

“Know anybody who owns a plane? I need to beta-test my air traffic control program.”

These kinds of problems lie beyond the interest horizon, and their numbers are legion. 

Existing software which is old or broken but has to be updated or fixed won’t clear the “interesting problem” hurdle. Boring software won’t acquire enough beta testers. Projects which require treating the user like an idiot (an oxymoron in the Open Source world, since scratching a personal itch turns the lead developer into User #1) won’t attract enough developers to take off. The overlapping categories of old, broken, boring and disdainful of the user’s intelligence currently describe a majority of the world’s software, an installed base that isn’t going away any time soon.

Given this interest horizon, what Open Source can do is create a market for software as service rather than as product. One way to get people to do things they wouldn’t do for love is to pay them. Who wants to sit at a desk all day, shuffle paper and answer phones as a hobby? Throw in $25 Gs and health insurance to sweeten the deal, though, and the answer is “A lot of people”. The same will be true of the interest horizon. There will never be an Open Source billing system for a water utility, but there will be Open Source accounting and report-generating packages which provide about 80% of the functionality, meaning that the water works only needs to do pay for the last 20% of the work, not all of it.

30 years ago, every IT department in this country was in the business of building custom products, and the software industry grew up around that assumption. Everyone solved their own problems in-house, leading to vast duplication of effort as thousands of nearly identical record keeping systems, billing systems and databases were created from scratch, over and over again.

15 years ago the rise of high-quality shrink-wrapped software began to change that, where people could buy products which had some core set of functionality which they could extend. Problems were no longer solved from scratch, but a huge pile of money still went to pay for the identical core functionality within each business.

Now, Open Source suggests an almost pure service model, where the basic functionality costs nothing, and all the money is in customization. Imagine an economy where the basic tools and serivces are free: in this world, once a company it knew what it wanted to build, it could put the basic software functionality in place for free and then either hire or outsource the talent to do whatever customizations were necessary.

This economy exists today, on the Web. Even a simple web site is a complicated piece of software, yet millions of individuals and small companies run Web sites, because the basic functionality of the Web server doesn’t need to be recreated from scratch every time. More surprising still, the Web does this so well that we hardly give it a moment’s thought.

Yet consider how far Open Source has gone to make this possible. In the 80’s, CompuServe, Prodigy and AOL all had the same rough set of goals and features, yet each of them had to build, from scratch, a data network, a set of interactive applications, and a software interface. It would have benefitted each of them enormously if they could have inherited basic functions, and then concentrated on their particular strengths, but there was no way in those days to do anything of the sort.

Now consider Yahoo, Geocities and Hotmail. All three use the same Open Source operating system (FreeBSD) and the same Open Source Web server (Apache), which means that their programmers and designers have been free to focus on making Yahoo be Yahoo, Geocities be Geocities, and Hotmail Hotmail – customization from the jump. From the same core of free functionality – net-connected operating system, web server, html interface, interactive scripts – we get three different sites, with three different interfaces, functions, business models, and audiences, yet each company spent much less money and energy, for much more gain, than the online services of old.

This is the future of Open Source. As the edges of the interest horizon become clear, the market for products will shrink and the market for customization will grow: Expect the Computer Associates of the world to begin training their employees in the use and customization of Open Source products. With Open Source software putting core set of functions in place, software companies will become more like consulting firms and consulting firms will become more like software companies, not duplicating the same basic functions over and over, but concentrating their efforts on the the work that lies over the interest horizon.

by Clay Shirky

Social Computing in Student Populations

What college students can tell us about Internet use.

Shortly after I sold my company (I was the Chief Technology Officer of a Web design firm in Manhattan’s Silicon Alley), Hunter College hired me to teach classes in Web design and uses of the Internet. One of the first things I did while setting up my lab was to drag an old computer into the hall outside my office, connect it to the Internet, and give students unfettered access to it. My thought was to bring the students into the present — little did I know that they would show me the future. 

For a few years now, there has been a deathwatch for the personal computer. It is an article of faith in much of my industry that every 15 years or so sees a major shift in computing hardware, and since the PC supplanted the mainframe in the early 80s, it’s time, or so the thinking goes, for the PC in turn to give way to the “Next Big Thing”, as these shifts are sometimes jokingly called. 

That shift is happening today — I can see it in the way my students are using the computer outside my office door — but unlike the previous revolution, this change is not about hardware but about patterns of use. The age of the personal computer is ending because computers are no longer personal, they’re social. My students are using the Web not just as a research tool but as a way to communicate with one another and to organize their lives. Web-based email, from companies like Hotmail and Yahoo, forms the center of their computer use; they use it for everything from communicating with professors to keeping in touch with family overseas to organizing forays to the pizza parlor. 

Furthermore, they’re not just using computers to run their social lives; they’ve begun to treat computers themselves as social objects, like phones. If you need to check your voice mail, it doesn’t matter to you what phone you use; any phone with an outside line is the same as any other. Even if its someone else’s phone, and they’ve programmed the speed dialer to suit themselves, you can still use it to get to your messages. In the same way, any computer with an “outside line” to the Internet is as good as any other for checking email, even if someone else owns that computer and they’ve got their own programs on the hard drive. From my students’ point of view, a computer is merely a way to get access to their stuff (email, web syllabi, online research, et al.) rather than being a place to store their stuff. Hotmail wins over Eudora because with web-based email, they can get their mail from any computer they happen to be at, whether they own it or not. 

This seemingly minor change has major ramifications for academic uses of computers. If a computer is for access to a network and not for storage or local computation, then the definition of a “good” computer changes. Any computer connected to the Internet is as good as any other computer connected to the Internet, and any computer not connected to the Internet is no good at all. Furthermore, the impact on student use is enormous. If computers are social objects like phones are, then any computer is their computer as long as their hands are on the keyboard. This is the real demise of the “personal” computer – many of my students don’t have their own computers, but that doesn’t stop them from running their lives via email. 

This isn’t just a question of affordability – the PC is coming unglued in many academic environments. An affluent student who has more than one computer has given up on the idea of the “personal” computer just as surely as the student who decides not to buy a computer at all. A student on a wired campus, with computers in the classroom and the dorm room, sooner or later realizes that it is easier to save their work on the network or email it to themselves than to carry disks or even laptops back and forth. Even faculty, usually the people with the most computing power on their desks, are seeing Web-accessible research and email discussion lists become as much a part of their academic work as the data stored on their hard drive. 

The personal has been taken out of the personal computer because the Internet has transformed computing into a social activity. With email from Hotmail, a free home page from Geocities, a personalized news page from my.yahoo, there is no need to own your own computer, lug a laptop from place to place, or pay $20 a month for Internet access. Computing is where you find it — campus labs, work, an Internet cafe — and the guts of any individual computer become unimportant. The old Macintosh vs. Windows debate is meaningless if they both run Netscape, since social computing turns the browser into the operating system and the Internet into the hard drive. 

Looking at my students instinctive sense of network use, I’ve boiled the elements of social computing down into three basic principles: Hardware is Boring. 

Who cares what kind of computer you have? The old concerns of the PC era – which operating system, what CPU speed, how much RAM – have been replaced by the concerns of connection – dial-up or ethernet, which connect speed, what browser version. If all hardware can connect, each individual piece is less important. 

Imagine that your institution plans to upgrade your computer, but that you have to choose: you can have either double the speed of your CPU, or double the speed of your Internet connection. Which would you choose? 

Computers are for Finding, not Keeping. 

Finding new information is more important than storing old information. A computer with a CD-ROM drive only becomes more valuable when you spend the time and money to buy a new CD, while a computer with network access becomes more valuable every time someone creates a new web site, which happens several times a day, for free. Computers used for keeping information lose value over time, as the information on them goes out of date. Computers used for finding information increase in value over time, as the breadth and depth of the accessible information increases with no additional investment. 

Imagine another choice: your institution will pay to quadruple the size of your hard drive, so you can store more stuff locally, or pay for you to get access to the online archives of the New York Times, the Wall Street Journal and the Encyclopedia Brittanica. Which would you choose? 

Information Leaks Out of its Containers. 

Information can no longer be trapped in objects. Multi-media is leaking out of CD-ROMs and into Web sites, music is leaking out of CDs and into Internet radio, text is leaking out of books and into online databases. At first, online academic materials were simply additions to existing research infrastructure, but they are starting to replace, rather than augment, traditional methods: MIT’s Sloan School of Business now only accepts electronic applications, and 1998 is the last year that Encyclopedia Brittanica will publish a paper edition. 

A third choice: imagine that your next computer can have either a CD-ROM drive or a connection to the Internet, but not both. Which would you choose? 

You may not face choices as stark as the ones outlined above, but your students do. Lacking an office, living on a budget, possibly balancing work and school, they will gravitate towards social computing wherever possible. Why spend money if they don’t have to? Why pay for a fast CPU if they have a slow modem, why pay for a large disk drive if TIME magazine stores all their articles online, why pay for a CD-ROM if the same material exists on the Web? 

The changes in a campus are which embraces social computing are far-reaching. Computers become less a product than a service, less an occasional and expensive purchase like office furniture, and more a steady stream of small purchases, like office supplies. PCs wither and networks bloom, and value is created by access instead of ownership. Anyone who has done the math for a ‘one computer per student’ policy sees millions of dollars tied up in machines that are unused most of the day, and that’s when the student body is affluent enough to afford such a requirement. For a student body already straining to meet tuition, the idea of personal computing is an absurdity, but social computing is not. 

If hardware is boring, then homogeneity of computing resources becomes unimportant. With the browser as the interface and the Internet as the hard drive, the differences between a 486 running Linux and a Pentium II running Windows 98 shrink in importance, and instead of a massive overhaul of computer labs every 3-5 years, it becomes possible to replace 20% of the machines annually. This takes advantage of continually rising quality and falling cost, raises the quality of the average machine every year, eliminates the problem of wheezing equipment at the tail end of a “complete replacement” cycle, and frees up low-end machines for use as hallway terminals or print servers as a side-effect. 

If computers are for finding instead of keeping, then then the role of the librarian changes drastically. With a computer as a finding tool, every library suddenly acquires a rapidly expanding global collection. Librarians will be increasingly freed from the part of their job that turns them into literate janitors, constantly reshelving paperback copies of The Iliad, while returning to their roles as taxonomists and sleuths, guiding students to useful resources and teaching them how to separate the good from the mediocre. 

If information leaks out of objects, then every professor becomes a publisher, and the separation between the classroom lecture and the textbook begins to fade. “Handouts” handed out over the web spare faculty the trip to the copy center (and the attendant cost), online syllabi don’t get out of date because the URL always points to the most recent copy, and one semester’s preparation means less editing to make it ready for future classes. An entire textbook can be built up online, one lecture at a time, and tested in the real world before it is published, rather than after. 

For us as members of the faculty, staff and administration of colleges and universities, the ramifications of the rise in social computing are clear. The amount of money and effort it takes for a “One personal computer per student” program puts it out of the reach of many institutions and many student bodies even as access to the Internet is becoming increasingly critical for both academic work and social life on college campuses. The rise in socal computing suggests a way to bridge that gap, by providing the advantages of access without the overhead of ownership. 

Our students are gravitating towards the Internet, using it in ways that aren’t intuitive for those of us still using personal computers. They will tend to prefer connectivity over storage and accessibility over raw processing power. If we aren’t sensitive to their preference for cheaper computers out where they can use them rather than expensive computers under lock and key, we will end up misallocating precious resources in ways that aren’t in our students best interests, or, ultimately, in our own.

by Clay Shirky

Short Takes on Linux

Brief theses about Linux and Open Source development.

Microsoft Is Not a Bad Company

Microsoft is not a bad company. Their problem is not systemic but situational – the same programmers with the same marching orders would focus on producing lightweight and useful code, as long as they weren’t working for a monopoly. It is Microsoft’s leverage from its desktop monopoly which makes them behave the way they do, not any inherent evil.
Microsoft, as a for-profit company, has one goal: make pots of money. The only way their current attempts to de-commoditize standards would help them do that was if they had a monopoly share of the market they were trying to affect. The same de-commoditizing behavior from a company which was competing for market share would lose them pots of money instead.

It is not necessary to “crush” Microsoft, it is merely necessary to get them to drop ‘Embrace and Extinguish’ in favor of actually advancing the state of the art. Linux is obviously Exhibit A, but Linux doesn’t have to get 50% desktop market share to defeat Microsoft, it merely needs to get a 15% market share to defeat Microsoft’s monopoly.

With a credible alternative taking as little as an eighth of the market share, Microsoft will no longer own the desktop. And as with IBM, once Microsoft no longer owns the space, they will undergo a painful retrenchment, and then emerge as a force for innovation instead of strangulation.

NT Competes with Linux, but Linux Does Not Compete with NT

Given the Bazaar model of development fostered by Open Source, Linux will always be under development. Accepting that is part of the cultural change necessary to work with OSS, even as an end user. There is a sense in some parts of the Open Source community that the current attention Linux is getting means that this is its chance to change the world, and that if it fails to do so quickly, there will be no second chance.
As long as programmers continue to improve it, Linux will have all the second chances in the world. The for-profit model needs end users for the positive externalities of network reinforcement (you use what other people at your job use, so its easy to find help, advice, etc.) and cash flow to pay programmers. OSS is a DIY model, which means that it can continue to advance without having a single “end user”.

The Linux market share issue is a tricky one – NT is competing with Linux, but Linux is not competing with NT. If a casual user switches from NT to Linux, MS loses cash flow, mind share, and the illusion of inevitability. If the same user switches from Linux to NT, it affects Linux’s development not at all.

Free Software Creates New Value

Thinking about the recent conversations about Linux ‘market share’, it occurred to me that some of us may be making the same mistake (self included) that Thomas Watson made when he predicted a worldwide market for 5 computers, or when Ken Olsen said there would be no market for computers in the home, i.e. assuming that the future will be much like the present.
Looking at Linux vs. NT in terms of present market share hides that group of people for whom the choice is not Linux vs. NT but Linux vs. nothing. Linux users don’t just come from the ranks of people who would otherswise use NT or Win9[58] — for people who only have access to 386 or 486 computers, and can’t afford anything new, Linux offers them the only possibility for running a modern OS, both because it runs well on minimal hardware and because it is free.

Watching large corporations dance around the idea of running Linux is interesting, but to switch to Linux from NT always involves some short term cost that raises the barrier to such a move. The real change in computing ecology could come from another quarter altogether – that group of people who either haven’t owned computers in the past, or who have had computers which have not been able to run web servers et al. Linux’s biggest growth in market share may come from places inaccessible to for-profit companies at all.

To Use Linux Without Criticizing It Is To Betray It

The greatest strength of Linux is the relentless, targetted criticism of its users. It is that, and not any largesse from a corporation which brings about improvement in the code. When the MS faithful lob ‘linux sux!!!1!” stink bombs into the Linux community, the natural reaction is to circle the wagons and praise Linux to the stars.
The Microsoft folks are right in many particulars, however: for desktop users, Linux is currently poorly documented and intimidating to intall. To let their hostility blind us to that fact robs us of our greatest strength – our ability not merely to criticize Linux, but, since we have the source before us and they do not, to criticize and improve Linux. This is not about two operating systems at this particular snapshot in their respective developments, this is about the long haul, and Linux gets better faster than any MS product ever has.

To use Linux without criticizing it is to betray it.

by Clay Shirky

View Source… Lessons from the Web’s Massively Parallel Development

The Web considered as a massive Open Source project.

First published April 1998.

How did Web grow so quickly?

In the 36 months from January of 1993 to December of 1995, HTML went from being an unknown protocol to being to being the pre-eminent tool for designing electronic interfaces, decisively displacing almost all challengers and upstaging online services, CD-ROMs, and a dozen expensive and abortive experiments with interactive TV, and it did this while having no coordinated center, no central R&D effort, and no discernible financial incentive for the majority of its initial participants.

Ragtag though it was, the early Web, with its endless compendia of ‘Cool Bands’, badly scanned photos of pets, and ubiquitous “Favorite Links” lists pointing to more of same, was a world-building excercise which provided an unequalled hothouse lab for structuring information and designing interfaces. By the time the corporate players and media companies considered getting involved, many of the basic interface issues – the idea of Home pages, buttons and toolbars, the standards for placement of things like logos, navigational elements, help menus and the like had already been worked out within a vast body of general practice generated by the pioneers.

More than any other factor, this ability to allow ordinary users to build their own Web sites, without requiring that they be software developers or even particularly savvy software users, caused its rapid rise. Web design, and in particular Web site design with its emphasis on architecture, interactivity, and structuring data in an implicit order or set of orders, is not graphic design but rather constitutes a special kind of low-level engineering. Prior to the invention of HTML, all software packages which implemented any sort of hypertext required the user to learn a programming language of some sort (Director, Hypercard). HTML, in contrast, allows users to create links without requiring an ablity to write code.

INTERFACE ENGINEERING

Call HTML’s practitioners interface designers, information architects or, in keeping with the idea of world-building, even citizen-engineers, these are people who can think creatively about information as media, without necessarily being comfortable looking down the business end of a compiler. This becomes Principle 0 for this essay:

#0. Web site design is a task related to, but not the same as, software engineering.

Any project which offers a limitless variety of ways in which information can be presented and stuctured can happen both faster and better if there is a way for designers and engineers to collaborate without requiring either group to fundamentally alter their way looking at the world. While it can’t hurt to have a designer see source code or a developer open a copy of Photoshop once in a while, environments which let each group concentrate on their strengths grow more rapidly than environments which burden designers with engineering concerns or force their engineers to design an interface whenever they want to alter an engine. The Web exploded in the space of a few years in part because it effected this separation of functions so decisively.

As opposed to software which seeks a tight integration between creation, file format, and display (e.g. Excel, Lotus Notes), HTML specifies nothing about the tools needed for its creation, validation, storage, delivery, interpretation, or display. By minimizing the part of the interface that is the browser to a handful of commands for navigating the Web, while maximizing the part of the interface that is in the browser (the HTML files themselves), the earliest experiments with the Web took most of the interface out of the browser code and put it in the hands of any user who could learn HTML’s almost babyish syntax.

With Web design separated from the browser that was displaying it, countless alternative ways of structuring a site could be experimented with – sites for online newspapers, banks, stores, magazines, interfaces for commerce, document delivery, display, propaganda, art – all without requiring that the designers arrange their efforts with the browser engineers or even with one another.

This separation of interface and engineering puts site design in a fundamentally new relationship to the software that displays it:

#1. An interface can be integrated with the information it is displaying, instead of with the display software itself.

This transfer of the interface from ‘something that resides in the software and is applied to the data’ to ‘something that resides in the data and is applied to the software’ is the single most important innovation of the early Web.

REDUCING REQUIRED COORDINATION

Site designers with training in visual or conceptual aspects of organizing and presenting information can design for a Web browser without working with, or even needing to talk to, the people who designed the browser itself. Newsweek has to coordinate with America Online in order to make its content available through AOL, but it does not have to coordinate with Netscape (or the end-user’s ISP) to make its content available through Netscape.

This move away from centralized coordination and towards lateral development is good for the Web, and that some basic principles about the ways this is acheived can be derived from looking at the HTML/Browser split as one instance of a general class of “good” tools. The basic rule of good tools is:

#2. Good tools are transparent.

Web design is a conversation of sorts between designers, with the Web sites themselves standing in for proposal and response. Every page launched carries an attached message from its designer which reads “I think this is a pretty good way to design a Web page”, and every designer reacting to that will be able to respond with their own work. This is true of all design efforts – cars, magazines, coffee pots, but on the Web, this conversation is a riot, swift and wide ranging.

THE CENTRALITY OF OPEN HTML SOURCE TO THE WEB’S SUCCESS

The single factor most responsible for this riot of experimentation is transparency – the ability of any user to render into source code the choices made by any other designer. Once someone has worked out some design challenge, anyone else should be able to adopt, modify it, and make that modified version available, and so on.

Consider how effortless it would have been for Tim Berners-Lee or Marc Andreeson to treat the browser’s “View Source…” as a kind of debugging option which could have been disabled in any public release of their respective browsers, and imagine how much such a ‘hidden source’ choice would have hampered this feedback loop between designers. Instead, with this unprecedented transparency of the HTML itself, we got an enormous increase in the speed of design development. When faced with a Web page whose layout or technique seems particularly worth emulating or even copying outright, the question “How did they do that?” can be answered in seconds. 

“X DOES Y”, NOT “X IS FOR Y”

This ability to coordinate laterally, for designers to look at one anothers work and to experiment with it without asking permission from a central committee, is critical to creating this speed of innovation. Once the tools for creating Web pages are in a designers hand, there is no further certification, standardization or permission required. Put another way:

#3. Good tools tell you what they do, not what they’re for.

Good, general-purpose tools specify a series of causes and effects, nothing more. This is another part of what allowed the Web to grow so quickly. When a piece of software specifies a series of causes and effects without specifying semantic values (gravity makes things fall to the ground, but gravity is not for keeping apples stuck to the earth’s surface, or for anything else for that matter), it maximises the pace of innovation, because it minimizes the degree to which an effect has to be planned in advance for it to be useful.

The best example of this was the introduction of tables, first supported in Netscape 1.1. Tables were originally imagined to be just that – a tool for presenting tabular data. Their subsequent adoption by the user community as the basic method for page layout did not have to be explicit in the design of either HTML or the browser, because once its use was discovered and embraced, it no longer mattered what tables were originally for, since they specified causes and effects that made them perfectly suitable in their new surroundings.

A corollary to rule #3 is:

#3b. Good tools allow users to do stupid things.

A good tool, a tool which maximizes the possibilities for unexpected innovation from unknown quarters, has to allow the creation of everything from brilliant innovation through workmanlike normalcy all the way through hideous dreck. Tools which try to prevent users from making mistakes enter into a tar pit, because this requires that in addition to cause and effect, a tool has to be burdened with a second, heuristic sense of ‘right’ and ‘wrong’. In the short run, average quality can be raised if a tool intervenes to prevent legal but inefficent uses, but in the long haul, that strategy ultimately hampers development by not letting users learn from their mistakes.

THE INDEPENDENT RATE OF DEVLOPMENT

The browser/HTML combination as described not only increases the number of people who can work on different functions by reducing the amount of cross-training and coordination needed between designers and engineers (or between any two designers), it also changes the speed at which things happen by letting designers develop different parts of the system at independant rates.

By separating Web site design so completely from browser engineering, the rate of development of Web sites became many times faster than the rate of browser-engineering or protocol design.

Reflecting on these extremes is instructive, considered in light of the traditional software release schedule, which has alterations to engineering and interface change appearing together, in relatively infrequent versions. In the heat of structuring a site, a Web design team may have several people altering and re-examining an interface every few minutes. At the opposite extreme, in the three years from 1993 to 1995, the http protocol was not changed once, and in fact, even in 1998, the majority of Web transactions still use http 1.0. In between these extremes come rates of change from more to less rapid: changes in site structure (the relation between file storage and pointers), new versions of Web browsers, and new specifications for HTML itself. Designers could create, display, alter, beg, borrow, and steal interfaces, as fast as they liked, without any further input from any other source.

CONCLUSION

The Web grew as quickly as it did because the independent rate of site design, freed from the dictates of browser engineering, was much faster than even its inventors had predicted. Tools that allowed designers to do anything that rendered properly, that allowed for lateral conversations through the transparency of the HTML source, and removed the need for either compiling the results or seeking permission, certifcation or registration from anyone else led to the largest example of parallel development seen to date, and the results have been world-changing.

Furthermore, while there were certainly aspects of that revolution which will not be easily repeated, there are several current areas of inquiry – multi-player games (e.g. Half Life, Unreal), shared 3D worlds (VRML, Chrome), new tagset proposals (XML, SMIL), new interfaces (PilotOS, Linux), which will benefit from examination in light of the remarkable success of the Web. Any project with an interface likely to be of interst to end users (create your own avatar, create your own desktop) can happen both faster and better if these principles are applied.

by Clay Shirky

Playfulness in 3-D Spaces

Why Quake is better than VRML, and what it means for software design.

First published in ACM, 03/98.

“Put the trodes on and [you] were out there, all the data in the world stacked up like a big neon city, so you could cruise around and have a kind of grip on it, visually anyway, because if you didn’t, it was too complicated, trying to find your way to a particular piece of data you needed.” 

Every six months or so, I find a reason to survey the state of VRML (Virtual Reality Modeling Language; more info at http://www.vrml.org/) and 3-D ‘shared worlds.’ I started these semi-annual forays in 1994, while researching a book, “Voices From the Net” (Ziff-Davis, 1994) , about online communities. In that book, I concluded that the classic model of cyberspace, quoted above from William Gibson’s “Mona Lisa Overdrive”, was not possible in this century (a bit of a grandiose cheat, since the century amounted to but a half-dozen years by that time), if at all. 

Woe betide the prognosticator who puts a far-off date on computer revolutions — as the visual components of networking grew by leaps and bounds, it looked like my predictions might be off the mark. I kept hearing that VRML was at the edge of usability, and every now and again would go exploring to see for myself. The improvements from year to year were impressive, although they were due in part to the rising tide of clock speed, which lifts all the boats. However, they never seemed any closer to being something that someone would use for anything other than a “Now, instead of just looking up the info you want, you can actually fly through your company’s parts catalog!” trade show demo. It was always better than last time, but never enough of a jump to actually be, in Alan Kay’s famous words, “good enough to criticize.” 

The Streets of San Francisco

VRML applications are the videophones of the Net — something that is possible in theory, but which excites designers far more than it excites potential users. At one point, the height of VRML applications was a 3-D Yellow Pages for San Francisco, the one town whose 3-D shape was uniquely ill-suited to such an already questionable endeavor (“Now, instead of simply looking up your dry cleaner’s phone number, you can search for it over hill and dale!”). Thus, I concluded, my Nostradamus-like powers were in no danger of being upended by unkind reality. Any real example of 3-D space that could be shared by multiple users and didn’t involve an unacceptable trade-off between appearance and speed, any 3-D space that had any sort of visceral effect on the user, was still years away. 

There was only one problem with that conclusion: I was wrong. 

I have now found what I was looking for, and found it, to my chagrin, years late. A networked 3-D engine with excellent shared space capabilities, open source code, a well-published world-building specification, an active development community, and a project showing the kind of rapid improvement year after year that comes from the best massively parallel design efforts, such as the Web. 

It is called Quake. 

I know, I know. As they say on Usenet, “Quake? That’s so 5 minutes ago!” 

Quake (http://www.idsoftware.com), for those of you like me who don’t follow the PC gaming world, is a “First Person Action” game, meaning roughly that the screen is your view of the world, and that you pilot your digital self, or “avatar,” through that world by maneuvering through what you see on the screen. I missed out on Quake when it was launched in the mid-’90s because as a general all-around Serious Person, I don’t play games, in particular games whose ideal customer is less than half my age. I have now seen the error of my ways. Quake solves the shared 3-D space problem far better than any VRML implementation — not despite being a game, but because it is a game. 

The Quake 3-D engine is for something, while VRML is a technology in search of a use. The Quake engine’s original use is immaterial; once you have a world-modeling protocol that lets you wade through a subterranean stream, picking up shotgun shells and blowing up other people who are picking up rocket launchers and trying to blow up you, you’ve got the whole ball of wax — spaces, movement, objects, avatars, even a simple physics. It would be easier to turn that into a tool for shared corporate meeting spaces than it would be to build the same thing in VRML, because once a tool is good for one thing, it’s easy to see how to make it good for another thing. 

On The Cutting Edge

This is one of the first axioms of creativity in this realm: Solve something first, then apply that solution elsewhere. It doesn’t matter if the first stone was sharpened to kill animals or cut wood, because once you have the principle down, making a knife into a spear or a scimitar or a scalpel is just application-level development. I have always been skeptical of any real possibility of viscerally engaging 3-D spaces, but I believe that I see in Quake — and its subsequent imitators and descendants — the sharpened stone, the thing that indicates the beginning of a new way of using the network. 

Quake is the real thing. It’s a local solution to a local problem (how to make a 3-D space realistic enough so that you can accurately aim a double-barreled shotgun at a chainsaw-wielding ogre, while both of you are running) that has global ramifications. I have seen a lot of 3-D spaces over the years, and I have never run into anything that gives as palpable a sense of place as this does. (Before the objection is raised that Quake is software and VRML is a protocol, consider that Quake, like VRML, always runs in client-server mode and reads external data files, even locally, and that the Quake client is one-fifth the size of the benchmark VRML client.) 

After playing with Quake for only a few hours, I found myself in possession of an absolute and rigorous sense of the spaces it contained: There was a moment, while I was off thinking about something else altogether, when I realized that in one particularly sticky part of the game, I could jump off a ramp and swim through some water under a trap located in a hallway. This was as matter-of-fact a thought as if I’d realized I could bring a big package up the service elevator at my office; in other words, I had a large, accurate map of an entire space in my head, not from having studied it, but from having experienced it. I had derived, for the first time ever, a sense of space — not an ‘as if’ sense or a sense of a metaphorical cyberspace, but of real space — from a screen. 

Here’s why I think this matters: Though Quake isn’t necessarily the right solution for the 3-D shared space problem, it feels right in a way that nothing in the VRML world does. Once you’ve seen it in action, much of what’s wrong with VRML snaps into focus: Quake does something well instead of many things poorly. 

This matters more than I can say. Despite all the theoretical underpinnings we now use as a basis for our work as computing reaches the half-century mark and networking turns 30, world-shifting developments still often come from practical solutions to simple problems. 

The VRML community has failed to come up with anything this compelling — not despite the community’s best intentions, but because of them. Every time VRML practitioners approach the problem of how to represent space on the screen, they have no focused reason to make any particular trade-off of detail versus rendering speed, or making objects versus making spaces, because VRML isn’t for anything except itself. Many times, having a particular, near-term need to solve brings a project’s virtues into sharp focus, and gives it enough clarity to live on its own. Quake puts tools in the hands of the users. 

Id software, producers of Quake, started allowing its games to be edited by users, and published a world-building spec so that users could write their own editors as well. As with HTML, one can build a variety of 3-D editors that can create a Quake world by simply writing to the spec. This has led to a riot of sophisticated tools for creating Quake “levels” like BSP and WorldCraft that would make any VRML partisan drool, and has spawned a community of user-engineers who are creating compelling worlds, objects, and sometimes whole alternate worlds at a pace and with an intensity that should make any VRML adherent faint. VRML is older than Quake, and yet it has fewer tools and fewer users, and still exists at the stage where the best examples of the technology are produced by the companies selling it, not the users using it. Quake has a simple model of compliance. 

The problem of representing space, surfaces, objects and worlds is almost infinitely vast. There was for some time in the AI community a proposal to develop a ‘naive physics’ where every aspect of physical organization of the world would be worked out in a way that would let computers deduce things about their surroundings. For instance, the naive physics of liquids included all sorts of rules about flow and how it was affected by being bounded by floors, walls and ceilings. This kind of elaborate modeling has been inherited in spirit by VRML, which is so complex that rendering a simple piece of furniture can bog down an average desktop computer. 

Quake, on the other hand, adopts a breathtakingly simple attitude toward physics. Quake has shapes like blocks, wedges and columns, and it has materials like stone, metal and water. If you make a freestanding column of water, so be it. The column will appear in any space you create; you can walk into it, swim to the top, walk out at the top and then fall down again. It does this simply because a physics of liquids is too complex to solve right now, and because Quake does not try to save users from themselves by adding layers of interpretation outside the engine itself. If it compiles, it complies. 

Hacking X for Y 

This willingness to allow users to do stupid things, on the assumption that they will learn more quickly if the software doesn’t try to second-guess them, has given Quake a development curve that looks like HTML in the early days, where a user’s first efforts were often lousy but the progress between lousy and good was not blocked by the software itself. Quake encourages experimentation and incrementalism in the tradition of the best tools out there. 

I acknowledge, however, that the Quake format lacks the rigorous throughput into VRML — the comparison here is not architecture to architecture, but effect to effect. I am not seriously suggesting that Quake or one of its children like Quake II (http://www.idsoftware.com), Half-Life (http://www.valvesoftware.com) or Unreal (http://www.unreal.com) be used as-is as a platform from which to build other 3-D shared spaces (well, maybe I am suggesting that half-seriously). More important, I am suggesting that there is a problem because the 3-D game people aren’t talking enough to the VRML people, and vice versa. Networking has come so far so fast partly because we have been willing to say, over and over again, “Well, we never thought of using X for Y, but now that you mention it, why not?” 

The Net is too young, and the 3-D space problem too tricky, to shoot for perfection right now. What worries me most about my belated discovery of Quake is that in years of looking at 3-D spaces, no one ever mentioned it in the same breath as VRML and I never thought of it myself, simply because of the stigma attached to things that are “merely” games. Many of us, myself very much included, have come to believe our own press about the seriousness of our current endeavors. We have been adversely affected by the commercial applications of the network, and have weaned ourselves off the playfulness that has always produced insights and breakthroughs. 

It will be a terrible loss if those of us doing this work start to take ourselves too seriously. Round the clock and round the world, there are hundreds of people chasing one another through rooms and tunnels and up and down staircases that don’t exist anywhere in the real world, a model of electronic space closer to Gibson’s ideal than anything else that exists out there, and we shouldn’t write those people off. Their game playing may look frivolous to us Serious Persons, but they know a lot about 3-D shared worlds, and it will be a terrible waste of everybody’s time if we don’t find a way to tap into that.

by Clay Shirky

In Praise of Evolvable Systems

Central Planning vs. Evolution.

(First appeared in the ACM’s net_worker, 1996)

Why something as poorly designed as the Web became The Next Big Thing, and what that means for the future.

If it were April Fool’s Day, the Net’s only official holiday, and you wanted to design a ‘Novelty Protocol’ to slip by the Internet Engineering Task Force as a joke, it might look something like the Web:
The server would use neither a persistent connection nor a store-and-forward model, thus giving it all the worst features of both telnet and e-mail.
The server’s primary method of extensibility would require spawning external processes, thus ensuring both security risks and unpredictable load.
The server would have no built-in mechanism for gracefully apportioning resources, refusing or delaying heavy traffic, or load-balancing. It would, however, be relatively easy to crash.
Multiple files traveling together from one server to one client would each incur the entire overhead of a new session call.
The hypertext model would ignore all serious theoretical work on hypertext to date. In particular, all hypertext links would be one-directional, thus making it impossible to move or delete a piece of data without ensuring that some unknown number of pointers around the world would silently fail.
The tag set would be absurdly polluted and user-extensible with no central coordination and no consistency in implementation. As a bonus, many elements would perform conflicting functions as logical and visual layout elements.
HTTP and HTML are the Whoopee Cushion and Joy Buzzer of Internet protocols, only comprehensible as elaborate practical jokes. For anyone who has tried to accomplish anything serious on the Web, it’s pretty obvious that of the various implementations of a worldwide hypertext protocol, we have the worst one possible.

Except, of course, for all the others.

MAMMALS VS. DINOSAURS

The problem with that list of deficiencies is that it is also a list of necessities — the Web has flourished in a way that no other networking protocol has except e-mail, not despite many of these qualities but because of them. The very weaknesses that make the Web so infuriating to serious practitioners also make it possible in the first place. In fact, had the Web been a strong and well-designed entity from its inception, it would have gone nowhere. As it enters its adolescence, showing both flashes of maturity and infuriating unreliability, it is worth recalling what the network was like before the Web.

In the early ’90s, Internet population was doubling annually, and the most serious work on new protocols was being done to solve the biggest problem of the day, the growth of available information resources at a rate that outstripped anyone’s ability to catalog or index them. The two big meta-indexing efforts of the time were Gopher, the anonymous ftp index; and the heavy-hitter, Thinking Machines’ Wide Area Information Server (WAIS). Each of these protocols was strong — carefully thought-out, painstakingly implemented, self-consistent and centrally designed. Each had the backing of serious academic research, and each was rapidly gaining adherents.

The electronic world in other quarters was filled with similar visions of strong, well-designed protocols — CD-ROMs, interactive TV, online services. Like Gopher and WAIS, each of these had the backing of significant industry players, including computer manufacturers, media powerhouses and outside investors, as well as a growing user base that seemed to presage a future of different protocols for different functions, particularly when it came to multimedia.

These various protocols and services shared two important characteristics: Each was pursuing a design that was internally cohesive, and each operated in a kind of hermetically sealed environment where it interacted not at all with its neighbors. These characteristics are really flip sides of the same coin — the strong internal cohesion of their design contributed directly to their lack of interoperability. CompuServe and AOL, two of the top online services, couldn’t even share resources with one another, much less somehow interoperate with interactive TV or CD-ROMs.

THE STRENGTH OF WEAKNESS AND EVOLVABILITY

In other words, every contender for becoming an “industry standard” for handling information was too strong and too well-designed to succeed outside its own narrow confines. So how did the Web manage to damage and, in some cases, destroy those contenders for the title of The Next Big Thing? Weakness, coupled with an ability to improve exponentially.

The Web, in its earliest conception, was nothing more than a series of pointers. It grew not out of a desire to be an electronic encyclopedia so much as an electronic Post-it note. The idea of keeping pointers to ftp sites, Gopher indices, Veronica search engines and so forth all in one place doesn’t seem so remarkable now, but in fact it was the one thing missing from the growing welter of different protocols, each of which was too strong to interoperate well with the others.

Considered in this light, the Web’s poorer engineering qualities seem not merely desirable but essential. Despite all strong theoretical models of hypertext requiring bi-directional links, in any heterogeneous system links have to be one-directional, because bi-directional links would require massive coordination in a way that would limit its scope. Despite the obvious advantages of persistent connections in terms of state-tracking and lowering overhead, a server designed to connect to various types of network resources can’t require persistent connections, because that would limit the protocols that could be pointed to by the Web. The server must accommodate external processes or it would limit its extensibility to whatever the designers of the server could put into any given release, and so on.

Furthermore, the Web’s almost babyish SGML syntax, so far from any serious computational framework (Where are the conditionals? Why is the Document Type Description so inconsistent? Why are the browsers enforcement of conformity so lax?), made it possible for anyone wanting a Web page to write one. The effects of this ease of implementation, as opposed to the difficulties of launching a Gopher index or making a CD-ROM, are twofold: a huge increase in truly pointless and stupid content soaking up bandwidth; and, as a direct result, a rush to find ways to compete with all the noise through the creation of interesting work. The quality of the best work on the Web today has not happened in spite of the mass of garbage out there, but in part because of it.

In the space of a few years, the Web took over indexing from Gopher, rendered CompuServe irrelevant, undermined CD-ROMs, and now seems poised to take on the features of interactive TV, not because of its initial excellence but because of its consistent evolvability. It’s easy for central planning to outperform weak but evolvable systems in the short run, but in the long run evolution always has the edge. The Web, jujitsu-like, initially took on the power of other network protocols by simply acting as pointers to them, and then slowly subsumed their functions.

Despite the Web’s ability to usurp the advantages of existing services, this is a story of inevitability, not of perfection. Yahoo and Lycos have taken over from Gopher and WAIS as our meta-indices, but the search engines themselves, as has been widely noted, are pretty lousy ways to find things. The problem that Gopher and WAIS set out to solve has not only not been solved by the Web, it has been made worse. Furthermore, this kind of problem is intractable because of the nature of evolvable systems.

THREE RULES FOR EVOLVABLE SYSTEMS

Evolvable systems — those that proceed not under the sole direction of one centralized design authority but by being adapted and extended in a thousand small ways in a thousand places at once — have three main characteristics that are germane to their eventual victories over strong, centrally designed protocols.

  • Only solutions that produce partial results when partially implemented can succeed. The network is littered with ideas that would have worked had everybody adopted them. Evolvable systems begin partially working right away and then grow, rather than needing to be perfected and frozen. Think VMS vs. Unix, cc:Mail vs. RFC-822, Token Ring vs. Ethernet.
  • What is, is wrong. Because evolvable systems have always been adapted to earlier conditions and are always being further adapted to present conditions, they are always behind the times. No evolving protocol is ever perfectly in sync with the challenges it faces.
  • Finally, Orgel’s Rule, named for the evolutionary biologist Leslie Orgel — “Evolution is cleverer than you are”. As with the list of the Web’s obvious deficiencies above, it is easy to point out what is wrong with any evolvable system at any point in its life. No one seeing Lotus Notes and the NCSA server side-by-side in 1994 could doubt that Lotus had the superior technology; ditto ActiveX vs. Java or Marimba vs. HTTP. However, the ability to understand what is missing at any given moment does not mean that one person or a small central group can design a better system in the long haul.

Centrally designed protocols start out strong and improve logarithmically. Evolvable protocols start out weak and improve exponentially. It’s dinosaurs vs. mammals, and the mammals win every time. The Web is not the perfect hypertext protocol, just the best one that’s also currently practical. Infrastructure built on evolvable protocols will always be partially incomplete, partially wrong and ultimately better designed than its competition.

LESSONS FOR THE FUTURE

And the Web is just a dress rehearsal. In the next five years, three enormous media — telephone, television and movies — are migrating to digital formats: Voice Over IP, High-Definition TV and Digital Video Disc, respectively. As with the Internet of the early ’90s, there is little coordination between these efforts, and a great deal of effort on the part of some of the companies involved to intentionally build in incompatibilities to maintain a cartel-like ability to avoid competition, such as DVD’s mutually incompatible standards for different continents.

And, like the early ’90s, there isn’t going to be any strong meta-protocol that pushes Voice Over IP, HDTV and DVD together. Instead, there will almost certainly be some weak ‘glue’ or ‘scaffold’ protocol, perhaps SMIL (Synchronized Multimedia Integration Language) or another XML variant, to allow anyone to put multimedia elements together and synch them up without asking anyone else’s permission. Think of a Web page with South Park in one window and a chat session in another, or The Horse Whisperer running on top with a simultaneous translation into Serbo-Croatian underneath, or clickable pictures of merchandise integrated with a salesperson using a Voice Over IP connection, ready to offer explanations or take orders.

In those cases, the creator of such a page hasn’t really done anything ‘new’, as all the contents of those pages exist as separate protocols. As with the early Web, the ‘glue’ protocol subsumes the other protocols and produces a kind of weak integration, but weak integration is better than no integration at all, and it is far easier to move from weak integration to strong integration than from none to some. In 5 years, DVD, HDTV, voice-over-IP, and Java will all be able to interoperate because of some new set of protocols which, like HTTP and HTML, is going to be weak, relatively unco-ordinated, imperfectly implemented and, in the end, invincible.