Author / Clay Shirky

by Clay Shirky

Time to Open Source the Human Genome

Patent law and human genetics.

9/8/1999

The news this week that a researcher has bred a genetically smarter mouse is another precursor to a new era of genetic manipulation. The experiment, performed by Dr. Joe Tsien of Princeton, stimulated a gene called NR2B, producing mice with greatly heightened intelligence and memory. The story has generated a great deal of interest, especially as human brains may have a similar genetic mechanism, but there is a behind- the-scenes story which better illuminates the likely shape of 21st century medicine. Almost from the moment that the experiment was publicized, a dispute broke out about ownership of NR2B itself, which is already patented by another lab. Like an information age land-grab, whole sections of genetic information are being locked behind pharmaceutical patents as fast as they are being identified. If the current system of private ownership of genes continues, the majority of human genes could be owned in less than two years. The only thing that could save us from this fate would be an open- source movement in genetics — a movement to open the DNA source code for the human genome as a vast public trust.

Fighting over mouse brains may seem picayune, but the larger issue is ownership of the genetic recipe for life. The case of the smarter mouse, patent pending, is not an
isolated one: among the patents granted for human genes are blindness (Axys
Pharmaceuticals), epilepsy (Progenitor), and Alzheimer’s (Glaxo Wellcome). Rights to the gene which controls arthritis will be worth millions, breast cancer, billions, and
the company that patents the genes that control weight loss can write their own ticket. As the genome code becomes an essential part of industries from bio-computing and cybernetic interfaces to cosmetics and nutrition, the social and economic changes it instigates are going to make the effects of the Internet seem inconsequential. Unfortunately for us, though, the Internet’s intellectual property is mostly in the public domain, while the genome is largely — and increasingly — private.

It didn’t have to be this way. Patents exist to encourage investment by guaranteeing a
discoverer time to recoup their investment before facing any competition, but patent
law has become increasingly permissive in what constitutes a patentable discovery. There are obviously patents to be had in methods of sequencing genes and for methods of using those sequences to cure disease or enhance capability. But to allow a gene itself to be patented makes a mockery of “prior art,” the term which covers unpatented but widely dispersed discoveries. It is prior art which keeps anyone from patenting fire, or the wheel, and in the case of genetic information, life itself is the prior art. It is a travesty of patent law that someone can have the gene for Alzheimer’s in every cell of their body, but that the patent for that gene is owned by Glaxo Wellcome.

The real action, of course, is not in mouse brains but in the human genome. Two teams, one public and one private, are working feverishly to sequence all of the 100,000 or so genes which lie within the 23 pairs of human chromosomes. The public consortium aims to release the sequence into the public domain, while the private group aims to patent much of the genome, especially the valuable list of mutations that cause genetic disease. The competition between these two groups has vastly accelerated the pace of the work — moving it from scheduled completion in 2005 to next year — but the irony is that this accelerated timetable won’t give the public time to grasp the enormous changes the project portends. By this time next year, the fate of the source code for life on earth — open or closed — will be largely finished, long before most people have even begun to understand what is at stake.

by Clay Shirky

The Internet and Hit-driven Industries

Why Hollywood is really going to hate the Web.

First published on SAR, 07/99.

ALERT! New community-based email virus on the loose!

This virus, called the “DON’T GO” virus, primarily targets major Hollywood studios, and is known to proliferate within 24 hours of the release of a mediocre movie. If you receive a piece of email from a friend with a subject line like “Notting Hill: Don’t Go”, do not open it! Its contents can cause part of your memory to be erased, replacing expensive marketing hype with actual information from movie-goers. If this virus is allowed to spread, it can cause millions of dollars of damage to a Hollywood studio in a single weekend.

Hit-driven industries (movies, books, music, et al.) are being radically transformed by Internet communities, because the way these industries make money is directly threatened by what Internet communities do best – move word of mouth at the speed of light. Internet communities are putting so much information in the hands of the audience so quickly that the ability of studios, publishing houses, and record labels to boost sales with pre-release hype are diminishing even as the costs of that hype are rising.

Consider Hollywood, the classic hit-driven industry. The financial realities can be summed up thusly:

Every year, there are several major flops. There are a horde of movies that range from mildly unprofitable to mildly profitable. There are a tiny core of wildly profitable hits. The hits are what pays for everything else.

This is true of all hit-driven businesses – Stephen King’s books more than earn back what Marcia Clark lost, the computer game Half-Life sells enough to pay for flops like Dominion, Boyzone recoups The Spice Girls latest album, and so on. Many individual works lose money, but the studios, publishers, or labels overall turn a profit. Obviously, the best thing Hollywood could do in this case would be to make all the movies worth watching, but as the perennial black joke goes, “There are three simple rules for making a successful movie, but unfortunately nobody knows what they are.” Thus the industry is stuck managing a product whose popularity they can’t predict in advance, and they’ve responded by creating a market where the hits more than pay for the flops.

For Hollywood, this all hinges on the moment when a movie’s quality is revealed: opening weekend. Once a movie is in the theaters, the audience weighs in and its fate is largely sealed. Opening weekend is the one time when the producers know more about the product than the audience — it isn’t until Monday morning water cooler talk begins that a general sense of “thumbs up” or “thumbs down” becomes widespread. Almost everything movie marketers do is to try to use the media they do control — magazine ads, press releases, commercials, talk show spots — to manipulate the medium they don’t control — word of mouth. The last thing a studio executive wants is to allow a movie to be judged on its merits — they’ve spent too much money to leave things like that to the fickle reaction of the actual audience. The best weapon they have in this fight is that advertising spreads quickly while word of mouth spreads slowly.

Enter the Internet. A movie audience is kind of loose community, but the way information is passed — phone calls, chance encounters at the mall, conversations in the office — makes it a community where news travels slow. No so with email, news groups, fan web pages — news that a movie isn’t worth the price of admission can spread through an Internet community in hours. Waiting til Monday morning to know about a movie’s fate now seems positively sluggish — a single piece of email can be forwarded 10 times, a newsgroup can reach hundreds or even thousands, a popular web site can reach tens of thousands, and before you know it, it’s Saturday afternoon and the people are staying away in droves.

This threat — that after many months and many millions of dollars the fate of a movie can be controlled by actual fan’s actual reactions — is Hollywood’s worst nightmare. There are two scenarios that can unfold in the wake of this increased community power: The first scenario (call it “Status Quo Plus”) is that the studios can do more of everything they’re already doing: more secrecy about the product, more pre-release hype, more marketing tie-ins, more theaters showing the movie on opening weekend. This has the effect of maximising revenues before people talk to their friends and neighbors about the film. This is Hollywood’s current strategy, having hit its current high-water mark with the marketing juggernaut of The Phantom Menace. The advantage of this strategy is that it plays to the strengths of the existing Hollywood marketing machine. The disadvantage of this strategy is that it won’t work.

Power has moved from the marketer to the audience, and there is no way to reverse that trend, because nothing is faster than the Internet. The Internet creates communities of affinity without regard to geography, and if you want the judgement of your peers you can now get it instantly. (Star Wars fan sites were posting reactions to “The Phantom Menace” within minutes of the end of the first showing.) Furthermore, this is only going to get worse, both because the Internet population is still rising rapidly and because Internet users are increasingly willing to use community recommendations in place of the views of the experts, and while experts can be bought, communities can’t be. This leaves the other scenario, the one that actually leverages the power of Internet communities: let the artists and the fans have more direct contact. If the audience knows instantly whats good and what isn’t, let the creators take their products directly to the audience. Since the creators are the ones making the work, and the community is where the work stands or falls, much of what the studio does only adds needless expense while inhibiting the more direct feedback that might help shape future work. Businesses that halve the marketing budget and double the community outreach will find that costs go down while profits from successful work goes up. The terrible price of this scenario, though, is that flops will fail faster, and the studio will have to share more revenue with the artist in return for asking them to take on more risk.

The technical issues of entertainment on the Internet are a sideshow compared to community involvement — the rise of downloadable video, MP3 or electronic books will have a smaller long-term effect on restructuring hit-driven industries than the fact that there’s no bullshitting the audience anymore, not even for as long as a weekend. We will doubtless witness an orgy of new marketing strategies in the face of this awful truth — coupons for everything a given movie studio or record label produces in a season, discounts for repeat viewings, Frequent Buyer Miles, on and on — but in the end, hit driven businesses will have to restructure themselves around the idea that Internet communities will sort the good from the bad at lightning speed, and only businesses that embrace that fact and work with word of mouth rather than against it will thrive in the long run.

by Clay Shirky

Language Networks

On the net, two countries border each other only if they share a language.

7/7/1999

The 21st Century is going to look a lot like the 19th century, thanks to the internet.
A recent study in the aftermath of the Asian financial storm (“Beyond The Crisis –
Asia’s Challenge for Recovery,” Dentsu Institute for Human Studies) found that citizens of Asian countries who speak English are far more likely to be online than those who don’t. The study, conducted in Tokyo, Beijing, Seoul, Bangkok, Singapore, and Jakarta, found that English speakers were between two and four times as likely to use the internet as their non-English speaking fellow citizens. Within each country, this is a familiar story of haves and have-nots, but in the connections between countries something altogether different is happening — the internet is creating an American version of the British Empire, with the English language playing the role of the Royal Navy.

This isn’t about TCP/IP — in an information economy the vital protocol is language,
written and spoken language. In this century, trade agreements have tended to revolve around moving physical goods across geographical borders: ASEAN, EU, OAS, NAFTA. In the next century, as countries increasingly trade more in information than hard goods, the definition of proximity changes from geographic to linguistic: two countries border one another if and only if they have a language they can use in common. The map of the world is being redrawn along these axes: traders in London are closer to their counterparts in New York than in Frankfurt, programmers in Sydney are closer to their colleagues in Vancouver than in Taipei. This isn’t an entirely English phenomenon: on the internet, Lisbon is closer to Rio than to Vienna, Dakar is closer to Paris than to Nairobi.

This linguistic map is vitally important for the wealth of nations — as the Dentsu
study suggests, the degree to which a country can plug into a “language network,”
especially the English network, will have much to do with its place in the 21st century
economy. These language networks won’t just build new connections, they’ll tear at
existing ones as well. Germany becomes a linguistic island despite its powerhouse
economy. Belgium will be rent in two as its French- and Flemish-speaking halves link
with French and Dutch networks. The Muslim world will see increasing connection among its Arabic-speaking nations — Iraq, Syria, Egypt — and decreasing connections with its non-Arabic-speaking members. (Even the translation software being developed reflects this bias: given the expense of developing translation software, only languages with millions of users — standard versions of English, French, Portuguese, Spanish, Italian, German — will make the cut.) And as we would expect of networks with different standards, gateways will arise; places where multi-lingual populations will smooth the transition between language networks. These gateways — Hong Kong, Brussels, New York, Delhi — will become economic centers in the 21st century because they were places where languages overlapped in the 19th.

There are all sorts of reasons why none of this should happen — why the Age of Empire shouldn’t be resurrected, why countries that didn’t export their language by force should suffer, why English shouldn’t become the Official Second Language of the 21st century — but none of those reasons will matter. We know from the 30-year history of the internet that when a new protocol is needed to continue internet growth, it’ll be acquired at any expense. What the internet economy demands more than anything right now is common linguistic standards. In the next 10 years, we will see the world’s languages sorted into two categories — those that form part of language networks will grow, and those that don’t will shrink, as the export of languages in the last century reshapes the map of the next one.

by Clay Shirky

Language, The Internet, and the Next Century

Human language is the next great challenge for the world’s networks.

Published in ACM, 12/1999.

The 21st Century is going to look a lot like the 19th century, thanks to the internet. A recent study in the aftermath of the Asian financial storm (“Beyond The Crisis – Asia’s Challenge for Recovery,” Dentsu Institute for Human Studies) found that citizens of Asian countries who speak English are far more likely to be online than those who don’t. The study, conducted in Tokyo, Beijing, Seoul, Bangkok, Singapore, and Jakarta, found that English speakers were between two and four times as likely to use the internet as their non-English speaking fellow citizens. Within each country, this is a familiar story of haves and have-nots, but in the connections between countries something altogether different is happening — the internet is creating an American version of the British Empire, with the English language playing the role of the Royal Navy.

This isn’t about TCP/IP — in an information economy the vital protocol is language, written and spoken language. In this century, trade agreements have tended to revolve around moving physical goods across geographical borders: ASEAN, EU, OAS, NAFTA. In the next century, as countries increasingly trade more in information than hard goods, the definition of proximity changes from geographic to linguistic: two countries border one another if and only if they have a language they can use in common. The map of the world is being redrawn along these axes: traders in London are closer to their counterparts in New York than in Frankfurt, programmers in Sydney are closer to their colleagues in Vancouver than in Taipei. This isn’t an entirely English phenomenon: on the internet, Lisbon is closer to Rio than to Vienna, Dakar is closer to Paris than to Nairobi.

This linguistic map is vitally important for the wealth of nations — as the Dentsu study suggests, the degree to which a country can plug into a “language network,” especially the English network, will have much to do with its place in the 21st century economy. These language networks won’t just build new connections, they’ll tear at existing ones as well. Germany becomes a linguistic island despite its powerhouse economy. Belgium will be rent in two as its French- and Flemish-speaking halves link with French and Dutch networks. The Muslim world will see increasing connection among its Arabic-speaking nations — Iraq, Syria, Egypt — and decreasing connections with its non-Arabic-speaking members. (Even the translation software being developed reflects
this bias: given the expense of developing translation software, only languages with millions of users — standard versions of English, French, Portuguese, Spanish, Italian, German — will make the cut.) And as we would expect of networks with different standards, gateways will arise; places where multi-lingual populations will smooth the transition between language networks. These gateways — Hong Kong, Brussels, New York, Delhi — will become economic centers in the 21st century because they were places where languages overlapped in the 19th.

There are all sorts of reasons why none of this should happen — why the Age of Empire shouldn’t be resurrected, why countries that didn’t export their language by force should suffer, why English shouldn’t become the Official Second Language of the 21st century — but none of those reasons will matter. We know from the 30-year history of the internet that when a new protocol is needed to continue internet growth, it’ll be acquired at any expense. What the internet economy demands more than anything right now is common linguistic standards. In the next 10 years, we will see the world’s languages sorted into two categories — those that form part of language networks will grow, and those that don’t will shrink, as the export of languages in the last century reshapes the map of the next one.

by Clay Shirky

Internet Use and National Identity

The effects of Internet adoption on national culture.

First published in FEED, 7/15/99.

The United Nations released its annual Human Development Report this week, including a section concerning the distribution of Internet use among the nations of the world. It painted a picture of massively unequal distribution, showing among other things that the United States has a hundred times more Internet users per capita than the Arab States, and that Europe has 70 times more users per capita than sub-Saharan Africa.
Surveying the adoption rates detailed in this report, anyone who has any contact with the Internet can only be left with one thought — “Well, duh.” There is some advantage to quantifying what is common knowledge, but the UN has muddied the issues here rather than clarifying them.

Is there really anybody who could be surprised that the country that invented the internet has more users per capita than Qatar? Is there really anyone who can work themselves up over the lack of MyYahoo accounts in nations that also lack clean water? The truth of the matter is that internet growth is not gradual, it is a phase change — when a country crosses some threshold of readiness, demand amongst its citizens explodes. Beneath that threshold, trying to introduce the internet by force is like pushing string — its is absurd to put internet access on the same plane as access to condoms and antibiotics.

Once a country reaches that threshold, though, there is one critical resource that drives internet adoption, and the UN desperately wants that resource to be money. Among the UN’s proposals is a “bit tax” (one penny per 100 emails) to build out telecommunications infrastructure in the developing world. While improving infrastructure is an admirable goal, it fudges the real issue: among countries who are ready for rapid internet adoption, the most important resource isn’t per capita income but per
capita freedom. Massive internet adoption of the sort the UN envisions will require an equally massive increase in political freedom, and the UN is in no position to say that part out loud.

The HDR report is hamstrung by the UN’s twin goals of advancing human rights and respecting national sovereignty. Where the internet is concerned, these goals are incompatible. The United Arab Emirates has a much better telecom infrastructure than Argentina, but a lower per capita use of the internet. Saudi Arabia has a higher per capita income than Spain but lower internet penetration. What Argentina has more of
than the UAE is neither infrastructure, nor money, but the right of the citizens to get information from a wide variety of sources, and their willingness to exercise that right. Among nations of relatively equal development, it will be the freer nations and not the richer ones that adopt the internet fastest.

The report addresses this issue by suggesting a toothless campaign to “…persuade national governments not to restrict access to the internet because of its tremendous potential for human development,” avoiding mentioning that the “potential for human development” is a death sentence for many of the world’s leaders. If the UN was serious
about driving internet adoption, the section on the internet would have started with the following declaration: “Attention all dictators: internet access is the last stop for your regime. You can try to pull into the station gradually, as China and Kuwait are trying to do, or you can wait to see what happens when you plow into the wall at full speed, like North Korea and Kenya, but the one thing you can’t do is keep going full steam ahead. Enjoy your ride.”

by Clay Shirky

Citizens and Customers

National Identity vs. Economic Identity.

6/15/1999

All countries are different; all customers are the same. That’s the lesson to be
learned from Freeserve ISP’s meteoric rise, and the subsequent reshaping of the UK
internet industry. Prior to Freeserve, the British adoption rate of the internet was
fairly sluggish, but Freeserve figured out how to offer free internet access by
subsidizing its service with for-fee tech support and a cut of local call revenues,
and in the six months since they’ve launched (and spawned over 50 copycat services),
the UK user base has grown from 6 to 10 million. Their main advantage over the other major ISP player, British Telecom, was the contempt BT has for the British public.

Wherever technology is concerned, there are a host of nationalistic prejudices: the
Americans are early adopters, for example, while the British are nation of shopkeepers, suspicious of technology and fearful of change. BT held this latter view, behaving as if Britain’s slow adoption of the internet was just another aspect of a national reticence about technology, and therefore treating the ISP business as an expensive service for elites rather than trying to roll it out cheaply to the masses.

This idea of national differences in the use of the internet is everywhere these days,
but this idea confuses content with form. There will be Czech content on the net, but
there won’t be a “Czech Way” of using the network, or a “Chinese Way” or a “Chilean Way.” The internet’s content is culturally determined, but its form is shaped by economics. Once a country gets sufficiently wired, the economic force of the internet has little to do with ethnicity or national sentiment and much to do with the unsurprising fact that given two offers of equal value, people all over the world will take the cheaper one, no matter who is offering it to them.

Unsurprising to consumers, that is; businesses all over the world are desperate to
convince themselves that national identity matters more than quality and price. (Remember the “Buy American” campaign that tried to get Americans to pay more for inferior cars? or the suggestion that corrupt business practices were part of “Asian Values”?) Freeserve’s genius was not to be swayed by the caricature of stodgy, technophobic Brits. British reticence about the internet turned out to be about price and not about national character at all — now that internet access has come in line with ordinary incomes, the British have been as keen to get connected as Americans are.

Patriotism is the last refuge of an unprofitable business. We’ve seen the internet take
off in enough countries to have some idea of the necessary preconditions: when a
literate population has phones at home, cheap PCs, and competitive telecom businesses, the value of connecting to the internet rises continually while the cost of doing so falls. In these countries, any business that expects national identity to provide some defense against competition is merely using a flag as a fig leaf. In the end, countries with wired populations will see national differences reduced in importance to the level of the Local Dish and Colorful Garb, because once a country passes some tipping point, its population starts behaving less like citizens of a particular place and more like global customers, making the same demands made by customers everywhere. Businesses that fill those demands, regardless of nationality, will thrive, and businesses that ignore those demands, regardless of nationality, will die.

by Clay Shirky

Why Smart Agents Are A Dumb Idea

“Smart Agents” are the wrong solution to the problem of info-liquidity.

Smart agents are a dumb idea. Like several of the other undead ideas floating around (e.g Digital Cash, Videophones), the idea of having autonomous digital agents that scour the net acting on your behalf seems so attractive that, despite a string of failures, agents enjoy periodic resurgances of interest. A new such surge seems to be beginning, with another round of stories in the press about how autonomous agents
equipped with instructions from you (and your credit card number) are going to shop for your CDs, buy and sell your stocks, and arrange your travel plans. The primary thing smart agents seem to have going for them is the ‘cool’ factor (as in ‘This will work because it would be cool if it did.’) The primary thing they have going against them is that they do not work and they never will work, and not just because they are
impractical, but because they have the wrong problems in their domain, and they solve them in the wrong order.

Smart agents — web crawling agents as opposed to stored preferences in
a database — have three things going against them:

  • Agents’ performance degrades with network growth
  • Agents ask people to do what machines are good at (waiting) and machines to do what people are good at (thinking).
  • Agents make the market for information less efficient rather than more

These three barriers render the idea of agents impractical for almost all of the duties they are supposedly going to perform.

Consider these problems in context; the classic scenario for the mobile agent is the business trip. You have business in Paris (or, more likely, Peoria) and you need a flight, a hotel and a rental car. You instruct your agent about your dates, preferences, and price limits, and it scours the network for you putting together the ideal package based on its interpretatino of your instructions. Once it has secured this package, it makes the purchases on your behalf, and presents you with the completed travel package, dates, times and confirmation numbers in one fell swoop.

A scenario like this requires a good deal of hand waving to make it seem viable, to say nothing of worthwhile, because it assumes that the agent’s time is more valuable than your time. Place that scenario in a real world context – your boss tells you you need to be in Paris (Peoria) at the end of the week, and could you make the arrangements before you go home? You fire up your trusty agent, and run into the
following problems:

  • Agents’ performance degrades with network growth

Upon being given its charge, the agent needs to go out and query all the available sources of travel information, issue the relevant query, digest the returned information and then run the necessary weighting of the results in real time. This is like going to Lycos and asking it to find all the resources related to Unix and then having it start indexing the Web. Forget leaving your computer to make a pot of coffee – you could leave your computer and make a ship in a bottle.

One of the critical weaknesses in the idea of mobile agents is that the time taken to run a query improves with processor speed (~2x every 18 months) but degrades with the amount of data to be searched (~2x every 4 months). A back of the envelope calculation comparing Moore’s Law vs. traffic patterns at public internet interconnect points suggests that an autonomous agent’s performance for real-time requests should suffer by roughly an order of magnitude annually. Even if you make optimistic assumptions about algorithm design and multi-threading and assume that
data sources are always accessible, mere network latency in an exploding number sources prohibits real-time queries. The right way to handle this problem is the mundane way – gather and index the material to be queried in advance.

  • Agents ask people to do what machines are good at (waiting) and machines to do what people are good at (thinking).

The usual answer to this problem with real-time queries is to assume that people are happy to ask a question hours or days in advance of needing the answer, a scenario that occurs with a frequency of approximately never. People ask questions when they want to know the answer – if they wanted the answer later they would have asked the
question later. Agents thus reverse the appropriate division of labor between humans and computers — in the agent scenario above, humans do the waiting while agents do the thinking. The humans are required to state the problem in terms rigorous enough to be acted on by a machine, and be willing to wait for the answer while the machine applies the heuristics. This is in keeping with the Central Dream of AI, namely that humans can be relegated to a check-off function after the machines have done the thinking.

As attractive as this dream might be, it is far from the realm of the possible. When you can have an agent which understands why 8 hours between trains in Paris is better than 4 hours between trains in Frankfurt but 8 hours in Peoria is worse than 4 hours in Fargo, then you can let it do all the work for you, but until then the final step in the process is going to take place in your neural network, not your agent’s.

  • Agents make the market for information less efficient

This is the biggest problem of all – agents rely on a wrong abstraction of the world. In the agent’s world, their particular owner is at the center, and there are a huge number of heterogenous data sources scattered all around, and one agent makes thousands of queries outwards to perform one task. This ignores the fact that the data is neither static nor insensitive to the agent’s request. The agent is not just
importing information about supply, it is exporting information about demand at the same time, thus changing the very market conditions it is trying to record. The price of a Beanie Baby rises as demand rises since Beanie Babies are an (artificially) limited resource, while the price of bestsellers falls with demand, since bookstores can charge lower prices in return for higher volume. Airline prices are updated thousands of times a day, currency exchange rates are updated tens of thousands of times a day. Net-crawling agents are completely unable to deal with markets for information like these; these kind of problems require the structured data to be at the center, and for a huge number of heterogenous queries to be made inwards towards the centralized data, so that information about supply and demand are all captured in one place, something no autonomous agent can do.

Enter The Big Fat Webserver

So much of the history of the Internet, and particularly of the Web, has been about decentralization that the idea of distributing processes has become almost reflexive. Because the first decade of the Web has relied on PCs, which are by their very nature decentralized, it is hard to see that much of the Web’s effect has been in the opposite direction, towards centralization, and centralization of a particular kind – market-making.

The alternative for the autonomous mobile agent is the Big Fat Webserver, and while its superiority as a solution has often been overlooked next to the sexier idea of smart agents, B.F.Ws are A Good Thing for the same reasons markets are A Good Thing – they are the best way of matching supply with demand in real time. What you would really do when Paris (Peoria) beckons is go to Travelocity or some similar
B.F.W. for travel planning. Travelocity runs on that unsexiest of hardware (the mainframe) in that unsexiest of architectures (centralized) and because of that, it works well everywhere the agent scenario works badly. You log into Travelocity and ask it a question about plane flights, get an answer right then, and decide.

B.F.Ws performance scales with database size, not network size

The most important advantage of B.F.Ws over agents is that BFWs acquire and structure the data before a request comes in. Net-crawling agents are asked to identify sources, gather data and then query the results all at once, even though these functions require completely different strategies. By gathering and structuring data in advance, B.F.W.s remove the two biggest obstacles to agent performance before any request is issued.

B.F.W.s let computer do what computers are good at (gathering, indexing) and people do what people are good at (querying, deciding).

Propaganda to the contrary, when given a result set of sufficiently narrow range (a dozen items, give or take), humans are far better at choosing between different options than agents are. B.F.W.s provide the appropriate division of labor, letting the machine do the coarse-grained sorting, which has mostly to do with excluding the worst options, while letting the humans make the fine-grained choices at the end.

B.F.W.s make markets

This is the biggest advantage of B.F.W.s over agents — databases open to heterogenous requests are markets for information. Information about supply and demand are handled at the same time, and the transaction takes place as close to real time as database processing plus network latency can allow.

For the next few years, B.F.W.s are going to be a growth area. They solve the problems previously thought to be in the agents domain, and they solve them better than agents every could. Where the agents make the assumption of a human in the center, facing outward to a heterogenous collection of data which can be gathered asynchronously, B.F.Ws make the asusmption that is more in line with markets (and
reality) – a source of data (a market, really) in the center, with a collection of humans facing inwards and making requests in real time. Until someone finda abetter method of matching supply with demand than real-time markets, B.F.W.s are a better answer than agents every time.

by Clay Shirky

Pretend vs. Real Economy

The difference between the stock market and the economy.

First published in FEED, 06/99.

The Internet happened to Merrill Lynch last week, and it cost them a couple billion dollars — when Merrill announced its plans to open an online brokerage after years of deriding the idea, its stock price promptly fell by a tenth, wiping out $2 billion in its market capitalization. The internet’s been happening like that to a lot of companies lately — Barnes and Noble’s internet stock is well below its recent launch price, Barry Diller’s company had to drop its Lycos acquisition because of damage to the stock prices of both companies, and both Borders and Compaq dumped their CEOs after it became clear that they were losing internet market share. In all of these cases, those involved learned the hard way that the internet is a destroyer of net value for traditional businesses because the internet economy is fundamentally at odds with the market for internet stocks.

The internet that the stock market has been so in love with (call it the “Pretend Internet” for short) is all upside — it enables companies to cut costs and compete without respect to geography. The internet that affects the way existing goods and services are sold, on the other hand (call it the “Real Internet”), forces companies to cut profit margins, and exposes them to competitors without respect to geography. On the Pretend Internet, new products will pave the way for enormous profitability arising from unspecified revenue streams. Meanwhile, on the Real Internet, prices have fallen and they can’t get up. There is a rift here, and its fault line appears wherever offline companies like Merrill tie their stock to their internet offerings. Merrill currently pockets a hundred bucks every time it executes a trade, and when investors see that Merrill online is only charging $30 a trade, they see a serious loss of revenue. When they go on to notice that $30 is something like three times the going rate for an internet stock trade, they see more than loss of revenue, they see loss of value. When a company can cut its prices 70% and still be three times as expensive as its competitors, something has to give. Usually that is the company’s stock price.

The internet is the locus of the future economy, and its effect is the wholesale transfer of information and choice (read: power and leverage) from producer to consumer. Producers (and the stock market) prefer one-of-a-kind businesses who can force their customers to accept continual price increases for the same products. Consumers, on the other hand, prefer commodity businesses where prices start low and keep falling. On the internet, consumers have the upper hand, and as a result, anybody who profited from offline inefficiencies — it used to be hard work to distribute new information to thousands of people every day, for example — are going to see much of their revenue destroyed with no immediate replacement in sight.

This is not to say that the internet produces no new value — on the contrary, it produces enormous value every day. Its just that most of the value is concentrated in the hands of the consumer. Every time someone uses the net to shop on price (cars, plane tickets, computers, stock trades) the money they didn’t spend is now available for other things. The economy grows even as profit margins shrink. In the end, this is what Merrill’s missing market cap tells us — the internet is now a necessity, but there’s no way to use the internet without embracing consumer power, and any business which profits from inefficiency is going to find this embrace more constricting than comforting. The effects of easy price comparison and global reach are going to wring inefficiency (read: profits) out of the economy like a damp dishrag, and as the market comes to terms with this equation between consumer power and lower profit margins, $2 billion of missing value is going to seem like a drop in the bucket.

by Clay Shirky

Who Are You Paying When You’re Paying Attention?

Converting attention to cash.

First published in ACM, 06/99.

Two columns ago, in “Help, the Price of Information Has Fallen and It Can’t Get Up”, I argued that traditional pricing models for informational goods (goods that can theoretically be transmitted as pure data – plane tickets, stock quotes, classified ads) fall apart on the net because so much of what’s actually being paid for when this data is distributed is not the content itself, but its packaging, storage and transportation. This content is distributed either as physical packages, like books or newspapers, or on closed (pronounced ‘expensive’) networks, like Lexis/Nexis or stock tickers, and its cost reflects both these production and distribution expenses and the scaricty that is created when only a few companies can afford to produce and distribute said content. 

The net destroys both those effects, first by removing the need for either printing and distributing physical objects (online newspapers, e-tickets and electronic greeting ‘cards’ are all effortless to distribute relative to their physical counterparts) and by removing many of the barriers to distribution (only a company with access to a printing press can sell classified ads offline, but on the network all it takes is a well-trafficked site), so that many more companies can compete with one another. 

The net effect of all this, pun intended, is to remove the ability to charge direct user fees for many kinds of online content which people are willing to shell out for offline. This does not mean that the net is valueless, however, or that users can’t be asked to pay for content delivered over the Internet. In fact, most users willingly pay for content now. The only hitch is that what they’re paying isn’t money. They’re paying attention. 

THE CURRENCY EXCHANGE MODEL

Much of the current debate surrounding charging user fees on the Internet assumes that content made available over the network follows (or should follow) the model used in the print world – ask users to pay directly for some physical object which contains the content. In some cases, the whole cost of the object is borne by the users, as with books, and in other cases users are simply subsidizing the part of the cost not paid for by advertisements, as with newspapers and magazines. There is, however, another model, one more in line with the things the net does well, where the user pays no direct fees but the providers of the content still get paid – the television model. 

TV networks are like those currency exchange booths for tourists. People pay attention to the TV, and the networks collect this attention and convert it into money at agreed upon rates by supplying it in bulk to their advertisers, generally by calculating the cost to the advertiser of reaching a thousand viewers. The user exchanges their attention in return for the content, and the TV networks exchange this attention for income. These exchange rates rise and fall just like currency markets, based on the perceived value of audience attention and the amount of available cash from the advertiser. 

This model, which generates income by making content widely available over open networks without charging user fees, is usually called ‘ad-supported content’, and it is currently very much in disfavor on the Internet. I believe however, that not only can ad-supported content work on the Internet, I believe it can’t not work. It’s success is guaranteed by the net’s very makeup – the net is simply too good at gathering communities of interest, too good at freely distributing content, and too lousy at keeping anything locked inside subscription networks, for it to fail. Like TV, the net is better at getting people to pay attention than anything else. 

OK SHERLOCK, SO IF THE IDEA OF MAKING MONEY ON THE INTERNET BY CONVERTING ATTENTION INTO INCOME IS SO BRILLIANT, HOW COME EVERYONE ELSE THINKS YOU’RE WRONG?

Its a question of scale and time horizons. 

One of the reasons for the skepticism about applying the TV model to the Internet is the enormous gulf between the two media. This is reflected in both the relative sizes of their audiences and the incomes of those businesses – TV is the quintessential mass medium, commanding tens of millions more viewers than the net does. TV dwarfs the net in both popularity and income. 

Skeptics eyeing the new media landscape often ask “The Internet is fine as a toy, but when will it be like TV?” By this they generally mean ‘When will the net have a TV-style audience with TV-style profits?’ The question “When will the net be like TV” is easy to answer – ‘Never’. The more interesting question is when will TV be like the net, and the answer is “Sooner than you think”. 

A BRIEF DIGRESSION INTO THE BAD OLD DAYS OF NETWORK TV

Many people have written about the differences between the net and television, usually focussing on the difference between broadcast models and packet switched models like multicasting and narrowcasting, but these analyses, while important, overlook one of the principal differences between the two media. The thing that turned TV into the behemoth we know today isn’t broadcast technology but scarcity. 

From the mid-1950s to the mid-1980s, the US national TV networks operated at an artificially high profit. Because the airwaves were deemed a public good, their use was heavily regulated by the FCC, and as a consequence only three companies got to play on a national level. With this FCC-managed scarcity in place, the law of supply and demand worked in the TV networks favor in ways that most industries can only dream of – they had their own private government created and regulated cartel. 

It is difficult to overstate the effect this had on the medium. With just three players, a TV show of merely average popularity would get a third of the available audience, so all the networks were locked in a decades long three-way race for the attention of the ‘average’ viewer. Any business which can get the attention of a third of its 100 million+ strong audience by producing a run-of-the-mill product, while being freed for any other sort of competition by the government, has a license to print money and a barrel of free ink. 

SO WHAT HAPPENED?

Cable happened – the TV universe has been slowly fracturing for the last 20 years or so, with the last 5 years seeing especially sharp movement. With a growing competition from cable (and later sattelite, microwave, a 4th US TV network, and most recently the Internet draining TV watching time), the ability of television to command a vast audience with average work has suffered badly. The two most popular US shows of this year each struggle to get the attention of a fifth of the possible viewers, called a “20 share’ in TV parlance, where 20 is the percentage of the possible audience tuning in. 

The TV networks used to cancel shows with a 20 share, and now that’s the best they can hope for from their most popular shows, and its only going to get worse. As you might imagine, this has played hell with the attention-to-cash conversion machine. When the goal was a creating multiplicity of shows for the ‘average’ viewer, pure volume was good, but in the days of the Wind-Chill Channel and the Abe Vigoda Channel, the networks have had to turn to audience segmentation, to not just counting numbers but counting numbers of women, or teenagers, or Californians, or gardeners, who are watching certain programs. 

The TV world has gone from three channels of pure mass entertainment to tens or even hundreds of interest-specific channels, with attention being converted to cash based not solely on the total number people they attract, but also on how many people with specific interests, or needs, or characteristics. 

Starting to sound a bit like the Web, isn’t it? 

THE TV PEOPLE ARE GOING TO RUE THE DAY THEY EVER HEARD THE WORD ‘DIGITAL’

All this is bad enough from the TV networks point of view, but its a mere scherzo compared to the coming effects of digitality. As I said in an earlier column, apropos CD-ROMs, “Information has been decoupled from objects. Forever.”, and this is starting to be true of information and any form of delivery. A TV is like a book in that it is both the mechanism of distribution and of display – the receiver, decoder and screen travel together. Once television becomes digital, this is over, as any digital content can be delivered over any digital medium. “I just saw an amazing thing on 60 Minutes – here, I’ll mail it to you”, “Whats the URL for ER again?”, “Once everybody’s here, we’ll start streaming Titanic”. Digital Baywatch plus frame relay is the end of ‘appointment TV’. 

Now I am not saying that the net will surpass TV in size of audience or income anytime soon, or that the net’s structure, as is, is suitable for TV content, as is. I am saying that the net’s method of turning attention into income, by letting audience members select what they’re intersted in and when, where and how to view it, is superior to TV’s, and I am saying that as the net’s bandwidth and quality of service increases and television digitizes, many of the advantages TV had move over to the network. 

The Internet is a massive medium, but it is not a mass medium, and this gives it an edge as the scarcity that TV has lived on begins to seriously erode. The fluidity with which the net apportions content to those interested in it without wasting the time of those not interested in it makes it much more suited in the long run for competing for attention in the increasingly fractured environment for television programming, or for any content delivery for that matter. 

In fact, most of the experiments with TV in the last decade – high-definition digital content, interactive shopping and gaming, community organization, and the evergreen ‘video on demand’ – are all things that can be better accomplished by a high bandwidth packet switched network than by traditional TV broadcast signals. 

In the same way that ATT held onto over 2/3rds of its long-distance market for a decade after the breakup only to see it quickly fall below 50% in the last three years, the big 3 TV networks have been coasting on that same kind of intertia. Only recently, prodded by cable and the net, is the sense memory of scarcity starting to fade, and in its place is arising a welter of competition for attention, one that the Internet is poised to profit from enormously. A publicly accessible two-way network that can accomodate both push and pull and can transmit digital content with little regard to protocol has a lot of advantages over TV as an ‘attention to income’ converter, and in the next few years those advantages will make themselves felt. I’m not going to bet on when overall net income surpasses overall TV income, but in an arena where paying attention is the coin of the realm, the net has a natural edge, and I feel confident in predicting that revenues from content will continue to double annually on the net for the forseeable future, while network TV will begin to stagnate, caught flat in a future that looks more like the Internet than it does like network TV.

by Clay Shirky

An Open Letter to Jakob Nielsen

An argument against design standards on the web.

First published on ACM, 06/99.

[For those not subscribing to CACM, Jakob Nielsen and I have come down on opposite sides of a usability debate. Jakob believes that the prevalence of bad design on the Web is an indication that the current method of designing Web sites is not working and should be replaced or augmented with a single set of design conventions. I believe that the Web is an adaptive system and that the prevalence of bad design is how evolving systems work.

Jakob’s ideas are laid out in “User Interface Directions for the Web”, CACM, January 1999.

My ideas are laid out in “View Source… Lessons from the Web’s Massively Parallel Development”, networker, December 1998, and http://www.shirky.com/writings/view-source

Further dialogue between the two of us is in the Letters section of the March 1999 CACM.]

Jakob,

I read your response to my CACM letter with great interest, and while I still disagree, I think I better understand the disagreement, an will try to set out my side of the argument in this letter. Let me preface all of this by noting what we agree on: the Web is host to some hideous dreck; things would be better for users if Web designers made usability more of a priority; and there are some basics of interface usability that one violates at one’s peril.

Where we disagree, however, is on both attitude and method – for you, every Web site is a piece of software first and foremost, and therefore in need of a uniform set of UI conventions, while for me, a Web site’s function is something only determined by its designers and users – function is as function does. I think it presumptous to force a third party into that equation, no matter how much more “efficient” that would make things.

You despair of any systemic fix for poor design and so want some sort of enforcement mechanism for these external standards. I believe that the Web is an adaptive system, and that what you deride as “Digital Darwinism” is what I would call a “Market for Quality”. Most importantly, I believe that a market for quality is in fact the correct solution for creating steady improvements in the Web’s usability.

Let me quickly address the least interesting objection to your idea: it is unworkable. Your plan requires both centralization and force of a sort it is impossible to acheive on the Internet. You say

“…to ensure interaction consistency across all sites it will be necessary to promote a single set of design conventions.”

and

“…the main problem lies in getting Web sites to actually obey any usability rules.”

but you never address who you are proposing to put in the driver’s seat – “it will be necessary” for whom? “[T]he main problem” is a problem for whom? Not for me – I am relieved that there is no authority who can make web site designers “obey” anything other httpd header validity. That strikes me as the Web’s saving grace. With the Web poised to go from 4 million sites to 100 million in the next few years, as you note in your article, the idea of enforcing usability rules will never get past the “thought experiment” stage.

However, as you are not merely a man of action but also a theorist, I want to address why I think enforced conformity to usability standards is wrong, even in theory. My objections break out into three rough categories: creating a market for usability is better than central standards for reasons of efficency, innovation, and morality.

EFFICENCY

In your letter, you say “Why go all the way to shipping products only to have to throw away 99% of the work?” I assume that you meant this as a rhetorical question – after all, how could anybody be stupid enough to suggest that a 1% solution is good enough? The Nielsen Solution – redesign for everybody not presently complying with a “single set of design conventions” – takes care of 100% of the problem, while the Shirky Solution, let’s call it evolutionary progress for the top 1% of sites, well what could that possibly get you?

1% gets you a surprising amount, actually, if it’s properly arranged.

If only the top 1% most trafficked Web sites make usability a priority, those sites would nevertheless account for 70% of all Web traffic. You will recognize this as your own conclusion, of course, as you have suggested on UseIt (http://www.useit.com/alertbox/9704b.html) that Web site traffic is roughly a Zipf distribution, where the thousandth most popular site only sees 1/1000th the traffic of the most popular site. This in turn means that a very small percentage of the Web gets the lion’s share of the traffic. If you are right, then you do not need good design on 70% of the web sites to cover 70% of user traffic, you only need good design on the top 1% of web sites to reach 70% of the traffic.

By ignoring the mass of low traffic sites and instead concentrating on making the popular sites more usable and the usable sites more popular, a market for quality is a more efficient way of improving the Web than trying to raise quality across the board without regard to user interest.

INNOVATION

A market for usability is also better for fostering innovation. As I argue in “View Source…”, good tools let designers do stupid things. This saves overhead on the design of the tools, since they only need to concern themselves with structural validity, and can avoid building in complex heuristics of quality or style. In HTML’s case, if it renders, it’s right, even if it’s blinking yellow text on a leopard-skin background. (This is roughly the equivalent of letting the reference C compiler settle arguments about syntax – if it compiles, it’s correct.)

Consider the use of HTML headers and tables as layout tools. When these practices appeared, in 1994 and 1995 respectively, they infuriated partisans of the SGML ‘descriptive language’ camp who insisted that HTML documents should contain only semantic descriptions and remain absolutely mute about layout. This in turn led to white-hot flamefests about how HTML ‘should’ and ‘shouldn’t’ be used.

It seems obvious from the hindsight of 1999, but it is worth repeating: Everyone who argued that HTML shouldn’t be used as a layout language was wrong. The narrowly correct answer, that SGML was designed as a semantic language, lost out to the need of designers to work visually, and they were able to override partisan notions of correctness to get there. The wrong answer from a standards point of view was nevertheless the right thing to do.

Enforcing any set of rules limits the universe of possibilities, no matter how well intentioned or universal those rules seem. Rules which raise the average quality by limiting the worst excesses risk ruling out the most innovative experiments as well by insisting on a set of givens. Letting the market separate good from bad leaves the door open to these innovations.

MORALITY

This is the most serious objection to your suggestion that standards of usability should be enforced. A web site is an implicit contract between two and only two parties – designer and user. No one – not you, not Don Norman, not anyone, has any right to enter into that contract without being invited in, no matter how valuable you think your contribution might be.

IN PRAISE OF EVOLVEABLE SYSTEMS, REDUX

I believe that the Web is already a market for quality – switching costs are low, word of mouth effects are both large and swift, and redesign is relatively painless compared to most software interfaces. If I design a usable site, I will get more repeat business than if I don’t. If my competitor launches a more usable site, it’s only a click
away. No one who has seen the development of Barnes and Noble and Amazon or Travelocity and Expedia can doubt that competition helps keep sites focussed on improving usability. Nevertheless, as I am a man of action and not just a theorist, I am going to suggest a practical way to improve the workings of this market for usability –
lets call it usable.lycos.com.

The way to allocate resources efficently in a market with many sellers (sites) and many buyers (users) is competition, not standards. Other things being equal, users will prefer a more usable site over its less usable competition. Meanwhile, site owners prefer more traffic to less, and more repeat visits to fewer. Imagine a search engine that weighted usability in its rankings, where users knew that a good way to find a usable site was by checking the “Weight Results by Usability” box and owners knew that a site could rise in the list by offering a good user experience. In this environment, the premium for good UI would align the interests of buyers and sellers around increasing quality. There is no Commisar of Web Design here, no International Bureau of Markup Standards, just an implicit and ongoing compact between users and designers that improvement will be rewarded.

The same effect could be created in other ways – a Nielsen/Norman “Seal of Approval”, a “Usability” category at the various Web awards ceremonies, a “Usable Web Ring”. As anyone who has seen “Hamster Dance” or an emailed list of office jokes can tell you, the net is the most efficent medium the world has ever known for turning user preference into widespread awareness. Improving the market for quality simply harnesses that effect.

Web environments like usable.lycos.com, with all parties maximizing preferences, will be more efficent and less innovation-dampening than the centralized control which would be necessary to enforce a single set of standards. Furthermore, the virtues of such a decentralized system mirrors the virtues of the Internet itself rather than fighting them. I once did a usability analysis on a commerce site which had fairly ugly graphics but a good UI nevertheless. When I queried the site’s owner about his design process, he said “I didn’t know anything when I started out, so I just put up a site with an email link on every page, and my customers would mail me suggestions.”

The Web is a marvelous thing, as is. There is a dream dreamed by engineers and designers everywhere that they will someday be put in charge, and that their rigorous vision for the world will finally overcome the mediocrity around them once and for all. Resist this idea – the world does not work that way, and the dream of centralized control is only pleasant for the dreamer. The Internet’s ability to be adapted slowly, imperfectly, and in many conflicting directions all at once is precisely what makes it so powerful, and the Web has taken those advantages and opened them up to people who don’t know source code from bar code by creating a simple interface design language.

The obvious short term effect of this has been the creation of an ocean of bad design, but the long term effects will be different – over time bad sites die and good sites get better, so while those short-term advantages seem tempting, we would do well to remember that there is rarely any profit in betting against the power of the marketplace in the long haul.