10/16/2000

In the aftermath of LinuxWorld, the open source conference that took place in San
Jose, Calif., in August, we’re now being treated with press releases announcing Linux
as Almost Ready for the Desktop.

It is not.

Even if Linux were to achieve double-digit penetration among the world’s PC users, it
would be little more than an also-ran desktop OS. For Linux, the real action is
elsewhere. If you want to understand why Linux is the most important operating system in the world, ignore the posturing about Linux on the desktop, and pay attention to the fact that IBM has just ported Linux to a wristwatch, because that is the kind of news that illustrates Linux’s real strengths.

At first glance, Linux on a wristwatch seems little more than a gimmick–cellphone
displays and keypads seem luxurious by comparison, and a wristwatch that requires you to type “date” at the prompt doesn’t seem like much of an upgrade. The real import of the Linux wristwatch is ecological, though, rather than practical, because it
illustrates Linux’s unparalleled ability to take advantage of something called
“adaptive radiation.”

Let’s radiate

Adaptive radiation is a biological term that describes the way organisms evolve to
take advantage of new environments. The most famous example is Darwin’s finches. A single species of finch blew off of the west coast of South America and landed on the Galapagos Islands, and as these birds took advantage of the new ecological niches offered by the islands, they evolved into several separate but closely related species.

Adaptive radiation requires new environments not already crowded with competitors and organisms adaptable enough to take advantage of those environments. So it is with Linux–after a decade of computers acting as either clients or servers, new classes of devices are now being invented almost weekly–phones, consoles, PDAs–and only Linux is adaptable enough to work on most of them.

In addition to servers and the occasional desktop, Linux is being modified for use in
game machines (Indrema), Internet appliances (iOpener, IAN), handhelds (Yopy, iPAQ), mainframes (S/390), supercomputers (Los Lobos, a Beowulf cluster), phones (Japan Embedded Linux Consortium), digital VCRs (TiVO), and, of course, wristwatches. Although Linux faces fierce competition in each of these categories, no single competitor covers every one. Furthermore, given that each successful porting effort increases Linux’s overall plasticity, the gap between Linux’s diversity and that of its competitors will almost inevitably increase.

Where ‘good’ beats ‘best’

In a multidevice world, the kernel matters more than the interface. Many commentators (including Microsoft) have suggested that Linux will challenge Microsoft’s desktop monopoly, and among this camp it is an article of faith that one of the things holding Linux back is its lack of a single standardized interface. This is not merely wrong, it’s backward–the fact that Linux refuses to constrain the types of interfaces that are wrapped around the kernel is precisely what makes Linux so valuable to the individuals and companies adapting it for new uses. (The corollary is also true–Microsoft’s attempt to simply repackage the Windows interface for PDAs rendered early versions of WinCE unusable.)

Another lesson is that being merely good enough has better characteristics for adaptive
radiation, and therefore for long-term survival, than being Best of Breed.

Linux is not optimized for any particular use, and it is improved in many small
increments rather than large redesigns. Therefore, the chances that Linux will become a better high-availability server OS than Solaris, say, in the next few years, is tiny. Although not ideal, Linux is quite a good server, whereas Solaris is unusable for game consoles, digital VCRs, or wristwatches. This will keep Linux out of the best of breed competition because it is never perfectly tailored to any particular environment, but it also means that Linux avoids the best of breed trap. For any given purpose, best of breed products are either ideal or useless. Linux’s ability to adapt to an astonishing array of applications means that the chances of it being able to run on any new class of device are superior to a best of breed product.

The real action

The immediate benefits of Linux’s adaptive radiation ability are obvious to the Linux
community. Since nothing succeeds like success, every new porting effort increases both the engineering talent pool and the available code base. The potential long-term benefit, though, is even greater. If a Linux kernel makes interoperation easier, each new Linux device can potentially accelerate a network effect, driving Linux adoption still faster.

This is not to say that Linux will someday take over everything, or even a large subset
of everything. There will always be a place for “Best of Breed” software, and Linux’s
use of open protocols means its advantage is always in ease of use, never in locking out the competition. Nevertheless, only Linux is in a position to become ubiquitous across most kinds of devices. Pay no attention to the desktop sideshow–in the operating system world, the real action in the next couple of years is in adaptive radiation.

First published on Biz2.0, 09/00.

The Internet is a wonderful source of technical jargon and a bubbling cauldron of alphabet soup. FTP, TCP, DSL, and a host of additional TLAs (three-letter acronyms) litter the speech of engineers and programmers. Every now and then, however, one of those bits of jargon breaks away, leaving the world of geek-speak to become that most sought-after of technological developments: a Magic Problem Solver.

A Magic Problem Solver is technology that non-technologists believe can dissolve stubborn problems on contact. Just sprinkle a little Java or ODBC or clustering onto your product or service, and, voila, problems evaporate. The downside to Magic Problem Solvers is that they never work as advertised. In fact, the unrealistic expectations created by asserting that a technology is a Magic Problem Solver may damage its real technological value: Java, for example, has succeeded far beyond any realistic expectations, but it hasn’t succeeded beyond the unrealistic expectations it spurred early on.

Today’s silver bullet

The Magic Problem Solver du jour is XML, or Extensible Markup Language, a system for describing arbitrary data. Among people who know nothing about software engineering, XML is the most popular technology since Java. This is a shame since, although it really is wonderful, it won’t solve half the problems people think it will. Worse, if it continues to be presented as a Magic Problem Solver, it may not be able to live up to its actual (and considerably more modest) promise.

XML is being presented as the ideal solution for the problem of the age: interoperability. By asserting that their product or service uses XML, vendors everywhere are inviting clients to ignore the problems that arise from incompatible standards, devices, and formats, as if XML alone could act as a universal translator and future-proofer in the post-Babel world we inhabit.

The truth is much more mundane: XML is not a format, it is a way of making formats, a set of rules for making sets of rules. With XML, you can create ways to describe Web-accessible resources using RDF (Resource Description Framework), syndicated content using ICE (Information Content Exchange), or even customer leads for the auto industry using ADF (Auto-lead Data Format). (Readers may be led to believe that XML is also a TLA that generates additional TLAs.)

Notice, however, that using XML as a format-describing language does not guarantee that the result will be well designed (XML is no more resistant to “Garbage In, Garbage Out” than any other technology), that it will be adopted industry-wide (ICE and RDF are overlapping attempts to describe types of Internet-accessible data), or even that the format is a good idea (Auto-lead Data Format?). If two industry groups settle on XML to design their respective formats, they’re no more automatically interoperable than are two languages that use the same alphabet–no more “interoperable,” for example, than are English and French.

Three sad truths

When it meets the real world, this vision of XML as a pain-free method of describing and working with data runs into some sad truths:

Sad XML Truth No. 1: Designing a good format using XML still requires human intelligence. The people selling XML as a tool that makes life easy are deluding their customers–good XML takes more work because it requires a rigorous description of the problem to be solved, and its much vaunted extensibility only works if the basic framework is sound.

Sad XML Truth No. 2: XML does not mean less pain. It does not remove the pain of having to describe your data; it simply front-loads the pain where it’s easier to see and deal with. The payoff only comes if XML is rolled out carefully enough at the start to lessen day-to-day difficulties once the system is up and running. Businesses that use XML thoughtlessly will face all of the upfront trouble of implementing XML, plus all of the day-to-day annoyances that result from improperly described data.

Sad XML Truth No. 3: Interoperability isn’t an engineering issue, it’s a business issue. Creating the Web — HTTP plus HTML — was probably the last instance where standards of global importance were designed and implemented without commercial interference. Standards have become too important as competitive tools to leave them where they belong, in the hands of engineers. Incompatibility doesn’t exist because companies can’t figure out how to cooperate with one another. It exists because they don’t want to cooperate with one another.

XML will not solve the interoperability problem because the difficulties faced by those hoping to design a single standard and the difficulties caused by the existence of competing standards have not gone away. The best XML can do is to ensure that data formats can be described with rigor by thoughtful and talented people capable of successfully completing the job, and that the standard the market selects can easily be spread, understood, and adopted. XML doesn’t replace standards competition, in other words, but if it is widely used it might at least allow for better refereeing and more decisive victories. On the other hand, if XML is oversold as a Magic Problem Solver, it might fall victim to unrealistically high expectations, and even the modest improvement it promises will fail to materialize.

First published in FEED, 1/6/2000.

The Open Source movement got a Christmas present at the end of 1999, some assembly required. John Carmack of id software, arguably the greatest games programmer ever, released the source code for Quake, the wildly popular shoot-em-up. Quake, already several years old, has maintained popularity because it allows players to battle one another over the internet, with hundreds of servers hosting round-the-clock battles. It jives with the Open Source ethos because it has already benefitted enormously from player-created ‘mods’ (or modifications) to the game’s surface appearance. By opening up the source code to the game’s legion of fanatical players, id is hoping to spur a new round of innovation by allowing anyone interested in creating these modifications to be able to alter any aspect of the program, not just its surface. Within minutes of id’s announcement, gamers and hackers the world over were downloading the source code. Within hours they were compiling new versions of the game. Within days people began using their new knowledge of the games inner workings to cheat. A problem new to the Open Source movement began to surface: what to do when access to the source code opens it up to abuse.

Quake works as a multi-player game where each player has a version of Quake running on his or her own PC, and it is this local copy of the game that reports on the player’s behavior — running, shooting, hiding — to a central Quake server. This server then collates all the players’ behaviors and works out who’s killed whom. With access to the Quake source code, a tech-savvy player can put themselves on electronic steroids by altering their local version of the game to give themselves superhuman speed, accuracy, or force, simply by over-reporting their skill to the server. This would be like playing tennis against someone with an invisible racket a yard wide.

All of this matters much more than you would expect a game to matter. With Open Source now associated with truth, justice, and the Internet Way, and with Carmack revered as a genius and a hero, the idea that the combination of these two things could breed anything so mundane as cheating caught people by surprise. One school of thought has been simply to deny that there is a problem by noting that if Quake had been Open Source to begin with, this situation would never have arisen. This is true, as far as it goes, but a theory which doesn’t cover real world cases isn’t much use. id’s attempt to open the source for some of its products while keeping others closed is exactly the strategy players like Apple, IBM, and Sun are all testing out, and if the release of Quake fails to generate innovation, mere ideological purity will be cold comfort.

As so often in the digital world, what happens to the gaming industry has ramifications for the computing industry as a whole. Players in a game are simultaneously competing and co-operating, and all agree to abide by rules that sort winners from losers, a process with ramifications for online economics, education, even auctions. If Quake, with its enormous audience of tech-savvy players and its history of benefitting from user modifications, can’t make the transition from closed source to open source easily, then companies with a less loyal user base might think twice about opening their products, so id’s example is going to be watched very closely. The Quake release marks a watershed — if the people currently hard at work on the Quake cheating problem find a solution, it will be another Open Source triumph, but if they fail, the Quake release might be remembered as the moment that cheating robbed the Open Source movement of its aura of continuous progress.

First published in FEED, 10/7/1999.

If imitation is the sincerest form of flattery, the Open Source movement should be blushing from head to toe by now. Its most recent admirer is Sun Microsystems, who has just announced that it will make the source code to its Solaris operating system available under something it calls a “Community Source” license. In making such a bold move (Solaris is their core product) Sun is embracing everything that has made the Open Source movement such a success. Everything, that is, except that bit about opening up their source code.

Sun is attempting a quasi-Open Source move because while they want what Linux has (an army of talented developers working for free) they also want what Linux doesn’t have (commercial control, patents on intellectual property, and a steady stream of income). They are trying to split the difference by creating a license which only allows you access to Solaris source code if you promise not to make any money from it, and only if you agree to co-ordinate any changes you make with Sun. In order to stave off criticism of these restrictions (which don’t exist under real Open Source licenses), Sun has wrapped itself in the flag of community — after all, what could possibly be wrong with a “Community Source” license?

In net-speak, “community” no longer means “a gathering of likeminded people,” it is just a reductive variable written into every dot-com business plan. The problem with Sun’s “Community Source” is that Open Source software is not written by communities, it is written by individuals — thousands of individual developers, each sharing their work with the others. The difference is a subtle but important one; Sun is betting that Open Source software works as well as it does because of community feeling and collaboration, when in fact it is mostly driven by individual selfishness. In an Open Source project, new features come not from what a developer imagines some hypothetical client might possibly want to do someday, it comes from what the developer him or herself wants to do right now. Like a game of “Which of these things is not like the others?,” commercially developed software is starkly different from Open Source projects: Linux exists because Linus Torvalds wanted a Unix clone that ran on cheap hardware, Apache exists because Brian Behelendorf wanted a good web server, Perl exists because Larry Wall wanted to make writing reports easier, Solaris exists because Sun wanted to make money.

Most of the valuable aspects of open source software — its cleanliness in implementation, its compact size, its ability to run on cheap hardware — are a direct result of this selfishness, since developers are making the software they want to use. Like the paradox of the free market, where reducing central management increases economic efficiency, the paradox of Open Source is that by reducing commercial control, software can actually improve faster. What Sun doesn’t understand is that this core aspect of Open Source is indivisible: The thing that makes Linux so desirable for developers is the very thing that gives commercial software companies indigestion — it transfers control of the software to the individual. There’s no way to split the difference, because there’s no difference to split.

Sun’s license guts the very freedoms that drive open source adoption in the first place, because without real transfer of power, there is no incentive to use open source software. And this problem is not Sun’s alone — giants like Microsoft and Oracle will also need strategies for dealing with Open Source, and sooner rather than later. As with so much about the internet, Open Source is about removing the middleman and transferring control to the individual: Linux is the net’s first disintermediated operating system. No matter how many iterations of its “Community Source” license it goes through, in the end Sun is going to discover that the only way to get the advantages which the Open Source movement enjoys is to open its source.

First published in FEED, 8/18/1999.

As the software industry cracks down on its customers, the software itself is opening up. Open Source software, the freely available alternative to commercial software, is making inroads in the corporate world because of its superior flexibility, adaptability, and cost. Despite this competition, the commercial software industry has decided that now’s the time to make licensed software less flexible, less adaptable, and above all, more expensive. A proposed new law, called the Uniform Computer Information Transactions Act (UCITA) would give software manufacturers enormous new powers to raise prices and to control their software even after it is in the hands of their customers. By giving the software industry additional leverage over its customers, UCITA will have two principal effects: first, it will turn software from a product you buy to a service you pay for again and again. Second, its restrictions will greatly accelerate the corporate adoption of Open Source software.

All trade groups aspire to become like OPEC, the cartel that jacks up oil prices by controlling supply. The UCITA working group is no exception: UCITA is designed to allow software companies relief from competition (companies could forbid publishing the results of software comparisons), and to artificially limit supply (any company which was acquired by a larger company could be forced to re-license all its software). The most startling part of UCITA, though, is the smugly named “self-help” clause, which would allow a software company to remotely disable software at the customer’s site, even after it has been sold. This clause could be invoked with only 15 days notice if a software developer felt its licensing terms were being violated, making the customer guilty until proven innocent. UCITA’s proponents disingenuously suggest that the use of “self-help” would be rare, as it would make customers unhappy — what they are not as quick to point out is that the presence of “self-help” as a credible threat would give software companies permanent leverage over their customers in negotiating all future contracts.

Unfortunately for cartel-minded software firms, the OPEC scenario is elusive because software isn’t like oil. Software has no physical scarcity, and the people who know how to create good software can’t be isolated or controlled the way oil wells can. Where UCITA sets out to make software a controlled substance, the Open Source movement sets out to take advantage of software’s innate flexibility of distribution. By making software freely available and freely modifiable, Open Source takes advantage of everything UCITA would limit — Open Source software is easy to get, easy to modify, and easy to share. If UCITA becomes law, the difference between Open Source and commercial software will become even more stark. Expect Open Source to do very well in these circumstances.

Economics 101 tells us that people make economic decisions “on the margin” — a calculation not of total worth to total cost, but of additional worth for additional cost. For someone who wants a watch for telling time but not for status, for example, the choice between a $20 Timex and a $20,000 Rolex is clear — the $19,980 marginal cost of the Rolex knocks it out of the running. In the case of Open Source vs. commercial software, the differences in cost can be equally vast — in many cases (such as the Apache web server) the Open Source solution is both cheaper and better. Cartels only work if there is no competition, a fact the UCITA group seems not to have grasped. If UCITA becomes law — that could happen as soon as December — the commercial software industry will be sending its customers scrambling for Linux, Apache, and the other Open Source products faster than they already are.

9/8/1999

The news this week that a researcher has bred a genetically smarter mouse is another precursor to a new era of genetic manipulation. The experiment, performed by Dr. Joe Tsien of Princeton, stimulated a gene called NR2B, producing mice with greatly heightened intelligence and memory. The story has generated a great deal of interest, especially as human brains may have a similar genetic mechanism, but there is a behind- the-scenes story which better illuminates the likely shape of 21st century medicine. Almost from the moment that the experiment was publicized, a dispute broke out about ownership of NR2B itself, which is already patented by another lab. Like an information age land-grab, whole sections of genetic information are being locked behind pharmaceutical patents as fast as they are being identified. If the current system of private ownership of genes continues, the majority of human genes could be owned in less than two years. The only thing that could save us from this fate would be an open- source movement in genetics — a movement to open the DNA source code for the human genome as a vast public trust.

Fighting over mouse brains may seem picayune, but the larger issue is ownership of the genetic recipe for life. The case of the smarter mouse, patent pending, is not an
isolated one: among the patents granted for human genes are blindness (Axys
Pharmaceuticals), epilepsy (Progenitor), and Alzheimer’s (Glaxo Wellcome). Rights to the gene which controls arthritis will be worth millions, breast cancer, billions, and
the company that patents the genes that control weight loss can write their own ticket. As the genome code becomes an essential part of industries from bio-computing and cybernetic interfaces to cosmetics and nutrition, the social and economic changes it instigates are going to make the effects of the Internet seem inconsequential. Unfortunately for us, though, the Internet’s intellectual property is mostly in the public domain, while the genome is largely — and increasingly — private.

It didn’t have to be this way. Patents exist to encourage investment by guaranteeing a
discoverer time to recoup their investment before facing any competition, but patent
law has become increasingly permissive in what constitutes a patentable discovery. There are obviously patents to be had in methods of sequencing genes and for methods of using those sequences to cure disease or enhance capability. But to allow a gene itself to be patented makes a mockery of “prior art,” the term which covers unpatented but widely dispersed discoveries. It is prior art which keeps anyone from patenting fire, or the wheel, and in the case of genetic information, life itself is the prior art. It is a travesty of patent law that someone can have the gene for Alzheimer’s in every cell of their body, but that the patent for that gene is owned by Glaxo Wellcome.

The real action, of course, is not in mouse brains but in the human genome. Two teams, one public and one private, are working feverishly to sequence all of the 100,000 or so genes which lie within the 23 pairs of human chromosomes. The public consortium aims to release the sequence into the public domain, while the private group aims to patent much of the genome, especially the valuable list of mutations that cause genetic disease. The competition between these two groups has vastly accelerated the pace of the work — moving it from scheduled completion in 2005 to next year — but the irony is that this accelerated timetable won’t give the public time to grasp the enormous changes the project portends. By this time next year, the fate of the source code for life on earth — open or closed — will be largely finished, long before most people have even begun to understand what is at stake.

Commercial software has its limits, and it runs into them regularly. Any commercial developer has a “resource horizon” – some upper boundary of money or programmer time which limits how much can be done on any given project. Projects which lie over this horizon either can’t be accomplished (very few companies could write a new operating system from scratch), or, once started, can’t be completed because of their size or scope (as with the Denver Airport baggage handling system).

Open Source software has no such resource horizon – programmers creating Open Source software do it because they’re interested in it, not because they’re paid to do it. Furthermore, since there’s no requirement to keep the source code secret, a much larger of number of programmers can work on any given project, and they don’t have to work for the same company, or even live in the same country.

However, the lack of this resource horizon does not mean that Open Source software has no limits, it just means that the limits aren’t financial. Commercial companies make software for money, so money is the limiting factor. Open Source developers make software for the love of the thing, so love becomes the limiting factor as well. Unloved software can’t be built using Open Source methods.

Open Source software trades the resource horizon for an interest horizon, the horizon over which a problem simply isn’t interesting enough to attract a group of developers who burn to spend their Friday nights wrestling with, say, meter-reading software for a natural gas pipeline. Consider, from the seminal description of the Open Source process contained in “The Cathedral and the Bazaar” , how many of the principles assume that programmers find problems, rather than vice-versa:

Every good work of software starts by scratching a developer’s personal itch.
…
If you have the right attitude, interesting problems will find you.
…
Often, the most striking and innovative solutions come from realizing that your concept of the problem was wrong.
…
To solve an interesting problem, start by finding a problem that is interesting to you.
…
Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

This is geek nirvana, where interest and innovation have more weight than profit or productivity. Because the articulation of these principles is so new (though the way of working is at least as old as the Internet itself), and because the acheivements of Linux, Apache et al. are so unprecedented, Open Source currently has an aura of unstoppability, a sense that “Maybe Open Source will take over everything!”

It won’t. Wherever Open Source is a good way of working, it is more effective than closed source, for the same reason that open markets are more effective than central planning. The very principles of personal interest outlined above, however, indicate where Open Source isn’t a good way of working in the first place.

Consider the converse of those principles: what if you have a problem that doesn’t scratch some core developer’s personal itch?

“I stayed up all weekend rewriting this awesome time-sheet processing system!” or where innovative solutions aren’t an option in an existing system? “We could make the package much simpler if we just bill them every year instead of every month.”

or where you can’t have a large enough group of beta testers?

“Know anybody who owns a plane? I need to beta-test my air traffic control program.”

These kinds of problems lie beyond the interest horizon, and their numbers are legion. 

Microsoft Is Not a Bad Company

Microsoft is not a bad company. Their problem is not systemic but situational – the same programmers with the same marching orders would focus on producing lightweight and useful code, as long as they weren’t working for a monopoly. It is Microsoft’s leverage from its desktop monopoly which makes them behave the way they do, not any inherent evil.
Microsoft, as a for-profit company, has one goal: make pots of money. The only way their current attempts to de-commoditize standards would help them do that was if they had a monopoly share of the market they were trying to affect. The same de-commoditizing behavior from a company which was competing for market share would lose them pots of money instead.

It is not necessary to “crush” Microsoft, it is merely necessary to get them to drop ‘Embrace and Extinguish’ in favor of actually advancing the state of the art. Linux is obviously Exhibit A, but Linux doesn’t have to get 50% desktop market share to defeat Microsoft, it merely needs to get a 15% market share to defeat Microsoft’s monopoly.

With a credible alternative taking as little as an eighth of the market share, Microsoft will no longer own the desktop. And as with IBM, once Microsoft no longer owns the space, they will undergo a painful retrenchment, and then emerge as a force for innovation instead of strangulation.

NT Competes with Linux, but Linux Does Not Compete with NT

Given the Bazaar model of development fostered by Open Source, Linux will always be under development. Accepting that is part of the cultural change necessary to work with OSS, even as an end user. There is a sense in some parts of the Open Source community that the current attention Linux is getting means that this is its chance to change the world, and that if it fails to do so quickly, there will be no second chance.
As long as programmers continue to improve it, Linux will have all the second chances in the world. The for-profit model needs end users for the positive externalities of network reinforcement (you use what other people at your job use, so its easy to find help, advice, etc.) and cash flow to pay programmers. OSS is a DIY model, which means that it can continue to advance without having a single “end user”.

The Linux market share issue is a tricky one – NT is competing with Linux, but Linux is not competing with NT. If a casual user switches from NT to Linux, MS loses cash flow, mind share, and the illusion of inevitability. If the same user switches from Linux to NT, it affects Linux’s development not at all.

Free Software Creates New Value

Thinking about the recent conversations about Linux ‘market share’, it occurred to me that some of us may be making the same mistake (self included) that Thomas Watson made when he predicted a worldwide market for 5 computers, or when Ken Olsen said there would be no market for computers in the home, i.e. assuming that the future will be much like the present.
Looking at Linux vs. NT in terms of present market share hides that group of people for whom the choice is not Linux vs. NT but Linux vs. nothing. Linux users don’t just come from the ranks of people who would otherswise use NT or Win9[58] — for people who only have access to 386 or 486 computers, and can’t afford anything new, Linux offers them the only possibility for running a modern OS, both because it runs well on minimal hardware and because it is free.

Watching large corporations dance around the idea of running Linux is interesting, but to switch to Linux from NT always involves some short term cost that raises the barrier to such a move. The real change in computing ecology could come from another quarter altogether – that group of people who either haven’t owned computers in the past, or who have had computers which have not been able to run web servers et al. Linux’s biggest growth in market share may come from places inaccessible to for-profit companies at all.

To Use Linux Without Criticizing It Is To Betray It

The greatest strength of Linux is the relentless, targetted criticism of its users. It is that, and not any largesse from a corporation which brings about improvement in the code. When the MS faithful lob ‘linux sux!!!1!” stink bombs into the Linux community, the natural reaction is to circle the wagons and praise Linux to the stars.
The Microsoft folks are right in many particulars, however: for desktop users, Linux is currently poorly documented and intimidating to intall. To let their hostility blind us to that fact robs us of our greatest strength – our ability not merely to criticize Linux, but, since we have the source before us and they do not, to criticize and improve Linux. This is not about two operating systems at this particular snapshot in their respective developments, this is about the long haul, and Linux gets better faster than any MS product ever has.

To use Linux without criticizing it is to betray it.

First published April 1998.

How did Web grow so quickly?

In the 36 months from January of 1993 to December of 1995, HTML went from being an unknown protocol to being to being the pre-eminent tool for designing electronic interfaces, decisively displacing almost all challengers and upstaging online services, CD-ROMs, and a dozen expensive and abortive experiments with interactive TV, and it did this while having no coordinated center, no central R&D effort, and no discernible financial incentive for the majority of its initial participants.

Ragtag though it was, the early Web, with its endless compendia of ‘Cool Bands’, badly scanned photos of pets, and ubiquitous “Favorite Links” lists pointing to more of same, was a world-building excercise which provided an unequalled hothouse lab for structuring information and designing interfaces. By the time the corporate players and media companies considered getting involved, many of the basic interface issues – the idea of Home pages, buttons and toolbars, the standards for placement of things like logos, navigational elements, help menus and the like had already been worked out within a vast body of general practice generated by the pioneers.

More than any other factor, this ability to allow ordinary users to build their own Web sites, without requiring that they be software developers or even particularly savvy software users, caused its rapid rise. Web design, and in particular Web site design with its emphasis on architecture, interactivity, and structuring data in an implicit order or set of orders, is not graphic design but rather constitutes a special kind of low-level engineering. Prior to the invention of HTML, all software packages which implemented any sort of hypertext required the user to learn a programming language of some sort (Director, Hypercard). HTML, in contrast, allows users to create links without requiring an ablity to write code.

INTERFACE ENGINEERING

Call HTML’s practitioners interface designers, information architects or, in keeping with the idea of world-building, even citizen-engineers, these are people who can think creatively about information as media, without necessarily being comfortable looking down the business end of a compiler. This becomes Principle 0 for this essay:

#0. Web site design is a task related to, but not the same as, software engineering.

Any project which offers a limitless variety of ways in which information can be presented and stuctured can happen both faster and better if there is a way for designers and engineers to collaborate without requiring either group to fundamentally alter their way looking at the world. While it can’t hurt to have a designer see source code or a developer open a copy of Photoshop once in a while, environments which let each group concentrate on their strengths grow more rapidly than environments which burden designers with engineering concerns or force their engineers to design an interface whenever they want to alter an engine. The Web exploded in the space of a few years in part because it effected this separation of functions so decisively.

As opposed to software which seeks a tight integration between creation, file format, and display (e.g. Excel, Lotus Notes), HTML specifies nothing about the tools needed for its creation, validation, storage, delivery, interpretation, or display. By minimizing the part of the interface that is the browser to a handful of commands for navigating the Web, while maximizing the part of the interface that is in the browser (the HTML files themselves), the earliest experiments with the Web took most of the interface out of the browser code and put it in the hands of any user who could learn HTML’s almost babyish syntax.

With Web design separated from the browser that was displaying it, countless alternative ways of structuring a site could be experimented with – sites for online newspapers, banks, stores, magazines, interfaces for commerce, document delivery, display, propaganda, art – all without requiring that the designers arrange their efforts with the browser engineers or even with one another.

This separation of interface and engineering puts site design in a fundamentally new relationship to the software that displays it:

#1. An interface can be integrated with the information it is displaying, instead of with the display software itself.

This transfer of the interface from ‘something that resides in the software and is applied to the data’ to ‘something that resides in the data and is applied to the software’ is the single most important innovation of the early Web.

REDUCING REQUIRED COORDINATION

Site designers with training in visual or conceptual aspects of organizing and presenting information can design for a Web browser without working with, or even needing to talk to, the people who designed the browser itself. Newsweek has to coordinate with America Online in order to make its content available through AOL, but it does not have to coordinate with Netscape (or the end-user’s ISP) to make its content available through Netscape.

This move away from centralized coordination and towards lateral development is good for the Web, and that some basic principles about the ways this is acheived can be derived from looking at the HTML/Browser split as one instance of a general class of “good” tools. The basic rule of good tools is:

#2. Good tools are transparent.

Web design is a conversation of sorts between designers, with the Web sites themselves standing in for proposal and response. Every page launched carries an attached message from its designer which reads “I think this is a pretty good way to design a Web page”, and every designer reacting to that will be able to respond with their own work. This is true of all design efforts – cars, magazines, coffee pots, but on the Web, this conversation is a riot, swift and wide ranging.

THE CENTRALITY OF OPEN HTML SOURCE TO THE WEB’S SUCCESS

The single factor most responsible for this riot of experimentation is transparency – the ability of any user to render into source code the choices made by any other designer. Once someone has worked out some design challenge, anyone else should be able to adopt, modify it, and make that modified version available, and so on.

Consider how effortless it would have been for Tim Berners-Lee or Marc Andreeson to treat the browser’s “View Source…” as a kind of debugging option which could have been disabled in any public release of their respective browsers, and imagine how much such a ‘hidden source’ choice would have hampered this feedback loop between designers. Instead, with this unprecedented transparency of the HTML itself, we got an enormous increase in the speed of design development. When faced with a Web page whose layout or technique seems particularly worth emulating or even copying outright, the question “How did they do that?” can be answered in seconds. 

“X DOES Y”, NOT “X IS FOR Y”

This ability to coordinate laterally, for designers to look at one anothers work and to experiment with it without asking permission from a central committee, is critical to creating this speed of innovation. Once the tools for creating Web pages are in a designers hand, there is no further certification, standardization or permission required. Put another way:

#3. Good tools tell you what they do, not what they’re for.

Good, general-purpose tools specify a series of causes and effects, nothing more. This is another part of what allowed the Web to grow so quickly. When a piece of software specifies a series of causes and effects without specifying semantic values (gravity makes things fall to the ground, but gravity is not for keeping apples stuck to the earth’s surface, or for anything else for that matter), it maximises the pace of innovation, because it minimizes the degree to which an effect has to be planned in advance for it to be useful.

The best example of this was the introduction of tables, first supported in Netscape 1.1. Tables were originally imagined to be just that – a tool for presenting tabular data. Their subsequent adoption by the user community as the basic method for page layout did not have to be explicit in the design of either HTML or the browser, because once its use was discovered and embraced, it no longer mattered what tables were originally for, since they specified causes and effects that made them perfectly suitable in their new surroundings.

A corollary to rule #3 is:

#3b. Good tools allow users to do stupid things.

A good tool, a tool which maximizes the possibilities for unexpected innovation from unknown quarters, has to allow the creation of everything from brilliant innovation through workmanlike normalcy all the way through hideous dreck. Tools which try to prevent users from making mistakes enter into a tar pit, because this requires that in addition to cause and effect, a tool has to be burdened with a second, heuristic sense of ‘right’ and ‘wrong’. In the short run, average quality can be raised if a tool intervenes to prevent legal but inefficent uses, but in the long haul, that strategy ultimately hampers development by not letting users learn from their mistakes.

THE INDEPENDENT RATE OF DEVLOPMENT

The browser/HTML combination as described not only increases the number of people who can work on different functions by reducing the amount of cross-training and coordination needed between designers and engineers (or between any two designers), it also changes the speed at which things happen by letting designers develop different parts of the system at independant rates.

By separating Web site design so completely from browser engineering, the rate of development of Web sites became many times faster than the rate of browser-engineering or protocol design.

Reflecting on these extremes is instructive, considered in light of the traditional software release schedule, which has alterations to engineering and interface change appearing together, in relatively infrequent versions. In the heat of structuring a site, a Web design team may have several people altering and re-examining an interface every few minutes. At the opposite extreme, in the three years from 1993 to 1995, the http protocol was not changed once, and in fact, even in 1998, the majority of Web transactions still use http 1.0. In between these extremes come rates of change from more to less rapid: changes in site structure (the relation between file storage and pointers), new versions of Web browsers, and new specifications for HTML itself. Designers could create, display, alter, beg, borrow, and steal interfaces, as fast as they liked, without any further input from any other source.

CONCLUSION

The Web grew as quickly as it did because the independent rate of site design, freed from the dictates of browser engineering, was much faster than even its inventors had predicted. Tools that allowed designers to do anything that rendered properly, that allowed for lateral conversations through the transparency of the HTML source, and removed the need for either compiling the results or seeking permission, certifcation or registration from anyone else led to the largest example of parallel development seen to date, and the results have been world-changing.

Furthermore, while there were certainly aspects of that revolution which will not be easily repeated, there are several current areas of inquiry – multi-player games (e.g. Half Life, Unreal), shared 3D worlds (VRML, Chrome), new tagset proposals (XML, SMIL), new interfaces (PilotOS, Linux), which will benefit from examination in light of the remarkable success of the Web. Any project with an interface likely to be of interst to end users (create your own avatar, create your own desktop) can happen both faster and better if these principles are applied.

First published in ACM, 03/98.

“Put the trodes on and [you] were out there, all the data in the world stacked up like a big neon city, so you could cruise around and have a kind of grip on it, visually anyway, because if you didn’t, it was too complicated, trying to find your way to a particular piece of data you needed.” 

Every six months or so, I find a reason to survey the state of VRML (Virtual Reality Modeling Language; more info at http://www.vrml.org/) and 3-D ‘shared worlds.’ I started these semi-annual forays in 1994, while researching a book, “Voices From the Net” (Ziff-Davis, 1994) , about online communities. In that book, I concluded that the classic model of cyberspace, quoted above from William Gibson’s “Mona Lisa Overdrive”, was not possible in this century (a bit of a grandiose cheat, since the century amounted to but a half-dozen years by that time), if at all. 

Woe betide the prognosticator who puts a far-off date on computer revolutions — as the visual components of networking grew by leaps and bounds, it looked like my predictions might be off the mark. I kept hearing that VRML was at the edge of usability, and every now and again would go exploring to see for myself. The improvements from year to year were impressive, although they were due in part to the rising tide of clock speed, which lifts all the boats. However, they never seemed any closer to being something that someone would use for anything other than a “Now, instead of just looking up the info you want, you can actually fly through your company’s parts catalog!” trade show demo. It was always better than last time, but never enough of a jump to actually be, in Alan Kay’s famous words, “good enough to criticize.” 

The Streets of San Francisco

VRML applications are the videophones of the Net — something that is possible in theory, but which excites designers far more than it excites potential users. At one point, the height of VRML applications was a 3-D Yellow Pages for San Francisco, the one town whose 3-D shape was uniquely ill-suited to such an already questionable endeavor (“Now, instead of simply looking up your dry cleaner’s phone number, you can search for it over hill and dale!”). Thus, I concluded, my Nostradamus-like powers were in no danger of being upended by unkind reality. Any real example of 3-D space that could be shared by multiple users and didn’t involve an unacceptable trade-off between appearance and speed, any 3-D space that had any sort of visceral effect on the user, was still years away. 

There was only one problem with that conclusion: I was wrong. 

I have now found what I was looking for, and found it, to my chagrin, years late. A networked 3-D engine with excellent shared space capabilities, open source code, a well-published world-building specification, an active development community, and a project showing the kind of rapid improvement year after year that comes from the best massively parallel design efforts, such as the Web. 

It is called Quake. 

I know, I know. As they say on Usenet, “Quake? That’s so 5 minutes ago!” 

Quake (http://www.idsoftware.com), for those of you like me who don’t follow the PC gaming world, is a “First Person Action” game, meaning roughly that the screen is your view of the world, and that you pilot your digital self, or “avatar,” through that world by maneuvering through what you see on the screen. I missed out on Quake when it was launched in the mid-’90s because as a general all-around Serious Person, I don’t play games, in particular games whose ideal customer is less than half my age. I have now seen the error of my ways. Quake solves the shared 3-D space problem far better than any VRML implementation — not despite being a game, but because it is a game. 

The Quake 3-D engine is for something, while VRML is a technology in search of a use. The Quake engine’s original use is immaterial; once you have a world-modeling protocol that lets you wade through a subterranean stream, picking up shotgun shells and blowing up other people who are picking up rocket launchers and trying to blow up you, you’ve got the whole ball of wax — spaces, movement, objects, avatars, even a simple physics. It would be easier to turn that into a tool for shared corporate meeting spaces than it would be to build the same thing in VRML, because once a tool is good for one thing, it’s easy to see how to make it good for another thing. 

On The Cutting Edge

This is one of the first axioms of creativity in this realm: Solve something first, then apply that solution elsewhere. It doesn’t matter if the first stone was sharpened to kill animals or cut wood, because once you have the principle down, making a knife into a spear or a scimitar or a scalpel is just application-level development. I have always been skeptical of any real possibility of viscerally engaging 3-D spaces, but I believe that I see in Quake — and its subsequent imitators and descendants — the sharpened stone, the thing that indicates the beginning of a new way of using the network. 

Quake is the real thing. It’s a local solution to a local problem (how to make a 3-D space realistic enough so that you can accurately aim a double-barreled shotgun at a chainsaw-wielding ogre, while both of you are running) that has global ramifications. I have seen a lot of 3-D spaces over the years, and I have never run into anything that gives as palpable a sense of place as this does. (Before the objection is raised that Quake is software and VRML is a protocol, consider that Quake, like VRML, always runs in client-server mode and reads external data files, even locally, and that the Quake client is one-fifth the size of the benchmark VRML client.) 

After playing with Quake for only a few hours, I found myself in possession of an absolute and rigorous sense of the spaces it contained: There was a moment, while I was off thinking about something else altogether, when I realized that in one particularly sticky part of the game, I could jump off a ramp and swim through some water under a trap located in a hallway. This was as matter-of-fact a thought as if I’d realized I could bring a big package up the service elevator at my office; in other words, I had a large, accurate map of an entire space in my head, not from having studied it, but from having experienced it. I had derived, for the first time ever, a sense of space — not an ‘as if’ sense or a sense of a metaphorical cyberspace, but of real space — from a screen. 

Here’s why I think this matters: Though Quake isn’t necessarily the right solution for the 3-D shared space problem, it feels right in a way that nothing in the VRML world does. Once you’ve seen it in action, much of what’s wrong with VRML snaps into focus: Quake does something well instead of many things poorly. 

This matters more than I can say. Despite all the theoretical underpinnings we now use as a basis for our work as computing reaches the half-century mark and networking turns 30, world-shifting developments still often come from practical solutions to simple problems. 

The VRML community has failed to come up with anything this compelling — not despite the community’s best intentions, but because of them. Every time VRML practitioners approach the problem of how to represent space on the screen, they have no focused reason to make any particular trade-off of detail versus rendering speed, or making objects versus making spaces, because VRML isn’t for anything except itself. Many times, having a particular, near-term need to solve brings a project’s virtues into sharp focus, and gives it enough clarity to live on its own. Quake puts tools in the hands of the users. 

Id software, producers of Quake, started allowing its games to be edited by users, and published a world-building spec so that users could write their own editors as well. As with HTML, one can build a variety of 3-D editors that can create a Quake world by simply writing to the spec. This has led to a riot of sophisticated tools for creating Quake “levels” like BSP and WorldCraft that would make any VRML partisan drool, and has spawned a community of user-engineers who are creating compelling worlds, objects, and sometimes whole alternate worlds at a pace and with an intensity that should make any VRML adherent faint. VRML is older than Quake, and yet it has fewer tools and fewer users, and still exists at the stage where the best examples of the technology are produced by the companies selling it, not the users using it. Quake has a simple model of compliance. 

The problem of representing space, surfaces, objects and worlds is almost infinitely vast. There was for some time in the AI community a proposal to develop a ‘naive physics’ where every aspect of physical organization of the world would be worked out in a way that would let computers deduce things about their surroundings. For instance, the naive physics of liquids included all sorts of rules about flow and how it was affected by being bounded by floors, walls and ceilings. This kind of elaborate modeling has been inherited in spirit by VRML, which is so complex that rendering a simple piece of furniture can bog down an average desktop computer. 

Quake, on the other hand, adopts a breathtakingly simple attitude toward physics. Quake has shapes like blocks, wedges and columns, and it has materials like stone, metal and water. If you make a freestanding column of water, so be it. The column will appear in any space you create; you can walk into it, swim to the top, walk out at the top and then fall down again. It does this simply because a physics of liquids is too complex to solve right now, and because Quake does not try to save users from themselves by adding layers of interpretation outside the engine itself. If it compiles, it complies. 

Hacking X for Y 

This willingness to allow users to do stupid things, on the assumption that they will learn more quickly if the software doesn’t try to second-guess them, has given Quake a development curve that looks like HTML in the early days, where a user’s first efforts were often lousy but the progress between lousy and good was not blocked by the software itself. Quake encourages experimentation and incrementalism in the tradition of the best tools out there. 

I acknowledge, however, that the Quake format lacks the rigorous throughput into VRML — the comparison here is not architecture to architecture, but effect to effect. I am not seriously suggesting that Quake or one of its children like Quake II (http://www.idsoftware.com), Half-Life (http://www.valvesoftware.com) or Unreal (http://www.unreal.com) be used as-is as a platform from which to build other 3-D shared spaces (well, maybe I am suggesting that half-seriously). More important, I am suggesting that there is a problem because the 3-D game people aren’t talking enough to the VRML people, and vice versa. Networking has come so far so fast partly because we have been willing to say, over and over again, “Well, we never thought of using X for Y, but now that you mention it, why not?” 

The Net is too young, and the 3-D space problem too tricky, to shoot for perfection right now. What worries me most about my belated discovery of Quake is that in years of looking at 3-D spaces, no one ever mentioned it in the same breath as VRML and I never thought of it myself, simply because of the stigma attached to things that are “merely” games. Many of us, myself very much included, have come to believe our own press about the seriousness of our current endeavors. We have been adversely affected by the commercial applications of the network, and have weaned ourselves off the playfulness that has always produced insights and breakthroughs. 

It will be a terrible loss if those of us doing this work start to take ourselves too seriously. Round the clock and round the world, there are hundreds of people chasing one another through rooms and tunnels and up and down staircases that don’t exist anywhere in the real world, a model of electronic space closer to Gibson’s ideal than anything else that exists out there, and we shouldn’t write those people off. Their game playing may look frivolous to us Serious Persons, but they know a lot about 3-D shared worlds, and it will be a terrible waste of everybody’s time if we don’t find a way to tap into that.