Commercial software has its limits, and it runs into them regularly. Any commercial developer has a “resource horizon” – some upper boundary of money or programmer time which limits how much can be done on any given project. Projects which lie over this horizon either can’t be accomplished (very few companies could write a new operating system from scratch), or, once started, can’t be completed because of their size or scope (as with the Denver Airport baggage handling system).

Open Source software has no such resource horizon – programmers creating Open Source software do it because they’re interested in it, not because they’re paid to do it. Furthermore, since there’s no requirement to keep the source code secret, a much larger of number of programmers can work on any given project, and they don’t have to work for the same company, or even live in the same country.

However, the lack of this resource horizon does not mean that Open Source software has no limits, it just means that the limits aren’t financial. Commercial companies make software for money, so money is the limiting factor. Open Source developers make software for the love of the thing, so love becomes the limiting factor as well. Unloved software can’t be built using Open Source methods.

Open Source software trades the resource horizon for an interest horizon, the horizon over which a problem simply isn’t interesting enough to attract a group of developers who burn to spend their Friday nights wrestling with, say, meter-reading software for a natural gas pipeline. Consider, from the seminal description of the Open Source process contained in “The Cathedral and the Bazaar” , how many of the principles assume that programmers find problems, rather than vice-versa:

Every good work of software starts by scratching a developer’s personal itch.
…
If you have the right attitude, interesting problems will find you.
…
Often, the most striking and innovative solutions come from realizing that your concept of the problem was wrong.
…
To solve an interesting problem, start by finding a problem that is interesting to you.
…
Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

This is geek nirvana, where interest and innovation have more weight than profit or productivity. Because the articulation of these principles is so new (though the way of working is at least as old as the Internet itself), and because the acheivements of Linux, Apache et al. are so unprecedented, Open Source currently has an aura of unstoppability, a sense that “Maybe Open Source will take over everything!”

It won’t. Wherever Open Source is a good way of working, it is more effective than closed source, for the same reason that open markets are more effective than central planning. The very principles of personal interest outlined above, however, indicate where Open Source isn’t a good way of working in the first place.

Consider the converse of those principles: what if you have a problem that doesn’t scratch some core developer’s personal itch?

“I stayed up all weekend rewriting this awesome time-sheet processing system!” or where innovative solutions aren’t an option in an existing system? “We could make the package much simpler if we just bill them every year instead of every month.”

or where you can’t have a large enough group of beta testers?

“Know anybody who owns a plane? I need to beta-test my air traffic control program.”

These kinds of problems lie beyond the interest horizon, and their numbers are legion. 

Microsoft Is Not a Bad Company

Microsoft is not a bad company. Their problem is not systemic but situational – the same programmers with the same marching orders would focus on producing lightweight and useful code, as long as they weren’t working for a monopoly. It is Microsoft’s leverage from its desktop monopoly which makes them behave the way they do, not any inherent evil.
Microsoft, as a for-profit company, has one goal: make pots of money. The only way their current attempts to de-commoditize standards would help them do that was if they had a monopoly share of the market they were trying to affect. The same de-commoditizing behavior from a company which was competing for market share would lose them pots of money instead.

It is not necessary to “crush” Microsoft, it is merely necessary to get them to drop ‘Embrace and Extinguish’ in favor of actually advancing the state of the art. Linux is obviously Exhibit A, but Linux doesn’t have to get 50% desktop market share to defeat Microsoft, it merely needs to get a 15% market share to defeat Microsoft’s monopoly.

With a credible alternative taking as little as an eighth of the market share, Microsoft will no longer own the desktop. And as with IBM, once Microsoft no longer owns the space, they will undergo a painful retrenchment, and then emerge as a force for innovation instead of strangulation.

NT Competes with Linux, but Linux Does Not Compete with NT

Given the Bazaar model of development fostered by Open Source, Linux will always be under development. Accepting that is part of the cultural change necessary to work with OSS, even as an end user. There is a sense in some parts of the Open Source community that the current attention Linux is getting means that this is its chance to change the world, and that if it fails to do so quickly, there will be no second chance.
As long as programmers continue to improve it, Linux will have all the second chances in the world. The for-profit model needs end users for the positive externalities of network reinforcement (you use what other people at your job use, so its easy to find help, advice, etc.) and cash flow to pay programmers. OSS is a DIY model, which means that it can continue to advance without having a single “end user”.

The Linux market share issue is a tricky one – NT is competing with Linux, but Linux is not competing with NT. If a casual user switches from NT to Linux, MS loses cash flow, mind share, and the illusion of inevitability. If the same user switches from Linux to NT, it affects Linux’s development not at all.

Free Software Creates New Value

Thinking about the recent conversations about Linux ‘market share’, it occurred to me that some of us may be making the same mistake (self included) that Thomas Watson made when he predicted a worldwide market for 5 computers, or when Ken Olsen said there would be no market for computers in the home, i.e. assuming that the future will be much like the present.
Looking at Linux vs. NT in terms of present market share hides that group of people for whom the choice is not Linux vs. NT but Linux vs. nothing. Linux users don’t just come from the ranks of people who would otherswise use NT or Win9[58] — for people who only have access to 386 or 486 computers, and can’t afford anything new, Linux offers them the only possibility for running a modern OS, both because it runs well on minimal hardware and because it is free.

Watching large corporations dance around the idea of running Linux is interesting, but to switch to Linux from NT always involves some short term cost that raises the barrier to such a move. The real change in computing ecology could come from another quarter altogether – that group of people who either haven’t owned computers in the past, or who have had computers which have not been able to run web servers et al. Linux’s biggest growth in market share may come from places inaccessible to for-profit companies at all.

To Use Linux Without Criticizing It Is To Betray It

The greatest strength of Linux is the relentless, targetted criticism of its users. It is that, and not any largesse from a corporation which brings about improvement in the code. When the MS faithful lob ‘linux sux!!!1!” stink bombs into the Linux community, the natural reaction is to circle the wagons and praise Linux to the stars.
The Microsoft folks are right in many particulars, however: for desktop users, Linux is currently poorly documented and intimidating to intall. To let their hostility blind us to that fact robs us of our greatest strength – our ability not merely to criticize Linux, but, since we have the source before us and they do not, to criticize and improve Linux. This is not about two operating systems at this particular snapshot in their respective developments, this is about the long haul, and Linux gets better faster than any MS product ever has.

To use Linux without criticizing it is to betray it.

First published April 1998.

How did Web grow so quickly?

In the 36 months from January of 1993 to December of 1995, HTML went from being an unknown protocol to being to being the pre-eminent tool for designing electronic interfaces, decisively displacing almost all challengers and upstaging online services, CD-ROMs, and a dozen expensive and abortive experiments with interactive TV, and it did this while having no coordinated center, no central R&D effort, and no discernible financial incentive for the majority of its initial participants.

Ragtag though it was, the early Web, with its endless compendia of ‘Cool Bands’, badly scanned photos of pets, and ubiquitous “Favorite Links” lists pointing to more of same, was a world-building excercise which provided an unequalled hothouse lab for structuring information and designing interfaces. By the time the corporate players and media companies considered getting involved, many of the basic interface issues – the idea of Home pages, buttons and toolbars, the standards for placement of things like logos, navigational elements, help menus and the like had already been worked out within a vast body of general practice generated by the pioneers.

More than any other factor, this ability to allow ordinary users to build their own Web sites, without requiring that they be software developers or even particularly savvy software users, caused its rapid rise. Web design, and in particular Web site design with its emphasis on architecture, interactivity, and structuring data in an implicit order or set of orders, is not graphic design but rather constitutes a special kind of low-level engineering. Prior to the invention of HTML, all software packages which implemented any sort of hypertext required the user to learn a programming language of some sort (Director, Hypercard). HTML, in contrast, allows users to create links without requiring an ablity to write code.

INTERFACE ENGINEERING

Call HTML’s practitioners interface designers, information architects or, in keeping with the idea of world-building, even citizen-engineers, these are people who can think creatively about information as media, without necessarily being comfortable looking down the business end of a compiler. This becomes Principle 0 for this essay:

#0. Web site design is a task related to, but not the same as, software engineering.

Any project which offers a limitless variety of ways in which information can be presented and stuctured can happen both faster and better if there is a way for designers and engineers to collaborate without requiring either group to fundamentally alter their way looking at the world. While it can’t hurt to have a designer see source code or a developer open a copy of Photoshop once in a while, environments which let each group concentrate on their strengths grow more rapidly than environments which burden designers with engineering concerns or force their engineers to design an interface whenever they want to alter an engine. The Web exploded in the space of a few years in part because it effected this separation of functions so decisively.

As opposed to software which seeks a tight integration between creation, file format, and display (e.g. Excel, Lotus Notes), HTML specifies nothing about the tools needed for its creation, validation, storage, delivery, interpretation, or display. By minimizing the part of the interface that is the browser to a handful of commands for navigating the Web, while maximizing the part of the interface that is in the browser (the HTML files themselves), the earliest experiments with the Web took most of the interface out of the browser code and put it in the hands of any user who could learn HTML’s almost babyish syntax.

With Web design separated from the browser that was displaying it, countless alternative ways of structuring a site could be experimented with – sites for online newspapers, banks, stores, magazines, interfaces for commerce, document delivery, display, propaganda, art – all without requiring that the designers arrange their efforts with the browser engineers or even with one another.

This separation of interface and engineering puts site design in a fundamentally new relationship to the software that displays it:

#1. An interface can be integrated with the information it is displaying, instead of with the display software itself.

This transfer of the interface from ‘something that resides in the software and is applied to the data’ to ‘something that resides in the data and is applied to the software’ is the single most important innovation of the early Web.

REDUCING REQUIRED COORDINATION

Site designers with training in visual or conceptual aspects of organizing and presenting information can design for a Web browser without working with, or even needing to talk to, the people who designed the browser itself. Newsweek has to coordinate with America Online in order to make its content available through AOL, but it does not have to coordinate with Netscape (or the end-user’s ISP) to make its content available through Netscape.

This move away from centralized coordination and towards lateral development is good for the Web, and that some basic principles about the ways this is acheived can be derived from looking at the HTML/Browser split as one instance of a general class of “good” tools. The basic rule of good tools is:

#2. Good tools are transparent.

Web design is a conversation of sorts between designers, with the Web sites themselves standing in for proposal and response. Every page launched carries an attached message from its designer which reads “I think this is a pretty good way to design a Web page”, and every designer reacting to that will be able to respond with their own work. This is true of all design efforts – cars, magazines, coffee pots, but on the Web, this conversation is a riot, swift and wide ranging.

THE CENTRALITY OF OPEN HTML SOURCE TO THE WEB’S SUCCESS

The single factor most responsible for this riot of experimentation is transparency – the ability of any user to render into source code the choices made by any other designer. Once someone has worked out some design challenge, anyone else should be able to adopt, modify it, and make that modified version available, and so on.

Consider how effortless it would have been for Tim Berners-Lee or Marc Andreeson to treat the browser’s “View Source…” as a kind of debugging option which could have been disabled in any public release of their respective browsers, and imagine how much such a ‘hidden source’ choice would have hampered this feedback loop between designers. Instead, with this unprecedented transparency of the HTML itself, we got an enormous increase in the speed of design development. When faced with a Web page whose layout or technique seems particularly worth emulating or even copying outright, the question “How did they do that?” can be answered in seconds. 

“X DOES Y”, NOT “X IS FOR Y”

This ability to coordinate laterally, for designers to look at one anothers work and to experiment with it without asking permission from a central committee, is critical to creating this speed of innovation. Once the tools for creating Web pages are in a designers hand, there is no further certification, standardization or permission required. Put another way:

#3. Good tools tell you what they do, not what they’re for.

Good, general-purpose tools specify a series of causes and effects, nothing more. This is another part of what allowed the Web to grow so quickly. When a piece of software specifies a series of causes and effects without specifying semantic values (gravity makes things fall to the ground, but gravity is not for keeping apples stuck to the earth’s surface, or for anything else for that matter), it maximises the pace of innovation, because it minimizes the degree to which an effect has to be planned in advance for it to be useful.

The best example of this was the introduction of tables, first supported in Netscape 1.1. Tables were originally imagined to be just that – a tool for presenting tabular data. Their subsequent adoption by the user community as the basic method for page layout did not have to be explicit in the design of either HTML or the browser, because once its use was discovered and embraced, it no longer mattered what tables were originally for, since they specified causes and effects that made them perfectly suitable in their new surroundings.

A corollary to rule #3 is:

#3b. Good tools allow users to do stupid things.

A good tool, a tool which maximizes the possibilities for unexpected innovation from unknown quarters, has to allow the creation of everything from brilliant innovation through workmanlike normalcy all the way through hideous dreck. Tools which try to prevent users from making mistakes enter into a tar pit, because this requires that in addition to cause and effect, a tool has to be burdened with a second, heuristic sense of ‘right’ and ‘wrong’. In the short run, average quality can be raised if a tool intervenes to prevent legal but inefficent uses, but in the long haul, that strategy ultimately hampers development by not letting users learn from their mistakes.

THE INDEPENDENT RATE OF DEVLOPMENT

The browser/HTML combination as described not only increases the number of people who can work on different functions by reducing the amount of cross-training and coordination needed between designers and engineers (or between any two designers), it also changes the speed at which things happen by letting designers develop different parts of the system at independant rates.

By separating Web site design so completely from browser engineering, the rate of development of Web sites became many times faster than the rate of browser-engineering or protocol design.

Reflecting on these extremes is instructive, considered in light of the traditional software release schedule, which has alterations to engineering and interface change appearing together, in relatively infrequent versions. In the heat of structuring a site, a Web design team may have several people altering and re-examining an interface every few minutes. At the opposite extreme, in the three years from 1993 to 1995, the http protocol was not changed once, and in fact, even in 1998, the majority of Web transactions still use http 1.0. In between these extremes come rates of change from more to less rapid: changes in site structure (the relation between file storage and pointers), new versions of Web browsers, and new specifications for HTML itself. Designers could create, display, alter, beg, borrow, and steal interfaces, as fast as they liked, without any further input from any other source.

CONCLUSION

The Web grew as quickly as it did because the independent rate of site design, freed from the dictates of browser engineering, was much faster than even its inventors had predicted. Tools that allowed designers to do anything that rendered properly, that allowed for lateral conversations through the transparency of the HTML source, and removed the need for either compiling the results or seeking permission, certifcation or registration from anyone else led to the largest example of parallel development seen to date, and the results have been world-changing.

Furthermore, while there were certainly aspects of that revolution which will not be easily repeated, there are several current areas of inquiry – multi-player games (e.g. Half Life, Unreal), shared 3D worlds (VRML, Chrome), new tagset proposals (XML, SMIL), new interfaces (PilotOS, Linux), which will benefit from examination in light of the remarkable success of the Web. Any project with an interface likely to be of interst to end users (create your own avatar, create your own desktop) can happen both faster and better if these principles are applied.

First published in ACM, 03/98.

“Put the trodes on and [you] were out there, all the data in the world stacked up like a big neon city, so you could cruise around and have a kind of grip on it, visually anyway, because if you didn’t, it was too complicated, trying to find your way to a particular piece of data you needed.” 

Every six months or so, I find a reason to survey the state of VRML (Virtual Reality Modeling Language; more info at http://www.vrml.org/) and 3-D ‘shared worlds.’ I started these semi-annual forays in 1994, while researching a book, “Voices From the Net” (Ziff-Davis, 1994) , about online communities. In that book, I concluded that the classic model of cyberspace, quoted above from William Gibson’s “Mona Lisa Overdrive”, was not possible in this century (a bit of a grandiose cheat, since the century amounted to but a half-dozen years by that time), if at all. 

Woe betide the prognosticator who puts a far-off date on computer revolutions — as the visual components of networking grew by leaps and bounds, it looked like my predictions might be off the mark. I kept hearing that VRML was at the edge of usability, and every now and again would go exploring to see for myself. The improvements from year to year were impressive, although they were due in part to the rising tide of clock speed, which lifts all the boats. However, they never seemed any closer to being something that someone would use for anything other than a “Now, instead of just looking up the info you want, you can actually fly through your company’s parts catalog!” trade show demo. It was always better than last time, but never enough of a jump to actually be, in Alan Kay’s famous words, “good enough to criticize.” 

The Streets of San Francisco

VRML applications are the videophones of the Net — something that is possible in theory, but which excites designers far more than it excites potential users. At one point, the height of VRML applications was a 3-D Yellow Pages for San Francisco, the one town whose 3-D shape was uniquely ill-suited to such an already questionable endeavor (“Now, instead of simply looking up your dry cleaner’s phone number, you can search for it over hill and dale!”). Thus, I concluded, my Nostradamus-like powers were in no danger of being upended by unkind reality. Any real example of 3-D space that could be shared by multiple users and didn’t involve an unacceptable trade-off between appearance and speed, any 3-D space that had any sort of visceral effect on the user, was still years away. 

There was only one problem with that conclusion: I was wrong. 

I have now found what I was looking for, and found it, to my chagrin, years late. A networked 3-D engine with excellent shared space capabilities, open source code, a well-published world-building specification, an active development community, and a project showing the kind of rapid improvement year after year that comes from the best massively parallel design efforts, such as the Web. 

It is called Quake. 

I know, I know. As they say on Usenet, “Quake? That’s so 5 minutes ago!” 

Quake (http://www.idsoftware.com), for those of you like me who don’t follow the PC gaming world, is a “First Person Action” game, meaning roughly that the screen is your view of the world, and that you pilot your digital self, or “avatar,” through that world by maneuvering through what you see on the screen. I missed out on Quake when it was launched in the mid-’90s because as a general all-around Serious Person, I don’t play games, in particular games whose ideal customer is less than half my age. I have now seen the error of my ways. Quake solves the shared 3-D space problem far better than any VRML implementation — not despite being a game, but because it is a game. 

The Quake 3-D engine is for something, while VRML is a technology in search of a use. The Quake engine’s original use is immaterial; once you have a world-modeling protocol that lets you wade through a subterranean stream, picking up shotgun shells and blowing up other people who are picking up rocket launchers and trying to blow up you, you’ve got the whole ball of wax — spaces, movement, objects, avatars, even a simple physics. It would be easier to turn that into a tool for shared corporate meeting spaces than it would be to build the same thing in VRML, because once a tool is good for one thing, it’s easy to see how to make it good for another thing. 

On The Cutting Edge

This is one of the first axioms of creativity in this realm: Solve something first, then apply that solution elsewhere. It doesn’t matter if the first stone was sharpened to kill animals or cut wood, because once you have the principle down, making a knife into a spear or a scimitar or a scalpel is just application-level development. I have always been skeptical of any real possibility of viscerally engaging 3-D spaces, but I believe that I see in Quake — and its subsequent imitators and descendants — the sharpened stone, the thing that indicates the beginning of a new way of using the network. 

Quake is the real thing. It’s a local solution to a local problem (how to make a 3-D space realistic enough so that you can accurately aim a double-barreled shotgun at a chainsaw-wielding ogre, while both of you are running) that has global ramifications. I have seen a lot of 3-D spaces over the years, and I have never run into anything that gives as palpable a sense of place as this does. (Before the objection is raised that Quake is software and VRML is a protocol, consider that Quake, like VRML, always runs in client-server mode and reads external data files, even locally, and that the Quake client is one-fifth the size of the benchmark VRML client.) 

After playing with Quake for only a few hours, I found myself in possession of an absolute and rigorous sense of the spaces it contained: There was a moment, while I was off thinking about something else altogether, when I realized that in one particularly sticky part of the game, I could jump off a ramp and swim through some water under a trap located in a hallway. This was as matter-of-fact a thought as if I’d realized I could bring a big package up the service elevator at my office; in other words, I had a large, accurate map of an entire space in my head, not from having studied it, but from having experienced it. I had derived, for the first time ever, a sense of space — not an ‘as if’ sense or a sense of a metaphorical cyberspace, but of real space — from a screen. 

Here’s why I think this matters: Though Quake isn’t necessarily the right solution for the 3-D shared space problem, it feels right in a way that nothing in the VRML world does. Once you’ve seen it in action, much of what’s wrong with VRML snaps into focus: Quake does something well instead of many things poorly. 

This matters more than I can say. Despite all the theoretical underpinnings we now use as a basis for our work as computing reaches the half-century mark and networking turns 30, world-shifting developments still often come from practical solutions to simple problems. 

The VRML community has failed to come up with anything this compelling — not despite the community’s best intentions, but because of them. Every time VRML practitioners approach the problem of how to represent space on the screen, they have no focused reason to make any particular trade-off of detail versus rendering speed, or making objects versus making spaces, because VRML isn’t for anything except itself. Many times, having a particular, near-term need to solve brings a project’s virtues into sharp focus, and gives it enough clarity to live on its own. Quake puts tools in the hands of the users. 

Id software, producers of Quake, started allowing its games to be edited by users, and published a world-building spec so that users could write their own editors as well. As with HTML, one can build a variety of 3-D editors that can create a Quake world by simply writing to the spec. This has led to a riot of sophisticated tools for creating Quake “levels” like BSP and WorldCraft that would make any VRML partisan drool, and has spawned a community of user-engineers who are creating compelling worlds, objects, and sometimes whole alternate worlds at a pace and with an intensity that should make any VRML adherent faint. VRML is older than Quake, and yet it has fewer tools and fewer users, and still exists at the stage where the best examples of the technology are produced by the companies selling it, not the users using it. Quake has a simple model of compliance. 

The problem of representing space, surfaces, objects and worlds is almost infinitely vast. There was for some time in the AI community a proposal to develop a ‘naive physics’ where every aspect of physical organization of the world would be worked out in a way that would let computers deduce things about their surroundings. For instance, the naive physics of liquids included all sorts of rules about flow and how it was affected by being bounded by floors, walls and ceilings. This kind of elaborate modeling has been inherited in spirit by VRML, which is so complex that rendering a simple piece of furniture can bog down an average desktop computer. 

Quake, on the other hand, adopts a breathtakingly simple attitude toward physics. Quake has shapes like blocks, wedges and columns, and it has materials like stone, metal and water. If you make a freestanding column of water, so be it. The column will appear in any space you create; you can walk into it, swim to the top, walk out at the top and then fall down again. It does this simply because a physics of liquids is too complex to solve right now, and because Quake does not try to save users from themselves by adding layers of interpretation outside the engine itself. If it compiles, it complies. 

Hacking X for Y 

This willingness to allow users to do stupid things, on the assumption that they will learn more quickly if the software doesn’t try to second-guess them, has given Quake a development curve that looks like HTML in the early days, where a user’s first efforts were often lousy but the progress between lousy and good was not blocked by the software itself. Quake encourages experimentation and incrementalism in the tradition of the best tools out there. 

I acknowledge, however, that the Quake format lacks the rigorous throughput into VRML — the comparison here is not architecture to architecture, but effect to effect. I am not seriously suggesting that Quake or one of its children like Quake II (http://www.idsoftware.com), Half-Life (http://www.valvesoftware.com) or Unreal (http://www.unreal.com) be used as-is as a platform from which to build other 3-D shared spaces (well, maybe I am suggesting that half-seriously). More important, I am suggesting that there is a problem because the 3-D game people aren’t talking enough to the VRML people, and vice versa. Networking has come so far so fast partly because we have been willing to say, over and over again, “Well, we never thought of using X for Y, but now that you mention it, why not?” 

The Net is too young, and the 3-D space problem too tricky, to shoot for perfection right now. What worries me most about my belated discovery of Quake is that in years of looking at 3-D spaces, no one ever mentioned it in the same breath as VRML and I never thought of it myself, simply because of the stigma attached to things that are “merely” games. Many of us, myself very much included, have come to believe our own press about the seriousness of our current endeavors. We have been adversely affected by the commercial applications of the network, and have weaned ourselves off the playfulness that has always produced insights and breakthroughs. 

It will be a terrible loss if those of us doing this work start to take ourselves too seriously. Round the clock and round the world, there are hundreds of people chasing one another through rooms and tunnels and up and down staircases that don’t exist anywhere in the real world, a model of electronic space closer to Gibson’s ideal than anything else that exists out there, and we shouldn’t write those people off. Their game playing may look frivolous to us Serious Persons, but they know a lot about 3-D shared worlds, and it will be a terrible waste of everybody’s time if we don’t find a way to tap into that.

(First appeared in the ACM’s net_worker, 1996)

Why something as poorly designed as the Web became The Next Big Thing, and what that means for the future.

If it were April Fool’s Day, the Net’s only official holiday, and you wanted to design a ‘Novelty Protocol’ to slip by the Internet Engineering Task Force as a joke, it might look something like the Web:
The server would use neither a persistent connection nor a store-and-forward model, thus giving it all the worst features of both telnet and e-mail.
The server’s primary method of extensibility would require spawning external processes, thus ensuring both security risks and unpredictable load.
The server would have no built-in mechanism for gracefully apportioning resources, refusing or delaying heavy traffic, or load-balancing. It would, however, be relatively easy to crash.
Multiple files traveling together from one server to one client would each incur the entire overhead of a new session call.
The hypertext model would ignore all serious theoretical work on hypertext to date. In particular, all hypertext links would be one-directional, thus making it impossible to move or delete a piece of data without ensuring that some unknown number of pointers around the world would silently fail.
The tag set would be absurdly polluted and user-extensible with no central coordination and no consistency in implementation. As a bonus, many elements would perform conflicting functions as logical and visual layout elements.
HTTP and HTML are the Whoopee Cushion and Joy Buzzer of Internet protocols, only comprehensible as elaborate practical jokes. For anyone who has tried to accomplish anything serious on the Web, it’s pretty obvious that of the various implementations of a worldwide hypertext protocol, we have the worst one possible.

Except, of course, for all the others.

MAMMALS VS. DINOSAURS

The problem with that list of deficiencies is that it is also a list of necessities — the Web has flourished in a way that no other networking protocol has except e-mail, not despite many of these qualities but because of them. The very weaknesses that make the Web so infuriating to serious practitioners also make it possible in the first place. In fact, had the Web been a strong and well-designed entity from its inception, it would have gone nowhere. As it enters its adolescence, showing both flashes of maturity and infuriating unreliability, it is worth recalling what the network was like before the Web.

In the early ’90s, Internet population was doubling annually, and the most serious work on new protocols was being done to solve the biggest problem of the day, the growth of available information resources at a rate that outstripped anyone’s ability to catalog or index them. The two big meta-indexing efforts of the time were Gopher, the anonymous ftp index; and the heavy-hitter, Thinking Machines’ Wide Area Information Server (WAIS). Each of these protocols was strong — carefully thought-out, painstakingly implemented, self-consistent and centrally designed. Each had the backing of serious academic research, and each was rapidly gaining adherents.

The electronic world in other quarters was filled with similar visions of strong, well-designed protocols — CD-ROMs, interactive TV, online services. Like Gopher and WAIS, each of these had the backing of significant industry players, including computer manufacturers, media powerhouses and outside investors, as well as a growing user base that seemed to presage a future of different protocols for different functions, particularly when it came to multimedia.

These various protocols and services shared two important characteristics: Each was pursuing a design that was internally cohesive, and each operated in a kind of hermetically sealed environment where it interacted not at all with its neighbors. These characteristics are really flip sides of the same coin — the strong internal cohesion of their design contributed directly to their lack of interoperability. CompuServe and AOL, two of the top online services, couldn’t even share resources with one another, much less somehow interoperate with interactive TV or CD-ROMs.

THE STRENGTH OF WEAKNESS AND EVOLVABILITY

In other words, every contender for becoming an “industry standard” for handling information was too strong and too well-designed to succeed outside its own narrow confines. So how did the Web manage to damage and, in some cases, destroy those contenders for the title of The Next Big Thing? Weakness, coupled with an ability to improve exponentially.

The Web, in its earliest conception, was nothing more than a series of pointers. It grew not out of a desire to be an electronic encyclopedia so much as an electronic Post-it note. The idea of keeping pointers to ftp sites, Gopher indices, Veronica search engines and so forth all in one place doesn’t seem so remarkable now, but in fact it was the one thing missing from the growing welter of different protocols, each of which was too strong to interoperate well with the others.

Considered in this light, the Web’s poorer engineering qualities seem not merely desirable but essential. Despite all strong theoretical models of hypertext requiring bi-directional links, in any heterogeneous system links have to be one-directional, because bi-directional links would require massive coordination in a way that would limit its scope. Despite the obvious advantages of persistent connections in terms of state-tracking and lowering overhead, a server designed to connect to various types of network resources can’t require persistent connections, because that would limit the protocols that could be pointed to by the Web. The server must accommodate external processes or it would limit its extensibility to whatever the designers of the server could put into any given release, and so on.

Furthermore, the Web’s almost babyish SGML syntax, so far from any serious computational framework (Where are the conditionals? Why is the Document Type Description so inconsistent? Why are the browsers enforcement of conformity so lax?), made it possible for anyone wanting a Web page to write one. The effects of this ease of implementation, as opposed to the difficulties of launching a Gopher index or making a CD-ROM, are twofold: a huge increase in truly pointless and stupid content soaking up bandwidth; and, as a direct result, a rush to find ways to compete with all the noise through the creation of interesting work. The quality of the best work on the Web today has not happened in spite of the mass of garbage out there, but in part because of it.

In the space of a few years, the Web took over indexing from Gopher, rendered CompuServe irrelevant, undermined CD-ROMs, and now seems poised to take on the features of interactive TV, not because of its initial excellence but because of its consistent evolvability. It’s easy for central planning to outperform weak but evolvable systems in the short run, but in the long run evolution always has the edge. The Web, jujitsu-like, initially took on the power of other network protocols by simply acting as pointers to them, and then slowly subsumed their functions.

Despite the Web’s ability to usurp the advantages of existing services, this is a story of inevitability, not of perfection. Yahoo and Lycos have taken over from Gopher and WAIS as our meta-indices, but the search engines themselves, as has been widely noted, are pretty lousy ways to find things. The problem that Gopher and WAIS set out to solve has not only not been solved by the Web, it has been made worse. Furthermore, this kind of problem is intractable because of the nature of evolvable systems.

THREE RULES FOR EVOLVABLE SYSTEMS

Evolvable systems — those that proceed not under the sole direction of one centralized design authority but by being adapted and extended in a thousand small ways in a thousand places at once — have three main characteristics that are germane to their eventual victories over strong, centrally designed protocols.

• Only solutions that produce partial results when partially implemented can succeed. The network is littered with ideas that would have worked had everybody adopted them. Evolvable systems begin partially working right away and then grow, rather than needing to be perfected and frozen. Think VMS vs. Unix, cc:Mail vs. RFC-822, Token Ring vs. Ethernet.
• What is, is wrong. Because evolvable systems have always been adapted to earlier conditions and are always being further adapted to present conditions, they are always behind the times. No evolving protocol is ever perfectly in sync with the challenges it faces.
• Finally, Orgel’s Rule, named for the evolutionary biologist Leslie Orgel — “Evolution is cleverer than you are”. As with the list of the Web’s obvious deficiencies above, it is easy to point out what is wrong with any evolvable system at any point in its life. No one seeing Lotus Notes and the NCSA server side-by-side in 1994 could doubt that Lotus had the superior technology; ditto ActiveX vs. Java or Marimba vs. HTTP. However, the ability to understand what is missing at any given moment does not mean that one person or a small central group can design a better system in the long haul.

Centrally designed protocols start out strong and improve logarithmically. Evolvable protocols start out weak and improve exponentially. It’s dinosaurs vs. mammals, and the mammals win every time. The Web is not the perfect hypertext protocol, just the best one that’s also currently practical. Infrastructure built on evolvable protocols will always be partially incomplete, partially wrong and ultimately better designed than its competition.

LESSONS FOR THE FUTURE

And the Web is just a dress rehearsal. In the next five years, three enormous media — telephone, television and movies — are migrating to digital formats: Voice Over IP, High-Definition TV and Digital Video Disc, respectively. As with the Internet of the early ’90s, there is little coordination between these efforts, and a great deal of effort on the part of some of the companies involved to intentionally build in incompatibilities to maintain a cartel-like ability to avoid competition, such as DVD’s mutually incompatible standards for different continents.

And, like the early ’90s, there isn’t going to be any strong meta-protocol that pushes Voice Over IP, HDTV and DVD together. Instead, there will almost certainly be some weak ‘glue’ or ‘scaffold’ protocol, perhaps SMIL (Synchronized Multimedia Integration Language) or another XML variant, to allow anyone to put multimedia elements together and synch them up without asking anyone else’s permission. Think of a Web page with South Park in one window and a chat session in another, or The Horse Whisperer running on top with a simultaneous translation into Serbo-Croatian underneath, or clickable pictures of merchandise integrated with a salesperson using a Voice Over IP connection, ready to offer explanations or take orders.

In those cases, the creator of such a page hasn’t really done anything ‘new’, as all the contents of those pages exist as separate protocols. As with the early Web, the ‘glue’ protocol subsumes the other protocols and produces a kind of weak integration, but weak integration is better than no integration at all, and it is far easier to move from weak integration to strong integration than from none to some. In 5 years, DVD, HDTV, voice-over-IP, and Java will all be able to interoperate because of some new set of protocols which, like HTTP and HTML, is going to be weak, relatively unco-ordinated, imperfectly implemented and, in the end, invincible.

First published in ACM, 11/97.

A funny thing happened on the way from the router. 

We are all such good students of Moore’s Law, the notion that processor speeds will double every year and a half or so, that in any digital arena, we have come to treat it as our ‘c’, our measure of maximum speed. Moore’s Law, the most famously accurate prediction in the history of computer science, is treated as a kind of inviolable upper limit: the implicit idea is that since nothing can grow faster than chip speed, and chip speed is doubling evey 18 months, that necessarily sets the pace for everything else we do. 

Parallelling Moore’s law is the almost equally rapid increase in storage density, with the amount of data accessible on any square inch of media growing by a similar amount. These twin effects are contantly referenced in ‘gee-whiz’ articles in the computer press: “Why, just 7 minutes ago, a 14Mhz chip with a 22K disk cost eleventy-seven thousand dollars, and now look! 333 Mhz and a 9 gig drive for $39.95!” 

All this breath-taking period doubling makes these measurements into a kind of physics of our world, where clock speeds and disk densities become our speed of light and our gravity – the boundaries that determine the behavior for everything else. All this is well and good for stand-alone computers, but once you network them, a funny thing happens on the way from the router: this version of the speed of light is exceeded, and from a most improbable quarter. 

It isn’t another engineering benchmark that is outstripping the work at Intel and IBM, its the thing that often gets shortest shrift in the world of computer science – the users of the network. 

Chip speeds and disk densities may be doubling every 18 months, but network population is doubling roughly annually, half again as fast as either of those physical measurements. Network traffic, measured in packets, is doubling semi-annually (last year MAE-East, a major East Coast internet interconnect point) was seeing twice the load every 4 months, or an 8-fold annualized increase). 

There have always been internal pressures for better, faster computers – weather modelling programs and 3-D rendering, to name just two, can always consume more speed, more RAM, more disk – but the Internet, and particularly the Web and its multi-media cousins of java applications and streaming media, present the first externalpressure on computers, where Moore’s law simply can’t keep up and will never catch up. The network can put more external pressure on individual computers than they handle, now and for the forseeable future. 

IF YOU SUCCEED, YOU FAIL. 

This leads to a curious situation on the Internet, where any new service risks the usual failure if there is not enough traffic, but also risks failure if there is too much traffic. In a literal update of Yogi Berra’s complaint about a former favorite hang-out, “Nobody goes there anymore. Its too crowded”, many of the Web sites covering the 1996 US Presidential election crashed on election night, the time when they would have been most valuable, because so many people thought they were a good idea. We might dub this the ‘Icarus Effect’ – fly too high and you crash. 

What makes this ‘Icarus Effect’ more than just an engineering oversight is the relentless upward pressure on both population and traffic – given the same scenario in the 2000 election, computers will be roughly 8 times better equipped to handle the same traffic, but they will be asked to handle roughly 16 times the traffic. (More traffic than that even, much more, if the rise in number of users is accompanied by the same rise in time spent on the net by each user that we’re seeing today.) 

This is obviously an untenable situation – computing limits can’t be allowed to force entrepreneurs and engineers to hope for only middling success, and yet everywhere I go, I see companies excercising caution whenever they are comtemplating making any moves which will increase traffic, even if that would be make for a better site or service. 

FIRST, FIX THE PROBLEM. NEXT, EMBRACE FAILURE. 

We know what happens when the need for computing power outstrips current technology – its a two-step process, which first beefs up the current offering by improving performance and fighting off failure, and then, when that line of development hits a wall (as it inevitably does), embracing the imperfection of individual parts and adopting parallel development to fill the gap. 

Ten years ago, Wall St. had a similar problem to the Web today, except it wasn’t web sites and traffic, it was data and disk failure. When you’re moving trillions of dollars around the world in real time, a disk drive dying can be a catastrophic loss, and a backup that can get online ‘in a few hours’ does little to soften the blow. The first solution is to buy bigger and better disk drives, moving the Mean Time Between Failure from say, 10,000 hours to 30,000 hours. This is certainly better, but in the end, the result is simply spreading the pain of catastrophic failure over a longer average period of time. When the failure does come, it is the same catastrophe as before. 

Even more disheartening, the price/performance curve is exponential, putting the necessary order-of-magnitude improvements out of reach. It would cost far more to go from 30K/hrs MTBF to 90K/hrs than it did to go from 10 to 30, and going from 90 to 270 would be unthinkably expensive. 

Enter the RAID, the redundant array of inexpensive disks. Instead of hoping for the Platonic ‘ideal disk’, the RAID accepts that each disk is prone to failure, howsobeit rare, and simply groups them together in such a way that the failure of any one disk isn’t catastrophic, because the other disks contain all of the failed disk’s data in a matrix shared among the remaining disks. As long as a new working disk is put in place of the failed drive, the theoretical MTBF of a RAID made of ordinary disks, where two disks failed at precisely the same time, would be something like 900 million hours. 

A similar path of development happened with the overtaking of the supercomputer by the parallel processor, where the increasingly baroque designs of single CPU supercomputers was facing the same uphill climb that building single reliable disks did, and where the notion of networking cheaper, slower CPUs proved a way out that bottleneck. 

THE WEB HITS THE WALL. 

I believe that with the Web we are now seeing the beginning of one of those uphill curves – there is no way that chip speed and storage density can keep up with exploding user base, and this problem will not abate in the forseeable future. Computers, individual computers, are now too small, slow and weak to handle the demand of a popular web site, and the current solution to the demands of user traffic – buy a bigger computer – are simply postponing the day when those solutions also fail. 

What I can see in the outlines of in current web site development is what might be called a ‘RAIS’ strategy – redundant arrays of inexpensive servers. Just as RAIDs accept the inadequacy of any individual disk, a RAIS would accept that servers crash when overloaded, and that when you are facing 10% more traffic than you can handle, having to buy a much bigger and more expensive server is a lousy solution. RAIS architecture comes much closer to the necessary level of granularity for dealing with network traffic increases. 

If you were to host a Web site on 10 Linux boxes instead of one big commercial Unix server, you could react to a 10% increase in traffic with 10% more server for 10% more money. Furthermore, one server dying would only inconvenience the users who were mid-request on that particular box, and they could restart their work on one of the remaining servers immediately. Contrast this with the current norm, a 100% failure for the full duration of a restart in cases where a site is served by a single server. 

The initial RAISs are here in sites like C|NET and ESPN, where round-robin DNS configurations spread the load across multiple boxes. However, these solutions are just the beginning – their version of redundancy is often simply to mirror copies of the Web server. A true RAIS architecture will spread not only versions of the site, but will also spread functionality: images, a huge part of network traffic, are ‘read only’ – a server or group of servers optimized to handle only images could be served from WORM drives and serve the most popular images from RAM. Incoming CGI data, on the other hand, can potentially be ‘write only’ simply recording information on removable medai which can be imported into a database at a later date, on another computer, and so on. 

This kind of development will ultimately dissolve the notion of discrete net servers, and will lead to server networks, where an individual network address does not map to a physical computer but rather to a notional source of data. Requests to and from this IP address will actually be handled not by individual computers, whether singly or grouped into clusters of mirroring machines, but by a single-address network, a kind of ecosystem of networked processors, disks and other devices, each optimized for handling certain aspects of the request – database lookups, image serving, redirects, etc. Think of the part of the site that handles database requests as an organ, specialized to its particular task, rather than as a seperate organism pressed into that particular service. 

THE CHILD IS FATHER TO THE MAN 

It has long been observed that in the early days of ARPANet, packet switching started out by piggy-backing on the circuit-switched network, only to overtake it in total traffic, which will happen this year, and almost certainly to subsume it completely within a decade. I beleive a similar process is happening to computers themselves: the Internet is the first place where we can see that cumulative user need outstrips the power of individual computers, even taking Moore’s law into account, but it will not be the last. In the early days, computers were turned into networks, with the cumulative power of the net rising with the number of computers added to it. 

In a situation similar to the packet/circuit dichotomy, I believe that we are witnessing another such tipping point, where networks are brought into individual computers, where all computing resources, whether cycles, RAM, storage, whatever, are mediated by a network instead of being bundled into discrete boxes. This may have been the decade where the network was the computer, but in the next decade the computer will be the network, and so will everything else.