First published on O’Reilly’s OpenP2P.

“Whatever else you think about, think about interoperability. Don’t think about standards yet.”

Nothing I said at the O’Reilly P2P conference in San Francisco has netted me more flak than that statement. To advocate interoperability while advising caution on standards seems oxymoronic — surely standards and interoperability are inextricably linked?

Indeed, the coupling of standards and interoperability is the default for any widely dispersed technology. However, there is one critical period where interoperability is not synonymous with standardization, and that is in the earliest phases of work, when it is not entirely clear what, if anything, should be standardized.

For people working with hardware, where Pin 5 had better carry voltage on all plugs from the get-go, you need a body creating a priori standards. In the squishier field of software, however, the history of RFCs demonstrates a successful model where standards don’t get created out of whole cloth, but ratify existing practice. “We reject kings, presidents and voting. We believe in rough consensus and running code,” as David Clarke put it. Standardization of software can’t proceed in a single giant hop, but requires some practical solution to point to first.

I take standardization to be an almost recursive phenomena: a standard is any official designation of a protocol that is to be adopted by any group wanting to comply with the standard. Interoperability, meanwhile, is much looser: two systems are interoperable if a user of one system can access even some resources or functions of the other system.

Because standardization requires a large enough body of existing practice to be worth arguing over, and because P2P engineering is in its early phases, I believe that a focus on standardization creates two particular dangers: risk of premature group definition and damage to meaningful work. Focusing on the more modest goals of interoperability offers a more productive alternative, one that will postpone but improve the eventual standards that do arise.

Standardization and Group Definition

A standard implies group adoption, which presupposes the existence of a group, but no real P2P group exists yet. (The P2P Working Group is an obvious but problematic candidate for such a group.) The only two things that genuinely exist in the P2P world right now are software and conversations, which can can be thought of as overlapping circles:

• There is a small set of applications that almost anyone thinking about P2P regards as foundational — Napster, ICQ and SETI@Home seem to be as close to canonical as we’re likely to get.
• There is a much larger set of applications that combine or extend these functions, often with a view to creating a general purpose framework, like Gnutella, Jabber, Aimster, Bitzi, Allcast, Groove, Improv, and on and on.
• There is a still larger set of protocols and concepts that seem to address the same problems as these applications, but from different angles — on the protocol front, there are attempts to standardize addressing and grid computing with things like UDDI, XNS, XML-RPC, and SOAP, and conceptually there are things like the two-way Web, reputation management and P2P journalism.
• And covering all of these things is a wide-ranging conversation about something called P2P that, depending on your outlook, embraces some but probably not all of these things.

What is clear about this hodge-podge of concepts is that there are some powerful unlocking resources at the edges of the Internet and democratizing the Internet as a media channel. 

Does P2P even need standards? What should the work of the P2P Working Group be?

Tell us what you think.

What is not clear is which of these things constitute any sort of group amenable to standards. Should content networks use a standard format for hashing their content for identification by search tools? Probably. Would the distributed computation projects having a standard client engine to run code? Maybe. Should the people who care about P2P journalism create standards for all P2P journalists to follow. No.

P2P is a big tent right now, and it’s not at all clear that there is any one thing that constitutes membership in a P2P group, nor is there any reason to believe (and many reasons to disbelieve) that there is any one standard, other than eventually resolving to IP addresses for nodes, that could be adopted by even a large subset of companies who describe themselves as “P2P” companies.

Standardization and Damage to Meaningful Work

Even if at this point, P2P were a crystal-clear definition –within which it was clear which sub-groups should be adopting standards — premature standardization risks destroying meaningful work.

This is the biggest single risk with premature standardization — the loss of that critical period of conceptualization and testing that any protocol should undergo before it is declared superior to its competitors. It’s tempting to believe that standards are good simply because they are standard, but to have a good standard, you first need a good protocol, and to have a good protocol, you need to test it in real-world conditions.

Imagine two P2P companies working on separate metadata schemes; call them A and B. For these two companies to standardize, there are only two options: one standard gets adopted by both groups, or some hybrid standard is created.

Now if both A and B are in their 1.0 versions, simply dropping B in favor of A for the sole purpose of having a standard sacrifices any interesting or innovative work done on B, while the idea of merging A and B could muddy both standards, especially if the protocols have different design maxims, like “lightweight” vs. “complete.”

This is roughly the position of RSS and ICE, or XML-RPC and SOAP. Everyone who has looked at these protocols has had some sense that these pairs of protocols solve similar problems, but as it is not immediately obvious which one is better (and better here can mean “most lightweight” or “most complete,” “most widely implemented” or “most easily extensible,” and so on) the work goes on both of them.

This could also describe things like Gnutella vs. Freenet, or further up the stack, BearShare vs. ToadNode vs. Lime Wire. What will push these things in the end will be user adoption — faced with more than one choice, the balance of user favor will either tip decisively in one direction, as with the fight over whether HTML should include visual elements, or else each standard will become useful for particular kinds of tasks, as with Perl and C++.

Premature standardization is a special case of premature optimization, the root of all evil, and in many cases standardization will have to wait until something more organic happens: interoperability.

Interoperability Can Proceed by Pairwise Cooperation

Standardization requires group definition — interoperability can proceed with just a handshake between two teams or even two individuals — and by allowing this kind of pairwise cooperation, interoperability is more peer-to-peer in spirit than standardization is. By growing out of a shared conversation, two projects can pursue their own design goals, while working out between themselves only those aspects of interoperability both consider important.

This approach is often criticized because it creates the N² problem, but the N² problem is only a problem for large values of N. Even the largest P2P category in the O’Reilly P2P directory — file sharing — contains only 50 entries, and it’s obvious that many of these companies, like Publius, are not appropriate targets for standardization now, and may not even be P2P.

For small numbers of parallel engineering efforts, pairwise cooperation maximizes the participation of each member of the collaboration, while minimizing bureaucratic overhead.

Interoperability Can Proceed Without Pairwise Cooperation

If a protocol or format is well-documented and published, you can also create interoperability without pairwise cooperation. The OpenNAP servers adopted the Napster protocol without having to coordinate with Napster; Gnutella was reverse-engineered from the protocol used by the original binary; and after Jabber published its messaging protocol, Roku adopted it and built a working product without ever having to get Jabber’s sign-off or help.

Likewise, in what is probably the picture-perfect test case of the way interoperability may grow into standards in P2P, the P2P conference in San Francisco was the site of a group conversation about adopting SHA1 instead of MD5 as the appropriate hash for digital content. This came about not because of a SHA1 vs MD5 Committee, but because Bitzi and OpenCOLA thought it was a good idea, and talked it up to Freenet, and to Gnutella, and so on. It’s not clear how many groups will eventually adopt SHA1, but it is clear that interoperability is growing, all without standards being sent down from a standards body.

Even in an industry as young as ours, there is a tradition of alternative interfaces to file-sharing networks for things like Mac, Linux and Java clients being created by groups who have nothing more than access to publicly published protcols. There is widespread interoperability for the Napster protocol, which is a standard in all but name, and it has approached this state of de facto standardhood without any official body to nominate it.

Interoperability Preserves Meaningful Work

The biggest advantage of pursuing interoperability is that it allows for partial or three-layer solutions, where interested parties agree to overlap in some but not all places, or where an intermediate layer that speaks to both protocols is created. In the early days, when no one is sure what will work, and user adoption has not yet settled any battles, the kludgy aspects of translation layers can, if done right, be more than offset by the fact that two protocols can be made interoperable to some degree without having to adjust the core protocols themselves. 

What Is Needed

To have standards, you need a standards body. To have interoperability, you just need software and conversations, which is good news, since that’s all we have right now.

The bad news is that the conversations are still so fragmented and so dispersed. 

There are only a handful of steady sources for P2P news and opinion: this site, Peerprofits.com,, the decentralization@yahoogroups.com mailing list, the P2P Working Group and a handful of people who have been consistently smart and public about this stuff — Dan Bricklin, Doc Searls, Dan Gillmor, Dave Winer and Jon Udell. While each of these sources is interesting, the conversation carried on in and between them is far from being spread widely enough to get the appropriate parties talking about interoperability.

As a quick sampling, Openp2p.com’s P2P directory and Peerprofit.com’s P2P directory list about 125 projects, but only 50 groups appear on both lists. Likewise, the Members List at the P2P Working Group is heavy on participating technology companies, but does not include Freenet, Gnutella, OpenCola or AIMster.

The P2P Working Group is one logical place to begin public conversations about interoperability, but it may be so compromised by its heritage as a corporate PR site that it can never perform this function. That in itself is a conversation we need to have, because while it may be premature to have a “Standards Body,” it is probably not premature to have a place where people are tying to hammer out rough consensus about running code. 

The decentralization list is the other obvious candidate, but with 400 messages a month recently, it may be too much for people wanting to work out specific interoperability issues.

But whatever the difficulties in finding a suitable place or places to have these conversations, now is the time for it. The industry is too young for standards, but old enough for interoperability. So don’t think about standards yet, but whatever else you think about, think about interoperability.