My, how times have changed.

Today, depending on how many regional, local and "fringe" feeds you choose to carry, Usenet has grown from a mere couple of dozen to more than 22,000 total newsgroups. At its inception, it carried fewer than a dozen new messages a day. Now it accounts for 8 gigabytes or so of daily traffic. Then, it was transmitted via 300 baud acoustically-coupled modems, using UUCP (Unix-to-Unix-CoPy). Now it's distributed at Ethernet and higher speeds courtesy of NNTP (Network News Transfer Protocol)--and, oddly enough, UUCP for efficiency's sake.

The content, architecture, and administration of Usenet have also all changed radically over the nearly two decades of its existence. What once was a compact selection of topics almost exclusively scientific and technical in nature has evolved into a bewildering sprawl of categories so vast as to overwhelm the senses and challenge the courage of any responsible News administrator. Even the original set of top-level hierarchies has joined the dodo, the buggy whip and Joe Camel in what Leon Trotsky called "the dustbin of history." In terms of sheer volume of data, the ochlocratic alt. hierarchy has become the tail that wags the Netnews dog and the once and future Usenet Backbone Cabal has given way to the "Internet standard model" of government by consensual anarchy.

Marx would have said it was historically inevitable.

Then again, Marx was a nitwit.

A Crowd of People Stood and Stared..

It's hard to imagine, but in the early years, the Internet--then known by its original moniker, "ARPAnet"--was a pretty exclusive place. Simply being physically able to connect to it wasn't nearly sufficient grounds to justify your admittance to this rarified environment. You not only had to be a research laboratory with defense contracts, but your users often had to be individually cleared for access to this Defense Department-funded network. Often, that meant some of your staff had access and some didn't.

And that meant you had a problem. The Captain in Cool Hand Luke called it a "failure to communicate." Still worse, the medium of choice for mass communication on the ARPAnet was the mailing list, with all the overhead, relatively high bandwidth and duplication of effort that entails.

In 1979, a group of computer scientists at Duke University and the University of North Carolina (primarily Steve Bellowin, Stephen Daniel, Jim Ellis, and Tom Truscott) came up with a better way, courtesy of the then-new Unix version 7 and a handy new utility called UUCP. The Duke/UNC crew realized that UUCP, despite its poor documentation, made possible communication between Unix hosts via dial-up modem lines. Unix now also offered store-and-forward and remote execution capabilities that hadn't existed prior to version 7. Those two new functions meant that the UNC lights could build a low-speed, UUCP-based communications network separate from the ARPAnet--one to which any organization willing to spend the money on modems and Unix boxes could belong.

Now, in 1979, modems were agonizingly slow. Factor in long-distance charges and it was clear that a more efficient model than the ARPAnet mailing lists was going to be needed, if this alternate communications net was ever to work. In an act of collective brilliance born of necessity, the distributed computer BBS came into being. It found its first formal expression in a package (authored mainly by the Duke contributors) called A News.

A News was clunky. The mail relayer daemon and the newsreader were one, monolithic piece of software. The user interface sucked and the article format wasn't readily extensible and it didn't quite conform to the ARPA mail spec, so interoperation was a problem.

But, what the heck, it worked, more or less. And after Jim Ellis gave a talk to attendees at the Winter 1980 Usenix conference on what he and his co-conspirators at Duke and UNC had cooked up, the Unix community--most of which was still on the outside of the ARPAnet, looking wistfully in--rushed to embrace it.

Made The Bus In Seconds Flat..

Usenet--a play on Usenix--then exploded in popularity. It swiftly grew from 3 initial sites in 1979 to include more than 600 by the end of 1983. A News was hopelessly--and increasingly--inadequate for the rapidly-expanding load. Beginning in 1981, it was extensively rewritten by Matt Glickman and Mark Horton at U.C. Berkeley, who named their creation B News. They goosed the daemon's performance and reworked the article format to make it slightly more extensible. They also made it more nearly compliant with the ARPA mail standard, somewhat easing the interoperability problem.

Once he finished B News, Horton authored Internet RFC 850, Standard for Interchange of Usenet Messages to let ARPAnauts know how Usenet worked. He was careful to include a disclaimer that RFC 850 did not "specify an Internet standard," but, even then, the handwriting on the wall was pretty clear.

Even B News didn't last very long. The server/client bundle was unwieldy and there was still considerable room for improvement in performance of both the server and the transport. Thus, in 1985, Geoff Collyer and Henry Spencer of the University of Toronto rewrote B News, dropping the bundled news reader and tweaking the server and transport. They called the result C News and, subject to bug fixes and incremental twiddling, it's still the standard Internet news software today--although INN (which was developed from scratch in 1992) gives it a run for its money (they're both freeware) and D News is steadily picking up adherents.

What C News didn't do was to mess with the article structure that Horton and Glickman had established with B News. In 1987, Rick Adams reworked Mark Horton's RFC 850 to reflect the changes that the last implementation of B News had wrought in the structure of Usenet articles and posted the codified update as Internet RFC 1036. That RFC--which continued the fiction that it did not reflect an Internet standard--is still current, despite its age and the modern infiltration of MIME inclusions into Usenet articles.

Perhaps the single most important technical change of the mid-1980s was the development by Brian Kantor and Phil Lapsley (with help from Erik Fair, Steven Grady, and Mike Meyer, among others) at the University of California in early 1986 of the Network News Transfer Protocol Reference Implementation, which accompanied Usenet's integration into the Internet proper. Although NNTP suffers from its own set of performance problems, (which is why UUCP is still in widespread use as a news transport mechanism, even in the pure Ethernet environment of the Internet proper,) the Reference Implementation became and remains the standard.

In June 1994, Henry Spencer attempted to update RFC 1036 in a draft formally known as News Article Format and Transmission, but he eventually gave up on the effort. Even so, the de facto current best practices working standard is generally taken to be a combination of his draft and RFC 1036.

In July 1997, a Usenet Article Standard Update Working Group was formed with a charter to formalize and document the current and proposed extensions to the Usenet article format in a standards-track successor to RFC 1036. Among other things, this effort would address signing and authentication of both regular articles and control messages, (particularly cancel requests) the use of non-ASCII characters in headers and bodies, standardize article body conventions in general and the use of MIME in particular and to revise upward the 60-kilobyte limit on article size established in RFC 1036. If they meet their announced timetable, their draft will be submitted to the Internet Engineering Standards Group (IESG) sometime in mid-1998.

Four Thousand Holes in Blackburne-Lancashire..

Meanwhile, it became increasingly clear that the original three Usenet newsgroup hierarchies--net, fa and mod (respectively meaning Usenet-wide, from ARPAnet and moderated)--were simply too general to be usefully meaningful as the number of subgroups and the range of subjects they addressed grew and grew and grew. In 1986, the Backbone Cabal, (so-called by virtue of its members' willingness to foot the bill for the long-distance charges that distributing Usenet feeds around the globe incurred,) which made all the important decisions about Usenet, decided it was time to restructure. The result, which took until 1987 to fully take hold, was known as the Great Renaming, and it resulted in the creation of the seven "official" Usenet global hierarchies we know and love today: comp, misc, news, rec, sci, soc and talk.

The Great Renaming was all very well and good, but the "official" hierarchies evinced an increasingly unacceptable degree of conservatism that was even more strongly reflected in the policies by which the Cabal approved the creation of new newsgroups. Even though a somewhat democratic procedure for new group creation by interested parties formally existed, the members of the Cabal reserved the right of veto over even the most popular proposals via the simple mechanism of refusing to carry new groups of whose charters they disapproved.

In early 1986, the Cabal turned down rec.music.rock-n-roll. It refused to have anything to do with rec.drugs. And, of course, sex was the real bugaboo. The poor Cabalistas were all terrified that one of the suits upon whose favor their budgets depended might discover rec.sex, for instance and raise a stink about misuse of organizational resources, embarrassing individual Cabalistas and perhaps even getting them shut down. So they refused to carry soc.sex, even though the popular vote was overwhelmingly in favor of its creation.

That's when John Gilmore, employee number five at Sun Microsystems, co-founder of the Electronic Frontier Foundation and original host of the cypherpunk mailing list, stepped in to create (with help from Brian Reid and Gordon Moffett) the alt hierarchy. Originally, the alt groups were distributed via a UUCP network completely separate from the one the Cabal maintain--and which Gilmore financed most of out of his pocket. Sex found its home there, as did drugs and rock-n-roll, along with gourmet cooking, discussions of alien abduction and a host of other topics too weird, too controversial or just plain too much fun for the stuffy confines of the seven "official" hierarchies. Eventually, demand for the alt groups grew to the point that, today, their inclusion in a basic news feed is more-or-less a given.

Gilmore's original injunction was to "use common sense" in alt newgroup creation, but nowadays, it has become the wild card in the Usenet deck. Although theoretically newgroup proposals are supposed to be submitted to alt.config to test their support first, in practice, any Usenet admin can create an alt group. That's how alt.swedish.chef.bork.bork.bork came into existence--as a joke--and became the first one of a flood of supposedly humorous alt group names.

The proliferation of joke newgroups led some thoroughly cheesed-off news admins to begin issuing rmgroup control messages to try and prune the alt thicket. That, in turn, led other News admins of a more anarchic bent to re-send those same messages as newgroup control messages under the names of the wanna-be net.cops who had issued the originals. That game quickly ended in a stalemate which inevitably left the joke groups intact--although largely unvisited by anyone except the massive influx of Ponzi schemers that coincided with America Online's opening of its Usenet gateway in early 1996 and the sex site operators who have relentlessly spammed the alt groups since reliable mechanisms for securely accepting credit card information first came into widespread use in mid-1996.

Of course, the alt.sex family now gets by far the most spam. That pumps up the volume of alt posts considerably. So, too, does the prodigious traffic in the alt.binaries tribe--which once upon a time was all uuencoded, but is now increasingly MIME-encoded.

I'd Love To Turn You On..

Some of this stuff I've known for years. Some of it I only recently learned, courtesy of Managing Usenet by Henry Spencer and David Lawrence (copyright 1998 by O'Reilly & Associates, $32.95, ISBN 1-56592-198-4). It's a great book that covers the history of Usenet in both capsule form in the Introduction and in greater depth in Chapters 16 and 17. Managing Usenet also goes into considerable detail on how to install and configure both C News and INN, the two major Unix-based Net news relayers, and it nicely explains the architecture and administration of Usenet itself. I recommend it without reservation for Unix News admins--and for aficionados of other operating systems with the admonition that you'll have to go elsewhere for instructions and advice on installing relayers for other platforms.

Speaking of which, you can get the C News relayer and NNTP Reference Implementation at cs.toronto.edu/pub/cnews.tar.Z. INN is available at www.isc.org/inn.html. (For INN, you'll need a lot of RAM, recent versions of egrep and awk and the latest build of Perl, too.) Other Unix relayers you might consider are Cyclone, the very fast Diablo or Netscape's Collabra Server, which is also available for Windows NT.

Since that takes us to the subject of NT, Microsoft's Internet Information Server includes a News server--although you should be aware that this "free" software will only install to NT Server 4.0, rather than the less-expensive Workstation version. MetaInfo's NewsChannel 2.0 also runs on NT. Alternatively, Ukiahsoft's NetRoad NewsServer runs as an NLM on NetWare 3.x, 4.x and IntraNetWare servers, and D News comes in versions that run on a dozen flavors of Unix, as well as on NT, NetWare, OS/2, the MacOS and Vaxen.

Having read the book, you'll just have to look at some of the groups that deal with various aspects of Usenet itself, such as: news.announce.newusers, news.announce.newgroups, news.groups, news.admin.misc, news.answers and (for information about the alt hierarchy) alt.config and alt.answers.

Goo-goo-ga-jub.

(Copyright© 1998 by Thom Stark--all rights reserved)