Project Xanadu: Even More Hindsight

Retrospective on Project Xanadu’s success and failure: a lack of design iteration, meaningful use-cases, or practicality stopped a valuable vision from maturing into something useful. (And contrasted with my approach.) In December 2024 during a visit to San Francisco, I was lucky enough to be invited at the last minute to a party that could only have happened there: a celebration of the 50th anniversary of hypermedia visionary Ted Nelson’s 197452ya manifesto, Computer Lib/Dream Machines, extolling the vision of Project Xanadu hypertext. I’ve contributed to English Wikipedia for 20 years now, and I’ve been working on Gwern.net on and off for 15 years now, so I could not possibly miss an entire party of people with strong opinions on hypertext. Our host, James, had arranged for a surprisingly extensive collection of Nelson memorabilia: not just copies of that book (far larger and more impressive in person than I had realized, similar to a Compact OED in requiring a magnifying glass, also provided), or Nelson’s 199729ya book The Future of Information, but also copies of his Swarthmore College mimeograph magazines, and most impressive of all—several vintage computers running copies of various Xanadu implementations. Ted Nelson was still alive at age 87, but unfortunately could not attend. Fortunately, one of the attendees was one of the former Xanadu programmers from the Autodesk era (c. 1988–5199333ya), and we could listen to some of his stories. They were a reminder of how, while we romanticize earlier eras of computing, the hardware constraints were really quite severe, and made productive development difficult. I think some disappointments in past software systems become more comprehensible when we remember how much time and ingenuity is spent working around the limitations—eg. McIlroy 1982 is justifiably proud of the months of clever algorithm design & optimization he did to get a useful spellchecker to fit in RAM in under 1MB that today we would write in a few lines of JavaScript (and done by an LLM). For example, he described how they prototyped Xanadu in Smalltalk (which made sense, as a highly-productive, pleasant language/OS whose object-oriented model matches hypermedia beautifully), but then had to cross-compile it to C++ and compile that… which took about a week to compile. Not a minute, or an hour, or even a day, but a week. (And I thought that Gwern.net’s multi-hour compile-times were bad for my development speed!) He also had to waste a lot of time dealing with C++ silliness and compile issues.1 Getting this to run at all for the PCs we saw was a challenge, and one reason for the party. I had, I must admit, sometimes wondered how the Xanadu years at Autodesk could have so little to show for multiple fulltime man-years, when implementing the various client-side transclusion or popup features on Gwern.net were typically a few days of work for Said Achmiz. But hearing some war stories from the horse’s mouth helped put things in proper perspective, and remind me just how incredibly compute-impoverished people were at that time. (Perhaps worse than the CPU was the storage: I take for granted being able to host any PDF or PostScript or HTML file I need, but a meaningful fraction of my hosted documents exceed the total hard drive space of a mid-range 199036ya PC, which might have a 50 MB hard drive. Meanwhile, the Markdown source files of the Gwern.net essays are themselves ~40MB, the annotations twice that, and the final site is 221,438 MB!) The college magazines were a surprise and entertaining to leaf through: young Ted already liked to write, quite a lot, and dispense his advice. Someone mentioned that Nelson had dreamed of being a Hollywood director and regretted that he went into technology instead; I thought that made sense and explained some things about his auteur approach to software development (like his insistence he is “not a programmer”, 50 years later) or use of film-editing metaphors like “edit decision lists”. I also had never sat down to read Computer Lib/Dream Machines properly, and leafed through a few pages. It was interesting to see such a large book, in multiple columns to get as much in as possible: Nelson can’t assume the reader knows anything about computers and has to start from scratch to explain the basic concepts like bytes or files. (You really would need the magnifying glass if you were older.) The later (and much more obscure) book Future of Information was also interesting for an unusual structure of chapters, where you could read in multiple orders, with a central summary chapter. Yuxi Liu points out that Xanadu resembles another famous long-running boil-the-ocean project with close connections to databases & AI with a charismatic leader who never changed his mind, averse to open-source, and which showed similar signs of ‘pathological science’: Douglas Lenat’s Cyc. And thinking about it and re-reading Xanadu materials, I agree. I briefly poked at the Xanadu PCs, impressed that they were running at all, but I and most of the party-goers bounced off them. The UI was too alien. We really needed to see James demo them, or something like that: hypertext systems do not lend themselves to immediate exploration, especially when they are running in OSes on computers no one there has used in 25 years, if ever. But I didn’t need to use them much to look at the screen displaying the stereotypical Project Xanadu demo, the opening of the Book of Genesis with its famous lines zig-zagging off to the right to denote transclusions or commentary on passages, and have a sudden realization: “Oh my god—It’s completely unreadable.” The lines were confusing clutter, especially as they crisscrossed (a perennial problem in sidenotes layout, made far worse by the outlines). None of the ‘sidenotes’ were readable because the screen was so small. Even as you simply scrolled, for many possible positions, due to the lines you were unable to read anything! How could a document UI where often, you could read nothing, have ever seemed like a good idea?2 The UI was just terrible—it could never have worked. Even on a large screen like my 4k monitor, I wouldn’t want that. And then I thought about the choice of text, and I realized that the UI wasn’t the real problem; and the problem, for all these many decades, wasn’t the team either. The whole concept of side-by-side range transclusions is a solution in search of a problem. The range-specific transclusion & commentary made sense for the Book of Genesis, where there are detailed commentaries on every line, and much Biblical criticism sorting out how it’s redacted from multiple contradictory texts (like the famously self-contradicting multiple stories of creation), but as I thought to myself about how “hey, we can do sidenotes and range transcludes/commentaries with bidirectional backlinks on Gwern.net too, and we do do it in my ”Suzanne Delage” short story analysis!”3, I suddenly realized: we can, but we mostly don’t, because no one really needs to do that. This is especially true if we look at Ted Nelson’s “Examples of Parallel Documents”: textual criticism again (the Bible, Hamlet, “Rashomon” vs the movie Rashomon), the stretched example of the Virginia Declaration of Rights influencing the United States Bill of Rights, and then very dubious examples—lists of saints, telephone phone-calls inside an organization, and comedy TV episodes…? Hardly any text in the world actually needs to be fisked, or have specific lines or paragraphs transcluded; hardly anyone is doing Talmudic commentary, nested layer upon layer. Most real instances of citations are citations to the target as a whole. And even with those, we wouldn’t want to see 99% of them, and we wouldn’t know how to organize the good 1% anyway because they may each have a different purpose. (Just look at online comments sometime: do we sort them by date, by popularity or some sort of karma or ‘page rank’, by length, by whether the page author responded directly to them…?) In retrospect, I think it’s telling that in neither Lib/Dream nor Future did I see any instances of ‘transclusion’ on paper. Nelson could have used any amount of side-by-side layout or range-transclusion in his books, because software is no obstacle: he was laying them out by hand, and could draw or illustrate or copy anything he wanted in any arrangement. But he didn’t, because… it’s just not that useful for books. Not even his. (You don’t need transclusions when you can just cite an earlier passage.)