At the Foothills of Eccentricity

Pipedream: an incomplete review of the literature

Previously on this blog, I introduced the idea of Pipedream, a novel system for creating and understanding software1. This post is meant to sketch out some of the historical work on similar systems, in order to help understanding of the new ideas.


Yes. On with it!

Values and visions

If I thought that the way programming was typically done was mostly OK, I wouldn’t need to make something like Pipedream. Without the following influences, I wouldn’t be mad enough about programming sucking to try to make it.

Bret Victor

One of the driving ideas or values that motivate Pipedream, and in particular its structural editor, The Language of Making, is the notion that manipulating raw software entities in a direct and tangible way should be possible, and that “code” is not software, but merely one possible representation of it. A lot of the credit for my ideas or values along those lines goes to Bret Victor, who either articulated, amplified, or engendered them or their seeds. The first big splash he made in the programming UI/UX community I was aware of was from a talk in 2012 called “Inventing on Principle” where he demonstrated some tools he’d made in order to make the effects of changes in software code be more immediately understandable and extensible, essentially by creating custom visualization tools for functions or datatypes and hooking that into the code editor. There were also some affordances shown for more directly manipulating values within the code (jump to right around the 5-minute mark in the video linked for an example).

This lead to the discovery of other essays or talks on his site, like “Learnable Programming” or “Up and Down the Ladder of Abstraction”, both of which emphasized the need for the ability of the user of the software system to be able to directly manipulate elements within the system, and to get immediate feedback on the concrete effects of whatever action was just taken. These talks inspired a couple popular tools, first with Lighttable, an interactive code editor, and then the more ambitious Eve; see more below. The other big project that was directly inspired by these ideas is Apple’s Swift Playgrounds, which implement flows almost directly as seen in “Inventing on Principle”.2

He’s consistently pushed on the ideas of real-time interactive dynamic feedback on changes to whatever you’re doing with the computer (either writing software or creating an interactive visualization, as with “Drawing Dynamic Visualizations” ). “Media for Thinking the Unthinkable” is a talk and set of interactive demos that do a decent job of summing up his main influences on the kinds of interactions Pipedream is meant to enable. But my favorite talk is called called “The Future of Programming”.

“The Future of Programming” is a talk given in 2013 that pretends to be in 1973. Bret Victor walks up to the stage wearing a 70s nerd outfit and proceeds to use a digital scanner/projector in order to show a series of transparencies that discuss the innovative ideas in human/computer interaction from the last ten years (1963-1973), and admonishes the budding “computer scientists” in the room to be sure to not let the broken paradigms of files and teletexts and tedious imperative programming to become entrenched in computing over the next forty years. There’s too much good stuff to go into any detail, so just go ahead and watch it; here’s an embedded version for your convenience!

It’s only thirty-three minutes, and if you’ve ever had to use a computer to create software, even with a spreadsheet, you’ll likely walk away amazed and depressed at the mistakes made over the last forty years in nearly equal measure, but also maybe you’ll be inspired. I was. The tragedy on display in this talk is the lack of our computers keeping track of the books for us in a way that we can understand.

Currently, Bret is creating a building-sized immersive computing environment called Dynamicland whose details are a little out of scope for Pipedream, but it is extremely cool. If you’re in Oakland, you should try to figure out how to get a tour.


The Eve project, mentioned above, has shut down, but it was open source, so their work is not totally lost. Their dev blog has a lot of great videos and gifs and whatnot, and a lot of the ideas for the concrete interface and experience within Pipedream (and specifically within the Language of Making, the structural editor component of Pipedream) are strongly influenced by Eve, especially like such as e.g.

The main developer of Lighttable, and one of the founding developers of Eve, is named Chris Granger, and he has a great post from 2014, before Eve was announced, that goes into some detail regarding the problems with the way that programming is done in the mainstream.

Alan Kay, and Smalltalk and Self

Meta-talk for a moment: in the course of trying to find a link to Alan Kay’s canonical home on the web, I realized that he looms so large in the history of personal computing that the idea of a canonical, single place for him is ludicrous. Also, he’s still alive, so even if you could pin him down for a moment, whatever it was you’d been pointing at would be out of date. So with that in mind, here’s a reasonable place to start, for a giant list of his papers about what interacting with computers could be like:; you can filter by author there, and he’s prolific.

OK, for the impatient, here’s his Wikipedia entry. But among his greatest hits are the Dynabook, a “personal computer for children of all ages”, and the software required to power it, Smalltalk. Bee-tee-dubs, Smalltalk makes an appearance in “The Future of Programming”, so if you still haven’t given in and watched it yet, now’s a good time!

One of the big ideas in Smalltalk is that the canonical source of a piece of software is not a text file of “code”, but rather, a living process called an image, that was basically the current state of the memory of the Smalltalk virtual machine. You program in Smalltalk within a self-contained world whose tools are more like a video game. The entire Smalltalk development environment is itself a Smalltalk program, and is available for inspection, modification, and appropriation for other programs, at all times.

This was in direct support of the mission of the Dynabook, which was the 1972 version of the iPad. It was a tablet computer, though being the 70s, it still had a keyboard. However, unlike the iPad, it was not made by an entity that hated and mistrusted its users2, so instead of running iOS where nothing can be inspected or modified, it ran Smalltalk, where the opposite was true, as just described. One of the key Pipedream-related take-aways here is the idea that a really good and humane interface for software probably requires some special hardware; the Dynabook for Smalltalk; cameras, mics, and touchscreens for Pipedream.

Smalltalk still exists, and even has a growing community of users. The most popular is called Pharo. Actually getting it up and running, and then playing with it, is not super easy to figure out from just the text, but there’s a whole online class for using it, with videos and exercises and stuff. In a later post, I’ll address, “Why not Smalltalk?” for “Why Pipedream?”, but not today.

Smalltalk itself inspired further exploration along the spectrum of “turtles all the way down” conceptual simplicity, a live and dynamic environment, and total transparency/reflectivity within the system, in the form of Self, a version of Smalltalk based on prototypes instead of classes. From the linked paper3:

The Self system attempts to integrate intellectual and non-intellectual aspects of programming to create an overall experience. The language semantics, user interface, and implementation each help create this integrated experience. The language semantics embed the programmer in a uniform world of simple objects that can be modified without appealing to definitions of abstractions. In a similar way, the graphical interface puts the user into a uniform world of tangible objects that can be directly manipulated and changed without switching modes. The implementation strives to support the world-of-objects illusion by minimizing perceptible pauses and by providing true source-level semantics without sacrificing performance. As a side benefit, it encourages factoring. Although we see areas that fall short of the vision, on the whole, the language, interface, and implementation conspire so that the Self programmer lives and acts in a consistent and malleable world of objects.

The lesson for Pipedream here, similarly to the one with the Dynabook and Smalltalk, is that the act of programming is subject to the rigors of user experience, and there’s more to a programming system than just the language’s syntax, or environment, or semantics, and all those factors (and more!) should be considered fair game for coherent criticism or integration.

Hoo… there’s kind of a lot there.

Believe me, I know. But that’s the end of the first part, so here’s a picture of my cat, The Cheat, taking a nap in the dirt. He’s really old4, so I think he’s practicing for his last big one.

The Cheat, taking a dirt-nap like a dirty dirt-monster


My pleasure! Let’s get into the second part:


Pipedream is meant to be a real, physical system made of hardware and software5. The Pipedream software is meant to take unique advantage of not just the keyboard and mouse, but also, microphones and cameras and touchscreens, in order to allow users and the system to fluently interact with each other, in order to automate as much of the software creation process as possible, while still fully empowering and enlightening the user. To recap from the previous post, there are four main pieces to Pipedream:

  • The Language of Making, a structural editor for software, that exposes your software as a high-level abstract graph, enabling automated generation of low-level coding details that is inspectable and modifiable by the user;
  • The Magic Mirror, a multi-sensory input system using cameras, microphones, touchscreens, mouse, and keyboard in order to provide gesture-driven UIs, where gestures are things like waving your hands, speaking, pressing a key combination on a keyboard (eg, double-tapping both shift buttons quickly), moving your head, moving a mouse, clicking, grunting, frowning, etc.;
  • Jeens, non-character-based intelligent agents that observe you through Magic Mirror, and communicate with you in context-appropriate, idiomatic ways, like by causing an element on the screen to pulsate softly, or by modulating the order of options being presented to you; the main idea is that they’re not like cartoons who are trying to talk to you;
  • Passing the Pipe, the integrated sub-system for sharing and finding software that Pipedream knows about, built on a distributed content-addressable storage system, that includes web-of-trust creation and exploration for verifying publication, and a system for creating and sharing tag-based ontologies6.

Each one of those quite a bit of historical precedent, and as I discovered tonight while writing this, the original Clippy was meant to be pretty much exactly what Jeens are meant to be; I’ll show the video below. But first, let’s take a look at the Language and the Mirror.

The True Language of Making

I know, I know, there’s a lot of flowery metaphor in that first post, and this one’s a mouthful. But, the official documentation will be skinnable to let you come up with a better name for it, so in the meantime, it’s the Language for short. It’s the main visible interface for Pipedream, and the most fundamental component.

Like with most ideas, this one is kind of in the air lately. The benefits of structural editing are many, including the software code being correct by construction, meaning, never ever have compiler errors. Structural manipulation means that you’re free to wildly move components around, and their regular structure makes them amenable to being manipulated by machines, who might be working on your behalf.

Anyway, there’s a whole ton of content available for systems like the Language, and the next blog post will be a walk-through of using it in Pipedream.

The Magic Mirror

Terminator Vision

Pictured above is the output of a simple program that uses the webcam on my laptop in order to estimate the orientation (“pose”) of my head. Within Pipedream, the Magic Mirror would be responsible for analyzing that video stream in order to get the pose estimation, and then use that information to control a type of secondary pointer within Pipedream. In general, interaction in Pipedream is via gestures, as previously described. A Pipedream jeen may perform some action, present you with a sutble indication, and wait for you to say “yes” or nod your head; it may monitor what emotion your face is indicating and use that to train itself on what it thought you wanted.

It’s hard to talk about the Mirror without getting into the soft details of the UX. Those are important, but also, hard to describe in text. An in-depth look at it will be forthcoming once you can see a movie of it in action.


OK, back in that first post, I joked that Clippy was exactly not what Jeens were supposed to be like.

Then I read this page on the research that was supposed to drive Clippy, and saw this video (see below for a walk-through):

and my jaw hit the floor. Here’s the little summary of the video from the page in the first link here:

The Bayesian inference engine demonstrated in the above video works like a charm monitoring the user’s behavior, inferring his intent, and providing help in a contextual and personalized fashion.

And that’s exactly right! This demo of Clippy is awesome, and I want to walk through it a little more explicitly, so let’s follow along because this is pretty much exactly how you’d see the Jeens in Pipedream behaving.

  • 1:43 the dude “menu surfs” in order to indicate confusion. A Jeen could behave the same, or menu surfing could be explicitly interpreted by the Magic Mirror as the “I need help” gesture
  • note that it’s popping up help right away, not asking if it should bring up a help resource, and the system knows from its experience with the user that these are the things they need help with
  • 1:59 shows off the controls for the help system itself, in terms of setting thresholds for how much it should make itself be seen, demonstrating that the user is truly in control
  • 2:44 the system understands the domain of need for expertise (Excel), and since it’s been watching you, it has some idea of what you might be looking for
  • 3:04 you have insight into what the system thinks of you and why
  • 5:20 the system is sensitive to how expert the current user is; in Pipedream, your Jeens would know who you were and what your history was, so you wouldn’t need to tell it your level of experience
  • 6:09 the guy doing the demo is attempting to show natural speech searching, but really, doing some simple tag-based queries over the subject matter would probably be easier and more powerful
  • 6:50 the system can show you how much it thinks you need help; it does this by analyzing your actions within the the program, but a Jeen could use the Magic Mirror in order to see if you’re frowning, for example, and attempt to correlate changes in your expressed mood with changes to the state of your software as you’re working on it, and modulate the experience accordingly

So, you’ll have noticed during that demonstration that for the most part, there was no anthropomorphic paperclip, but at the end, the guy talks about advanced versions that present a character-based interface. I think that would be a huge mistake, and this “simpler” demo is exactly the kind of assistance interaction we should want from our systems.

The “AI” in that demo is also pretty simple, just a probabilistic model based on historical usage patterns. But hey, it was 1995, and that was pretty state of the art. But it’s nearly 25 years later now, and AI has gotten a lot better and simpler to incorporate into your project.

Wrapping up

Well, I hope that was enlightening! I know I’m totally skipping going over the mechanisms of Passing the Pipe, but just imagine that all Pipedream software is available in your personal environment via integrated bittorrent client, and that’s pretty close to the picture, if not the reality. Next is a walk-through of using the Language of Making in order to create some software. After that is a video demo, so hang onto your butts.

There’s a bunch of things I didn’t even really get to. In the “culture of programming” column, I can’t recommend Houyhnhnm computing (“huNAM computing”) series strongly enough:

Whereas Humans view computers as tools below them to which they give orders and that do their bidding, Houyhnhnms view computing as an interaction within a system around them that extends their consciousness. Humans articulate their plans primarily in terms of things: the logical and physical devices they build (sometimes including tools to make more tools), in the lower realms of software and hardware. Houyhnhnms weave their conversations foremost in terms of processes: the interactions they partake in, that they attempt to automate (including these conversations themselves), which always involves wetware first. In short, Humans have computer systems, Houyhnhnms have computing systems.

The Cheat is saying goodbye now. It’s also what he looks like when he’s saying good morning.

good morning from The Cheat

  1. In fact, I called it an ideal system for creating software. [return]
  2. Apple, Inc is a rent-seeking parasite on human software developers whose hegemony is a serious threat to global human freedom. Fuck Apple. [return]
  3. There are a bunch of papers on Self at and relatively new releases of Self, ie, [return]
  4. He was a full-grown adult cat when we got him and topically named him in 2003, which was fifteen years ago. [return]
  5. In very particular, Pipedream is being developed on my Thinkpad X1 Yoga laptop that’s running Linux, and being developed primarily in the Rust programming language. [return]
  6. There’s a lot going on in this pipe-passing; it’s probably a sign that the scope of it is too large. [return]