What’s going on here?

Hi, I’m Joe, and I’m describing Pipedream, taking inspiration from the Houyhnhnm Computing chronicles.

Pipedream?

Pipedream is my currently-mostly-vaporware software project for creating software. It’s a coherent fusion of multi-modal, multi-sensory interface, functionality, and integrated idiomatic automation whose sum total is meant to make creating software orders of magnitude faster and easier. This post is meant to sketch out its form in advance of it existing fully, as a way to help me stay on point, and to help others understand what I’m talking about.

OK, but… what IS it?

The mature system is based on four pieces, arranged in a loose hierarchy, meant to support investigation-based development, and allow fluent synchronous and asynchronous collaboration with one’s self, other developers, and automated assistant entities called Jeens.

Like Clippy?

No, Clippy sucks. Everyone knows Clippy was terrible, because it was an awkward metaphor over-extended and inappropriately literalized. Unlike the convoluted and over-extended metaphors I’m about to introduce below, Clippy was an attempt to represent content assistant programs as though they were servants or butlers¹. You interact with the Jeens more like the way a shepherd interacts with sheep dogs working together to manage a herd. We don’t expect the dog to speak human to the shepherd, or the shepherd to pretend to have a tail to wag at the dog; each partner in the project communicates in its own natural way with the other. But I’m getting a little ahead of myself.

If Jeens are one of the four pieces, what are the other three?

The True Language of Making, Passing the Pipe, and the Magic Mirror. My mythical and poetic referents are all over the place, and this is a bit of a hodge-podge, but what’s cooler: the structural editor, or, The True Language of Making? Anyway, I’m still workin’ out the aesthetic.

Yeah, these names aren’t exactly descriptive.

I know. Here they are in more detail.

The True Language of Making

The most fundamental piece of Pipedream is a structural software editor, known as the True Language of Making, after the speech in Ursula LeGuin’s Earthsea books. Only true statements can be spoken in LeGuin’s Old Speech, and in the True Language of Making, only syntactically valid entities can be constructed². You can’t forget a semicolon and accidentally change the semantics of your program without knowing it, or have to hunt down an obtuse syntax error, because instead of telling the compiler indirectly that you wish for, say, variable binding or reference to be created by entering a sequence of lexical characters, you directly instantiate an abstract entity known as a Bud, that you can choose to Blossom into a more concrete entity, such as that variable or reference. All character-level changes to the source code to accomplish this semantic transformation are done by the True Language of Making, and are guaranteed to be valid at all times.³

To think about this more concretely, it’s useful to consider how traditional software editors work, and how that’s different from the True Language of Making (henceforth shortened to the Language). The typical software editor is not actually aware of the structure and meaning of your program as you’re creating it; it’s focused on directly manipulating meaningless text. It may offer such things as “autocompletion”, but even that is fundamentally based on the system recognizing strings of characters that the programmer is entering, and the system is not considering that the intent of the programmer is not really to manipulate character strings, but rather, to tell the computer what or how to do something. In the Language, you express your intent much more directly, and when you’ve done so, you look directly at the effects of what you just did. It’s like looking at a pine tree or a mink stole, vs. looking at some postcards with the words “fir” and “fur”, and trying to keep track of where each one was in your environment.

Which is the easier depiction of a tree to work with?

This traditional editor’s ignorance of the semantic meaning of the software we’re crafting extends to other tools, too, like our version control systems (eg, git, etc.), and bleeds into how we must reckon with how our software evolves. If you’re doing a code review of something, the questions you want to answer are, “What is the impact of this change on resource consumption? On interactivity? Does it make a component too hard to understand, and if so, can it be refactored in a way that eases comprehension?” However, in practice, if the change is peer-reviewed at al, there’s focus on things like, “Have the naming conventions been followed?” or, “Can you add a comment here for why we’re doing this weird thing with casting?” or things that should be checked by the compiler like, “Have you handled all cases?” Questions like the last are addressed by using safety-oriented languages like Rust, but Rust itself provides no native help to the user/programmer in constructing those things; the programmer must first guess at what the correct sequence of characters will result in their being constructed, and then ask the compiler if they were right. In the Language, if it is written, it is properly made, and its nature is obvious in the UI.

Speaking in technical terms, the True Language of Making is an interactive editor for a control-flow graph, where each node is a syntax object (a Bud), with viewable and editable properties, such as “display name”, or, “concrete type”. Each node can potentially be a nested graph itself, and interaction scope can be fluently altered during use. If a Bud has been specified enough to be concretely instantiated by the Language as a functional, concrete syntax object in the underlying implementation language, then it is considered to be a Blossom. Each Bud, and the graph in which it’s embedded, is canonically addressable by content-based hash, or, True Name. If you know an entity’s True Name, you may summon and manifest it, and it is yours to command. Examples of partially Blossomed Buds are software constructs like bindings, references, scopes, etc. Full Blossoms would be things like, in Rust, let x = 5;, let y = &x;, { // ... }, etc., respectively. Each Blossom has a distinctive visual representation in the Language, avoiding the toll of constantly checking whether something says “fir” or “fur” or “fire” or whatever, and a Bud can only Blossom into an entity compatible with its context. The Language assists the programmer/user in discovering and understanding what can Blossom in any given position in a graph. As an implementation detail, the Language uses syntax quoting macros in Rust to allow Blossoms, which are in-memory concrete Rust syntax objects, to be turned into actual files of Rust source code on disk that can be compiled, but multiple native implementations of the Language should be relatively simple to create.

By treating software as content-addressed nodes with metadata, in a content-addressed topology, it becomes much easier for history exploration tools to focus on questions of change and evolution that are relevant, rather than tedious. It’s easier to reason about changes at the level of the Language, the True Language of Making, than it is to at the level of, say, Rust.

Passing the Pipe

Although a tool such as the True Language of Making would be useful to one person working on their own, most people want to collaborate and share their work with others, or in the case of revision control, with themselves. Passing the Pipe, or sharing the software you’ve created, is done by publishing its definition (as defined in the True Language of Making) into the Indra⁴, along with the software’s True Name (which is the content-based hash of the Blossoms and their topology) any way you want. The Indra is a content-addressed, distributed, peer-to-peer filesystem⁵, and True Names, of course, are the content-based hashes of entities in the Language. Which is to say, if the following things are true, you have access to a copy of the software:

1. You know the True Name of the software;
2. You have a network connection;
3. The contents that hash to the True Name you know have been published to the Indra.

then you can instantiate a copy of that software so Named. Because the True Name is based on the contents of the entity, it also specifies a version in time. Two Names that represent two versions of the same software can be related by a piece of metadata that asserts that these Names are so related.

Every entity in the Language has a True Name, no matter how large or small it is. This allows you to share software at an incredibly granular level; in practice, it’s reasonable to think about sharing or searching for software at the half-page-of-code function size. True Names can also be associated with rich metadata describing in human or machine-parsable ways what the True Name actually is, so the concept of “version” becomes more fluid. It’s much more like, “This is the version that does Foo,” instead of, “This is version 2.1.17.”

So, sharing your code?

Among other fun activities, yes, that’s what Passing the Pipe is. And it’s smoothly integrated into the UI (you’ll have to take my word for that at this point).

The Magic Mirror

Moving further away from technical details now, the Magic Mirror is the collection of tools and algorithms that allow the Pipedream system to more fully engage with the human programmer by watching the programmer work (like literally watch, with a video camera and microphone), and present synthetic views of Pipedream objects. It allows you to use your head as a pointer⁶, or see what your real-time collaborators are looking at. It refers to having the system watching your face and performing sentiment analysis for continuous feedback on what it is doing, which brings us to…

Jeens

Finally, the Jeens. I was originally thinking of calling them “djin”, but I didn’t want to be appropriative and colonial. So, Jeens aren’t really intelligent, the way that horses or dogs are not, though obviously they have some cunning and agency. Jeens use output from Magic Mirror modules in order to perform higher-level activities, like offering documentation in-line, or suggesting changes or additions to your software. They’re meant to communicate using the same UI affordances available to the human user, such as by controlling a read-only cursor (so that the human might know what the Jeen is working on), or by causing a Bud or Blossom to glow or animate. Jeens are typically expected to be implemented with modern AI/ML techniques, including but not limited to deep convolutional neural networks, Bayesian classifiers, hierarchical temporal memory systems, etc.

Is that it?

I guess that’s a reasonable overview. There are some gaps in there, like how deeply integrated gestures are.

What’s a gesture?

Anything you can do with your body that Pipedream can recognize. Examples include:

1. hand gestures
2. whistling tone sequences
3. drumming patterns on the table
4. nodding or shaking your head
5. saying a keyword
6. executing a keyboard chord, like hitting both shift keys at the same time
7. etc.

They can mean anything you want, and are context sensitive; you can imagine that a common UI/UX pattern would be a Jeen indicating a proposed action, and the programmer shaking or nodding her head in response.

OK! Is that it?

Ha ha, no! Right now, I’m working on the True Language of Making, and the Magic Mirror. Once those are in an acceptable minimally functional state, which I’m hoping will be soon, I’ll start working on Passing the Pipe and Jeens. But I was carrying all of this (and more!) around in my head for the last couple years while I refined the concepts, and I needed to get it out on paper, as it were. I apologize for not having an implementation you can play with, but that’s coming soon.

I have more blog posts to write about this. Next up is a literature review for some of the ideas here, and then some deep-dive on different aspects of the system. I haven’t really even gotten to what is meant by “investigation-driven development”, though it shouldn’t be too surprising.