What happened to all the visual images in this PDF?

Looks like the linked PDF was a draft (see Brad's comment) Here is the link to paper with figures http://www.cs.cmu.edu/~bam/papers/chi86vltax.pdf

I found the final version: http://www.cs.cmu.edu/~bam/papers/VLtax2-jvlc-1990.pdf

Thanks for your interest in my old article! That version is a draft update of this article: Brad A. Myers. "Visual Programming, Programming by Example, and Program Visualization; A Taxonomy," Proceedings SIGCHI '86: Human Factors in Computing Systems. Boston, MA. April 13-17, 1986. pp. 59-66. http://www.cs.cmu.edu/~bam/papers/chi86vltax.pdf

Have you been made aware of any interesting later developments? How do you view the current landscape, eg. minecraft, scratch, blockly, kodu, etc.?

I do try to follow the field. There are lots of modern VPs and some commercial successes. The key "large-scale" VPs are LabView from National Instruments, and OutSystems. There are a lot of what nowadays are called "No Code" or "Low Code" environments, that are mostly VPs as well - see https://en.wikipedia.org/wiki/No-code_development_platform

Hey Brad! Can you believe it's been 29 years since we worked together when I was visiting the Garnet group at CMU? Thank you for that wonderful opportunity, I really learned a lot from it. It's such a rare treat to actually be paid to write Lisp code!

About four years ago I mentioned Garnet on HN and linked to an article I wrote about it called "Constraints and Prototypes in Garnet and Laszlo", and your "All the Widgets" video, here:

https://news.ycombinator.com/item?id=11232154

Yay Garnet! ;) I worked on Garnet with Brad Myers at CMU, on the PostScript printing driver. Brad is really into mineral acronyms, and I came up with an acronym he liked: "GLASS: Graphical Layer And Server Simplifier".

Constraints and Prototypes in Garnet and Laszlo (updated link)

https://web.archive.org/web/20070422104545/http://www.donhop...

All the Widgets (Fixed v2) - 1990

https://www.youtube.com/watch?v=9qtd8Hc90Hw

Thank you for inviting me to give a guest lecture to your UI class about pie menus:

https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=...

Just yesterday I cited Richard Potter's "Just-in-Time Programming" paper in "Watch What I Do: Programming by Demonstration" that you co-edited, as being relevant to playing Factorio!

https://news.ycombinator.com/item?id=26053703

Playing Factorio requires starting out by performing a few tasks by hand, then thinking about how to incrementally automate factories with trains, drones, combinator circuits, and blueprints, with high level modular repeatable patterns. And estimating where the break-even point is between completing a few tasks by hand, and spending the time to automate more common tasks.

That evokes an article that Richard Potter wrote about "Just-in-Time Programming" in Alan Cypher's classic book (which is now online for free), "Watch What I Do: Programming by Demonstration", about when "the user attempts to write a program for a task that is already in progress":

http://acypher.com/wwid

Watch What I Do: Programming by Demonstration. Edited by Allen Cypher. Co-edited by Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky. 1993. The MIT Press, Cambridge, Massachusetts, London, England.

http://acypher.com/wwid/Chapters/27JITP.html

Chapter 27: Just-in-Time Programming. Richard Potter.

[...]

I was lucky to have the pleasure of working with Richard Potter at Ben Shneiderman's Human Computer Interaction Lab at the University of Maryland, while he was developing an early version of Triggers, which he also wrote about in that book:

http://acypher.com/wwid/Chapters/17Triggers.html

Morgan Dixon and James Fogarty, who were working with James Landay at the University of Washington, also did some wonderful work along those lines on Prefab, which I've written about here:

https://news.ycombinator.com/item?id=11520967

Also just yesterday, I enjoyed watching Dan Ingalls' talk on "Pronto: Toward a Live Designer's Notebook", in which he demonstrated how he's been exploring Programming by Demonstration in Lively!

https://www.youtube.com/watch?v=if72CFsF_SY&ab_channel=YOW%2...

Also, here are some other great papers about visual programming I've linked to before on HN, in which I mentioned your brilliant acronym C32:

https://news.ycombinator.com/item?id=18496880

I really enjoyed this paper “A Taxonomy of Simulation Software: A work in progress” from Learning Technology Review by Kurt Schmucker at Apple. It covered many of my favorite systems.

http://donhopkins.com/home/documents/taxonomy.pdf

It reminds me of the much more modern an comprehensive "Gadget Background Survey" that Chaim Gingold did at HARC, which includes Alan Kay's favorites, Rockey’s Boots and Robot Odyssey, and Chaim's amazing SimCity Reverse Diagrams and lots of great stuff I’d never seen before:

http://chaim.io/download/Gingold%20(2017)%20Gadget%20(1)%20S...

I've also been greatly inspired by the systems described in the classic books “Visual Programming” by Nan C Shu, and “Watch What I Do: Programming by Demonstration” edited by Alan Cypher.

https://archive.org/details/visualprogrammin00shu_2pf

https://archive.org/details/watchwhatido00alle

Brad Myers wrote several articles in that book about his work on PERIDOT and GARNET, and he also developed C32:

C32: CMU's Clever and Compelling Contribution to Computer Science in CommonLisp which is Customizable and Characterized by a Complete Coverage of Code and Contains a Cornucopia of Creative Constructs, because it Can Create Complex, Correct Constraints that are Constructed Clearly and Concretely, and Communicated using Columns of Cells, that are Constantly Calculated so they Change Continuously, and Cancel Confusion

http://www.cs.cmu.edu/~bam/acronyms.html

Also, here's an interesting paper about Fabrik:

https://donhopkins.com/home/Fabrik%20PE%20paper.pdf

Danny Ingalls, one of the developers of Fabrik at Apple, explains:

"Probably the biggest difference between Fabrik and other wiring languages was that it obeyed modular time. There were no loops, only blocks in which time was instant, although a block might ’tick’ many times in its enclosing context. This meant that it was real data flow and could be compiled to normal languages like Smalltalk (and Pascal for Apple at the time). Although it also behaved bidirectionally (e.g. temp converter), a bidirectional diagram was really only a shorthand for two diagrams with different sources (this extended to multidirectionality as well)"

By the way, something I always meant to ask you, Brad:

How does "C32" fit into your acronym theme of gemstones and rocks? Is it a teeny tiny 4x4x2 carbon atom block of diamond? How many carats would that be?

Brad Myers wrote several articles in that book about his work on PERIDOT and GARNET, and he also developed C32:

C32: CMU's Clever and Compelling Contribution to Computer Science in CommonLisp which is Customizable and Characterized by a Complete Coverage of Code and Contains a Cornucopia of Creative Constructs, because it Can Create Complex, Correct Constraints that are Constructed Clearly and Concretely, and Communicated using Columns of Cells, that are Constantly Calculated so they Change Continuously, and Cancel Confusion

Spreadsheet-like system that allows constraints on objects to be specified by demonstration. Intelligent cut and paste. Implemented using Garnet.

http://www.cs.cmu.edu/~bam/acronyms.html

That's a great acronym, but IMHO the acronym I came up with for the first iteration of FBP was also pretty good: DOORMAT (Data-Oriented Organization Running Multiple Asynchronous Tasks)! However, it was deemed to be lacking in gravitas!

C32 doesn't follow the general convention of my group of naming after gemstones or kinds of minerals, but I think it is fun. Very few of our systems strive for gravitas!

Does anyone feel we are still in the silent film era of computer programming? Will we ever advance from text editors into something else and leave Vim, Emacs etc. forever behind for good?

The 80-column IBM punch card was invented in 1928 and here we are almost 100 years later writing programs as a sequence of 80-column lines of text. It seems pretty obvious to me that we're stuck in a local maximum. I don't think visual programming is the solution but there must be another alternative. Sort of like how databases are based on text but are much more than text.

I do! I am currently in the very early stages of creating a VR-based language. I believe that if we can represent programs more intuitively to our spatial, visual, and tactile senses, we can unlock more programming potential in more people.

spatial, visual, and tactile senses

What about our olfactory senses, for code smells?

You are designing language from scratch? Is it too early to show screenshot or a sketch?

Often I get the urge to make spatial development environment, but I always envision it as lisp with 3D rendering/manipulation.

I hope not. The 'advancement' from text editing to something else would in all likelihood not be an advancement at all but a retardation instead. There is a reason that the invention of writing is the basis of civilization. The only way for leaving text editing to be left behind is if all these big companies forbid ordinary users to do any programming besides a very dumbed-down/graphical version of it. I am sure they would very much like this but it would be a form of tyranny.

Think about it: why are you writing a textual comment right now and not sending a picture, video, sound clip of your message ? That would be much less "silent film"-ish, no ?

Text is a good medium for communicating stories and ideas, but software systems are primarily models.

Other engineering professions make heavy use of visual modeling tools. Oddly, the industry that builds these tools is itself "the barefoot shoemaker."

No, software isn't models.

Software is text written in a mathematical language.

Doing a visual interpretation of math (or interpretative dance, or a song, etc) isn't going to end in anything useful.

Individual programs are text, but useful software systems are models.

Software is currently described by written text that is transformed by other process (compilers/interpreters) to a different state and then executed. Often the execution does not work as expected due to undefined conditions or states within the "mathematical language", or misunderstanding of the scope or memory allocation or release, etc.

There certainly could and should be visual representations of state, memory utilization, registers, variable values and execution sequence based on the textual conditions described. Contrasts of expected execution vs actual.

Perhaps using GPT-3 that is taught each programming language and then a designer can specify the behavior and GPT-X* can build a solution that can be formally verified as provable. Then generate test cases and show the flow and proof of correct execution based on test conditions.

Brad Myers' paper answers the age-old argument about whether or not spreadsheets are visual programming languages!

https://news.ycombinator.com/item?id=20425821

DonHopkins on July 13, 2019 | on: I was wrong about spreadsheets (2017)

Google sheets (and other google docs) can be programmed in "serverless" JavaScript that runs in the cloud somewhere. It's hellishly slow making sheets API calls, though. Feels like some kind of remote procedure call. (Slower than driving Excel via OLE Automation even, and that's saying something!) Then it times out on a wall clock (not cpu time) limit, and breaks if you take too long.

A CS grad student friend of mine was in a programming language class, and the instructor was lecturing about visual programming languages, and claimed that there weren't any widely used visual programming languages. (This was in the late 80's, but some people are still under the same impression.)

He raised his hand and pointed out that spreadsheets qualified as visual programming languages, and were pretty darn common.

They're quite visual and popular because of their 2D spatial nature, relative and absolute 2D addressing modes, declarative functions and constraints, visual presentation of live directly manipulatable data, fonts, text attributes, background and foreground colors, lines, patterns, etc. Some even support procedural scripting languages whose statements are written in columns of cells.

Maybe "real programmers" would have accepted spreadsheets more readily had Lotus named their product "Lotus 012"? (But then normal people would have hated it!)

I Was Wrong About Spreadsheets And I'm Sorry:

https://www.reifyworks.com/writing/2017-01-25-i-was-wrong-ab...

HN Discussion:

https://news.ycombinator.com/item?id=20417967

Excerpt from "Taxonomies of Visual Programming and Program Visualization", by Brad A Myers, 1990/3/1, Journal of Visual Languages & Computing, Volume 1, Issue 1, pages 97-123:

Spreadsheets, such as those in VisiCalc or Lotus 1-2-3, were designed to help nonprogrammers manage finances. Spreadsheets incorporate programming features and can be made to do general purpose calculations [71] and therefore qualify as a very-high level Visual Programming Language. Some of the reasons that spreadsheets are so popular are (from [43] and [1]):

1. the graphics on the screen use familiar, concrete, and visible representation which directly maps to the user's natural model of the data,

2. they are nonmodal and interpretive and therefore provide immediate feedback,

3. they supply aggregate and high-level operations,

4. they avoid the notion of variables (all data is visible),

5. the inner world of computation is suppressed,

6. each cell typically has a single value throughout the computation,

7. they are nondeclarative and typeless,

8. consistency is automatically maintained, and

9. the order of evaluation (flow of control) is entirely derived from the declared cell dependencies.

The first point differentiates spreadsheets from many other Visual Programming Languages including flowcharts which are graphical representations derived from textual (linear) languages. With spreadsheets, the original representation in graphical and there is no natural textual language.

Action Graphics [41] uses ideas from spreadsheets to try to make it easier to program graphical animations. The 'Forms' system [43] uses a more conventional spreadsheet format, but adds sub-sheets (to provide procedural abstraction) which can have an unbounded size (to handle arbitrary parameters).

A different style of system is SIL-ICON [49], which allows the user to construct 'iconic sentences' consisting of graphics arranged in a meaningful two-dimensional fashion, as shown in Figure 5. The SIL-ICON interpreter then parses the picture to determine what it means. The interpreter itself is generated from a description of the legal pictures, in the same way that conventional compilers can be generated from BNF descriptions of the grammar.

Cool stuff. Wish I found this a few years ago when I dug into this topic for a thesis. Back then I analyzed this incredible resource (warning: clicking this link will load looots of images): http://blog.interfacevision.com/design/design-visual-progarm... and came up with my own provisional taxonomy where major categories were "Line-connected block-based", "Snap-together block-based", "List-based", "Enhanced text", and a few others.

Looking back I still think that the most promising direction when it comes to real-world applications of visual programming is the hybrid approach: make a language which has "real-time" interchangeable text and visual representations -- you get the best of both worlds this way (that was basically the topic of my thesis[1]).

I thought it was an original idea when I started out with my research, but sure enough there were attempts in the past. And towards the end (4+ years ago) I discovered a fresh one which seems to be still going strong, albeit rebranded: https://github.com/enso-org Last time I checked they were making what I consider a bad design choice by leaving the structuring of the visual representation entirely up to the user, but maybe that's fixed now.

Anyway I think it's a cool topic and worth keeping an eye on.

[1] If anyone is interested it's available here: https://djedr.github.io/masters_thesis.pdf

@djder thanks for referring to us (Enso)! The topic you mentioned - leaving the graph structure (layout?) entirely up to the user - why do you feel it is a bad choice? In our experience (strongly influenced by VFX tools, like Sidefx Houdini), people are designing the 2D workspace similar to how you structure your code. For example, "on left top corner we will have database filtering", "on right bottom space we will have credit risk analysis". This way authors of workflows are able to navigate trough hierarchical graphs really, really fast, knowing where are things they have designed. With auto-generated layouts, the layout can change drastically when you apply even small semantic changes (like when you pass a new parameter between two nodes), and thus, it will break this mental space-mapping between nodes positions and their responsivities.

Anyway, we are building Enso in a completely open way, allowing third-party auto-layouting algorithms to manage the graph visual layout, and we are very open to investigate it further - that's why I asked this question :)

Hey, thanks for the response. :)

I see it as analogous to structuring source code text. The sweet spot IMO lies in the place where the editor offers guidance for common use-cases (think auto-indent, on-demand auto-format), but allows you to arrange code in a non-standard way when appropriate. That I think would be ideal.

Not all good programmers care about maintaining perfect visual order and I think that's fine. They would appreciate the automation and would be discouraged by the lack of it. Same as in text languages/IDEs.

Of course it's totally fine if your audience does not care about such features or they are not a priority.

But for a good general-purpose system I think that would be a must.

Now the sweet spot is not trivial to achieve and I haven't completely solved it, but I feel it can be done and I'd invest in researching it.

PS I hope your project succeeds!

Soon after starting work on what later became Flow-Based Programming (FBP), I decided that I should leave the block layout entirely up to the application developer... A few years later, James Martin (1981) was working on what he called 4th-generation languages, and the blocks were prone to moving around unpredictably as the text was changed... Very confusing, as people do like things to stay where they are put! This actually fits in with the idea of "loci" alluded to elsewhere in this page. In FBP, "stepwise refinement" is a key part of the design process, so you get a hierarchy of simple structures... I agree that you could add a function to "prettify" your diagram(s), but this should still be under the developer's control. And if you make it simple to move blocks around (as in DrawFBP), it's probably not worth the trouble!

Paul, its such an honor to see your reply here! I'm a big fan of your work and your Flow-Based Programming book. It was one of the most important inspirations to what I'm doing in my life now (Enso: https://www.youtube.com/channel/UC4oMK7cL1ElfNR_OhS-YQAw). I truly love the arguments that you use there and how you describe how visual programming can unlock the communication between users of different backgrounds (business people, developers, etc).

To be honest, I was thinking about contacting you, Paul, for a long time already, however, I always felt like we are not yet in the stage of the development to bother you. Right now, when we are very close to Enso 2.0 public release, I would truly love to spend an hour with you on a call – I'll write an email to you about it and I hope you'll agree to chat with us! :)