Project: Concatenative JavaScript

After I explored Forth, I wondered if I could apply some of what I learned to my comfort zone of JavaScript in a browser environment.

The TypeScript source code for the language lives here.

Tests

If you see ❌ or 🚧, then that test has failed.

This paragraph contains some tests which don't make much sense in the table below. First test: 🚧. Leading and extra whitespace don't change anything: 🚧. Look in the console to see if this logs a checkmark.

Logbook

Tue Feb 06 21:53:03 PST 2024

I reflected on Hyperscript and how it added a convenient layer on top of JavaScript. In my experience with Hyperscript, I was frustrated with extending the language. And that was one of the main strengths of what I'd learned of Forth. So I started this experiment as a way to explore some universe where a library as convenient as Hyperscript was as extensible as Forth.

I started with a script on this page.

Wed Feb 7 10:46:08 AM PST 2024

I continued making progress in the time normally allotted for creative coding. Today's creative coding prompt was "some of my favorite things." JavaScript, exploring programming languages, and putting energy towards creative interfaces felt aligned with the prompt.

I succeeded in getting some small code on an HTML attribute to work how I expected! It was so fun to use JS's step debugger to walk through my code and see it work. I continued to make some more basic words, like dup, swap, and drop.

Thu Feb 22 08:49:18 AM PST 2024

There were two different things I wanted to implement in my language. Well, re-implement. I'd struggled thinking of things I'd like to use my language for as I was writing it, because I invested most of my interest and drive in that moment into the language itself.

Future: I wanted to reimplement some hyperscript I'd written recently in my typing game. The hyperscript was relatively simple, but there was a lot that this language was missing to get there.

Future: I wanted to reimplement a 5 minute timer I'd written in Forth.

Both of these ideas were pressing today since I wanted to give a presentation on this concept. I signed up to give the presentation to give me an accountability target for the day, and I had to admit that was motivating me!

I started taking incremental steps towards those things, to see how far I could get before the presentation. Since my hyperscript example relied heavily on variables, and I wasn't immediately sure how I'd implement the same thing in my language, I started with the forth example. I implemented now to get the current milliseconds and typeof to quickly test if the result was a number. Then, of course, I noticed that my Forth code also uses variables. But these are Forth-type variables, which are much more straightforward (now that I've read Starting Forth) than Hyperscript variables. So I pushed forward.

I got an implementation of variables that matched the most basic semantics from Forth. That is, my new words @ and ! could act successfully on the result of a variable. But pointer arithmetic would not work, since there was no linear memory behind it. In short, I faked it.

The next thing I had to tackle was waiting. Dot dot dot. Silence. Do nothing. Up to this point, my interpreter would execute every single bit of code as quickly as it could without pause or interruption. To do so, I'd have to hook my language in to JavaScript's event loop somehow. I wasn't sure how, but I created a dictionary word sleep to stare at where the implementation should go.

I got my basic test passing, and in the process I found a lot of inadequacies in my interpreter implementation. I had made many assumptions in order to move quickly towards the next incremental step. I felt good about the decisions I made up to this point, but also daunted by the challenges. And there was still yet more features to add! I still had my failing click event test, which now wasn't being executed at all.

Up to this point, I used the same interpreter for every distinct piece of code on the page. This worked until I introduced asynchronicity. The problem was that the interpreter was stateful, and each piece of code on the page (i.e. each c= HTML attribute) needed to set a different state. Previously this worked smoothly because each script would block until completion, and the next one could set the internal variables for itself without any conflict.

The solution I had in mind was to use a separate instance of each interpreter for each distinct script. And a pattern I'd already been using without understanding why became clear - I'd been passing around a "context object", ctx, where I was storing all the state. Well, not every piece of state, I hadn't been perfectly consistent. But if I refactored to make my use of the context object consistently, and make every other function entirely pure, then I could simply switch to a separate ctx for each script, and isolate all state therein. That worked!

I was missing a lot of the functionality still to replicate my timer in Forth, but I realized I could do a different version of the same thing with separate scripts in separate HTML elements.

I made a rough-and-tumble, ad-hoc area to present this at RC:

4 1 + . ." languages"

In my rush to prepare the above presentation, I accidentally made the timers all 10 times the duration, so one minute actually took 10 minutes to trigger. Whoops! Anyway, the presentation went very well.

Sun Feb 25 09:14:19 PM PST 2024

I wanted control flow via if and else, and loops like Forth's begin until. I struggled to understand how one does this, but this reddit comment unlocked it for me. The missing link for me was that Forth uses the parameter stack during compilation to save, recall, and point back to memory locations earlier in the definition. So during compilation, when if runs, it compiles some code that will jump to a place, but it also leaves an empty spot in the compiled code and puts the address of that empty space on the data stack. When later endif is compiled, it grabs the address off the data stack and sets the current compilation address back into it. Thus, if now has the address it needs to jump to if the condition is false. With that level of understanding, I tried to make it work.

My attempt at this revealed a handful of missing pieces in my design. For example, I had no way to run user-defined words containing multiple words. That would be a prerequisite to jumping around in a multi-word definition! So I had to implement that first.

As an aside, I found it very fun to implement a word debugger which had the same semantic effect as the debugger statement in JavaScript. I popped this word into the middle of a failing test and went exploring! And it helped me catch a significant bug as well. My previous isolation of the ctx object had not been as thorough as I'd imagined! I was still sharing a lot of data across different words. I also found a couple bugs where I was pushing data to the wrong ctx's parameter stack because of a typo. So fun to dive into my own code and watch it work!

Back to the main event, multi-word definitions were simple to implement in the case of synchronous code. In fact, I just had to loop through each definition. Asynchronous code was trickier, in the same way as I'd found when I implemented sleep in isolation, within the definition of a word it had to pause that loop and resume it later.

Unfortunately, I couldn't determine how to write a test which would fail accurately for my asynchronous code. That is, even though I knew the semantics of sleep inside a : definition were nonsense in my current implementation, the test I wrote passed! And I wasn't sure how to write a better test. The best I came up with was : sleepier ' ❌' 750 sleep ' ✅' ; sleepier me >text. But this passed incorrectly because the checkmark was immediately pushed to the stack! How could I design a test which would show the current flaw in my design? I considered that I could do a little math with now, so assert that I was sleeping for 750 milliseconds, and then test that now was different by at least 750 milliseconds! That should work!

Thu Feb 29 06:18:25 PM PST 2024

I wrote the test I described above and it failed successfully. That is, now 750 sleep now swap - 749 > was true for my sleep test at the top level, but it failed when sleep was used in a colon definition. Again, I expected this because I knew my implementation was incomplete.

The solution I imagined was for my colon definition implementation to change the state of the interpreter. As it was, the colon definition implementation simply looped synchronously. So I added a return stack and changed the interpreter as a state machine. I was utterly surprised at how effective it was!

Fri Mar 1 10:49:31 AM PST 2024

I traded the Markdown table for a HTML table, because a huge markdown table makes for really terrible git diffs - When you reformat to make the columns wider or skinnier, every line changes! Very confusing when mixed with semantic changes as well.

Sun Mar 3 03:53:28 PM PST 2024

I finally felt ready to implement if and endif. There were still some decisions to make about memory representation, but I felt I understood the problem enough to make those decisions as I went.

Future: I had an idea about compiling words. Perhaps I could append vanilla JS to a string to build a new function. So every time I run :, a global function is actually constructed. I'm not sure how I could do that with JavaScript return statements. So I think I'd have to do some careful management of return statements. However it works, it will be different from Forth because it has to compile JavaScript instead of assembly, and JS and assembly have very different operations.

Mon Mar 4 07:44:28 AM PST 2024

In Forth, the return stack and parameter stack hold arbitrary data of any type. In this language, I wanted similar semantics. And by now my parameter stack already worked this way. But the return stack was different, because it had one 'normal' kind of data for which the return type was consistent, but if users could push and pop any data from it, then those structured chunks would be interspersed with anything. Obviously, this made Typescript freak out. But that's the correct response to interspersing structured data with arbitrary data. One example of a nasty issue which might arise from this is that users could push things which kind of look like the structured return pointers but not really, and so bugs could get extremely confusing.

Fri Mar 29 02:39:42 PM EDT 2024

I started reading Jonesforth Implemented lit and compiled numeric primitives using it as Jonesforth does. I also renamed my internal dictionary variable to latest. Jonesforth is already an invaluable resource and I feel like I've only just begun to study it.

Wed Apr 24 10:29:00 AM PDT 2024

I've made a lot more progress I never described. I got control structures working and an idea for how to work with click handlers and much more.

I wanted to play with the library for a creative coding exercise, but I realized I know how to quickly include it on another page. It would be easy to include on another page on my website, but maybe not on the web-at-large? Or maybe I could just include the compiled JavaScript straight from my website? I tried this on a codepen. I noted a couple issues. First, the classic jQuery ready problem. I added an event listener for DOMContentLoaded to fix that. Then, because I was using a type="module" script tag, I didn't expose anything on the global object. That made it really hard to detect if it was loaded at all in codepen. So I added some of the utilities to the global scope. Then I pushed to deploy and test again on codepen.

After I pushed, it still didn't work. I checked how I was using codepen, and I was using it wrong! I never properly included my script in the first place. When I properly included it, it worked! I didn't know if it previously would have worked without my improvements, but I liked the improvements either way, so I didn't revert and check.