Diving deep on Raku regexes, and coming back with a better way for grammars to cooperate

After some excellent feedback my previous post on coordinating multiple grammars in Raku, I realized that I'd had fundamentally the wrong mental model of regexes in Raku(do). This flawed model hadn't stopped me from using regexes, but was nevertheless wrong – and it prevented me from correctly grasping some of the more complex behaviors involved in grammars. Now that I've corrected that misconception, everything makes so much more sense!

In this post, I'm going to briefly present the flawed view I had (hopefully in enough of an outline that you'll understand where I was coming from but not be tempted into the same confusion) and then present the correct (or at least, less wrong) mental model. Exploring this model will lead us to a significantly deeper understanding of how Raku's regexes work under the hood (or, at least, deeper than I had a week ago; YMMV, as they say). Next, I'll explain how this new understanding allowed me to build a trait – on that I believe will make composing multiple grammars much easier. And, finally, I'll quickly walk you through the ~100 lines of code involved in implementing that trait.

Let's learn a bit of Raku together   »ö«

Grammatical Actions: further thoughts on cooperative Raku grammars

Raku's grammars aren't always the right tool for the job – but when they are, they're so powerful that they feel almost like cheating. And one of the not-so-secret weapons that gives them that power is the ability to specify an action object to specify parse-time behavior. (Methods on this object are called whenever a token with the same name in the grammar matches, which lets you manipulate the syntax tree as you build it and avoids having to navigate a (potentially very deep!) match tree after parsing is complete.)

Over the past few weeks, there's been a fair bit of discussion about different ways to combine multiple grammars. Most notably, Mike Clark had an excellent blog post titled Multiple Co-operating Grammars in Raku. In that post, Mike showed two ways to combine two grammars – a simple way for times when you're fine with the grammars sharing an action object, and a more complex way to let each grammar have its own actions. And then, building on that post, Matthew Stuckwisch (guifa) opened a Problem Solving issue discussing different ways Raku could one day add syntax that better supports combining grammars (maybe in version 6.f).

Now, both of those posts were very interesting. But, being lazy, I wasn't too sure about Mike's second approach – it seemed like a lot of work, and a lot of bookkeeping. And, being impatient, I also wasn't too keen on waiting for a syntax-level solution in v6.f. After all, we haven't even finalized a release date for v6.e!

So I started thinking to myself, "self, would there be a way to do this with roles somehow?". And, after a bit of playing around, I replied in the affirmative. The rest of this post shows an example of using a Subgram role to easily combine multiple grammars, each with its own action object.

Further thoughts on Raku pattern matching

Brains sure are funny things. Or at least mine is; maybe I shouldn't speak for the rest of you.

Last night, I posted a few thoughts on pattern matching in Raku. A bit later, I saw a reply to that post suggesting that it'd be nice to add better pattern matching syntax with a Slang. I responded by saying that it probably wouldn't require a Slang, but that I'm not typically a fan of changing basic syntax purely for convenience (when Raku already gives us so much).

I didn't give the matter much more thought, at least consciously. But my subconscious mind must have been noodling around with the idea because, somehow, I woke up this morning absolutely convinced of three things.

1. A module should add support for better pattern matching.
2. It could do so with a regular sub, not a Slang or term.
3. Writing that sub would be easy.

And, for once at least, that all turned out to be true. Here's the function I came up with:

Let's try some pattern matching

Raku has extremely strong support for pattern matching in function/method signatures – you can match on literals, types, names, or pretty much anything at all and can conveniently destructure the value you're matching on into a set of variables that fit your needs.

But Raku also has a second type of pattern matching (or at least something very much like pattern matching): the ~~ smartmatch operator powered by the .ACCEPTS method. This form of matching is also very convenient; is has a slightly different use case from matching on a signature, but it's no less powerful on the whole. And, when it fits, it can be an even better/more lightweight solution to the same set of problems. In fact, I'd bet that when (which is powered by this sort of matching) is one of the keywords that shows up most often in my Raku code.

The problem

Since these two forms of pattern matching are different, there are some problems that are easier to solve with signature matching and others that are easier to solve with smartmatching. Fortunately, Raku makes it very easy to add smartmatching into a signature – you can easily smartmatch in a where clause, for example.

I've typically found going the other direction a bit more cumbersome, however. Consider the following code:

Raku's surprisingly good Lisp impression

Lisp is famous for having pretty much no syntax. Structure and Interpretation of Computer Programs – arguably the most well known intro to programming in Lisp – presents pretty much the entirety of the syntax in its first hour. And that's by no means the only thing SICP does in that hour.

Raku, on the other hand, has a bit more syntax.

Ok, that's an understatement. Raku is syntactically maximalist to exactly the same degree that Lisps are syntactically minimalist. Forget “syntax that fits on a postcard”; Raku's syntax struggles to fit on an A4 sheet of paper. Raku has the type of syntactic riches that inspire Rakoons to classify its operators into beautiful (though now sadly dated) Periodic Tables.

If you want to label your code, consider Label-ing your code

When organizing a program of any size, you'll obviously need to break your code up into smaller chunks. Often, it makes sense for these chunks to be factored out into their own functions – especially if they're good candidates for reuse. But it's possible to take this urge to factor out too far and it's often a better call to leave the code inline. Doing so makes it clear that the code isn't being reused anywhere else, and keeps your program's control flow more linear.

Indeed, John Carmack wrote an influential post a few years ago describing how he has shifted from a coding style that looked a lot like

void MinorFunction1( void ) {…}
 
void MinorFunction2( void ) {…}
 
void MinorFunction3( void ) {…}
 
void MajorFunction( void ) {
        MinorFunction1();
        MinorFunction2();
        MinorFunction3();
}

to one that looks more like

void MajorFunction( void ) {
        // MinorFunction1
 
        // MinorFunction2
 
        // MinorFunction3
}

I'm not trying to argue that this style is always better, but it is a good option to have.

When organizing my code like that, though, I don't tend to use // MinorFunction1 comments. Instead, I'm more likely to write a Label, so it'd look more like minor_function:. You might benefit from using labels too, if your language includes them – and if it's at all a C/Algol family language, it probably does. Labels are supported in at least JavaScript, C/C++, Perl, golang, and my own language of choice, Raku.

Open Source Keyboardio Atreus Keyboard – Six Week Review

As I write this, I've now had my Keyboardio Atreus for a full six weeks and have been using it pretty much exclusively that whole time. After a month and a half, I've given it a fair shake and can give you my full impressions. And my full impression is that it is amazing.

Atreus background

Before I tell you what I love about it, what is the Atreus? Well, it's a keyboard. It's a tiny keyboard.

More specifically, it's a keyboard with 44 keys – compared with 84 on my ThinkPad keyboard, and a full 108 on a traditional desktop keyboard. How does it get by with so few keys? More on that in a minute; for now, just focus on how small the thing is, without needing to cramp any of the individual keys in the least.

A Raku Manifesto, Part 3

In part 1 and part 2 I discussed my personal take on a Raku manifesto:

In those posts, I explained how Raku prioritizes each of the values on the left over each of the values on the right. I also explained that, in my view, the final value pair – prioritizing individual productivity over large-group productivity – is the most fundamental value driving Raku's design. All the other sets of priorities in my Raku manifesto play a supporting role in prioritizing individual productivity over large-group productivity. In this post, I explain why I'm convinced that Raku made the right call.

Why Raku is correct to prioritize individual productivity

I mentioned earlier that language designers have recognized the tension between individual productivity and large group productivity. If you look at what has been written on the topic – all the way from the original Programming-in-the-Large Versus Programming-in-the-Small paper through today – you might get the impression that small scale programming is a solved problem and that the only interesting/meaningful question is how we increase the productivity of large software teams. Certainly many of the programming languages that have gained popularity recently seem to reflect that view:

Language	Developer	Purpose
Golang	Google	build large software projects
Rust	Mozilla	build large software projects
TypeScript	Microsoft	build large software projects
Hask	Facebook	build large software projects

Looking at that list, you might conclude that we clearly need a programming language for large projects and that older languages do fine for smaller groups.

You might, but you shouldn't.

A Raku Manifesto, Part 2

If you recall from part 1 in this series, we're talking about my personal take on a Raku manifesto (modeled loosely after the Agile Manifesto):

In part 1, I discussed how and why I believe that Raku values expressive code over uniform code and rewarding mastery over providing a fast learning curve. Now, we're ready to move on to the third value pair.

Powerful code over unsurprising code

The principle of least astonishment is a fundamental maxim of computer science that states code is far clearer when its behavior can be easily predicted; code is far more head-scratchingly confusing when it triggers spooky action at a distance.

Rust provides a particularly clear example of this principle in action: Rust doesn't let you override existing operators for existing types; it doesn't let you declare a function without fully defining the types; it doesn't let you have default arguments; it doesn't let you call a macro without using the ! character that sets that call apart from function calls. Rust is powerful enough that it could let you break any of these rules, but it doesn't – enforcing those rules keeps the language less surprising, and thus makes it much easier to reason about.

And Rust is hardly alone in this regard. Indeed, the majority of programming languages don't allow operator overloading and even fewer allow the programmer to define new operators. Put differently, most languages are willing to deny programmers the considerable power of user-defined operators to keep the language less surprising.

How Raku values unsurprising code

I've already mentioned one way Raku helps prevent nasty surprises: eliminating the action-at-a-distance variables that were so sinful in Perl 5. Raku has also thoroughly embraced lexical scoping; even when you modify Raku's very syntax, your changes will be limited to your current lexical block. And, as I mentioned in part 1, Raku strongly supports both object-oriented and functional programming – each of which, in its own way, promotes predictable code.

A Raku Manifesto, Part 1

I'm not so sure that I'm a fan of agile software development (at least as it's commonly practiced). But I am sure that the Agile Manifesto got one thing really right: It not only stated what values were important but also stated which values (though important!) could be sacrificed. That is, the Agile Manifesto was explicit about what tradeoffs it was willing to make.

That's crucial because it's really easy to select some nice-sounding phrases and label them as "priorities" – who wouldn't agree that "readability counts"? But it's much harder to pick out important values that you're willing to sacrifice. And, because it's harder, it's also much more revealing. You'll often learn a lot more from knowing a project's non-goals than from knowing its goals.

In that spirit, I'd like to present a similar manifesto for the Raku programming language. Note: I said "a" manifesto, not "the" manifesto for Raku. I'm speaking only for myself; I'm sure many others would include different dichotomies on their version of a Raku manifesto. Additionally, this is very much a first draft. I've put considerable thought into this but haven't yet discussed my views broadly. I also artificially limited myself to four pairs of values (following the form set by the Agile Manifesto), and I could easily imagine changing my mind. If you disagree or just have a different perspective, I'd love to hear your thoughts on the r/rakulang thread for this post on the #raku IRC channel.

Before we get started, I want to remind you of a statement from the Agile Manifesto:

That is, while there is value in the items on the right, we value the items on the left more.

I feel the same way. All the items on the right side of the chart up above (the items that come after "over") are still really important. So, in this series of posts, I'm going to walk through the four value pairs in my Raku manifesto. I'll discuss the first two pairs in this post, and the next two in the second post. For each pair, I'll say why the value on the right side is important, and what Raku does to support it. Then I'll say why, in my view, the value on the left is even more important (for the language Raku is trying to be, anyway), and how Raku prioritizes it in ways that require sacrificing the value on the right. After discussing all four value pairs, I'll close with a post on why I believe that Raku's decision to prioritize the values on the left is absolutely the correct one.