(The TLDR: Nested lookahead was always broken in legacy JavaCC. This is finally fully addressed (as of 11/26/2022) in JavaCC 21. However, since this fix has a lot of potential to break existing code, for now, the fix is only in effect if you put LEGACY_GLITCHY_LOOKAHEAD=false at the top of your grammar(s). Meanwhile, you should …
JavaCC 21 now has assertions. There are actually two kinds of assertions: The assertion condition is expressed in Java code. The assertion condition is a lookahead expansion. The first kind of condition looks like this: ASSERT {someCondition()} Optionally, the assertion can have a message, as in: ASSERT {x!=y} : “At this point, x and y …
(To get some prerequisite understanding of this topic, it might be a good idea to read this earlier blog post on context-sensitive tokenization from three months ago.) The Lay of the Land There are two quite useful ideas that have been in JavaCC from the very beginning: lookahead (particularly syntactic lookahead) lexical states Syntactic lookahead …
Context-Sensitive Tokenizing, Part Deux: Lexical States Read More »
Thomas Hobbes famously said that in humankind’s natural state, a man’s life tends be "nasty, brutish, and short". I suppose the natural corollary of this is that in technologically advanced societies, life is comparatively "pleasant, peaceful, and long". Of those three things, it is the last one that can be measured most objectively; we see, …
When I say "fully supported", I mean all the new features that are categorized as "stable". In fact, the only new language feature that has been promoted to stable in JDK 14 is the new syntax for the Switch statement. This also allows a Switch statement to be used as an expression. In fact, the …
(N.B. Note added 13 June 2021: This article is useful in terms of understanding how to add token hooks to code. However, in terms of solving the specific problem outlined, the article is obsolete. See here for the updated solution.) Since I picked up my work on the JavaCC codebase at the end of 2019, …
“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization Read More »
Over the last days, I announced certain new features but then realized that they really needed certain key enhancements. I refer to the new FAIL statement and the up-to-here notation, the latter of which I only announced three days ago! The FAIL statement On first implementation, the FAIL statement allowed you to write something like: …
Revisiting LOOKAHEAD Redux Not so long ago, I had a sort of eureka moment when I realized that the legacy LOOKAHEAD construct was fundamentally half-baked or broken. Not only that, but I started narrowing in on how to definitely address the issue! Well, let’s get concrete. Suppose we have a production that looks kind of like this: FooBar : …
New Feature: The =>|| delimiter stands for “scan up to here” Read More »
The FAIL construct is really pretty dead simple. Here is an example: A. ( “foo” | “bar” | FAIL “Was expecting \”foo\” or \”bar\” here!” ) At first I thought the above was just syntactic sugar, since you can, of course, already write: B. ( “foo” | “bar” | {throw new ParseException(“Was expecting \”foo\”or “\bar\” …
If you liked LOOKAHEAD, then you’ll love LOOKBEHIND! Actually, I have not added LOOKBEHIND as a new keyword in JavaCC. It is just an informal way of referring to a new feature that I recently added. I implemented this because I realized that there was a very common thing people want to do in a JavaCC grammar …