Tips and Tricks

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization

Since I picked up my work on the JavaCC codebase at the end of 2019, various people have broached to me this question of strings that can (or should) be broken into tokens differently based on the context where they are encountered. I have to admit that it took me a while to grasp just …

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization Read More »

Token Hooks (CommonTokenAction) Revisited

In the beginning there was… CommonTokenAction Legacy JavaCC had (and still has) a means of applying whatever adjustments (a.k.a. kludges) to a Token just before it is handed off to the parser machinery. You could define a method called CommonTokenAction in your Lexer TokenManager class and this method is invoked when you get another Token …

Token Hooks (CommonTokenAction) Revisited Read More »

New Feature: FAIL Statement

The FAIL construct is really pretty dead simple. Here is an example: A. ( “foo” | “bar” | FAIL “Was expecting \”foo\” or \”bar\” here!” ) At first I thought the above was just syntactic sugar, since you can, of course, already write: B. ( “foo” | “bar” | {throw new ParseException(“Was expecting \”foo\”or “\bar\” …

New Feature: FAIL Statement Read More »

Straightforward LOOKAHEAD Enhancements

A couple of weeks ago, I implemented a solution to a longstanding Dmitry Dmitriyevich problem with LOOKAHEAD specifications in JavaCC. Dmitry Dmitriyevich (alternatively Ivan Ivanovich or Vladimir Vladimirovich) is my own personal terminology for situations where you are required to write highly repetitive code. (Patriotic Irishmen are welcome to call this the Patrick Fitzpatrick or Gerald …

Straightforward LOOKAHEAD Enhancements Read More »

Revisiting LOOKAHEAD in JavaCC, Attacking the Dmitry Dmitriyevich Problem

As a result of some private correspondence with one of the PMD developers (PMD makes extensive use of JavaCC) I started thinking about some issues that I had in the back of my mind to look into at a later point. Having recently ripped out the code in JavaCC 21 that supported putting LOOKAHEAD specifications …

Revisiting LOOKAHEAD in JavaCC, Attacking the Dmitry Dmitriyevich Problem Read More »