token hook

NFA stands for “Nondeterministic finite automaton”. (Does that make you nervous?)

The main goal of the JavaCC 21 project, soon to be renamed as CongoCC, is to develop a very capable, practical, usable parser generator tool. However, there is another overarching goal — perhaps not so much a goal, as a philosophy. That can be summed up in a single word: Demystification You see, somehow or …

NFA stands for “Nondeterministic finite automaton”. (Does that make you nervous?) Read More »

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization

(N.B. Note added 13 June 2021: This article is useful in terms of understanding how to add token hooks to code. However, in terms of solving the specific problem outlined, the article is obsolete. See here for the updated solution.) Since I picked up my work on the JavaCC codebase at the end of 2019, …

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization Read More »

Token Hooks (CommonTokenAction) Revisited

In the beginning there was… CommonTokenAction Legacy JavaCC had (and still has) a means of applying whatever adjustments (a.k.a. kludges) to a Token just before it is handed off to the parser machinery. You could define a method called CommonTokenAction in your Lexer TokenManager class and this method is invoked when you get another Token …

Token Hooks (CommonTokenAction) Revisited Read More »