Month: October 2020

A Bug’s Life

Thomas Hobbes famously said that in humankind’s natural state, a man’s life tends be "nasty, brutish, and short". I suppose the natural corollary of this is that in technologically advanced societies, life is comparatively "pleasant, peaceful, and long". Of those three things, it is the last one that can be measured most objectively; we see, …

A Bug’s Life Read More »

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization

Since I picked up my work on the JavaCC codebase at the end of 2019, various people have broached to me this question of strings that can (or should) be broken into tokens differently based on the context where they are encountered. I have to admit that it took me a while to grasp just …

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization Read More »

Token Hooks (CommonTokenAction) Revisited

In the beginning there was… CommonTokenAction Legacy JavaCC had (and still has) a means of applying whatever adjustments (a.k.a. kludges) to a Token just before it is handed off to the parser machinery. You could define a method called CommonTokenAction in your Lexer TokenManager class and this method is invoked when you get another Token …

Token Hooks (CommonTokenAction) Revisited Read More »