Error Recovery

A Glimpse of the Promised Land: Fault-tolerant parsing

For some time, it has been a goal of JavaCC 21 to provide the ability to generate fault-tolerant parsers. I started working on the problem about a year ago. However, I had not put in a comprehensive solution until now for several reasons. Basically these: The codebase, though already significantly refactored and cleaned up, was …

A Glimpse of the Promised Land: Fault-tolerant parsing Read More »

Context-Sensitive Tokenizing, Part Deux: Lexical States

(To get some prerequisite understanding of this topic, it might be a good idea to read this earlier blog post on context-sensitive tokenization from three months ago.) The Lay of the Land There are two quite useful ideas that have been in JavaCC from the very beginning: lookahead (particularly syntactic lookahead) lexical states Syntactic lookahead …

Context-Sensitive Tokenizing, Part Deux: Lexical States Read More »

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization

Since I picked up my work on the JavaCC codebase at the end of 2019, various people have broached to me this question of strings that can (or should) be broken into tokens differently based on the context where they are encountered. I have to admit that it took me a while to grasp just …

“You can’t get there from here!” — The Problem of Context-Sensitive Tokenization Read More »

New Feature: FAIL Statement

The FAIL construct is really pretty dead simple. Here is an example: A. ( “foo” | “bar” | FAIL “Was expecting \”foo\” or \”bar\” here!” ) At first I thought the above was just syntactic sugar, since you can, of course, already write: B. ( “foo” | “bar” | {throw new ParseException(“Was expecting \”foo\”or “\bar\” …

New Feature: FAIL Statement Read More »