Add cut operator (^) to grammar
#2104
Draft
+1,008
−547
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is the beginning of a new mdbook book that will house all of the guidelines for contributors. This is published via GitHub Pages.
This is just some light editing. I expect that this chapter will have larger edits in the future, but I want to defer that till later.
This is just a stub, with the expectation that it will be expanded/rewritten later.
This is just a stub, with the expectation that it will be expanded/rewritten later.
This has been superseded by the contributor guide.
This introduction should probably receive deeper edits to make it fit into the guide better.
traviscross
force-pushed
the
TC/add-cut-to-grammar
branch
from
8b74468 to
24690d2
Compare
The cut operator (`^`) is a backtracking fence. Once the expression to its left succeeds, we become committed to the alternative; the remainder of the expression must parse successfully or parsing will fail. See *Packrat Parsers Can Handle Practical Grammars in Mostly Constant Space*, Mizushima et al., <https://kmizu.github.io/papers/paste513-mizushima.pdf>. This operator solves a problem for us with C string literals. These literals cannot contain a null escape. But if we simply fail to lex the literal (e.g. `c"\0"`), we may instead lex it successfully as two separate tokens (`c "\0"), and that would be incorrect. As long as we only use cut to express constraints that can be expressed in a regular language and we keep our alternations disjoint, the grammar can still be mechanically converted to a CFG. Let's add the cut operator to our grammar and use it for C string literals and some similar constructs. In the railroad diagrams, we'll render the cut as a "no backtracking" box around the expression or sequence of expressions after the cut. The idea is that once you enter the box the only way out is forward.
traviscross
force-pushed
the
TC/add-cut-to-grammar
branch
from
24690d2 to
fc646a1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The cut operator (
^) is a backtracking fence. Once the expression to its left succeeds, we become committed to the alternative; the remainder of the expression must parse successfully or parsing will fail. See Packrat Parsers Can Handle Practical Grammars in Mostly Constant Space, Mizushima et al., https://kmizu.github.io/papers/paste513-mizushima.pdf.This operator solves a problem for us with C string literals. These literals cannot contain a null escape. But if we simply fail to lex the literal (e.g.
c"\0"), we may instead lex it successfully as two separate tokens (`c "\0"), and that would be incorrect.As long as we only use cut to express constraints that can be expressed in a regular language and we keep our alternations disjoint, the grammar can still be mechanically converted to a CFG.
Let's add the cut operator to our grammar and use it for C string literals and some similar constructs.
In the railroad diagrams, we'll render the cut as a "no backtracking" box around the expression or sequence of expressions after the cut. The idea is that once you enter the box the only way out is forward.
(H/t to @ehuss for suggesting the cut operator to solve this problem.)
cc @ehuss
This is stacked on #2097 and should merge after it.