Styjun

I'm trying to make a syntax highlighter in Atom for a toy language I'm working on. I'm at the stage where I'm defining the context free grammar. I've been building it up step by step and writing tests along the way. When I added the grammar for a for in loop it broke my test for parsing identifiers because the identifier started with "in". Here's the grammar as it stands now (Sorry for pasting so much code but I didn't know what might be relevant so I just added the whole thing):

module.exports = grammar(
 name: 'MooLang',

 rules: 
 source_file: $ => repeat($._declaration),

 _declaration: $ => choice(
 $.variable_declaration,
 $._statement
 ),

 variable_declaration: $ => seq(
 choice('var', 'let'),
 $.identifier,
 optional(seq(
 ':', $._type
 )),
 optional(seq(
 '=', $._expression
 )),
 $.eol
 ),

 _statement: $ => choice(
 $.for_statement,
 $.expression_statement
 ),

 for_statement: $ => prec(0, seq(
 'for',
 '(',
 choice(
 $.variable_declaration,
 $.expression_statement,
 ),
 'in',
 $._expression,
 ')',
 $._statement
 )),

 expression_statement: $ => prec(1, seq(
 $._expression,
 $.eol
 )),

 _expression: $ => choice(
 $.assignment,
 $.comparison_expression,
 $.addition_expression,
 $.multiplication_expression,
 $.unary_expression,
 prec(5, $.primary),
 prec(-1, $._type) // TODO:(Casey) Remove this
 ),

 assignment: $ => prec.right(0, seq(
 $.identifier,
 '=',
 $._expression
 )),

 comparison_expression: $ => prec.left(1, seq(
 $._expression,
 choice('<', '<=', '>', '>=', '==', '!='),
 $._expression
 )),

 addition_expression: $ => prec.left(2, seq(
 $._expression,
 choice('+', '-'),
 $._expression
 )),

 multiplication_expression: $ => prec.left(3, seq(
 $._expression,
 choice('*', '/', '%'),
 $._expression
 )),

 unary_expression: $=> prec.right(4, seq(
 choice('!', '-'),
 $.primary
 )),

 _type: $ => choice(
 $.primitive_type,
 $.list_type,
 $.map_type
 ),

 primitive_type: $ => choice(
 'bool', 'string',
 'int8', 'int16', 'int32', 'int64',
 'uint8', 'uint16', 'uint32', 'uint64',
 'float32', 'float64'
 ),

 list_type: $ => seq(
 '[',
 $._type,
 ']'
 ),

 map_type: $ => seq(
 '',
 $._type,
 ':',
 $._type,
 ''
 ),

 primary: $ => choice(
 $.bool_literal,
 $.list_literal,
 $.map_literal,
 $.parenthetical_expression,
 $.identifier,
 $.number
 ),

 bool_literal: $ => choice('true', 'false'),

 list_literal: $ => seq(
 '[',
 optional(seq(
 $._expression,
 repeat(seq(
 ',',
 $._expression
 )),
 optional(','),
 )),
 ']'
 ),

 map_literal: $ => seq(
 '',
 optional(seq(
 $._expression,
 ':',
 $._expression,
 repeat(seq(
 ',',
 $._expression,
 ':',
 $._expression,
 )),
 )),
 ''
 ),

 parenthetical_expression: $ => seq(
 '(',
 $._expression,
 ')'
 ),

 identifier: $ => prec(99, /[a-zA-Z_][a-zA-Z0-9_]*/),

 number: $ => prec(99, /d+(_d+)*(.d+)?/),

 eol: $ => 'n'
 
);

Here are the relevant tests:

==================
Identifier Tests
==================

13india

---

(source_file
 (expression_statement (primary (number)) (MISSING))
 (expression_statement (primary (identifier)) (eol))
)

==================
For Tests
==================

for (var a in people) a + 1

---

(source_file
 (for_statement (variable_declaration (identifier)) (primary (identifier)) (expression_statement (addition_expression (primary (identifier)) (primary (number))) (eol)))
)

Until I added the grammar for the for loop all of the Identifier Tests were passing but now I get this output:

My guess is that it finds an unexpected 'd' because it thinks this is the 'in' keyword. But I can't figure out why it would think that since it doesn't match anything else about the for loop.