FSharpCompiler.Lex 1.1.7

FSharpCompiler.Lex are a tool for generating the DFA for lexical analyzer.

The lexical analyzer reads a sequence of tokens as its input, and groups the tokens using the regular expression rules. If the input is valid, the end result is that the entire token sequence groups to a sequence of groupings, each grouping, which is represents a matching pattern, consists of a index and a list of tokens.

Unlike traditional Lex tools, FSharpCompiler.Lex does not require action in rules. Instead of explicitly actions a stream of groups is implicitly constructed during the analyzing. The specification of The Lex input file see: [The Lex Input File](Lex input file.md).

A function to generate DFA have following type:

Lex.generateDFA: input:string -> DFA<string>

Lex.generateDFA takes as input a regular expressions specification and produces a structural data that represent DFA of the regular expressions specification. the produced DFA's type definition is:

type DFA<'tag when 'tag: comparison> =
    {
        transitions  :Set<uint32*'tag*uint32>
        finalLexemes :(Set<uint32>*Set<uint32>) list
    }

'tag is token type that is string type typically.

transitions is a transition table.

finalLexemes is acceptable state list, every state pair of which is final state, and trailing state.

how to generate a lexical analyzer using FSharpCompiler.Lex , see the resolution: xp44mm/PolynomialExpressions

Lex usage Examples

Too many nullable nonterminal symbols can cause the number of BNFs to swell and even trigger reduce/reduce conflicts. The lexical analyzer can skip the extra separators so that the tokens entering the parser conform to the context-free grammar specification. For example, the following regular expression can find negligible line breaks. For example, the following regular expression can find negligible line breaks and process them (remove or merge) Before tokens enter the grammar parser.

BOF "\n"* "%%"? "\n"*
"\n"* EOF
"%%" "\n"+
"\n"+

To supplement the default operator before tokens enter the grammar parser. In this way, when the grammar parser looks forward the supplemented operator, it is enable to determine to reduce the top of the stack. For example, when two adjacent tokens are the end of the statement and the beginning of the statement, we can determine that the semicolon should be added here.

{last_token_of_stmt} / {first_token_of_stmt}

To supplement the holes in the array in JavaScript：

[ "[" "," ] / ","

To refine token, for example, both the two integers that connected by a colon are column addresses in Excel.

INTEGER ":" INTEGER