53.4 Building and using lexical analysers

Each lexer is associated to a lexer table and an input system. Any number of lexers, using any table, can be created using the same input system. Multiple lexers sharing the same input system can be invoked in turn to parse complex inputs.

For all the lexer makers: the input-system argument is the input system from which the analyser will take its input; it is mandatory to build the input system with the same counters: specification of the table handed to the maker.

Format agnostic lexer maker

The following bindings are exported from the library (vicare parser-tools silex lexer).

Function: make-lexer lexer-table input-system

Build and return a new lexical analyser: an analysis function which, when invoked with no arguments, returns the next token.

lexer-table must be a table generated by lex, in any format.

Decision tree lexer maker

The following bindings are exported from the library (vicare parser-tools silex tree-lexer-driver).

Function: make-tree-lexer lexer-table input-system

Build and return a new lexical analyser: an analysis function which, when invoked with no arguments, returns the next token.

lexer-table must be a table generated by lex using the decision-tree format.

Portable tree lexer maker

The following bindings are exported from the library (vicare parser-tools silex char-lexer-driver).

Function: make-char-lexer lexer-table input-system

Build and return a new lexical analyser: an analysis function which, when invoked with no arguments, returns the next token.

lexer-table must be a table generated by lex using the portable format.

Scheme code lexer maker

The following bindings are exported from the library (vicare parser-tools silex code-lexer-driver).

Function: make-code-lexer lexer-table input-system

Build and return a new lexical analyser: an analysis function which, when invoked with no arguments, returns the next token.

lexer-table must be a table generated by lex using the code format.