53.2 Creating lexer tables

The following bindings are exported by the (vicare parser-tools silex) library.

Syntax: lex ?clause …

Build a new lexer table from a lexer specification, which can be: loaded from a file, read from a textual input port, acquired from a Scheme string; Syntax of the specification.

The output is the lexer’s table, a Scheme vector representing the lexer automaton; it can be: saved to a file, written to a port, evaluated using the library (rnrs eval (6)) and returned as value.

The behaviour of this function is configured with the given ?clause arguments; see below for the list of supported options.

The table’s code must be evaluated in an environment with the following libraries:

(rnrs)
(vicare parser-tools silex input-system)
(vicare system $fx)

Auxiliary Syntax: input-string: ?spec-str

Instruct lex to build the lexer tables from the specification in the given Scheme string. This clause is mutually exclusive with input-file: and input-port:.

Auxiliary Syntax: input-port: ?port

Instruct lex to build the lexer tables from the specification read from the given textual input port. This clause is mutually exclusive with input-string: and input-file:.

Auxiliary Syntax: input-file: ?pathname

Instruct lex to build the lexer tables from the specification in the selected file. ?pathname must be an expression evaluating to a string representing an existent file pathname. This clause is mutually exclusive with input-string: and input-port:.

Auxiliary Syntax: library-spec: ?library-name

Instruct lex to build the output as a proper Scheme library. This means a Scheme string is built, representing a library form. The string may contain Scheme comments. When this clause is not used the output is the raw table vector or a define form.

The argument library-name must represent a valid library name; the following formats are accepted:

• A string, including the parentheses. Example:

  (library-spec: "(calc-lexer)")
  ; -> (library (calc-lexer) ---)
  

• A symbol, which will be converted to string and to which parentheses will be added. Example:

  (library-spec: 'calc-lexer)
  ; -> (library (calc-lexer) ---)
  

• A list of values, which will be simply converted to string. Example:

  (library-spec: '(calc-lexer))
  ; -> (library (calc-lexer) ---)
  

It is mandatory to use the clause table-name: along with library-spec:.

Auxiliary Syntax: library-language: ?lang

Select ?lang as language to use for the library; the language is the first import specification after import in the library form. By default the language is (rnrs).

Auxiliary Syntax: library-imports: ?import-list

Select a list of libraries which must be imported when using the generated lexer. ?import-list must be a list of library specifications; for example, if we want to include the (vicare language-extensions sentinels) and (vicare language-extensions variables) libraries we do:

(library-imports:
   '((vicare language-extensions sentinels)
     (vicare language-extensions variables)))

If the lexer table is written to a proper Scheme library: ?import-list is added to the import specification of the generated library. If the lexer table is directly evaluated: ?import-list is added to the environment which is handed to eval.

This clause is ignored when the selected output is neither a library, nor an evaluated form.

Auxiliary Syntax: table-name: ?name

Instruct lex to output a define form defining a binding between ?name and the table vector. ?name can be a string or symbol, and it must represent a valid identifier.

When this clause is used along with library-spec:, the library form will export the identifier ?name.

Auxiliary Syntax: output-value: #t

Auxiliary Syntax: output-value: #f

When the argument is #t, instruct lex to evaluate the vector table using the (rnrs eval (6)) library and to return the result, which is then directly usable.

The list of libraries selected by library-imports: is added to the environment used for the evaluation. By default, the environment always includes (rnrs), and also (vicare parser-tools silex lexer) when the selected lexer format is code.

This clause is mutually exclusive with table-name:, output-file: and output-port:.

Auxiliary Syntax: output-port: ?port

Instruct lex to write the output as string in the given textual output port.

This clause is mutually exclusive with output-value: and output-file:.

Auxiliary Syntax: output-file: ?pathname

Instruct lex to save the output in a file with given pathname; this is especially useful when the output is a proper library or define form. ?pathname must be an expression evaluating to a string representing the file pathname.

This clause is mutually exclusive with output-value: and output-port:.

Auxiliary Syntax: lexer-format: ?format

Instruct lex about the format of the lexer table. ?format can be one among the following symbols: decision-tree, ‘code’, ‘portable’; the default is ‘decision-tree’. Tables output format

Auxiliary Syntax: pretty-print: #t

Auxiliary Syntax: pretty-print: #f

Instruct lex to pretty–print the contents of the table. Normally, the table is displayed as a compact mass of characters fitting in about 75 columns. This clause is useful only for a developer of (vicare parser-tools silex). The Scheme code generated with the ‘code’ clause is always pretty–printed, the others are not by default.

Auxiliary Syntax: counters: ?which-ones

Instruct lex about which counters will be available to the lexer; counters are managed by the input system, and can be used by the lexer. The following values for ?which-ones are available:

all

Expect all the counters to be available from the input system: yyline, yycolumn, yyoffset.

line

Expect only the counter yyline to be available from the input system.

none

Expect no counter to be available.

‘line’ is the default.

Notice that the same counters: clause must be given to lex and to make-IS, a mismatch will result in undefined behaviour. This is because an input system is independent from a lexer table, and it is more efficient to build tables for a specific set of counters rather than to configure them at run time.