53.3 Input systems

An input system provides the buffering, the line counting and similar low level services. The following bindings are exported by the library (vicare parser-tools silex input-system) and reexported by the library (vicare parser-tools silex lexer).

Syntax: make-IS ?clauses …

Build and return a new input system. The behaviour of this function is configured with the given ?clauses; see below for the list of supported options.

Input characters can come from a string, a port or the return value of a procedure. When an input port is used by an input system, the program should avoid reading characters directly from the port. This is because the input system may have needed a look–ahead to do the analysis of the preceding token. The program would not find what it expects on the port. The input system provides safe functions to get characters from the input.

Auxiliary Syntax: string: ?string

Instruct make-IS to build an input system that will take characters from the supplied string. When the input system is initialized with a string, it takes a copy of it. This way, eventual mutations of the string do not affect the analysis.

This clause is mutually exclusive with port: and procedure:.

Auxiliary Syntax: port: ?port

Instruct make-IS to build an input system that will read characters from the supplied textual input port. The input system never closes itself the port it has received, this task is left to the program.

This clause is mutually exclusive with string: and procedure:.

Auxiliary Syntax: procedure: ?proc

Instruct make-IS to build an input system that will read characters invoking the supplied procedure.

The use of a function as character source allows the input system to parse any character stream, no matter how it is obtained. For example, the characters may come from the decompression or decryption of a huge file, the task being done lazily in order to save space.

The function must take no argument and return a character each time it is called. When the end of file (or its logical equivalent) is reached, the function must return an object that is not a character (for example, the symbol ‘eof’). After the function has returned an end of file indicator, it is not called again.

This clause is mutually exclusive with string: and port:.

Auxiliary Syntax: counters: ?which-ones

Instruct make-IS about which counters to make available to the lexer; counters are managed by the input system, and can be used by the lexer. The following values for ?which-ones are available:

all

Make available all the counters: yyline, yycolumn, yyoffset.

line

Make avilable only the counter yyline. Accessing any of the bindings yycolumn, yyoffset will raise an unbound identifier error.

none

Make available no counter. Accessing any of the bindings: yyline, yycolumn, yyoffset will raise an unbound identifier error.

‘line’ is the default. The more counters the input system maintains, the more it is slowed down.

Notice that the same counters: option must be given to lex and to make-IS, a mismatch will result in undefined behaviour. This is because an input system is independent from a lexer table, and it is more efficient to build tables for a specific set of counters rather than to configure them at run time.

(import (vicare)
  (prefix (vicare parser-tools silex)       lex.)
  (prefix (vicare parser-tools silex lexer) lex.))

;;Build table with no knowledge of the input system.
;;
(define table
  (lex.lex (lex.input-string: "...")
           (les.output-value: #t)
           (lex.counters:     'line))) ;!!!

;;Build input system with no knowledge of the table.
;;
(define IS
  (lex.make-IS (lex.string: "1+2+3")
               (lex.counters: 'all)))  ;!!!

;;Build lexer using the table and the input system.
;;Error!!!
;;
(define lexer
  (lex.make-lexer table IS))

Function: lexer-input-system? obj

Return #t if obj is an input system object, otherwise return #f.

Function: lexer-get-func-line input-system

Return a closure which, when invoked with no arguments, will return the current line number in input-system.

Function: lexer-get-func-column input-system

Return a closure which, when invoked with no arguments, will return the current column number in input-system.

Function: lexer-get-func-offset input-system

Return a closure which, when invoked with no arguments, will return the current offset in input-system.

Function: lexer-get-func-getc input-system

Return a closure which, when invoked with no arguments, will return the next character from input-system. The closure allows client code to perform a lookahead.

The returned character is not forgotten by the lexer, this function just increments by 1 a pointer into the internal buffer. Multiple invocations of this function will return the sequence of characters about to be analysed by the lexer. When there are no more characters, the return value is the EOF object.

The returned characters are skipped by the lexer, unless we put them back with the closure returned by lexer-get-func-ungetc.

Function: lexer-get-func-ungetc input-system

Return a closure which, when invoked with no arguments, will decrement a pointer into the buffer of input-system. This function puts back a character previously read by a closure returned by lexer-get-func-getc.

It is not possible to replace characters in the input system.