Marco's Bash Functions Library


Next: , Up: (dir)

Marco's Bash Functions Library

This document describes version 1.3b9 of MBFL, a library of functions for the GNU Bash shell. The package is distributed under the terms of the GNU Lesser General Public License (LGPL); the home page is at:

http://marcomaggi.github.com/mbfl.html

development takes place at:

http://github.com/marcomaggi/mbfl/

Copyright © 2003-2005, 2009, 2010, 2012 by Marco Maggi marco.maggi-ipsu@poste.it

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with Invariant Sections being “GNU Free Documentation License” and “GNU Lesser General Public License”, no Front–Cover Texts, and no Back–Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License”.

Appendices

Indexes

--- The Detailed Node Listing ---

Overview of the package

Using the script preprocessor

Manipulating files and pathnames

File names

File Commands

Parsing command line options

Using external programs

Interfaces to external programs

Manipulating strings

Manipulating variables

Interfacing with the system

Main function

Building test suites

Handling files in tests

Examples for sending email

Using gnutls-cli as connector


Next: , Previous: Top, Up: Top

1 Overview of the package

This package is an attempt to make GNU Bash a viable solution for medium sized scripts. A set of modules implementing common operations and a script template are offered by this package and the author has used them with success in implementing non–small scripts.

The philosophy of MBFL is to do the work as much as possible without external commands. For example: string manipulation is done using the special variable substitution provided by Bash, and no use is done of utilities like sed, grep and ed.

The library is better used starting from the template script examples/template.sh. This is because with MBFL some choices have been made to reduce the application dependent part of the script to the smallest dimension; if we follow another schema, MBFL modules may be inadequate. This is especially true for the options parsing module.


Next: , Up: overview

1.1 Quick run time loading

The easiest way to use the library is to include at run time the library file libmbfl.sh in the script. To do it, we install the package on the system and use this code in the scripts:

     source "${MBFL_LIBRARY:=$(mbfl-config)}"

after the service variables have been declared. Required user defined variables.

This code will read the full pathname of the library from the environment variable MBFL_LIBRARY; if this variable is not set: the script mbfl-config is invoked with no arguments to acquire the pathname. mbfl-config is installed by the package in the $(bindir) directory, which we must include in the PATH environment variable.


Next: , Previous: overview quick, Up: overview

1.2 Reliable run time loading

A more reliable way to load the library is:

     script_PROGNAME=this-script
     mbfl_INTERACTIVE=no
     mbfl_LOADED=no
     mbfl_INSTALLED=$(mbfl-config) &>/dev/null
     mbfl_HARDCODED=./some/dir/libmbfl.sh
     for item in \
         "$MBFL_LIBRARY" "$mbfl_HARDCODED" "$mbfl_INSTALLED"
     do
         test -n "$item" -a -f "$item" -a -r "$item" && {
             source "$item" &>/dev/null || {
                 printf '%s error: loading MBFL file "%s"\n' \
                     "$script_PROGNAME" "$item" >&2
                 exit 2
             }
         }
     done
     unset -v item
     test "$mbfl_LOADED" = yes || {
         printf '%s error: incorrect evaluation of MBFL\n' \
             "$script_PROGNAME" >&2
         exit 2
     }

where the optional ./some/dir/libmbfl.sh can be the pathname of a known location in which to find the library. This code:

  1. Looks for the pathname in the MBFL_LIBRARY variable, to allow us to explicitly select the file from the command line; example:
              $ MBFL_LIBRARY=/path/to/libmbfl.sh script.sh
    
  2. Looks in some known directory; for example a subdirectory of a software package that we have distributed.
  3. As a last resort tries to evaluate the mbfl-config script to pick up a library installed on the system.

We see that the output of the source command is discarded: this is because we assume that:

This code is included in the MBFL distribution in the src/lib/loader.sh file.


Next: , Previous: overview reliable, Up: overview

1.3 Direct inclusion

Another solution is to directly include the library in the script; this is easy if we preprocess our scripts with GNU m4. We only need to put the following in the script:

     m4_changequote([[,]])
     m4_include(libmbfl.sh)

then preprocess the script with:

     $ m4 --prefix-builtins --include=/path/to/library \
              script.sh.m4 >script.sh

easy to do in a Makefile. The installation directory pathname of the library (/path/to/library in the example) is the output of mbfl-config --libpath.

It is also interesting to process the script with the following rule of GNU Make: assuming that the source scripts are in the src/modules directory of the source tree:

     vpath	%.sh.m4		$(srcdir)/src/modules
     
     M4      = ...
     M4FLAGS = --prefix-builtins --include=/path/to/library
     
     %.sh: %.sh.m4
             $(M4) $(M4FLAGS) $(<) | \
             grep --invert-match -e '^#' -e '^$$' | \
             sed -e "s/^ \\+//" >$(@)

this will remove all the comments and blank lines, decreasing the size of the script significantly if one makes use of verbose comments; note that this will wipe out the #!/bin/bash first line, too.

Usually we want the script to begin with #!/bin/bash followed by a comment describing the license terms. We can do it by preparing a script like the following:

     #!/bin/bash
     # ... license ...
     
     m4_include(realscript.sh)
     
     ### end of file

and processing it with the following makefile rule:

     M4      = ...
     M4FLAGS = --prefix-builtins --include=/path/to/library
     
     script.sh: script.sh.m4 realscript.sh
             $(M4) $(M4FLAGS) $(<) >$(@)

realscript.sh can be processed as explained above.

At this point, though, it is better to use the MBFL preprocessor. Using the script preprocessor.


Previous: overview direct, Up: overview

1.4 Notes

When using MBFL we want to be aware of the following:


Next: , Previous: overview, Up: Top

2 Using the script preprocessor

The MBFL script preprocessor is a command line program named mbflpp.sh; it is itself a Bash script that makes use of the MBFL library. It can remove comments, blank lines, blank characters at the beginning of lines, and it can expand macros.

The macro preprocessing is done with the GNU m4 program, using a library of macros called preprocessor.m4 which is installed on the system with the MBFL package.

The use of the preprocessor is fully optional: every feature of the MBFL library can be used without the preprocessor.


Next: , Up: preprocessor

2.1 Invoking the preprocessor

The basic synopsis is:

     mbflpp.sh [options] <INFILE >OUTFILE
     mbflpp.sh [options] --outfile=OUTFILE [--] INFILE1 INFILE2 ...

all the INFILE pathnames are interpreted as files to concatenate in the specified order.

All the MBFL built in command line options are available, additionally the following options are supported.

--preserve-comments
Do not filter out comments.
--add-bash
Add #!$BASH at the beginning of the output, where BASH is the built–in variable of Bash, BASH.
--define=VALUE
Define a new symbol (m4 syntax). This is equivalent to the --define option of m4.
--include=VALUE
Add a search path for files. VALUE must be a directory pathname and it is handed to m4 to search for macro files. Can be used multiple times.
--library=VALUE
Include an m4 library. VALUE must be the name of an m4 macro file which is evaluated before the input files.
-oVALUE
--output=VALUE
Select an output file, - for stdout.
-e
--eval
If used evaluates the output script in bash, instead of printing it.


Previous: preprocessor invoking, Up: preprocessor

2.2 Macros library

The m4 preprocessor is invoked by mbflpp.sh with the --prefix-builtins option; so all the m4 built in macros and directives are available prefixed with m4_.

— Preprocessor Macro: mandatory_parameter varname number name

Define a variable local to a shell function holding an argument to the function. varname is the name of the variable; number is the positional parameter number; name is a description of the argument.

Example, the following:

          mandatory_parameter(PATHNAME, 2, file pathname)

is expanded to:

          local PATHNAME=${2:?"missing file pathname parameter to '$FUNCNAME'"}

Another example the following function:

          function message () {
            mandatory_parameter(PROGNAME, 1, program name)
            mandatory_parameter(STRING,   2, message string)
          
            printf '%s: %s\n' "$PROGNAME" "$STRING"
          }

is expanded to:

          function message () {
            local PROGNAME=${1:?"missing program name parameter to '$FUNCNAME'"}
            local STRING=${2:?"missing message string parameter to '$FUNCNAME'"}
          
            printf '%s: %s\n' "$PROGNAME" "$STRING"
          }

and so it is a function with two mandatory parameters.

— Preprocessor Macro: optional_parameter varname number default_value

Define a variable local to a shell function holding an argument to the function. varname is the name of the variable; number is the positional parameter number; default_value is the initialisation value for the variable if the argument is not used.

Example, the following:

          optional_parameter(COUNT, 2, 123)

is expanded to:

          local COUNT=${2:-123}"
— Preprocessor Macro: command_line_argument varname argindex

Define a variable local to a shell function holding an argument to the script. varname is the variable name, argindex is the argument index in the ARGV array.

Example, the following:

          command_line_argument(PATHNAME,3)

is expanded to:

          local PATHNAME="${ARGV[3]}"


Next: , Previous: preprocessor, Up: Top

3 Required user defined variables

The following variables are expected to be defined before the MBFL code is evaluated. They are used by MBFL to compute values for its own variables.

— Variable: script_PROGNAME

The name of the script.

— Variable: script_AUTHOR

The name of the script author(s).

— Variable: script_COPYRIGHT_YEARS

A comma separated list of years of copyright.

— Variable: script_VERSION

The version number of the script.

— Variable: script_LICENSE

The identifier of the license under which the script is released. Accepted values are: GPL or GPL2, GPL3, LGPL or LGPL2, LGPL3, BSD. It is used to select the appropriate value to be displayed when the user of the script selects the --license option. Predefined options

— Variable: script_USAGE

Must be a string describing the usage of the program.

— Variable: script_DESCRIPTION

One line string providing a brief description of the program. It is used in the help screen (the one echoed when the --help option is used) just after the content of script_USAGE.

— Variable: script_EXAMPLES

One or more lines of text to be displayed at the end of the help screen, after the options description. It should contain examples of common invocations for the script.

All the text in these variables is used as argument to the printf built–in command; in particular: script_DESCRIPTION and script_EXAMPLES are used as first argument to printf, so the escape sequences (in particular \t and \n) are expanded.

Examples

The following example shows how to declare the variables.

     script_PROGNAME=myscript.sh
     script_AUTHOR='Marco Maggi and Marco Maggi'
     script_COPYRIGHT_YEARS='2002, 2003, 2004'
     script_VERSION=1.0
     script_LICENSE=GPL3
     script_USAGE="usage: $script_PROGNAME [options] ..."
     script_DESCRIPTION='Does this and that.'
     script_EXAMPLES='Examples:
     
     \tmyscript.sh --do-something arg arg ...
     \tmyscript.sh --do-other arg arg ...'

Notice that script_DESCRIPTION and script_EXAMPLES do not end with a newline character.


Next: , Previous: service variables, Up: Top

4 Some base functions

— Function: mbfl_set_maybe name value

Set a variable named name to value, but only if name is not the empty string.

— Function: mbfl_read_maybe_null varname

Read a line from its stdin and store it in a variable named varname. If mbfl_option_null returns true: the null character is used as terminator, as in:

          IFS= read -d $'\x00'


Next: , Previous: base, Up: Top

5 Encoding and decoding strings

The purpose of this module is to let an external process invoke a Bash script with damned command line arguments: strings including blanks or strange characters that may trigger quoting rules.

This problem can arise when using scripting languages with some sort of eval command.

The solution is to encode the argument string in hexadecimal or octal format strings, so that all the damned characters are converted to “good” ones. The Bash script can convert them back.

— Function: mbfl_decode_hex hex_string

Decode a hex string and outputs it on stdout.

— Function: mbfl_decode_oct oct_string

Decode a oct string and outputs it on stdout.

Example:

     mbfl_decode_hex 414243
     -> ABC


Next: , Previous: encoding, Up: Top

6 Manipulating files and pathnames


Next: , Up: file

6.1 File names


Next: , Up: file names

6.1.1 Splitting a file name into its components

— Function: mbfl_file_extension pathname

Extract the extension from a file name. Search the last dot character in the argument string and echo to stdout the range of characters from the dot to the end, not including the dot. If a slash character or the beginning of the string is found first: echoes to stdout the empty string.

— Function: mbfl_file_dirname pathname

Extract the directory part from a fully qualified file name. Search the last slash character in the input string and echo to stdout the range of characters from the first to the slash, not including the slash.

If no slash is found: echo a single dot (the current directory).

If the input string begins with / or // with no slash characters after the first ones: echo a single slash.

— Function: mbfl_file_rootname pathname

Extract the root portion of a file name. Search the last dot character in the argument string and echo to stdout the range of characters from the beginning to the dot, not including the dot.

If a slash character is found first, or no dot is found, or the dot is the first character: echo the empty string.

— Function: mbfl_file_tail pathname

Extract the file portion from a fully qualified file name. Search the last slash character in the input string and echo to stdout the range of characters from the slash to the end, not including the slash. If no slash is found: echo the whole string.

— Function: mbfl_file_split pathname

Separate a file name into its components. One or more contiguous occurrences of the slash character is used as separator. The components are stored in an array named SPLITPATH, that may be declared local in the scope of the caller; the base index is zero. The number of elements in the array is stored in a variable named SPLITCOUNT. Return true.

— Function: mbfl_file_strip_trailing_slash pathname

If the last character in pathname is a slash (/): remove it and print the result on stdout.


Next: , Previous: file name parts, Up: file names

6.1.2 Handling relative pathnames

— Function: mbfl_file_normalise pathname
— Function: mbfl_file_normalise pathname prefix

Normalise a file name: remove all the occurrences of . and ...

If pathname is relative (according to mbfl_file_is_absolute) and prefix is not present or it is the empty string: the current process working directory is prepended to pathname.

If prefix is present and non empty, and pathname is relative (according to mbfl_file_is_absolute): prefix is prepended to pathname and normalised, too.

Echo to stdout the normalised file name. Return true.

— Function: mbfl_file_subpathname pathname basedir

If pathname is a subdirectory or file under basedir: print to stdout the subpathname portion. Example:

          mbfl_file_subpathname /a/b/c /a
          -> ./b/c

Both pathname and basedir must be full (normalised) pathnames for this function to work.

If pathname is recognised as subpathname of basedir: the return code is zero; else the return code is one.

— Function: mbfl_file_is_absolute pathname

Return true if the first character in pathname is a slash (/); else return false.

— Function: mbfl_file_is_absolute_dirname pathname

Return true if pathname is a directory according to mbfl_file_is_directory and an absolute pathname according to mbfl_file_is_absolute.

— Function: mbfl_file_is_absolute_filename pathname

Return true if pathname is a file according to mbfl_file_is_file and an absolute pathname according to mbfl_file_is_absolute.


Previous: file name path, Up: file names

6.1.3 Finding pathnames on the system

— Function: mbfl_file_find_tmpdir
— Function: mbfl_file_find_tmpdir PATHNAME

Find a value for a temporary directory according to the following rules:

  1. If PATHNAME argument is not used: it defaults to the current value of mbfl_option_TMPDIR.
  2. If PATHNAME is not null and it is a directory and it is writable: it is accepted as value.
  3. If PATHNAME argument is invalid: the value /tmp/$USER, where USER is the environment variable, is tried; finally the value /tmp is tried.

Echo the accepted value to stdout. Return true if a value is found, false otherwise.


Next: , Previous: file names, Up: file

6.2 File Commands


Next: , Up: file commands

6.2.1 Retrieving informations

— Function: mbfl_file_enable_listing

Declare the commands required to retrieve informations about files and directories. Declaring the intention to use a program

The programs are: ls, readlink.

— Function: mbfl_file_listing pathname

Execute ls with pathname as argument.

— Function: mbfl_file_long_listing pathname

Execute ls with pathname as argument and the flag -l.

— Function: mbfl_file_get_owner pathname

Print the owner of the file.

— Function: mbfl_file_get_group pathname

Print the group of the file.

— Function: mbfl_file_get_size pathname

Print the size of the file in bytes.

— Function: mbfl_file_normalise_link pathname

Make use of the readlink to normalise the pathname of a symbolic link (remember that a symbolic link references a file, never a directory). Echo to stdout the normalised pathname.

The command line of readlink is:

          readlink -fn "$pathname"
— Function: mbfl_file_read_link pathname

Make use of readlink to acquire the original pathname referenced by pathname, then print it.


Next: , Previous: file commands listing, Up: file commands

6.2.2 Creating directories

— Function: mbfl_file_enable_make_directory

Declare the commands required to create directories. Declaring the intention to use a program

The programs are: mkdir.

— Function: mbfl_file_make_directory pathname
— Function: mbfl_file_make_directory pathname permissions

Create a directory named pathname; all the non–existent parents are created, too. If permissions is present: it is the specification of directory permissions in octal mode.

This function does not test if the directory already exists: the command is always executed.

— Function: mbfl_file_make_if_not_directory pathname
— Function: mbfl_file_make_if_not_directory pathname permissions

Wrapper for mbfl_file_make_directory that creates the directory if it does not exist.

If a sudo user was requested: this function resets the request even if no command has been executed.


Next: , Previous: file commands mkdir, Up: file commands

6.2.3 Copying files

At present, copying of directories is not supported; we have to create the directory and then copy files into it.

— Function: mbfl_file_enable_copy

Declare the commands required to copy files and directories. Declaring the intention to use a program

The programs are: cp.

— Function: mbfl_file_copy source target ?...?

Copy the source, a file pathname, to target, a file pathname. Additional arguments are handed to the command unchanged.

If source does not exist, or if it is not a file, an error is generated and the return value is 1. If target exists an error is generated and the return value is 1.

— Function: mbfl_file_copy_to_directory source target ?...?

Copy the source, a file pathname, into the directory target. Additional arguments are handed to the command unchanged.

If source does not exist, or if it is not a file, an error is generated and the return value is 1. If target does not exist or it is not a directory: an error message is generated and the return value is 1.


Next: , Previous: file commands copy, Up: file commands

6.2.4 Moving files

— Function: mbfl_file_enable_move

Declare the commands required to move files and directories. Declaring the intention to use a program

The programs are: mv.

— Function: mbfl_file_move source target ?...?

Move the source, a file or directory, to target, a pathname. Additional arguments are handed to the command unchanged.

If source does not exist, or if it is not readable, an error is generated and the return value is 1. If target exists an error is generated and the return value is 1.

— Function: mbfl_file_move_to_directory source target ?...?

Move the source, a file or directory, into the directory target. Additional arguments are handed to the command unchanged.

If source does not exist, or if it is not readable, an error message is generated and the return value is 1. If target does not exist or it is not a directory: an error is generated and the return value is 1.


Next: , Previous: file commands move, Up: file commands

6.2.5 Removing files and directories

Files removal is forced: the --force option to rm is always used. It is responsibility of the caller to validate the operation before invoking these functions.

Some functions test the existence of the pathname before attempting to remove it: this is done only if test execution is disabled; if test execution is enabled the command line is echoed to stderr to make it easier to debug scripts.

— Function: mbfl_file_enable_remove

Declare the commands required to remove files and directories. Declaring the intention to use a program

The programs are: rm, rmdir.

— Function: mbfl_file_remove pathname

Remove pathname, no matter if it is a file or directory. If it is a directory: descend the sub-levels removing all of them. If an error occurs return 1.

— Function: mbfl_file_remove_file pathname

Remove the file or symbolic link selected by pathname. If the file does not exist or it is not a file or an error occurs: return 1.

— Function: mbfl_file_remove_symlink pathname

Remove the symbolic link selected by pathname. If the link does not exist or it is not a symbolic link or an error occurs: return 1.

— Function: mbfl_file_remove_file_or_symlink pathname

Remove the file or symbolic link selected by pathname. If the file does not exist or it is not a file or an error occurs: return 1.

— Function: mbfl_file_remove_directory pathname

Remove the directory selected by pathname. If the directory does not exist or an error occurs: return 1.

— Function: mbfl_file_remove_directory_silently pathname

Like mbfl_file_remove_directory, but do not print messages if the directory is not empty.


Next: , Previous: file commands removing, Up: file commands

6.2.6 Creating symbolic links

— Function: mbfl_file_enable_symlink

Declare the commands required to create symbolic links. Declaring the intention to use a program

The programs are: ln.

— Function: mbfl_file_symlink original_name symlink_name

Create a symbolic link.


Next: , Previous: file commands symlink, Up: file commands

6.2.7 Manipulating tar archives

Remember that when we execute a script with the --test option: the external commands are not executed: a command line is echoed to stdout. It is recommended to use this mode to fine tune the command line options required by tar.

— Function: mbfl_file_enable_tar

Declare the tar command. Declaring the intention to use a program

— Function: mbfl_tar_exec ?...?

Execute tar with whatever arguments are used. Return the return code of tar.

— Function: mbfl_tar_create_to_stdout directory ?...?

Create an archive and send it to stdout. The root of the archive is the directory. Files are selected with the . pattern. tar flags may be appended to the invocation to this function. In case of error return 1.

— Function: mbfl_tar_extract_from_stdin directory ?...?

Read an archive from stdin and extract it under directory. tar flags may be appended to the invocation to this function. In case of error return 1.

— Function: mbfl_tar_extract_from_file directory archive ?...?

Read an archive from a file and extract it under directory. tar flags may be appended to the invocation to this function. In case of error return 1.

— Function: mbfl_tar_create_to_file directory archive ?...?

Create an archive named archive holding the contents of directory. Before creating the archive, the process changes the current directory to directory and selects the files with the pattern .. tar flags may be appended to the invocation to this function. In case of error return 1.

— Function: mbfl_tar_archive_directory_to_file directory archive ?...?

Like mbfl_tar_create_to_file but archive all the contents of directory, including the directory itself (not its parents).

— Function: mbfl_tar_list archive ?...?

Print to stdout the list of files in archive. tar flags may be appended to the invocation to this function. In case of error return 1.


Next: , Previous: file commands tar, Up: file commands

6.2.8 Set/get file permissions

— Function: mbfl_file_enable_permissions

Declare the intention to use the programs required to get/set file permissions. Declaring the intention to use a program

The programs are: ls, chmod, cut.

— Function: mbfl_file_get_permissions pathname

Print the access permissions for pathname, in octal format.

To set permissions one may request the use of sudo: just request a user before invoking the following function. Executing a program

— Function: mbfl_file_set_permissions mode pathname

Set the access permissions for pathname; mode must be in a form accepted by chmod.


Previous: file commands perms, Up: file commands

6.2.9 Compressing files

This module has an internal state, stored in global variables. The state holds informations about:

Initialisation
— Function: mbfl_file_enable_compress

Declare the programs required to compress a file. Declaring the intention to use a program

The programs are: gzip, bzip2.

Configuration

The selection will affect all the future invocations of the compression/decompression functions.

— Function: mbfl_file_compress_select_gzip
— Function: mbfl_file_compress_select_bzip

Select a compressor program.

— Function: mbfl_file_compress_keep
— Function: mbfl_file_compress_nokeep

Select whether the compress program should keep the original file or not.

— Function: mbfl_file_compress_select_stdout
— Function: mbfl_file_compress_select_nostdout

Select if the output must be sent to stdout or a file. This takes precedence over the keep/no keep configuration: if the output is sent to stdout the original file is kept.

Actions

Additional arguments to the selected compressor may be appended to the invocation of the following functions and are handed to the compressor unchanged.

— Function: mbfl_file_compress PATHNAME ?...?

Compress PATHNAME, a file pathname, with the currently selected compressor program.

— Function: mbfl_file_decompress PATHNAME ?...?

Decompress PATHNAME, a file pathname, with the currently selected compressor program.


Next: , Previous: file commands, Up: file

6.3 Testing file existence and the like

When using the following functions: the optional argument PRINT_ERROR will cause an error message to be printed with mbfl_message_error if the test fails; the argument value must be print_error.

File functions
— Function: mbfl_file_exists pathname

Return true if pathname exists.

— Function: mbfl_file_is_file filename
— Function: mbfl_file_is_file filename PRINT_ERROR

Return true if filename is not the empty string and is a file.

— Function: mbfl_file_is_readable filename
— Function: mbfl_file_is_readable filename PRINT_ERROR

Return true if filename is not the empty string, is a file and is readable.

— Function: mbfl_file_is_writable filename
— Function: mbfl_file_is_writable filename PRINT_ERROR

Return true if filename is not the empty string, is a file and is writable.

— Function: mbfl_file_is_executable filename
— Function: mbfl_file_is_executable filename PRINT_ERROR

Return true if filename is not the empty string, is a file and is executable.

Directory functions
— Function: mbfl_file_is_directory directory
— Function: mbfl_file_is_directory directory PRINT_ERROR

Return true if directory is not the empty string and is a directory.

— Function: mbfl_file_directory_is_readable directory
— Function: mbfl_file_directory_is_readable directory PRINT_ERROR

Return true if directory is not the empty string, is a directory and is readable.

— Function: mbfl_file_directory_is_writable directory
— Function: mbfl_file_directory_is_writable directory PRINT_ERROR

Return true if directory is not the empty string, is a directory and is writable.

— Function: mbfl_file_directory_is_executable directory
— Function: mbfl_file_directory_is_executable directory PRINT_ERROR

Return true if directory is not the empty string, is a directory and is executable.

— Function: mbfl_file_directory_validate_writability directory description

Test directory existence and writability; return true if the directory exists and is writable. If the condition is not met: Print informative messages using description to refer to the directory.

Symbolic link functions
— Function: mbfl_file_is_symlink pathname
— Function: mbfl_file_is_symlink pathname PRINT_ERROR

Return true if pathname is not the empty string and is a symbolic link.

Generic pathname functions
— Function: mbfl_file_pathname_is_readable pathname
— Function: mbfl_file_pathname_is_readable pathname PRINT_ERROR

Return true if pathname is not the empty string and is readable.

— Function: mbfl_file_pathname_is_writable pathname
— Function: mbfl_file_pathname_is_writable pathname PRINT_ERROR

Return true if pathname is not the empty string and is writable.

— Function: mbfl_file_pathname_is_executable pathname
— Function: mbfl_file_pathname_is_executable pathname PRINT_ERROR

Return true if pathname is not the empty string and is executable.


Next: , Previous: file testing, Up: file

6.4 Reading and writing files with privileges

The following functions perform actions that can normally be done directly with the redirection operators of Bash:

     # write to a file
     printf '%s' "$string" >"$filename"
     
     # append to a file
     printf '%s' "$string" >>"$filename"
     
     # read a file, print contents
     printf '%s' "$(<$filename)"

The functions act differently in that they spawn a bash subprocess, by invoking mbfl_program_exec, and let it do the operation; this allows us to request the usage of sudo and so to read and write files with modified privileges, but only for the time needed to do the operation, not for the whole script.

— Function: mbfl_file_write string filename

Write string to filename, eventually creating it or overwriting old contents.

— Function: mbfl_file_append string filename

Append string to filename, eventually creating it.

— Function: mbfl_file_read filename

Read and print all the contents of filename.


Previous: file read and write, Up: file

6.5 Miscellaneous commands

— Function: mbfl_change_directory dirname ?...?

Change directory to dirname. Optional flags to cd may be appended.

— Function: mbfl_cd dirname ?...?

Wrapper for mbfl_change_directory. If verbose mode is on: print a message.


Next: , Previous: file, Up: Top

7 Parsing command line options

The getopt module defines a set of procedures to be used to process command line arguments with the following format:

-a
Brief option a with no value.
-a123
Brief option a with value 123.
--bianco
Long option bianco with no value.
--color=bianco
Long option color with value bianco.


Next: , Up: getopts

7.1 Arguments

The module contains, at the root level, a block of code like the following:

     ARGC=0
     declare -a ARGV ARGV1
     
     for ((ARGC1=0; $# > 0; ++ARGC1))
     do
         ARGV1[$ARGC1]=$1
         shift
     done

this block is executed when MBFL (and the script that loads it) is evaluated. Its purpose is to store command line arguments in the global array ARGV1 and the number of command line arguments in the global variable ARGC1.

The global array ARGV and the global variable ARGC are predefined and should be used by the mbfl_getopts_* functions to store non–option command line arguments.

Example:

     $ script --gulp wo --gasp=123 wa

if the script makes use of MBFL, the strings wo and wa will go into ARGV and ARGC will be set to 2. The option arguments are processed and some action is performed to register them.

We can access the non–option arguments with the following code:

     for ((i=0; $i < $ARGC; ++i))
     do
         # do something with ${ARGV[$i]}
     done

When using action arguments: the first non–option argument can be interpreted as special value that selects an action to be performed by the script. In this case the first argument is removed from the ARGV array, so that processing the other arguments is not affected.


Next: , Previous: getopts arguments, Up: getopts

7.2 Using the module

To use this module we have to declare a set of script options and optionally a set of action arguments. We declare a new script option with the function mbfl_declare_option and a new action argument with the function mbfl_declare_action_argument.

Option and action argument declarations should be done at the beginning of the script, before doing anything else; for example: right after the MBFL library code.

In the main block of the script, options are parsed by invoking mbfl_getopts_parse: this function will update global variables and invoke a script function for each option on the command line. It can also select a function to be invoked as the main action of the script.

Option with no argument

Example of option declaration:

     mbfl_declare_option ALPHA no a alpha noarg "enable alpha option"

this code declares an option with no argument having properties:

If the option is used: the function script_option_update_alpha is invoked (if it exists) with no arguments, after the variable script_option_ALPHA has been set to yes. Valid option uses are:

     $ script.sh -a
     $ script.sh --alpha
Option with argument

Example of option declaration:

     mbfl_declare_option BETA 123 b beta witharg "select beta value"

this code declares an option with argument having properties:

If the option is used: the function script_option_update_beta is invoked (if it exists) with no arguments, after the variable script_option_BETA has been set to the selected value. Valid option uses are:

     $ script.sh -b456
     $ script.sh --beta=456
Action options

A special option example:

     mbfl_declare_option ACTION_GAMMA \
        no g gamma noarg "do gamma action"
     mbfl_declare_option ACTION_DELTA \
        yes d delta noarg "do delta action"

this code declares two options with no arguments; the difference from the other declarations is that the keywords are prefixed with ACTION_: this prefix is recognised by the module and causes, if the option is used on the command line, the following code to be evaluated at argument parsing time:

     mbfl_main_set_main script_action_gamma

or:

     mbfl_main_set_main script_action_delta

where the argument script_action_gamma is built by prefixing the lower case version of the keyword with script_. The code selects a function as main function for the script. Driving script execution.

Additionally, if the default value is yes: the main function is selected at declaration time (that is by mbfl_declare_option); this is useful to declare an action option and select automatically the action function. In the example: the function script_action_delta is selected as main action function.

It is an error to declare a keyword prefixed with ACTION_ with an option with argument.

Action arguments

The following example declares two action arguments:

     mbfl_declare_action_argument gamma gamma no  no "do gamma action"
     mbfl_declare_action_argument delta delta yes no "do delta action"

an action argument is a non–option argument that is used on the command line as first parameter; its purpose is to select a function to be evaluated as main action of the script.

The first of the two declarations has the following properties:

If the first parameter to the script is gamma or delta, the following code is evaluated at argument parsing time:

     mbfl_main_set_main script_action_gamma

or:

     mbfl_main_set_main script_action_delta

where the argument script_action_gamma is built by prefixing the the keyword with script_action_. The code selects a function as main function for the script. Driving script execution.

Additionally, if the selection value is yes: the main function is selected at declaration time (that is by mbfl_declare_action_argument); this is useful to declare an action option and select automatically the action function. In the example: the function script_action_delta is selected as main action function.

Action arguments and action options

Notice that action arguments and action options implement the same feature and are fully compatible with each other. Both can be used to select an action for the script.

When declaring action options and action arguments selected by default: the last one wins. So the following:

     mbfl_declare_option ACTION_GAMMA \
       yes g gamma noarg "do gamma action"
     
     mbfl_declare_action_argument delta \
       delta yes no "do delta action"

defines a --gamma action option and a delta action argument, and the main function is set to script_action_delta.

However, it is bad interface design to use both of them; it is better to use action arguments or action options.


Next: , Previous: getopts usage, Up: getopts

7.3 Predefined options

A set of predefined options is recognised by the library and not handed to the user defined functions.

--tmpdir=DIR
Selects a directory for temporary files. The default value is the one in the environment variable TMPDIR, or /tmp/$USER if that variable is not set. The value is stored in the variable mbfl_option_TMPDIR.
--encoded-args
Signals to the library that the non–option arguments and the option values are encoded in hexadecimal strings. Encoding is useful to avoid quoting problems when invoking a script from another one.

If this option is used: the values are decoded by mbfl_getopts_parse before storing them in the ARGV array and before being stored in the option's specific global variables.

-v
--verbose
Turns on verbose messages. If this option is used: The function mbfl_option_verbose returns true. Printing messages to the console.
--silent
Turns off verbose messages. If this option is used: The function mbfl_option_verbose returns false.
--verbose-program
If used: The --verbose option is added to the command line of external programs that support it. The function mbfl_option_verbose_program returns true or false depending on the state of this option.
--show-program
Prints the command line of executed external programs.
--debug
Turns on debugging messages. Automatically turns on verbose messages and program showing. Printing messages to the console.
--test
Turns on test execution. Testing a script and external programs.
--null
Signals to the script that it has to use the null character to separate values, instead of the common newline. The global variable mbfl_option_NULL is set to yes.
-f
--force
Signals to the script that it does not have to query the user before doing dangerous operations, like overwriting files. The global variable mbfl_option_INTERACTIVE is set to no.
-i
--interactive
Signals to the script that it does have to query the user before doing dangerous operations, like overwriting files. The global variable mbfl_option_INTERACTIVE is set to yes.
--validate-programs
Validates the existence of all the programs needed by the script; then exits. The exit code is zero if all the programs were found, one otherwise.
--list-exit-codes
Prints a list of numerical exit codes and their associated names, as declared in the script. Declaring exit codes.
--print-exit-code=NAME
Prints the numerical exit code associated to NAME.
--print-exit-code-names=CODE
Prints the list of names associated to the numerical exit CODE.
--version
Prints to the standard output of the script the contents of the global variable mbfl_message_VERSION, then exits with code zero. The variable makes use of the service variables. Required user defined variables.
--version-only
Prints to the standard output of the script the contents of the global variable script_VERSION, then exits with code zero. Required user defined variables.
--license
Prints to the standard output of the script the contents of one of the global variables mbfl_message_LICENSE_*, then exits with code zero. The variable makes use of the service variables. Required user defined variables.
-h
--help
--usage
Prints to the standard output of the script: the contents of the global variable script_USAGE; a newline; the string options:; a newline; an automatically generated string describing the options declared with mbfl_declare_option; a string describing the MBFL default options; the contents of the global variable script_EXAMPLES. Then exits with code zero. Required user defined variables.
-H
--brief-help
--brief-usage
Prints to the standard output of the script: the contents of the global variable script_USAGE; a newline; the string options:; a newline; an automatically generated string describing the options declared with mbfl_declare_option. Then exits with code zero.

The difference with --help is that predefined options and usage examples are not displayed.

--print-options
Print all the long options with mbfl_getopts_print_long_switches, then exit the script with code zero.
--print-action-arguments
Print all the action arguments with mbfl_getopts_print_action_arguments, then exit the script with code zero.

The following functions may be used to set, unset and query the state of the predefined options.

— Function: mbfl_option_encoded_args
— Function: mbfl_set_option_encoded_args
— Function: mbfl_unset_option_encoded_args

Query/set/unset the encoded arguments option.

mbfl_option_encoded_args returns true if the option --encoded-args was used on the command line.

— Function: mbfl_option_verbose
— Function: mbfl_set_option_verbose
— Function: mbfl_unset_option_verbose

Query/set/unset the verbose messages option.

mbfl_option_verbose returns true if the option --verbose was used on the command line after all the occurrences of --silent; it returns false if the option --silent was used on the command line after all the occurrences of --verbose.

— Function: mbfl_option_verbose_program
— Function: mbfl_set_option_verbose_program
— Function: mbfl_unset_option_verbose_program

Query/set/unset verbose execution for external programs.

This option, of course, is supported only for programs that are known by MBFL (like rm): if a program is executed with mbfl_program_exec, it is responsibility of the caller to use the option.

— Function: mbfl_option_show_program
— Function: mbfl_set_option_show_program
— Function: mbfl_unset_option_show_program

Print the command line of executed external program. This does not disable program execution, it just prints the command line before executing it.

— Function: mbfl_option_test
— Function: mbfl_set_option_test
— Function: mbfl_unset_option_test

Query/set/unset the test execution option.

— Function: mbfl_option_debug
— Function: mbfl_set_option_debug
— Function: mbfl_unset_option_debug

Query/set/unset the debug messages option.

— Function: mbfl_option_null
— Function: mbfl_set_option_null
— Function: mbfl_unset_option_null

Query/set/unset the null list separator option.

— Function: mbfl_option_interactive
— Function: mbfl_set_option_interactive
— Function: mbfl_unset_option_interactive

Query/set/unset the interactive execution option.

mbfl_option_interactive returns true if the option --interactive was used on the command line after all the occurrences of --force; it returns false if the option --force was used on the command line after all the occurrences of --interactive.

The following are special option functions.

— Function: mbfl_option_test_save

Save the current state of the test option then invokes mbfl_unset_option_test.

— Function: mbfl_option_test_restore

Restore the state of the test option to the one before the invocation to mbfl_option_test_save.


Previous: getopts options, Up: getopts

7.4 Interface functions

Every declared option should have a long switch, the brief switch may be omitted.

— Function: mbfl_declare_option keyword default brief long hasarg description

Declare a new option. Arguments description follows.

keyword
A string identifying the option; internally it is used to build a function name and a variable name. It is safer to limit this string to the letters in the ranges a-z, A-Z and underscores.
default
The default value for the option. For an option with argument it can be anything; for an option with no argument: it must be yes or no.
brief
The brief option selector: a single character. It is safer to choose a single letter (lower or upper case) in the ASCII standard.
long
The long option selector: a string. It is safer to choose a sequence of letters in the ASCII standard, separated by underscores or dashes.
hasarg
Either witharg or noarg: declares if the option requires an argument or not.
description
A one–line string briefly describing the option.

— Function: mbfl_declare_action_argument keyword string selected skipopts description

Declare an action argument. Arguments description follows.

keyword
A string identifying the argument; internally it is used to build a function name. It must be a valid Bash function identifier.
string
The string that identifies the argument on the command line of the string. It has limitations in format:
  • Its first character must be a letter in the range a-z or A-Z.
  • All the characters but the first must be: letters in the range a-z or A-Z; numbers in the range 0-9; dash characters -.

selected
A boolean value that can be yes or no. If yes the function associated to this action argument is selected by default as main action of the script. The function name is built by concatenating script_action_ with keyword.
skipopts
A boolean value that can be yes or no. If no: command line options are parsed as usual; if yes when this action argument is found as first parameter to the script, the rest of the arguments are not parsed, but copied as is to the ARGV and ARGC variables. This allows to hand options not recognised by the script to some external program.
description
A one–line string briefly describing the argument.

— Function: mbfl_getopts_parse

Parse a set of command line options. The options are handed to user defined functions. The global array ARGV1 and the global variable ARGC1 are supposed to hold the command line arguments and the number of command line arguments. Non–option arguments are left in the global array ARGV, the global variable ARGC holds the number of elements in ARGV.

If an action argument is recognised as the first element in ARGV1: it is processed but not added to ARGV and not counted in ARGC.

— Function: mbfl_getopts_islong string
— Function: mbfl_getopts_islong string varname

Verify if a string has the format of a long option without argument. string is the string to validate. The optional varname is the name of a variable that this function will set to the option name from string, without the leading dashes.

Return with code zero if the string is a long option without argument, else returns with code one.

An option must be of the form --option, only characters in the ranges A-Z, a-z, 0-9 and the characters - and _ are allowed in the option name.

Usage examples:

          mbfl_getopts_islong --option            ⇒ 0
          mbfl_getopts_islong --option=123        ⇒ 1
          mbfl_getopts_islong gasp                ⇒ 1
— Function: mbfl_getopts_islong_with string
— Function: mbfl_getopts_islong_with string optname varname

Verify if a string has the format of a long option with argument. Arguments:

string
The string to validate.
optname
Optional name of a variable that this function will set to the option name from string, without the leading dashes.
varname
Optional name of a variable that this function will set to the option value from string.

Return with code zero if the string is a long option with argument, else return with code one.

An option must be of the form --option=value, only characters in the ranges A-Z, a-z, 0-9 and the characters - and _ are allowed in the option name.

If the argument is not an option with value, the variable names are ignored.

Usage examples:

          mbfl_getopts_islong_with --option=one   ⇒ 0
          mbfl_getopts_islong_with --option       ⇒ 1
          mbfl_getopts_islong_with wappa          ⇒ 1
— Function: mbfl_getopts_isbrief string
— Function: mbfl_getopts_isbrief string varname

Verify if a string has the format of a brief option without argument. string is the string to validate. The optional varname is the name of a variable that this function will set to the option name from string, without the leading dash.

Return with code zero if the argument is a brief option without argument, else return with code one.

A brief option must be of the form -a, only characters in the ranges A-Z, a-z, 0-9 are allowed as option letters.

Usage examples:

          mbfl_getopts_isbrief -o         ⇒ 0
          mbfl_getopts_isbrief -o123      ⇒ 1
          mbfl_getopts_isbrief gasp       ⇒ 1
— Function: mbfl_getopts_isbrief_with string
— Function: mbfl_getopts_isbrief_with string optname valname

Verify if a string has the format of a brief option with argument. Arguments:

string
The string to validate.
optname
Optional name of a variable that this function will set to the option name from string, without the leading dashes.
valname
Optional name of a variable that this function will set to the option value.

Return with code zero if the argument is a brief option without argument, else return with code one.

A brief option must be of the form -aV (a is the option, V is the value), only characters in the ranges A-Z, a-z, 0-9 are allowed as option letters.

Usage examples:

          mbfl_getopts_isbrief_with -o123         ⇒ 0
          mbfl_getopts_isbrief_with -o            ⇒ 1
          mbfl_getopts_isbrief_with --option      ⇒ 1
          mbfl_getopts_isbrief_with wappa         ⇒ 1
— Function: mbfl_getopts_is_action_argument string

Verify if a string has the format of an action argument. string is the string to validate. Return with code zero if the argument is an action argument, else return with code one.

Usage examples:

          mbfl_getopts_action_argument alpha              ⇒ 0
          mbfl_getopts_action_argument do-this            ⇒ 0
          mbfl_getopts_action_argument -o                 ⇒ 1
          mbfl_getopts_action_argument -o123              ⇒ 1
          mbfl_getopts_action_argument --option           ⇒ 1
          mbfl_getopts_action_argument --option=one       ⇒ 1
— Function: mbfl_wrong_num_args required present

Validate the number of arguments. required is the required number of arguments, present is the given number of arguments on the command line. If the number of arguments is different from the required one: print an error message and return with code one; else return with code zero.

— Function: mbfl_wrong_num_args_range min_required max_required argc

Validate the number of arguments. argc must be between min_required and max_required, inclusive.

— Function: exit_because_wrong_num_args

Exit with code 98.

— Function: mbfl_argv_from_stdin

If the ARGC global variable is set to zero: fills the global variable ARGV with lines read from stdin. If the global variable mbfl_option_NULL is set to yes: lines are read using the null character as terminator, else they are read using the standard newline as terminator.

This function may block waiting for input.

— Function: mbfl_argv_all_files

Check that all the arguments in ARGV are file names of existent files. Return with code zero if no errors, else print an error message and return with code 1.

— Function: mbfl_getopts_print_long_switches

Print all the long switches in a row, separated by spaces. This is useful to retrieve the option for Bash programmable completion.

— Function: mbfl_getopts_print_action_arguments

Print all the action arguments strings in a row, separated by spaces. This is useful to retrieve the option for Bash programmable completion.


Next: , Previous: getopts, Up: Top

8 Printing messages to the console

This module allows us to print messages on an output channel. Various forms of message are supported. All the function names are prefixed with mbfl_message_. All the messages will have the forms:

     <progname>: <message>
     <progname>: [error|warning]: <message>
— Function: mbfl_message_set_progname PROGNAME

Set the script official name to put at the beginning of messages. This value is initialised to script_PROGNAME.

— Function: mbfl_message_set_channel channel

Select the channel to be used to output messages. This value is initialised to 2, which is stderr.

— Function: mbfl_message_string string

Output a message to the selected output channel. Echo a string composed of: the selected program name, a colon, a space, string. No newline character is appended to the message. Escape characters supported by printf are allowed in string.

— Function: mbfl_message_verbose string

Output a message to the selected output channel, but only if the evaluation of the function mbfl_option_verbose returns true.

Echo a string composed of: the selected program name, a colon, a space, string. No newline character is appended to the message. Escape characters supported by printf are allowed in string.

— Function: mbfl_message_verbose_end string

Output a message to the selected output channel, but only if the evaluation of the function mbfl_option_verbose returns true. Echo the string. No newline character is appended to the message. Escape characters supported by printf are allowed in string.

— Function: mbfl_message_debug string

Output a message to the selected output channel, but only if the evaluation of the function mbfl_option_debug returns true. Echo a string composed of: the selected program name, a colon, a space, string. No newline character is appended to the message. Escape characters supported by printf are allowed in string.

— Function: mbfl_message_debug_printf string
— Function: mbfl_message_debug_printf pattern arg ...

Format the arguments in the same way printf would do; output the resulting string to the selected output channel, but only if the evaluation of the function mbfl_option_debug returns true.

Echo a string composed of: the selected program name, a colon, a space, the string debug, a colon, a space, the formatting result. No newline character is appended to the message.

— Function: mbfl_message_verbose_printf string
— Function: mbfl_message_verbose_printf pattern arg ...

Format the arguments in the same way printf would do; output the resulting string to the selected output channel, but only if the evaluation of the function mbfl_option_verbose returns true.

Echo a string composed of: the selected program name, a colon, a space, the formatting result. No newline character is appended to the message.

— Function: mbfl_message_warning string

Output a warning message to the selected output channel. Echo a string composed of: the selected program name, a colon, a space, the string warning, a colon, a space, string, a newline character. Escape characters supported by printf are allowed in string.

— Function: mbfl_message_warning_printf string
— Function: mbfl_message_warning_printf pattern arg ...

Format the arguments in the same way printf would do; output the resulting string to the selected output channel.

Echo a string composed of: the selected program name, a colon, a space, the string warning, a colon, a space, the formatting result. No newline character is appended to the message.

— Function: mbfl_message_error string

Output an error message to the selected output channel. Echo a string composed of: the selected program name, a colon, a space, the string error, a colon, a space, string, a newline character. Escape characters supported by printf are allowed in string.

— Function: mbfl_message_error_printf string
— Function: mbfl_message_error_printf pattern arg ...

Format the arguments in the same way printf would do; output the resulting string to the selected output channel.

Echo a string composed of: the selected program name, a colon, a space, the string error, a colon, a space, the formatting result. No newline character is appended to the message.


Next: , Previous: message, Up: Top

9 Using external programs


Next: , Up: program

9.1 Testing a script and running programs

MBFL allows a script to execute a “dry run”, that is: do not perform any operation on the system, just print messages describing what will happen if the script is executed with the selected options. This implies, in the MBFL model, that no external program is executed.

When this feature is turned on: mbfl_program_exec does not execute the program, instead it prints the command line on standard error and it returns true.

— Function: mbfl_set_option_test

Enable the script test option. After this: a script must not mutate the system in any way, it should just print messages describing the operations.

However, the script is allowed to acquire informations from the system; for example it can acquire the list of files in a directory or load the contents of a file.

This function is invoked when the predefined option --test is used on the command line.

— Function: mbfl_unset_option_test

Disable the script test option. After this a script must perform normal operations.

— Function: mbfl_option_test

Return true if test execution is enabled, else return false.


Next: , Previous: program testing, Up: program

9.2 Checking programs existence

The simpler way to test the availability of a program is to look for it just before it is used.

— Function: mbfl_program_find program

A wrapper for:

          type -ap program

that looks for a program in the current search path. It prints the full pathname of the program found, or an empty string if nothing is found.

program may be a program name with no directory part (examples: sed, grep), or an absolute or relative pathname (examples: /bin/sed, ../bin/grep). If program is a relative pathname and it is an executable file: the output of the function is exactly program, it is not normalised.

If an alias exists for a program, type -p program will return the empty string; that is why we have to use type -ap program, which will return the correct file pathname.

If the environment variable PATH holds the same directory more than once, or there exist programs with the same name in more than one PATH elements: type -ap $program will return more than one line of output, one for each executable file found. The first value (the first line) is the one selected by this function.

— Function: mbfl_program_check program ?program ...?

The use of this function is deprecated.

Check the availability of programs. All the pathnames on the command line are checked: if one is not executable an error message is printed on stderr. Return false if a program cannot be found, true otherwise.


Next: , Previous: program checking, Up: program

9.3 Executing a program

This module provides an API to execute a program under the privileges of the current user or under a more or less privileged user; it makes use of sudo, to allow one to execute a program as a different user (optionally without entering a password): refer to the sudo documentation for the required configuration.

The functions described here must be used in the following way:

Every time we execute a program with sudo: we have to select the user under which to execute it; if we do not do it: the internally registered user defaults to nosudo, which tells the function not to use sudo. So the following script works as commented:

     mbfl_program_enable_sudo
     
     # This is executed with the privileges of the user that
     # launched the script.
     mbfl_program_exec ls /bin
     
     mbfl_program_declare_sudo_user root
     # This is executed with root privileges.
     mbfl_program_exec ls /root
     
     # This is executed with the privileges of the user that
     # launched the script.
     mbfl_program_exec ls /bin
— Function: mbfl_program_exec program
— Function: mbfl_program_exec program arg ...

Evaluate a command line. program identifies an executable file: it can be the program name, or a relative or absolute pathname. The optional arg values are command line arguments that are handed to the program unchanged.

If usage of sudo was requested, the command is executed with it; then the sudo request is reset. This means that this function “consumes” a sudo request.

See below for the redirection of the standard error channel.

If the function mbfl_option_test returns true: instead of evaluation, the command line is sent to stderr.

If the function mbfl_option_show_program returns true: the command line is sent to stderr, then it is executed.

— Function: mbfl_program_execbg inchan ouchan program
— Function: mbfl_program_execbg inchan ouchan program arg ...

Does all the same things of mbfl_program_exec, running the given command line as:

          program arg ... <inchan >ouchan &

additionally: set the global variable mbfl_program_BGPID to the process id of the background process; that is: mbfl_program_BGPID is the value of $! right after the process execution.

Using this function is different from calling:

          mbfl_program_exec ls <inchan >ouchan &

because doing so puts in the background the function call (in a subshell) and then runs the program.

— Variable: mbfl_program_BGPID

Used by mbfl_program_execbg to store the process id of the program executed in background.

— Function: mbfl_program_enable_sudo

Declare the intention to use sudo and other commands required to use it. Declaring the intention to use a program.

The declared programs are: sudo, whoami.

— Function: mbfl_program_declare_sudo_user user

Register user as the user under which to execute the next program through sudo; the user will be selected using the -u option of sudo. The value nosudo means: do not use sudo.

When the time comes: if the selected user name equals the value printed by whoami, sudo is not used.

— Function: mbfl_program_reset_sudo_user

Reset the previously requested sudo user to a value that will cause sudo not to be used in the next program invocation. This is useful to abort a user request.

— Function: mbfl_program_sudo_user

Print the current sudo user.

— Function: mbfl_program_requested_sudo

Return true if the usage of sudo has been requested for the next command execution.

Executing a subshell
— Function: mbfl_program_bash arg ...

Execute bash with the arg arguments appended. The bash pathname is registered in the library at start up, from the built in variable BASH.

— Function: mbfl_program_bash_command command

Execute command in a bash subprocess, using the -c switch. The bash pathname is registered in the library at start up, from the built in variable BASH.

Redirecting the standard error channel

There are programs that output useful informations on their stderr channel (example: the at command).

— Function: mbfl_program_redirect_stderr_to_stdout

Just for the next invocation to mbfl_program_exec redirect stderr to stdout, that is: use the 2>&1 redirection for the executed program.

This is useful because redirecting the output of mbfl_program_exec:

     echo ls | \
         mbfl_program_exec at 'now +25 minutes' 2>&1 | \
         while read

redirects also the “show program” output (getopts options for the --show-program option explanation and see the above description of mbfl_program_exec). Instead By using:

     mbfl_program_redirect_stderr_to_stdout
     echo ls | \
         mbfl_program_exec at 'now +25 minutes' | \
         while read

the “show program” output goes to stderr and the stderr output of the at command is, internally, redirected to the stdout of mbfl_program_exec.


Previous: program executing, Up: program

9.4 Declaring the intention to use a program

To make a script model simpler, we assume that the unavailability of a program at the time of its execution is a fatal error. So if we need to execute a program and the executable is not there, the script must be aborted on the spot.

Functions are available to test the availability of a program, so we can try to locate an alternative or terminate the process under the script control. On a system where executables may vanish from one moment to another, no matter how we test a program's existence, there's always the possibility that the program is not “there” when we invoke it.

If we just use mbfl_program_exec to invoke an external program, the function will try and fail if the executable is unavailable: the return code will be false.

The vanishing of a program is a rare event: if it's there when we look for it, probably it will be there also a few moments later when we invoke it. For this reason, MBFL proposes a set of functions with which we can declare the intention of a script to use a set of programs.

A command line option is predefined to let the user test the availability of all the declared programs before invoking the script. Predefined options.

— Function: mbfl_declare_program program

Register program as the name of a program required by the script; mbfl_program_find is used to locate the program on the system. Checking programs existence.

If program is a file name with no directory part (examples: sed, grep) the selected program is the full pathname of the file in one of the directories of PATH.

If program is a relative pathname (examples: ../bin/sed, ./grep): the selected program is the full pathname of the file normalised by this function with respect to the current working directory (with a call to mbfl_file_normalise).

The return value is always zero.

— Function: mbfl_program_validate_declared

Validate the existence of all the declared programs. The return value is zero if all the programs are found, one otherwise.

This function is invoked by mbfl_getopts_parse when the --validate-programs option is used on the command line.

It may be a good idea to invoke this function at the beginning of a script, just before starting to do stuff, example:

          mbfl_program_validate_declared || \
             exit_because_program_not_found

If verbose messages are enabled: a brief summary is echoed to stderr; from the command line the option --verbose must be used before --validate-programs.

— Function: mbfl_program_found program

Print the pathname of the previously declared program. Return zero if the program was found, otherwise print an error message and exit the current (sub)shell by invoking exit_because_program_not_found.

This function should be used to retrieve the pathname of the program to be used as first argument to mbfl_program_exec:

          function program_wrapper () {
              local ARGUMENT PROGNAME FLAGS
          
              ARGUMENT=${1:?"missing 'ARGUMENT'"}
              shift
              PROGNAME=$(mbfl_program_found myprog) || exit $?
          
              mbfl_option_verbose_program && \
                 FLAGS="$FLAGS --verbose"
              mbfl_program_exec "$PROGNAME" \
                 $FLAGS "$ARGUMENT" "$@"
          }

Remember that we cannot use:

          local PROGNAME=$(mbfl_program_found 'myprog') || \
              exit $?

because local will return with code zero even if mbfl_program_found fails, so the error will not be reported.

— Function: exit_because_program_not_found

Terminate the script with exit code 99.


Next: , Previous: program, Up: Top

10 Interfaces to external programs


Up: interfaces

10.1 Scheduling jobs for later execution

This section documents the interface to the atd daemon; we may want to read the at(1) manual page. The at service allows a user to schedule commands to be executed at a later time.

This interface is suitable for scripts that define a unique simple policy to schedule commands; example: at each run they schedule a command in a fixed queue, to be executed at a fixed time in the future.

This is good to implement the logic: if a condition does not happen before time T, then execute command C.

Commands declaration
— Function: mbfl_at_enable

Declare the intention to use the at interface. Declaring the intention to use a program.

The declared programs are: at, atq, atrm.

Arguments validation
— Function: mbfl_at_validate_queue_letter letter

Return true if letter is a valid queue identifier, else return false.

— Function: mbfl_at_validate_selected_queue

Return true if the currently selected queue identifier is valid, else print an error message and return false. A false return code means that an internal error has corrupted the module state.

Commands scheduling
— Function: mbfl_at_select_queue letter

Select and register in an internal state a queue identifier; invoke mbfl_at_validate_queue_letter to validate the selection.

— Function: mbfl_at_schedule command time

Schedule command in the currently selected queue; the script will be executed at time.

If no error occurs: print to stdout the identifier of the scheduled job; the identifier can be used as argument to mbfl_at_drop.

The at command outputs some text (in which the job is embedded) on its stderr channel, so this function redirects stderr to stdout to return the value; this operation conflicts with the use of the “show program” feature of mbfl_program_exec.

command must be a string representing the invocation of an external executable program: it is sent unchanged to the at command. time is the argument to the at command, see the manual page for its description.

— Function: mbfl_at_drop identifier

Remove a job; the identifier of a job is unique in all the queues, so this function is not affected by the currently selected queue.

— Function: mbfl_at_queue_clean

Remove all the jobs from the currently selected queue.

Inspection
— Function: mbfl_at_queue_print_identifiers

Print all the job identifiers in the currently selected queue.

— Function: mbfl_at_queue_print_jobs

Print all the job descriptions in the currently selected queue.

— Function: mbfl_at_queue_print_queues

Print the letters identifying queues with pending jobs.

— Function: mbfl_at_print_queue

Print the currently selected queue letter.


Next: , Previous: interfaces, Up: Top

11 Catching signals

MBFL provides an interface to the trap builtin that allows the execution of more than one function when a signal is received; this may sound useless, but that is it.

— Function: mbfl_signal_map_signame_to_signum sigspec

Convert sigspec to the corresponding signal number, then print the number.

— Function: mbfl_signal_attach sigspec handler

Append handler to the list of functions that are executed whenever sigspec is received.

— Function: mbfl_signal_invoke_handlers signum

Invoke all the handlers registered for signum. This function is not meant to be used during normal scripts execution, but it may be useful to debug a script.


Next: , Previous: signal, Up: Top

12 Manipulating strings


Next: , Up: string

12.1 Quoted characters

— Function: mbfl_string_is_quoted_char string position

Return true if the character at position in string is quoted; else return false. A character is considered quoted if it is preceded by an odd number of backslashes (\). position is a zero–based index.

— Function: mbfl_string_is_equal_unquoted_char string position char

Return true if the character at position in string is equal to char and is not quoted (according to mbfl_string_is_quoted_char); else return false. position is a zero–based index.

— Function: mbfl_string_quote string

Print string with quoted characters. All the occurrences of the backslash character, \, are substituted with a quoted backslash, \\. Return true.


Next: , Previous: string quote, Up: string

12.2 Inspecting a string

— Function: mbfl_string_index string index

Select a character from a string. Echo to stdout the selected character. If the index is out of range: the empty string is echoed to stdout, that is: a newline is echoed to stdout.

— Function: mbfl_string_first string char
— Function: mbfl_string_first string char begin

Search characters in a string. Arguments: string, the target string; char, the character to look for; begin, optional, the index of the character in the target string from which the search begins (defaults to zero).

Print an integer representing the index of the first occurrence of char in string. If the character is not found: nothing is sent to stdout.

— Function: mbfl_string_last string char
— Function: mbfl_string_last string char begin

Search characters in a string starting from the end. Arguments: string, the target string; char, the character to look for; begin, optional, the index of the character in the target string from which the search begins (defaults to zero).

Print an integer representing the index of the last occurrence of char in string. If the character is not found: nothing is sent to stdout.

— Function: mbfl_string_range string begin end

Extract a range of characters from a string. Arguments: string, the source string; begin, the index of the first character in the range; end, optional, the index of the character next to the last in the range, this character is not extracted. end defaults to the last character in the string; if equal to end: the end of the range is the end of the string. Echo to stdout the selected range of characters.

— Function: mbfl_string_equal_substring string position pattern

Return true if the substring starting at position in string is equal to pattern; else return false. If position plus the length of pattern is greater than the length of string: the return value is false, always.


Next: , Previous: string inspection, Up: string

12.3 Splitting a string

— Function: mbfl_string_chars string

Split a string into characters. Fill an array named SPLITFIELD with the characters from the string; the number of elements in the array is stored in a variable named SPLITCOUNT. Both SPLITFIELD and SPLITCOUNT can be declared local in the scope of the caller.

The difference between this function and using ${STRING:$i:1}, is that this function detects backslash characters, \, and treats them as part of the following character. So, for example, the sequence \n is treated as a single char.

Example of usage for mbfl_string_chars:

     string="abcde\nfghilm"
     mbfl_string_chars "${string}"
     # Now:
     # "${#string}" = $SPLITCOUNT
     #  a = "${SPLITFIELD[0]}"
     #  b = "${SPLITFIELD[1]}"
     #  c = "${SPLITFIELD[2]}"
     #  d = "${SPLITFIELD[3]}"
     #  e = "${SPLITFIELD[4]}"
     #  \n = "${SPLITFIELD[5]}"
     #  f = "${SPLITFIELD[6]}"
     #  g = "${SPLITFIELD[7]}"
     #  h = "${SPLITFIELD[8]}"
     #  i = "${SPLITFIELD[9]}"
     #  l = "${SPLITFIELD[10]}"
     #  m = "${SPLITFIELD[11]}"
— Function: mbfl_string_split string separator

Split string into fields using separator. Fill an array named SPLITFIELD with the characters from the string; the number of elements in the array is stored in a variable named SPLITCOUNT. Both SPLITFIELD and SPLITCOUNT can be declared local in the scope of the caller.


Next: , Previous: string splitting, Up: string

12.4 Converting between upper and lower case

— Function: mbfl_string_toupper string

Output string with all the occurrences of lower case ASCII characters (no accents) turned into upper case.

— Function: mbfl_string_tolower string

Output string with all the occurrences of upper case ASCII characters (no accents) turned into lower case.


Next: , Previous: string case, Up: string

12.5 Matching a string with a class

— Function: mbfl_string_is_alpha_char char

Return true if char is in one of the ranges: a-z, A-Z.

— Function: mbfl_string_is_digit_char char

Return true if char is in the range: 0-9.

— Function: mbfl_string_is_alnum_char char

Return true if:

          mbfl_string_is_alpha_char char || \
             mbfl_string_is_digit_char char
— Function: mbfl_string_is_noblank_char char

Return true if char is none of the characters: , \n, \r, \f, \t. char is meant to be the unquoted version of the non–blank characters, the one obtained with:

          $'char'
— Function: mbfl_string_is_name_char char

Return true if mbfl_string_is_alnum_char returns true when applied to char or char is an underscore, _.

— Function: mbfl_string_is_alpha string
— Function: mbfl_string_is_digit string
— Function: mbfl_string_is_alnum string
— Function: mbfl_string_is_noblank string
— Function: mbfl_string_is_name string

Return true if the associated char function returns true for each character in string. As additional constraint: mbfl_string_is_name returns false if mbfl_string_is_digit returns true when applied to the first character of string.


Previous: string class, Up: string

12.6 Miscellaneous functions

— Function: mbfl_string_replace string pattern
— Function: mbfl_string_replace string pattern subst

Replace all the occurrences of pattern in string with subst, then print the result. If not used, subst defaults to the empty string.

— Function: mbfl_sprintf varname format ...

Make use of printf to format the string format with the additional arguments, then store the result in varname: If this name is local in the scope of the caller, this has the effect of filling the variable in that scope.

— Function: mbfl_string_skip string varname char

Skip all the characters in a string equal to char. varname is the name of a variable in the scope of the caller: Its value is the offset of the first character to test in string. The offset is incremented until a char different from char is found, then the value of varname is updated to the position of the different char. If the initial value of the offset corresponds to a char equal to char, the variable is left untouched. Return true.


Next: , Previous: string, Up: Top

13 Interacting with the user

— Function: mbfl_dialog_yes_or_no string
— Function: mbfl_dialog_yes_or_no string progname

Print the question string on the standard output and wait for the user to type yes or no in the standard input. Return true if the user has typed yes, false if the user has typed no.

The optional parameter progname is used as prefix for the prompt; if not given: It defaults to the value of script_PROGNAME. Required user defined variables.

— Function: mbfl_dialog_enable_programs

Declare the usage of the external program stty, which is used by mbfl_dialog_ask_password to turn of password echoing on the terminal.

— Function: mbfl_dialog_ask_password prompt

Print prompt followed by a colon and a space, then reads a password from the terminal. Print the password.


Next: , Previous: dialog, Up: Top

14 Manipulating variables


Next: , Up: variables

14.1 Manipulating arrays

Manipulating colon variables, for the use of the following functions.

— Function: mbfl_variable_find_in_array element

Search the array mbfl_FIELDS for a value equal to element. If it is found: Print the index and return true; else print nothing and return false.

mbfl_FIELDS must be filled with elements having subsequent indexes starting at zero.

— Function: mbfl_variable_element_is_in_array element

A wrapper for mbfl_variable_find_in_array that does not print anything.


Previous: variables arrays, Up: variables

14.2 Manipulating colon variables

— Function: mbfl_variable_colon_variable_to_array varname

Take varname's value, a colon separated list of string, and store each string in the array mbfl_FIELDS, starting with a base index of zero.

Example:

          VAR=a:b:c:d:e
          declare -a mbfl_FIELDS
          
          mbfl_variable_colon_variable_to_array VAR
          
          echo ${#mbfl_FIELDS[*]}       -| 5
          echo "${mbfl_FIELDS[0]}"      -| a
          echo "${mbfl_FIELDS[1]}"      -| b
          echo "${mbfl_FIELDS[2]}"      -| c
          echo "${mbfl_FIELDS[3]}"      -| d
          echo "${mbfl_FIELDS[4]}"      -| e
— Function: mbfl_variable_array_to_colon_variable varname

Store each value from the array mbfl_FIELDS (with base index zero) in varname as a colon separated list of strings.

Example:

          declare -a mbfl_FIELDS=(a b c d e)
          
          mbfl_variable_array_to_colon_variable VAR
          echo $VAR                       -| a:b:c:d:e
— Function: mbfl_variable_colon_variable_drop_duplicate varname

Take varname's value, a colon separated list of string, and remove duplicates. Reset varname to the result.


Next: , Previous: variables, Up: Top

15 Interfacing with the system

— Function: mbfl_system_enable_programs

Declare the intention to use the programs required by this module. Declaring the intention to use a program.

Required programs are: grep, cut.


Next: , Up: system

15.1 Converting user identifiers

— Function: mbfl_system_numerical_user_id_to_name id

Convert the numerical user id to the user name found in the /etc/passwd file.

— Function: mbfl_system_numerical_user_id_to_name name

Convert the symbolic user name to the numerical identifier found in the /etc/passwd file.


Previous: system user id, Up: system

15.2 Converting file permissions

— Function: mbfl_system_symbolic_to_octal_permissions mode

Convert three chars representing file permissions in a single octal digit.

— Function: mbfl_system_octal_to_symbolic_permissions mode

Convert a single octal digit representing file permissions into three chars.


Next: , Previous: system, Up: Top

16 Main function

MBFL declares a function to drive the execution of the script; its purpose is to make use of the other modules to reduce the size of scripts depending on MBFL. All the code blocks in the script, with the exception of global variables declaration, should be enclosed in functions.


Next: , Up: main

16.1 Driving script execution

— Function: mbfl_main

The invocation to this function must be the last line of code in the script. It does the following:

  1. Register the value of the variable script_PROGNAME in the message module using the function mbfl_message_set_progname.
  2. Invoke mbfl_main_create_exit_aliases. Declaring exit codes.
  3. If it exists: Invoke the function script_before_parsing_options.
  4. Parse command line options with mbfl_getopts_parse.
  5. If it exists: Invoke the function script_after_parsing_options.
  6. Invoke the function whose name is stored in the global variable mbfl_main_SCRIPT_FUNCTION, if it exists, with no arguments; if its return value is non–zero: Exit the script with the same code. The default value is main.
  7. Exit the script with the return code of the action function or zero.

— Function: mbfl_invoke_script_function funcname

If funcname is the name of an existing function: It is invoked with no arguments; the return value is the one of the function. The existence test is performed with:

          type -t FUNCNAME = function
— Function: mbfl_main_set_main funcname

Select the main function storing funcname into mbfl_main_SCRIPT_FUNCTION.

— Variable: mbfl_main_SCRIPT_FUNCTION

Global variable that holds the name of the custom main script function.


Previous: main function, Up: main

16.2 Declaring exit codes

Some functions and global variables are provided to declare script's exit codes.

— Function: mbfl_main_declare_exit_code code name

Declare an exit code with value code and identifier name.

— Function: mbfl_main_create_exit_functions

For each of the codes declared with mbfl_main_declare_exit_code: Create a function for the exit command using the numerical code. For example, if a code is declared as:

          mbfl_main_declare_exit_code 4 unexistent_file

a function is created with:

          function exit_because_unexistent_file () { exit 4; }

the name of the function is the string exit_because_ followed by the exit code name. The function may be used in the script to exit the process.

By default the exit code 0 is associated to the name success and the exit code 1 is associated to the name failure; so the following functions exist.

— Function: exit_because_success
— Function: exit_success

Exit the script with code zero.

— Function: exit_because_failure
— Function: exit_failure

Exit the script with code one.

— Function: mbfl_main_print_exit_code name

Print the code associated to name.


Next: , Previous: main, Up: Top

17 Building test suites

MBFL comes with a little library of functions that may be used to build test suites; its aim is at building tests for Bash functions, commands and scripts.

The ideas at the base of this library are taken from the tcltest package distributed with the TCL core 1; this package had contributions from the following people/entities: Sun Microsystems, Inc.; Scriptics Corporation; Ajuba Solutions; Don Porter, NIST; probably many many others.

The library tries to do as much as possible using functions and aliases, not variables; this is an attempt to let the user redefine functions to his taste.


Next: , Up: testing

17.1 A way to organise a test suite

A useful way to organise a test suite is to split it into a set of files: one for each module to be tested.

The file libmbfltest.sh must be sourced at the beginning of each test file. This means that the variables that you set may interfere with the ones in the library; this should not happen because the test library prefixes variable names with mbfl_ or dotest_, but one exception is TMPDIR: do not set it in your script, use dotest-echo-tmpdir to access that value. Handling files in tests.

A not so automated example

To understand how the library works lets examine a bare bones example.

The function dotest should be invoked at the end of each module in the test suite; each module should define functions starting with the same prefix. A module should be stored in a file, and should look like the following:

     # mymodule.test --
     
     source libmbfltest.sh
     source module.sh
     
     function module-featureA-1.1 () { ... }
     function module-featureA-1.2 () { ... }
     function module-featureA-2.1 () { ... }
     function module-featureB-1.1 () { ... }
     function module-featureB-1.2 () { ... }
     
     dotest module-
     
     ### end of file

the file should be executed with:

     $ bash mymodule.test

To test just "feature A":

     $ TESTMATCH=module-featureA bash mymodule.test

Remember that the source builtin will look for files in the directories selected by the PATH environment variables, so we may want to do:

     $ PATH=path/to/modules:${PATH} \
     TESTMATCH=module-featureA bash mymodule.test

It is better to put such stuff in a Makefile, with GNU Make:

     srcdir        = ...
     builddir      = ...
     BASH_PROGRAM  = bash
     MODULES       = moduleA moduleB
     
     testdir       = $(srcdir)/tests
     test_FILES    = $(foreach f,$(MODULES),$(testdir)/$(f).test)
     
     test_ENV      = PATH=$(builddir):$(testdir):$(PATH) \
                     TESTMATCH=$(TESTMATCH)
     test_CMD      = $(test_ENV) $(BASH_PROGRAM)
     
     .PHONY: test-modules
     
     test-modules:
             @$(foreach f,$(test_FILES),$(test_CMD) $(f);)


Next: , Previous: testing intro, Up: testing

17.2 A script to run tests

MBFL comes with a script that can be used to handle the execution of tests; it is called mbfltest.sh. Synopsis:

     mbfltest.sh [options] TESTFILE ...

supported options are all the MBFL generic ones (Predefined options) and additionally:

--start
Print start messages for tests, it is the same as invoking dotest-set-report-start.
--end
Print end messages for tests, it is the same as invoking dotest-set-report-success.
--match=VALUE
Select match pattern for tests, it is the same as setting the TESTMATCH variable.
--directory=VALUE
Change directory before executing tests, but after having located the test files on the file system.
--library=VALUE
Select a specific version of the MBFL library.

When running tests with the script: in the test modules we can omit the sourcing of MBFL and the MBFL test library, mbfltest.sh does this before sourcing the test module. Each test module is evaluated in a bash subprocess, so: there is no interference between modules; each module has to do its own initialisation and finalisation.

With GNU Make we can do:

     MBFLTEST        = mbfltest.sh
     MBFLTEST_FLAGS  = --end
     ifneq (,$(TESTMATCH))
     MBFLTEST_FLAGS  += --match=$(TESTMATCH)
     endif
     
     srcdir          = ...
     testdir         = $(srcdir)/tests
     TESTNAME        = *
     TESTFILES       = $(wildcard $(testdir)/$(TESTNAME).test)
     
     .PHONY: test tests
     
     ifneq ($(strip $(TESTFILES)),)
     test tests:
             $(MBFLTEST) $(MBFLTEST_FLAGS) $(TESTFILES)
     endif


Next: , Previous: testing shell, Up: testing

17.3 Configuring the package

— Function: dotest-set-verbose
— Function: dotest-unset-verbose

Set or unset verbose execution. If verbose mode is on: some commands output messages on stderr describing what is going on. Examples: files and directories creation/removal.

— Function: dotest-option-verbose

Return true if verbose mode is on, false otherwise.

— Function: dotest-set-test
— Function: dotest-unset-test

Set or unset test execution. If test mode is on: external commands (like rm and mkdir) are not executed, the command line is sent to stderr. Test mode is meant to be used to debug the test library functions.

— Function: dotest-option-test

Return true if test mode is on, false otherwise.

— Function: dotest-set-report-start
— Function: dotest-unset-report-start

Set or unset printing a message upon starting a function.

— Function: dotest-option-report-start

Return true if start function reporting is on; otherwise return false.

— Function: dotest-set-report-success
— Function: dotest-unset-report-success

Set or unset printing a message when a function execution succeeds. Failed tests always cause a message to be printed.

— Function: dotest-option-report-success

Return true if success function reporting is on; otherwise return false.


Next: , Previous: testing config, Up: testing

17.4 Running test functions

— Function: dotest pattern

Run all the functions matching pattern. Usually pattern is the first part of the name of the functions to be executed; the function names are selected with the following code:

          compgen -A function pattern

There's no constraint on function names, but they must be one–word names.

Before running a test function: the current process working directory is saved, and it is restored after the execution is terminated.

The return value of the test functions is used as result of the test: true, the test succeeded; false, the test failed. Remembering that the return value of a function is the return value of its last executed command, the functions dotest-equal and dotest-output, and of course the test command, may be used to return the correct value.

Messages are printed before and after the execution of each function, according to the mode selected with: dotest-set-report-success, dotest-set-report-start, ... Configuring the package

The following environment variables will influence the behaviour of dotest.

— Variable: TESTMATCH

Override the pattern argument to dotest.

— Variable: TESTSTART

If yes: It is equivalent to invoking dotest-set-report-start. If no: It is equivalent to invoking dotest-unset-report-start.

— Variable: TESTSUCCESS

If yes: It is equivalent to invoking dotest-set-report-success. If no: It is equivalent to invoking dotest-unset-report-success.


Next: , Previous: testing running, Up: testing

17.5 Validating results by comparing

— Function: dotest-equal expected got

Compare the two parameters and return true if they are equal; return false otherwise. In the latter case print a message showing the expected value and the wrong one. Must be used as last command in a function, so that its return value is equal to that of the function.

Example:

     function my-func () {
         echo $(($1 + $2))
     }
     function mytest-1.1 () {
         dotest-result 5 `my-func 2 3`
     }
     dotest mytest-

another example:

     function my-func () {
         echo $(($1 + $2))
     }
     function mytest-1.1 () {
         dotest-result 5 `my-func 2 3` && \
           dotest-result 5 `my-func 1 4` && \
           dotest-result 5 `my-func 3 2` && \
     }
     dotest mytest-


Next: , Previous: testing compare, Up: testing

17.6 Validating results by output

— Function: dotest-output
— Function: Function dotest-output string

Read all the available lines from stdin accumulating them into a local variable, separated by \n; then compare the input with string, or the empty string if string is not present, and return true if they are equal, false otherwise.

Example of test for a function that echoes its three parameters:

     function my-lib-function () {
         echo $1 $2 $3
     }
     function mytest-1.1 () {
         my-lib-function a b c | dotest-output "a b c"
     }
     dotest mytest

Example of test for a function that is supposed to print nothing:

     function my-lib-function () {
         test "$1" != "$2" && echo error
     }
     function mytest-1.1 () {
         my-lib-function a a | dotest-output
     }
     dotest mytest


Next: , Previous: testing output, Up: testing

17.7 Printing messages from test functions

— Alias: dotest-echo string ?...?
— Alias: dotest-debug string ?...?

Print the parameters on stderr. dotest-debug prints some * to make the message more visible.


Previous: testing messages, Up: testing

17.8 Handling files in tests

In this section are described functions to be used to create temporary files; it is a common task to write scripts to manipulate files and directories. All the files should be created under a temporary directory that must be removed after each test function is invoked; the library automatically invokes dotest-clean-files when exiting (using trap), but it is safer to invoke it at the end of each function that creates files.


Next: , Up: testing files

17.8.1 Directories

— Function: dotest-cd directory

Change the working directory. This is just a wrapper for cd; if verbose mode is on: print a message.

— Function: dotest-mkdir directory
— Function: dotest-mkdir directory prefix

Create directory under the temporary directory; directory must be a relative pathname (that is: it must not begin with a slash).

The optional prefix is a relative pathname that is prepended to directory: it is useful to prepend the name of a parent directory.

Print to stdout the full pathname of the directory.

Temporary directory
— Function: dotest-echo-tmpdir

Print the value of the temporary directory in which all the files and directories will be created. The value is prefixed with the value of the environment variable TMPDIR, or /tmp if not set.

— Function: dotest-cd-tmpdir

Change the working directory to the temporary directory.

— Function: dotest-mktmpdir

Create the temporary directory. This is automatically invoked by dotest-mkfile before creating files; dotest-mkdir creates the temporary directory automatically by using the --parents option of mkdir.


Next: , Previous: testing files directories, Up: testing files

17.8.2 Files

— Function: dotest-mkfile pathname
— Function: dotest-mkfile pathname prefix

Create an empty file. The optional prefix is a relative pathname that is prepended to pathname: It is useful to prepend the name of a parent directory. Print to stdout the full pathname of the file.

— Function: dotest-clean-files

Remove the temporary directory and all its children. Should be invoked at the end of each function that creates temporary files or directories.

Return the value of the last command executed before the invocation, that way it can be used right after dotest-output and dotest-equal without loosing the return value of the function.

Testing conditions
— Function: dotest-assert-file-exists file error_message

Test that file exists: If true returns with code zero; else print error_message, invoke dotest-clean-files and return with code one.

— Function: dotest-assert-file-unexists file error_message

Test that file does not exist: If true return with code zero; else print error_message, invoke dotest-clean-files and return with code one.


Previous: testing files files, Up: testing files

17.8.3 Examples

Examples of usage of dotest-clean-files:

     function mytest-1.1 () {
         local dir=$(dotest-mkdir a/b)
         local result=
     
         ...
         result=...
         dotest-equal 123 $result
         dotest-clean-files
     }
     function mytest-1.2 () {
         local dir=$(dotest-mkfile file.ext)
         local result=
     
         ...
         result=...
         dotest-equal 123 $result
         dotest-clean-files
     }
     
     dotest mytest-


Next: , Previous: testing, Up: Top

Appendix A Examples for sending email

In this appendix we review some example scripts that can be used to send email from a Bash script. All the scripts are in the MBFL distribution under the examples directory.

First we examine plain scripts (making no use of MBFL) to understand the basics of how to handle the SMTP protocol and how to “talk” to a process in background.

Then we see the documentation of a complex script, sendmail-mbfl.sh, which can be used to send mail with plain or encrypted connections.

Finally we see how to use the GNU Emacs interface to the script, sendmail-mbfl.el. sendmail script emacs.


Next: , Up: sendmail

A.1 How to compose a test email message

Here we discuss how to programmatically compose a minimal email message to be used in testing email scripts. Basically a message should look like this:

     Sender: marco@localhost
     From: marco@localhost
     To: root@localhost
     Subject: proof from sendmail-plain.sh
     Message-ID: <15704-6692-23464@this.hostname>
     Date: Tue, 28 Apr 2009 06:16:01 +0200
     
     This is a text proof from the sendmail-plain.sh script.
     --
     Marco

We have to remember that the SMTP server receiving the message may rewrite the addresses, for example: replacing localhost with the fully qualified local host name (the output of the command hostname --fqdn); so, when reading the delivered message, we do not have to be surprised to find changed addresses.

We want to notice the following:

In the end, we can use the following chunk of code to compose an email message:

     PROGNAME=${0##*/}
     FROM_ADDRESS=marco@localhost
     TO_ADDRESS=root@localhost
     
     function print_message () {
         local LOCAL_HOSTNAME DATE MESSAGE_ID MESSAGE
         LOCAL_HOSTNAME=$(hostname --fqdn) || exit 2
         DATE=$(date --rfc-2822) || exit 2
         MESSAGE_ID=$(printf '%d-%d-%d@%s' \
             $RANDOM $RANDOM $RANDOM "$LOCAL_HOSTNAME")
         MESSAGE="Sender: $FROM_ADDRESS
     From: $FROM_ADDRESS
     To: $TO_ADDRESS
     Subject: proof from $PROGNAME
     Message-ID: <$MESSAGE_ID>
     Date: $DATE
     
     This is a text proof from the $PROGNAME script.
     --\x20
     Marco
     "
         printf "$MESSAGE"
     }

notice that to put the required single white space character in the text/signature separator we use the escape sequence \x20 (where 20 is the hexadecimal value of the white space character in the ASCII encoding) and print the message with printf, which expands the escape sequences.

When sending the message to the SMTP server we have to:

So we can use an equivalent of the following chunk of code, assuming 3 is the file descriptor connected to the remote SMTP server:

     print_message | while IFS= read line
     do
         if test "${line:0:1}" = '.'
         then printf '.%s\r\n' "$line" >&3
         else printf  '%s\r\n' "$line" >&3
         fi
     done

notice that read is executed in an environment in which IFS is set to the empty string, this is to prevent unwanted modification of the message text. read splits the string it reads into words according to the current value of IFS, and this may lead to mutation of the input string; word splitting happens when there is a single output variable, too. To prevent word splitting, we set IFS to the empty string.


Next: , Previous: sendmail message, Up: sendmail

A.2 Just send an email message

The script below can be found in examples/sendmail-plain.sh. It just sends a hard–coded email message, from a hard–coded address to a hard–coded address. It makes no use of MBFL.

— Function: main

Drive the script controlling the SMTP protocol. It should be obvious what it does once we understand the following functions.

— Function: open_session hostname

Open a connection to the SMTP server using a fake device that Bash gives us as interface to the network. For the localhost, it ends up being:

          /dev/tcp/localhost/25

where 25 is the TCP port which is officially assigned to the SMTP service. To open the connection we use the idiom:

          exec 3<>/dev/tcp/localhost/25

which means: open a read and write connection to the selected hostname, using file descriptor number 3. There is nothing special in number 3, it is just the first free file descriptor number after 0 (standard input), 1 (standard output) and 2 (standard error).

The line:

          trap 'exec 3<&-' EXIT

means: close file descriptor 3 whenever the script terminates. This is redundant in such a simple script, it is there for completeness.

— Function: send string
— Function: send pattern arg ...

Send a string to the SMTP server. Use printf to format the string pattern with the optional arguments, then write the resulting string to file descriptor 3. The string written out is terminated with the sequence \r\n as mandated by the SMTP protocol.

— Function: read_and_send_message

Read an email message from stdin line by line (newline terminator), and rewrite it to file descriptor 3 terminating each line with the sequence carriage return/line feed. With the exception of the terminating sequence, the lines are left unchanged.

— Function: recv expected_code

Read a line (a sequence of characters up until the first \n) from file descriptor 3. The line is interpreted as a message from the SMTP server: the first three characters are a numeric code. If the code is different from expected_code, raise an error.

     #! /bin/bash
     #
     # Part of: Marco's Bash Functions Library
     # Contents: example script to send email
     # Date: Thu Apr 23, 2009
     #
     # Abstract
     #
     #       This script just sends a hardcoded email message from
     #       a hardcoded address to a hardcoded address.  It makes
     #       no use of MBFL.
     #
     #         The purpose of this script is to understand how to
     #       handle the SMTP protocol.
     #
     # Copyright (c) 2009, 2010 Marco Maggi <marcomaggi@gna.org>
     #
     # This  program  is free  software:  you  can redistribute  it
     # and/or modify it  under the terms of the  GNU General Public
     # License as published by the Free Software Foundation, either
     # version  3 of  the License,  or (at  your option)  any later
     # version.
     #
     # This  program is  distributed in  the hope  that it  will be
     # useful, but  WITHOUT ANY WARRANTY; without  even the implied
     # warranty  of  MERCHANTABILITY or  FITNESS  FOR A  PARTICULAR
     # PURPOSE.   See  the  GNU  General Public  License  for  more
     # details.
     #
     # You should  have received a  copy of the GNU  General Public
     # License   along   with    this   program.    If   not,   see
     # <http://www.gnu.org/licenses/>.
     #
     
     PROGNAME=${0##*/}
     
     function main () {
         local HOSTNAME=localhost
         local SMTP_PORT=25
         local FROM_ADDRESS=marco@localhost
         local TO_ADDRESS=root@localhost
         local LOGGING_TO_STDERR=yes
     
         open_session "$HOSTNAME"
         recv 220
         send 'HELO %s' 127.0.0.1
         recv 250
         send 'MAIL FROM:<%s>' "$FROM_ADDRESS"
         recv 250
         send 'RCPT TO:<%s>' "$TO_ADDRESS"
         recv 250
         send %s DATA
         recv 354
         print_message | read_and_send_message
         send %s .
         recv 250
         send %s QUIT
         recv 221
     }
     function print_message () {
         local LOCAL_HOSTNAME DATE MESSAGE_ID MESSAGE
         LOCAL_HOSTNAME=$(hostname --fqdn) || exit 2
         DATE=$(date --rfc-2822) || exit 2
         MESSAGE_ID=$(printf '%d-%d-%d@%s' \
             $RANDOM $RANDOM $RANDOM "$LOCAL_HOSTNAME")
         MESSAGE="Sender: $FROM_ADDRESS
     From: $FROM_ADDRESS
     To: $TO_ADDRESS
     Subject: proof from $PROGNAME
     Message-ID: <$MESSAGE_ID>
     Date: $DATE
     
     This is a text proof from the $PROGNAME script.
     --\x20
     Marco
     "
         printf "$MESSAGE"
     }
     function open_session () {
         local HOSTNAME=${1:?}
         local DEVICE=$(printf '/dev/tcp/%s/%d' "$HOSTNAME" $SMTP_PORT)
         exec 3<>"$DEVICE"
         trap 'exec 3<&-' EXIT
     }
     function recv () {
         local EXPECTED_CODE=${1:?}
         local line=
         IFS= read -t 5 line <&3
         test 127 -lt $? && {
             printf '%s: connection timed out\n' "$PROGNAME" >&2
             exit 2
         }
         test "$LOGGING_TO_STDERR" = yes && \
             printf '%s log: recv: %s\n' "$PROGNAME" "$line"
         test "${line:0:3}" = "$EXPECTED_CODE" || {
             send %s QUIT
             # It is cleaner to wait for the reply from the
             # server.
             IFS= read -t 5 line <&3
             test 127 -lt $? && {
                 printf '%s: connection timed out\n' "$PROGNAME" >&2
                 exit 2
             }
             test "$LOGGING_TO_STDERR" = yes && \
                 printf '%s log: recv: %s\n' "$PROGNAME" "$line"
             exit 2
         }
     }
     function send () {
         local pattern=${1:?}
         shift
         local line=$(printf "$pattern" "$@")
         printf '%s\r\n' "$line" >&3
         test "$LOGGING_TO_STDERR" = yes && \
             printf '%s log: sent: %s\n' "$PROGNAME" "$line"
     }
     function read_and_send_message () {
         local line
         local -i count=0
         while IFS= read line
         do
             if test "${line:0:1}" = '.'
             then printf '.%s\r\n' "$line" >&3
             else printf  '%s\r\n' "$line" >&3
             fi
             let ++count
         done
         test "$LOGGING_TO_STDERR" = yes && \
             printf '%s log: sent message (%d lines)\n' "$PROGNAME" $count
     }
     
     main
     
     ### end of file


Next: , Previous: sendmail plain, Up: sendmail

A.3 Send email through a process in background

The script below can be found in examples/sendmail-connector.sh. It just sends a hard–coded email message, from a hard–coded address to a hard–coded address. It makes no use of MBFL.

Bash version 4 introduced the new keyword coproc, which can be used to spawn processes in background and talk to them via pipes. This keyword in not used in this appendix.

The purpose of the script is to understand how to send a message through a process in background. It does the same things of the example described in Just send an email message. The main difference is that the single function open_session is replaced by the two functions open_session and connector.

What is important to understand, is how open_session runs connector in background and sets up two file descriptors to talk to it. In the real world we never use this technique with a function; this example script makes use of connector as a replacement for an external program that can establish sophisticated connections to remote hosts, for example using the TLS/SSL protocols.

— Function: open_session hostname

Open a connection to the SMTP server spawning a background process represented by the connector function. It makes use of two FIFOs (First In, First Out).

If we were to do it from a C language program: we would use the pipe() system function to create two pipes connecting script's process to the background process.

           ---------  out pipe  -----------  socket  --------
          | script  |--------->| connector |<======>| SMTP   |
          | process |<---------| process   |        | server |
           ---------  in pipe   -----------          --------

Bash has no way to create a pipe using the pipe() system function (up until version 4), so we use two FIFO channels created by the mkfifo program:

          : ${TMPDIR:=/tmp}
          local INFIFO=${TMPDIR}/in.$$
          local OUFIFO=${TMPDIR}/out.$$
          
          mkfifo --mode=0600 $INFIFO $OUFIFO

the script will use INFIFO to read characters from connector, and OUFIFO to send characters to connector.

           ---------  OUFIFO   -----------  socket  --------
          | script  |-------->| connector |<======>| SMTP   |
          | process |<--------| process   |        | server |
           ---------  INFIFO   -----------          --------

Once the FIFOs exist on the file system, we run connector in background, connecting its standard input and output to the FIFOs:

          connector $HOSTNAME $SMTP_PORT <$OUFIFO >$INFIFO &

be careful in selecting the redirections. Notice that, in this simple example, we ignore errors running connector.

Now we open file descriptors connecting them to the FIFOs:

          exec 3<>$INFIFO 4>$OUFIFO

the script will use file descriptor 3 to read characters from connector, and file descriptor 4 to send characters to connector. We open the input FIFO for both reading and writing, else exec will block waiting for the first char.

We have connected both the ends of both the FIFOs, so we can remove them from the file system:

          rm $INFIFO $OUFIFO

the FIFOs will continue to exist in the OS kernel until the file descriptors are closed.

Finally we register a clean up handler that closes the descriptors:

          trap 'exec 3<&- 4>&-' EXIT
— Function: connector hostname

Establish a connection to the SMTP server at hostname. It is not important here to fully understand how this function works; suffice it to say that it reads lines from stdin, and echoes them to the server; it reads lines from the server, and echoes them to stdout.

     #! /bin/bash
     #
     # Part of: Marco's Bash Functions Library
     # Contents: example script to send email using bg process
     # Date: Thu Apr 23, 2009
     #
     # Abstract
     #
     #       This script just sends a hardcoded email message from
     #       a hardcoded address to a hardcoded address.  It makes
     #       no use of MBFL.
     #
     #         The purpose of this script is to understand how to
     #       send a message through a process in background.
     #
     # Copyright (c) 2009, 2010 Marco Maggi <marcomaggi@gna.org>
     #
     # This  program  is free  software:  you  can redistribute  it
     # and/or modify it  under the terms of the  GNU General Public
     # License as published by the Free Software Foundation, either
     # version  3 of  the License,  or (at  your option)  any later
     # version.
     #
     # This  program is  distributed in  the hope  that it  will be
     # useful, but  WITHOUT ANY WARRANTY; without  even the implied
     # warranty  of  MERCHANTABILITY or  FITNESS  FOR A  PARTICULAR
     # PURPOSE.   See  the  GNU  General Public  License  for  more
     # details.
     #
     # You should  have received a  copy of the GNU  General Public
     # License   along   with    this   program.    If   not,   see
     # <http://www.gnu.org/licenses/>.
     #
     
     PROGNAME=${0##*/}
     : ${TMPDIR:=/tmp}
     
     function main () {
         local HOSTNAME=localhost
         local SMTP_PORT=25
         local FROM_ADDRESS=marco@localhost
         local TO_ADDRESS=root@localhost
         local LOGGING_TO_STDERR=yes
     
         open_session "$HOSTNAME"
         recv 220
         send 'HELO %s' 127.0.0.1
         recv 250
         send 'MAIL FROM:<%s>' "$FROM_ADDRESS"
         recv 250
         send 'RCPT TO:<%s>' "$TO_ADDRESS"
         recv 250
         send %s DATA
         recv 354
         print_message | read_and_send_message
         send %s .
         recv 250
         send %s QUIT
         recv 221
     }
     function print_message () {
         local LOCAL_HOSTNAME DATE MESSAGE_ID MESSAGE
         LOCAL_HOSTNAME=$(hostname --fqdn) || exit 2
         DATE=$(date --rfc-2822) || exit 2
         MESSAGE_ID=$(printf '%d-%d-%d@%s' \
             $RANDOM $RANDOM $RANDOM "$LOCAL_HOSTNAME")
         MESSAGE="Sender: $FROM_ADDRESS
     From: $FROM_ADDRESS
     To: $TO_ADDRESS
     Subject: proof from $PROGNAME
     Message-ID: <$MESSAGE_ID>
     Date: $DATE
     
     This is a text proof from the $PROGNAME script.
     --\x20
     Marco
     "
         printf "$MESSAGE"
     }
     function open_session () {
         local HOSTNAME=${1:?}
         local INFIFO=${TMPDIR}/in.$$
         local OUFIFO=${TMPDIR}/out.$$
         # Bash  has  no  operation  equivalent to  the  C  level
         # "pipe()" function, so we have to use FIFOs.
         mkfifo --mode=0600 $INFIFO $OUFIFO
         connector "$HOSTNAME" <$OUFIFO >$INFIFO &
         # Open the input FIFO for both reading and writing, else
         # "exec" will block waiting for the first char.
         exec 3<>$INFIFO 4>$OUFIFO
         # We have connected both the  ends of both the FIFOs, so
         # we  can remove them  from the  file system:  the FIFOs
         # will continue to exist  until the file descriptors are
         # closed.
         rm $INFIFO $OUFIFO
         trap 'exec 3<&- 4>&-' EXIT
     }
     function recv () {
         local EXPECTED_CODE=${1:?}
         local line=
         IFS= read line <&3
         test "$LOGGING_TO_STDERR" = yes && \
             printf '%s log: recv: %s\n' "$PROGNAME" "$line"
         if test "${line:0:3}" != "$EXPECTED_CODE"
         then
             send %s QUIT
             # It is cleaner to wait for the reply from the
             # server.
             IFS= read line <&3
             test "$LOGGING_TO_STDERR" = yes && \
                 printf '%s log: recv: %s\n' "$PROGNAME" "$line"
             exit 2
         fi
     }
     function send () {
         local pattern=${1:?}
         shift
         local line=$(printf "$pattern" "$@")
         printf '%s\r\n' "$line" >&4
         test "$LOGGING_TO_STDERR" = yes && \
             printf '%s log: sent: %s\n' "$PROGNAME" "$line"
     }
     function read_and_send_message () {
         local line
         local -i count=0
         while IFS= read line
         do
             if test "${line:0:1}" = '.'
             then printf '.%s\r\n' "$line" >&4
             else printf  '%s\r\n' "$line" >&4
             fi
             let ++count
         done
         test "$LOGGING_TO_STDERR" = yes && \
             printf '%s log: sent message (%d lines)\n' "$PROGNAME" $count
     }
     function connector () {
         local HOSTNAME=${1:?} query= answer= line=
         local DEVICE=$(printf '/dev/tcp/%s/%d' "$HOSTNAME" $SMTP_PORT)
         exec 3<>"$DEVICE"
         # Read the  greetings from the server, echo  them to the
         # client.
         IFS= read -t 5 answer <&3
         test 127 -lt $? && {
             printf '%s: connection timed out\n' "$PROGNAME" >&2
             exit 2
         }
         printf '%s\n' "$answer"
         # Read the query from the client, echo it to the server.
         while read query
         do
             printf '%s\r\n' "$query" >&3
             # Read the  answer from the  server, echo it  to the
             # client.
             IFS= read -t 5 answer <&3
             test 127 -lt $? && {
                 printf '%s: connection timed out\n' "$PROGNAME" >&2
                 exit 2
             }
             printf '%s\n' "$answer"
             # Test special queries.
             test "$query" = QUIT$'\r' && {
                 IFS= read -t 5 answer <&3
                 test 127 -lt $? && {
                     printf '%s: connection timed out\n' "$PROGNAME" >&2
                     exit 2
                 }
                 printf '%s\n' "$answer"
                 exit
             }
             test "$query" = DATA$'\r' && {
                 # Read data lines from  the client, echo them to
                 # the server up until ".\r" is read.
                 while IFS= read line
                 do
                     printf '%s\n' "$line" >&3
                     test "${line:0:2}" = .$'\r' && break
                 done
                 # Read the answer to  data from the server, echo
                 # it to the client.
                 IFS= read -t 5 answer <&3
                 test 127 -lt $? && {
                     printf '%s: connection timed out\n' "$PROGNAME" >&2
                     exit 2
                 }
                 printf '%s\n' "$answer"
             }
         done
         # We should never come here.
         exit 1
     }
     
     main
     
     ### end of file


Next: , Previous: sendmail connector, Up: sendmail

A.4 Using gnutls-cli as connector

GNU TLS is a library implementing the TLS protocol; it can be used to establish encrypted and authenticated connections to a remote host. The SMTP protocol has extensions to allow usage of a TLS layer.

A GNU TLS installation comes with a command line test program, gnutls-cli, that can be used to establish an encrypted connection by a shell script. This command can be used as the “connector” modeled in Send email through a process in background. It has a manual page, which we may want to read.

When handling an encrypted connection we have to know in advance how the remote SMTP server behaves. Let's see first the simpler example, using the a human driven interactive session; then we will describe a more complex interaction.

Notice that gnutls-cli has a --crlf option that will cause all the lines sent to the server to be terminated by a carriage return/line feed sequence (\r\n or \x0d\x0a). If we write a script that terminates by itself the lines with this sequence, for example:
     printf 'ehlo localhost.localdomain\r\n'

we must avoid this option, else SMTP protocol violation errors may occur. However, if we try a hand–driven interactive session, we want to use this option to send protocol–compliant lines.


Next: , Up: sendmail gnutls

A.4.1 Immediate encrypted bridge

We use as example the server relay.poste.it, port 465. You have to have an account there to use it; do not bomb this server with fake connections. This server requests us to build the encrypted bridge immediately after the connection has been established, without waiting for any line of greetings from the server.

So, we start the connector like this:

     $ gnutls-cli --port 465 relay.poste.it

if the connection succeeds: gnutls-cli prints a lot of message lines on its standard output explaining what is going on; at last comes the line of greetings from the server, which begins with code 220.

The server supports the AUTH LOGIN authentication mechanism, which requires the base64 encoding of the user name and password; we can perform it with the external program base64 (which comes with GNU Coreutils) like this:

     ENCODED_USERNAME=$(echo -n 'the-user-name' | base64)
     ENCODED_PASSWORD=$(echo -n 'the-pass-word' | base64)

GNU Emacs users can do it with:

     (setq my-usr (base64-encode-string "the-user-name"))
     (setq my-pwd (base64-encode-string "the-pass-word"))

The authentication dialogue goes like this:

  1. We send AUTH LOGIN, to start the authentication.
  2. It replies with 334 VXNlcm5hbWU6 which is the request for the username. The string VXNlcm5hbWU6 is the base64 encoding of the string Username: (without trailing newline); we can verify this with:
              $ echo -n Username: | base64
    

    or in the Emacs' scratch buffer:

              (base64-encode-string "Username:")
    
  3. We send the login user name encoded in base64.
  4. It checks the string and, if the format is correct, it replies with 334 UGFzc3dvcmQ6 (this should happen even if the username is unknown to the server). The string UGFzc3dvcmQ6 is the base64 encoding of the string Password: (without ending newline); we can verify this with:
              $ echo -n Password: | base64
    

    or in the Emacs' scratch buffer:

              (base64-encode-string "Password:")
    
  5. We send the login password encoded in base64.
  6. It checks it and, if correct, it replies with a line starting with code 235.

Beware that if we are not quick to send the encoded password after the encoded user name, the server may reset the authentication process as if we sent a wrong user name.

So we can do the SMTP dialogue reported below by hand (which is an edited log of a session under Emacs' eshell); lines starting with recv> are the ones received from the server, lines starting with send> are the ones we send to the server, the ellipses ... are replacements for server text we are not interested in.

     $ gnutls-cli --crlf --port 465 relay.poste.it
     
     recv> 220 ... ESMTP Service ...
     send> ehlo localhost.localdomain
     recv> 250-...
     recv> 250-DSN
     recv> 250-8BITMIME
     recv> 250-PIPELINING
     recv> 250-HELP
     recv> 250-AUTH=LOGIN
     recv> 250-AUTH LOGIN CRAM-MD5 DIGEST-MD5 PLAIN
     recv> 250-DELIVERBY 300
     recv> 250 SIZE
     send> auth login
     recv> 334 VXNlcm5hbWU6
     send> <the-base64-username>
     recv> 334 UGFzc3dvcmQ6
     send> <the-base64-password>
     recv> 235 login authentication successful
     send> mail from:<from-address@poste.it>
     recv> 250 MAIL FROM:<from-address@poste.it> OK
     send> rcpt to:<to-address@other-host.it>
     recv> 250 RCPT TO:<to-address@other-host.it> OK
     send> data
     recv> 354 Start mail input; end with <CRLF>.<CRLF>
     send> From: <from-address@poste.it>
     send> To: <to-address@other-host.it>
     send> Subject: interactive attempt
     send>
     send> Text for interactive attempt.
     send> --
     send> Marco
     send> .
     recv> 250 ... Mail accepted
     send> quit
     recv> 221 ... QUIT
     recv> - Peer has closed the GNUTLS connection


Previous: sendmail gnutls now, Up: sendmail gnutls

A.4.2 Delayed encrypted bridge

We use as example the server smtp.gmail.com, port 587. You have to have an account there to use it; do not bomb this server with fake connections. This server requests us to start an ESMTP dialogue, then issue the STARTTLS command and build the encrypted bridge; once the bridge is set up, we restart an ESMTP dialogue and do the authentication and the message delivery.

We start the connector like this:

     $ gnutls-cli --starttls --port 587 smtp.gmail.com

if the connection succeeds: gnutls-cli prints message lines on its standard output explaining what is going on; at last comes the line of greetings from the server, which begins with code 220.

The --starttls option tells gnutls-cli not to build the encrypted bridge immediately; rather, it waits for a SIGALRM signal, which we must deliver to it when we are ready. The quickest way to send such a signal, when there is only one gnutls-cli process running, is:

     $ kill -SIGALRM $(/sbin/pidof gnutls-cli)

beware that pidof may be installed in other places on your system.

The server supports the AUTH PLAIN authentication mechanism, which requires the base64 encoding of the user name and password stored in a special record; we can do it with the external program base64 (which comes with GNU Coreutils) like this:

     SECRETS=$(printf "\x00${LOGIN_NAME}\x00${PASSWORD}" | base64)

GNU Emacs users can do it with:

     (setq my-auth (base64-encode-string
        (format "%c%s%c%s" 0 "the-user-name" 0 "the-pass-word")))

The authentication dialogue goes like this:

  1. We send AUTH PLAIN followed by the encoded credentials.
  2. It checks the user name and password and, if correct, it replies with code 235.

So, we can do the SMTP dialogue reported below by hand (which is an edited log of a session under Emacs' eshell); lines starting with recv> are the ones received from the server, lines starting with send> are the ones we send to the server, the ellipses ... are replacements for server text we are not interested in.

     $ gnutls-cli --crlf --starttls --port 587 smtp.gmail.com
     
     recv> 220 ... ESMTP ...
     send> ehlo localhost.localdomain
     recv> 250-...
     recv> 250-SIZE 35651584
     recv> 250-8BITMIME
     recv> 250-STARTTLS
     send> 250-ENHANCEDSTATUSCODES
     recv> 250 PIPELINING
     send> starttls
     recv> 220 2.0.0 Ready to start TLS
     
     === here we deliver SIGALRM to the gnutls-cli process
     
     recv> *** Starting TLS handshake
     recv> - Certificate type: X.509
     recv>  - Got a certificate list of 1 certificates.
     recv>
     recv>  - Certificate[0] info:
     recv>  # The hostname in the certificate matches 'smtp.gmail.com'.
     recv>  # valid since: ..
     recv>  # expires at: ...
     recv>  # fingerprint: ...
     recv>  # Subject's DN: ...
     recv>  # Issuer's DN: ...
     recv>
     recv>
     recv> - Peer's certificate issuer is unknown
     recv> - Peer's certificate is NOT trusted
     recv> - Version: TLS1.0
     recv> - Key Exchange: RSA
     recv> - Cipher: ARCFOUR-128
     recv> - MAC: MD5
     recv> - Compression: NULL
     send> ehlo rapitore.luna
     recv> 250-...
     recv> 250-SIZE 35651584
     recv> 250-8BITMIME
     recv> 250-AUTH LOGIN PLAIN
     recv> 250-ENHANCEDSTATUSCODES
     recv> 250 PIPELINING
     send> auth plain <the-encoded-auth-credentials>
     recv> 235 2.7.0 Accepted
     send> mail from:<from-address@gmail.com>
     recv> 250 2.1.0 OK ...
     send> rcpt to:<to-address@poste.it>
     recv> 250 2.1.5 OK ...
     send> data
     recv> 354  Go ahead ...
     send> From: from-address@gmail.com
     send> To: to-address@poste.it
     send> Subject: interactive proof from gmail
     send>
     send> proof
     send> .
     recv> 250 2.0.0 OK ...
     send> quit
     recv> 221 2.0.0 closing connection ...


Next: , Previous: sendmail gnutls, Up: sendmail

A.5 Using openssl as connector

OpenSSL is a library implementing the SSL/TLS protocol; it can be used to establish encrypted and authenticated connections to a remote host.

An OpenSSL installation comes with a command line test program, openssl, that can be used to establish an encrypted connection by a shell script. This command can be used as the “connector” modeled in Send email through a process in background. It has a manual page, which we may want to read.

Here we see how we can use the openssl program in place of the gnutls-cli program described in Using gnutls-cli as connector. The two methods have a lot in common (the SMTP protocol is the same), we only have to understand the command line of the program.

Notice that openssl has a -crlf option that will cause all the lines sent to the server to be terminated by a carriage return/line feed sequence (\r\n or \x0d\x0a). If we write a script that terminates by itself the lines with this sequence, for example:
     printf 'ehlo localhost.localdomain\r\n'

we must avoid this option, else SMTP protocol violation errors may occur. However, if we try a hand–driven interactive session, we want to use this option to send protocol–compliant lines.


Next: , Up: sendmail openssl

A.5.1 Immediate encrypted bridge

To build the encrypted bridge right after the connection, without exchanging greetings with the server, we do:

     $ openssl s_client -quiet -connect relay.poste.it:465

then we start the SMTP dialogue as outlined in Immediate encrypted bridge.


Previous: sendmail openssl now, Up: sendmail openssl

A.5.2 Delayed encrypted bridge

To first exchange greetings, then send STARTTLS and finally build the encrypted bridge, we do:

     $ openssl s_client -quiet -starttls smtp \
         -connect smtp.gmail.com:587

then we start the SMTP dialogue as outlined in Immediate encrypted bridge. openssl knows how to start an SMTP dialogue, and it does it automatically.


Previous: sendmail openssl, Up: sendmail

A.6 Sending email

This section documents the example script examples/sendmail-mbfl.sh, which makes use of MBFL to send an email message. The script sends already composed messages using plain or TLS sessions, and it can use both OpenSSL and GNU TLS.


Next: , Up: sendmail script

A.6.1 Usage examples

Let's say we have a file named message.mail holding a fully composed email message:

     Sender: marco@localhost
     From: marco@localhost
     To: root@localhost
     Subject: server on fire?
     
     I noticed flames raising from the server room...
     --
     Marco

basically, to send it with sendmail-mbfl.sh we have to do:

     $ sendmail-mbfl.sh \
         --envelope-from=marco@localhost  \
         --envelope-to=root@localhost     \
         --message=message.mail

by default the SMTP server name is set to localhost and the TCP port to 25. The default session is plain, without TLS.

Port number 25 is officially assigned to the SMTP protocol; if the localhost uses a different TCP port, we can select it with the --port option:

     $ sendmail-mbfl.sh --port=587         \
         --envelope-from=marco@localhost   \
         --envelope-to=root@localhost      \
         --message=message.mail

To send mail to a remote SMTP server, we select its hostname with the --host option:

     $ sendmail-mbfl.sh \
         --host=smtp.gmail.com --port=587  \
         --envelope-from=marco@gmail.com   \
         --envelope-to=marco@spiffy.it     \
         --message=message.mail

Mail services may offer encrypted sessions to their SMTP servers. Encryption with the TLS protocol is supported by sendmail-mbfl.sh through external programs. Whether a server requires an encrypted session, can be specified using the --plain, --tls or --starttls options.

To establish an encrypted session, the script needs to acquire the credentials of the user. These can be stored in a configuration file named ~/.authinfo, which looks like this:

     machine smtp.gmail.com login marco@gmail.com password abcdefghilm
     machine relay.poste.it login marco@poste.it  password 0123456789

so that a line/record can be uniquely identified with values of the --host and --username options.

So we can do:

     $ sendmail-mbfl.sh --host=gmail       \
         --username=marco --starttls        \
         --envelope-from=marco@gmail.com   \
         --envelope-to=marco@spiffy.it     \
         --message=message.mail


Next: , Previous: sendmail script examples, Up: sendmail script

A.6.2 Command line options

The synopsis is:

     sendmail-mbfl.sh \
         --envelope-from=<ADDRESS>              \
         --envelope-to=<ADDRESS>                \
         [--message=<SOURCE> | --test-message]
         [options]

the script sends an email address, and it can do a plain session or use a connector. Options description follows.

-Faddress
--envelope-from=address
Select the MAIL FROM envelope address. If this option is used multiple times: the last one wins.
-Taddress
--envelope-to=address
Select the RCPT TO envelope address. This option can be used multiple times: each address is appended to a list of recipients.
-Msource
--message=source
Select the source of the whole email message. If source is a file pathname, that file is read and used as data; if source is -, the message is read from the standard input channel. It defaults to -.
--test-message
Send a test message with a hard–coded body part. This option supersedes --message.
--host=host
Select the SMTP server host name. It defaults to localhost.
--host-info=file
Select a file from which read host and port informations.
-pport
--port=port
Select the SMTP server TCP port; this option supersedes --host-info. It defaults to 25.
--plain
--tls
--starttls
Establish a plain session, TLS session with immediate bridge construction or TLS session with bridge construction after STARTTLS command.
--gnutls
Use gnutls-cli as connector for encrypted sessions. This is the default when --starttls or --delayed-starttls are used.
--openssl
Use gnutls-cli as connector for encrypted sessions.
--auth-info=file
Select the file, in netrc format, from which read the authorisation credentials. Defaults to ~/.authinfo.
--username=user
Specify a string used to select an account in the authorisation file.
--auth-none
--auth-plain
--auth-login
Perform no authorisation, the AUTH PLAIN authorisation or the AUTH LOGIN authorisation.


Next: , Previous: sendmail script invoking, Up: sendmail script

A.6.3 Reading host informations from file

For each SMTP server we need the following informations: the hostname, the port number, the session type, the authorisation method. There are two ways to specify these:

If --host is not used: the hostname defaults to localhost. If the selection of port, session type or authorisation type is left unspecified: The script automatically looks into the default hostinfo file. Informations from command line options supersede informations from the hostinfo file.

The default pathname for the hostinfo file is $HOME/.hostinfo and can be overridden by the --host-info option. The format of this file is line oriented: Blank lines are ignored, lines starting with a # character are comments, lines starting with machine are host records.

Each record line must have the format:

     machine <host> service <name> port <number> session <type> auth <type>

for example:

     # ~/.hostinfo --
     #
     
     # SMTP servers
     machine localhost service smtp port 25 session plain auth none
     machine relay.poste.it service smtp port 465 session tls auth login
     machine smtp.gmail.com service smtp port 587 session starttls auth plain
     
     # POP3 servers
     machine pop.tiscali.it service pop3 port 110 session plain auth userpass
     machine relay.poste.it service pop3 port 995 session tls auth userpass
     machine pop.googlemail.com service pop3 port 995  session tls auth userpass
     
     ### end of file

so that we can extract a record with the following script:

     file=~/.hostinfo
     host=gmail
     service=smtp
     
     rex='^[ \t]*'
     rex+='machine[ \t]\+.*%s.*[ \t]\+'
     rex+='service[ \t]\+%s[ \t]\+'
     rex+='port[ \t]\+[0-9]\+[ \t]\+'
     rex+='session[ \t]\+\(plain\|tls\|starttls\)[ \t]\+'
     rex+='auth[ \t]\+\(none\|plain\|login\)'
     rex+='[ \t]*$'
     rex=$(printf "$rex" $host $service)
     
     set -- $(grep "$rex" "$file")
     echo machine $2
     echo service $4
     echo port    $6
     echo session $8
     shift 9
     echo auth    $1

this is exactly what sendmail-mbfl.sh does with the file.

Notice that when using the hostinfo file, the value of the --host option can be a substring of the host name.


Next: , Previous: sendmail script hostinfo, Up: sendmail script

A.6.4 Reading authentication credentials from file

The only way the script has to acquire the user name and password to log into the remote server, is by reading the authinfo file. By default, its pathname is $HOME/.authinfo, it can be overridden with the --auth-info option.

Its format is a simplified version of the netrc file format: blank lines are ignored, lines starting with a # character are comments, lines starting with machine are host records.

Each record must have the format:

     machine <hostname> login <user-name> password <pass-word>

for example:

     machine smtp.gmail.com login one@gmail.com password abcdefghilm
     machine relay.poste.it login two@poste.it  password 0123456789

so that we can extract a record with the following script:

     file=~/.authinfo
     host=poste
     username=marco
     
     rex='^[ \t]*'
     rex+='machine[ \t]\+.*%s.*[ \t]\+'
     rex+='login[ \t]\+.*%s.*[ \t]\+'
     rex+='password[ \t]\+.*'
     rex+='[ \t]*$'
     rex=$(printf "$rex" $host $username)
     
     line=$(grep "$rex" $file)
     set -- $line
     echo machine  $2
     echo username $4
     echo password $6

and this is exactly what sendmail-mbfl.sh does.

The host name and the user name are selected by the command line options --host and --username. Notice that the values for these options can be substrings of the values in the authinfo file.


Previous: sendmail script authinfo, Up: sendmail script

A.6.5 Interfacing with GNU Emacs

This section documents an Emacs interface to the sendmail-mbfl.sh script that can be used to send mail. It consists of an Emacs Lisp library that allows us to send mail with all the methods supported by the script.

To use it we have to install the file examples/sendmail-mbfl.el in one of the directories in the load path of Emacs, then load it with:

     (require 'sendmail-mbfl)

The library assumes that message.el and sendmail.el are available on the system; recent installations of Emacs should have them.


Next: , Up: sendmail script emacs
A.6.5.1 Customisable variables

A customisation group called sendmail-mbfl is available to configure the library.

— Customisable Variable: sendmail-mbfl-program

A string representing the name of the MBFL shell script. By default set to sendmail-mbfl.sh.

— Customisable Variable: sendmail-mbfl-extra-args

A list of strings representing extra arguments for the command line of sendmail-mbfl.sh. By default set to:

          ("--verbose" "--debug")
— Customisable Variable: sendmail-mbfl-envelope-from-function

Select a function to call to acquire, from the current buffer, the envelope email address of the sender, to be used in the MAIL FROM SMTP command.

The function is invoked with no arguments and it must return a single string representing the email address. If no suitable address is found: it must raise an error. The function may be called multiple times for the same message buffer.

The selected function is used by send-mail-with-mbfl. By default it is set to sendmail-mbfl-envelope-from.

— Customisable Variable: sendmail-mbfl-envelope-to-function

Select a function to call to acquire, from the current buffer, the envelope email addresses of the receivers, to be used in the RCPT TO SMTP command.

The function is invoked with no arguments and it must return a list of strings representing email addresses. If no suitable address is found: it must raise an error. The function may be called multiple times for the same message buffer.

The selected function is used by send-mail-with-mbfl. By default it is set to sendmail-mbfl-envelope-to.

— Customisable Variable: sendmail-mbfl-extract-addresses-function

Select a function to call to extract a list of email addresses from an email header. It is invoked with no arguments and the buffer narrowed to the header to examine.

The function is invoked with no arguments and it must return a list of strings representing email addresses, or nil.

The selected function is used by sendmail-mbfl-envelope-from and sendmail-mbfl-envelope-to.

— Customisable Variable: sendmail-mbfl-hostname-function

Select a function to call to extract, from the current buffer, the hostname of the SMTP server to be used to send the message. The result is used as search key in the selected hostinfo file.

The function is invoked with no arguments and it must return a string representing the hostname; if it is unable to determine the hostname: it must raise an error.

The selected function is used by send-mail-with-mbfl. By default it is set to sendmail-mbfl-hostname.

— Customisable Variable: sendmail-mbfl-username-function

Select a function to call to extract, from the current buffer, the username with which to login to the SMTP server. The result is used as search key in the selected authinfo file.

The function is invoked with no arguments and it must return a string representing the username; if it is unable to determine the username: it must raise an error.

The selected function is used by send-mail-with-mbfl. By default it is set to sendmail-mbfl-username.

— Customisable Variable: sendmail-mbfl-host-info

The pathname of the file holding informations about known SMTP servers. Reading host informations from file.

By default it is set to ~/.hostinfo.

— Customisable Variable: sendmail-mbfl-auth-info

The pathname of the file holding informations about known accounts at SMTP servers. Reading authentication credentials from file.

By default it is set to ~/.authinfo.

— Customisable Variable: sendmail-mbfl-connector

Select the external program to use to establish the TLS transport layer. Valid values are the strings: gnutls, openssl. The default is openssl because it is more likely to be installed on any system.

— Customisable Variable: sendmail-mbfl-timeout

Select the timeout in seconds for reading answers from the SMTP server. The default is 5.


Next: , Previous: sendmail script emacs vars, Up: sendmail script emacs
A.6.5.2 Sending and delivering functions
— Function: send-mail-with-mbfl

Send the email message in the current buffer. This interactive function can be invoked directly by the user, or, better, used as value for message-send-mail-function. Sending involves:

  1. Normalising the message with sendmail-mbfl-normalise-message.
  2. Performing special deliveries with sendmail-mbfl-delivery.
  3. Posting the message to an MTA using sendmail-mbfl-post.

— Function: sendmail-mbfl-delivery

Perform special deliveries of the email message in the current buffer. If the message has an Fcc header, deliver is performed relying on mail-do-fcc from sendmail.el.

It is to be called after sendmail-mbfl-normalise-message or an equivalent normalisation has been applied to the message.

— Function: sendmail-mbfl-post

Post the email message in the current buffer using an MTA. Posting involves:

  1. Preparing the message with sendmail-mbfl-prepare-message-for-mta.
  2. Saving the message into a temporary file.
  3. Sending the file using the program selected with the customisable variable sendmail-mbfl-program.

It is to be called after sendmail-mbfl-normalise-message or an equivalent normalisation has been applied to the message.


Next: , Previous: sendmail script emacs send, Up: sendmail script emacs
A.6.5.3 Message inspection functions
— Function: sendmail-mbfl-envelope-from
— Function: sendmail-mbfl-envelope-from/message

sendmail-mbfl-envelope-from interprets the current buffer as an email message and searches the contents for an email address to be used as envelope sender.

It examines the headers From and Sender, in this order and it returns a single string representing the email address; if no suitable address is found: it raises an error. The address is extracted from the headers using the function selected by the customisable variable sendmail-mbfl-extract-addresses-function.

sendmail-mbfl-envelope-from/message is an interactive wrapper for sendmail-mbfl-envelope-from that prints the result to the *Message* buffer.

— Function: sendmail-mbfl-envelope-to
— Function: sendmail-mbfl-envelope-to/message

sendmail-mbfl-envelope-to interprets the current buffer as an email message and searches the contents for email addresses to be used as envelope receivers.

It examines the headers To, Cc and Bcc and it returns a list of strings representing email addresses; if no suitable address is found: it raises an error. The addresses are extracted from the headers using the function selected by the customisable variable sendmail-mbfl-extract-addresses-function.

sendmail-mbfl-envelope-to/message is an interactive wrapper for sendmail-mbfl-envelope-to that prints the result to the *Message* buffer.

— Function: sendmail-mbfl-extract-addresses

Extract a list of email addresses from the current buffer. It must be invoked with the buffer narrowed to the header to examine. Return a list of email addresses as strings, or nil if no address is found.

— Function: sendmail-mbfl-hostname
— Function: sendmail-mbfl-hostname/message

sendmail-mbfl-hostname extracts, from the current buffer, the hostname of the SMTP server to be used to send the message. The result is used as search key in the selected hostinfo file.

It returns a string representing the hostname else, if unable to determine the hostname, it raises an error. The hostname is the hostname part of the value returned by sendmail-mbfl-envelope-from.

sendmail-mbfl-hostname/message is an interactive wrapper for sendmail-mbfl-hostname that prints the result to the *Message* buffer.

— Function: sendmail-mbfl-username
— Function: sendmail-mbfl-username/message

sendmail-mbfl-username extracts, from the current buffer, the username with which login to the SMTP server. The result is used as search key in the selected authinfo file.

It returns a string representing the username else, if unable to determine the username, it raises an error. The username is the username part of the value returned by sendmail-mbfl-envelope-from.

sendmail-mbfl-username/message is an interactive wrapper for sendmail-mbfl-username that prints the result to the *Message* buffer.


Previous: sendmail script emacs insp, Up: sendmail script emacs
A.6.5.4 Miscellaneous functions
— Function: sendmail-mbfl-activate

Set message-send-mail-function so that functions from message.el send mail using send-mail-with-mbfl.

— Function: sendmail-mbfl-normalise-message

Normalise the email message in the current buffer so that it is ready to be posted or delivered. Scan the headers for invalid lines and try to fix them. Scan the message for mandatory headers and, if missing, add them; this may require querying the user for informations.

It is to be called before acquiring sender and receiver addresses from the headers. It is an interactive function: it can be explicitly applied to a buffer by the user any number of times.

This function does not remove the headers/body separator.

— Function: sendmail-mbfl-prepare-message-for-mta

Prepare the email message in the current buffer to be sent to a Mail Transport Agent. Headers like Fcc and Bcc are removed; the headers/body separator line is removed. It is to be called after acquiring sender and receiver addresses from the headers.

— Function: sendmail-mbfl-copy-message-buffer dst-buffer src-buffer

Copy an email message from src-buffer to dst-buffer. Set encoding and text representation properties of the destination buffer to be equal to the ones of the source buffer.


Next: , Previous: sendmail, Up: Top

Appendix B GNU LESSER GENERAL PUBLIC LICENSE

Version 3, 29 June 2007
     Copyright © 2007 Free Software Foundation, Inc. http://fsf.org/
     
     Everyone is permitted to copy and distribute verbatim copies of this
     license document, but changing it is not allowed.

This version of the GNU Lesser General Public License incorporates the terms and conditions of version 3 of the GNU General Public License, supplemented by the additional permissions listed below.

  1. Additional Definitions.

    As used herein, “this License” refers to version 3 of the GNU Lesser General Public License, and the “GNU GPL” refers to version 3 of the GNU General Public License.

    “The Library” refers to a covered work governed by this License, other than an Application or a Combined Work as defined below.

    An “Application” is any work that makes use of an interface provided by the Library, but which is not otherwise based on the Library. Defining a subclass of a class defined by the Library is deemed a mode of using an interface provided by the Library.

    A “Combined Work” is a work produced by combining or linking an Application with the Library. The particular version of the Library with which the Combined Work was made is also called the “Linked Version”.

    The “Minimal Corresponding Source” for a Combined Work means the Corresponding Source for the Combined Work, excluding any source code for portions of the Combined Work that, considered in isolation, are based on the Application, and not on the Linked Version.

    The “Corresponding Application Code” for a Combined Work means the object code and/or source code for the Application, including any data and utility programs needed for reproducing the Combined Work from the Application, but excluding the System Libraries of the Combined Work.

  2. Exception to Section 3 of the GNU GPL.

    You may convey a covered work under sections 3 and 4 of this License without being bound by section 3 of the GNU GPL.

  3. Conveying Modified Versions.

    If you modify a copy of the Library, and, in your modifications, a facility refers to a function or data to be supplied by an Application that uses the facility (other than as an argument passed when the facility is invoked), then you may convey a copy of the modified version:

    1. under this License, provided that you make a good faith effort to ensure that, in the event an Application does not supply the function or data, the facility still operates, and performs whatever part of its purpose remains meaningful, or
    2. under the GNU GPL, with none of the additional permissions of this License applicable to that copy.
  4. Object Code Incorporating Material from Library Header Files.

    The object code form of an Application may incorporate material from a header file that is part of the Library. You may convey such object code under terms of your choice, provided that, if the incorporated material is not limited to numerical parameters, data structure layouts and accessors, or small macros, inline functions and templates (ten or fewer lines in length), you do both of the following:

    1. Give prominent notice with each copy of the object code that the Library is used in it and that the Library and its use are covered by this License.
    2. Accompany the object code with a copy of the GNU GPL and this license document.
  5. Combined Works.

    You may convey a Combined Work under terms of your choice that, taken together, effectively do not restrict modification of the portions of the Library contained in the Combined Work and reverse engineering for debugging such modifications, if you also do each of the following:

    1. Give prominent notice with each copy of the Combined Work that the Library is used in it and that the Library and its use are covered by this License.
    2. Accompany the Combined Work with a copy of the GNU GPL and this license document.
    3. For a Combined Work that displays copyright notices during execution, include the copyright notice for the Library among these notices, as well as a reference directing the user to the copies of the GNU GPL and this license document.
    4. Do one of the following:
      1. Convey the Minimal Corresponding Source under the terms of this License, and the Corresponding Application Code in a form suitable for, and under terms that permit, the user to recombine or relink the Application with a modified version of the Linked Version to produce a modified Combined Work, in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source.
      2. Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (a) uses at run time a copy of the Library already present on the user's computer system, and (b) will operate properly with a modified version of the Library that is interface-compatible with the Linked Version.
    5. Provide Installation Information, but only if you would otherwise be required to provide such information under section 6 of the GNU GPL, and only to the extent that such information is necessary to install and execute a modified version of the Combined Work produced by recombining or relinking the Application with a modified version of the Linked Version. (If you use option 4d0, the Installation Information must accompany the Minimal Corresponding Source and Corresponding Application Code. If you use option 4d1, you must provide the Installation Information in the manner specified by section 6 of the GNU GPL for conveying Corresponding Source.)
  6. Combined Libraries.

    You may place library facilities that are a work based on the Library side by side in a single library together with other library facilities that are not Applications and are not covered by this License, and convey such a combined library under terms of your choice, if you do both of the following:

    1. Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities, conveyed under the terms of this License.
    2. Give prominent notice with the combined library that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work.
  7. Revised Versions of the GNU Lesser General Public License.

    The Free Software Foundation may publish revised and/or new versions of the GNU Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.

    Each version is given a distinguishing version number. If the Library as you received it specifies that a certain numbered version of the GNU Lesser General Public License “or any later version” applies to it, you have the option of following the terms and conditions either of that published version or of any later version published by the Free Software Foundation. If the Library as you received it does not specify a version number of the GNU Lesser General Public License, you may choose any version of the GNU Lesser General Public License ever published by the Free Software Foundation.

    If the Library as you received it specifies that a proxy can decide whether future versions of the GNU Lesser General Public License shall apply, that proxy's public statement of acceptance of any version is permanent authorization for you to choose that version for the Library.


Next: , Previous: Package License, Up: Top

Appendix C GNU Free Documentation License

Version 1.3, 3 November 2008
     Copyright © 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc.
     http://fsf.org/
     
     Everyone is permitted to copy and distribute verbatim copies
     of this license document, but changing it is not allowed.
  1. PREAMBLE

    The purpose of this License is to make a manual, textbook, or other functional and useful document free in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

    This License is a kind of “copyleft”, which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

    We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

  2. APPLICABILITY AND DEFINITIONS

    This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The “Document”, below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as “you”. You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

    A “Modified Version” of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

    A “Secondary Section” is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

    The “Invariant Sections” are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

    The “Cover Texts” are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

    A “Transparent” copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not “Transparent” is called “Opaque”.

    Examples of suitable formats for Transparent copies include plain ascii without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

    The “Title Page” means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, “Title Page” means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.

    The “publisher” means any person or entity that distributes copies of the Document to the public.

    A section “Entitled XYZ” means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as “Acknowledgements”, “Dedications”, “Endorsements”, or “History”.) To “Preserve the Title” of such a section when you modify the Document means that it remains a section “Entitled XYZ” according to this definition.

    The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

  3. VERBATIM COPYING

    You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

    You may also lend copies, under the same conditions stated above, and you may publicly display copies.

  4. COPYING IN QUANTITY

    If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

    If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

    If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

    It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

  5. MODIFICATIONS

    You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

    1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.
    2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.
    3. State on the Title page the name of the publisher of the Modified Version, as the publisher.
    4. Preserve all the copyright notices of the Document.
    5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.
    6. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.
    7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.
    8. Include an unaltered copy of this License.
    9. Preserve the section Entitled “History”, Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled “History” in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.
    10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the “History” section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.
    11. For any section Entitled “Acknowledgements” or “Dedications”, Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.
    12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.
    13. Delete any section Entitled “Endorsements”. Such a section may not be included in the Modified Version.
    14. Do not retitle any existing section to be Entitled “Endorsements” or to conflict in title with any Invariant Section.
    15. Preserve any Warranty Disclaimers.

    If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.

    You may add a section Entitled “Endorsements”, provided it contains nothing but endorsements of your Modified Version by various parties—for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

    You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

    The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

  6. COMBINING DOCUMENTS

    You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

    The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

    In the combination, you must combine any sections Entitled “History” in the various original documents, forming one section Entitled “History”; likewise combine any sections Entitled “Acknowledgements”, and any sections Entitled “Dedications”. You must delete all sections Entitled “Endorsements.”

  7. COLLECTIONS OF DOCUMENTS

    You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

    You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

  8. AGGREGATION WITH INDEPENDENT WORKS

    A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an “aggregate” if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

    If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

  9. TRANSLATION

    Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

    If a section in the Document is Entitled “Acknowledgements”, “Dedications”, or “History”, the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

  10. TERMINATION

    You may not copy, modify, sublicense, or distribute the Document except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense, or distribute it is void, and will automatically terminate your rights under this License.

    However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.

    Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.

    Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, receipt of a copy of some or all of the same material does not give you any rights to use it.

  11. FUTURE REVISIONS OF THIS LICENSE

    The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

    Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License “or any later version” applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation. If the Document specifies that a proxy can decide which future versions of this License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Document.

  12. RELICENSING

    “Massive Multiauthor Collaboration Site” (or “MMC Site”) means any World Wide Web server that publishes copyrightable works and also provides prominent facilities for anybody to edit those works. A public wiki that anybody can edit is an example of such a server. A “Massive Multiauthor Collaboration” (or “MMC”) contained in the site means any set of copyrightable works thus published on the MMC site.

    “CC-BY-SA” means the Creative Commons Attribution-Share Alike 3.0 license published by Creative Commons Corporation, a not-for-profit corporation with a principal place of business in San Francisco, California, as well as future copyleft versions of that license published by that same organization.

    “Incorporate” means to publish or republish a Document, in whole or in part, as part of another Document.

    An MMC is “eligible for relicensing” if it is licensed under this License, and if all works that were first published under this License somewhere other than this MMC, and subsequently incorporated in whole or in part into the MMC, (1) had no cover texts or invariant sections, and (2) were thus incorporated prior to November 1, 2008.

    The operator of an MMC Site may republish an MMC contained in the site under CC-BY-SA on the same site at any time before August 1, 2009, provided the MMC is eligible for relicensing.

ADDENDUM: How to use this License for your documents

To use this License in a document you have written, include a copy of the License in the document and put the following copyright and license notices just after the title page:

       Copyright (C)  year  your name.
       Permission is granted to copy, distribute and/or modify this document
       under the terms of the GNU Free Documentation License, Version 1.3
       or any later version published by the Free Software Foundation;
       with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
       Texts.  A copy of the license is included in the section entitled ``GNU
       Free Documentation License''.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.” line with this:

         with the Invariant Sections being list their titles, with
         the Front-Cover Texts being list, and with the Back-Cover Texts
         being list.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.


Next: , Previous: Documentation License, Up: Top

Appendix D Bibliography and references

See bash.

Lots of interesting stuff at the following site:

http://www.bash-hackers.org/wiki/


Next: , Previous: references, Up: Top

Appendix E An entry for each concept


Next: , Previous: concept index, Up: Top

Appendix F An entry for each function.


Previous: function index, Up: Top

Appendix G An entry for each variable.

Table of Contents


Footnotes

[1] TCL stands for Tool Command Language and it is a scripting language originally written by John Ousterhout, see: http://www.tcl.tk/.