Salang: The Sawmill Language

Salang is the Sawmill language, which is used throughout Sawmill for many purposes, from displaying pages like this one (or administrative or statistics pages) to writing log filters.

Salang is similar in some ways to perl, and borrows syntactic elements from perl, C, and other languages. Because Salang code is intended to be written inline, rather than as a stand-alone programs, it deviates from perl and C in various ways, including the use of parentheses instead of brackets to group statements and expressions (in this way, it distantly resembles LISP). Still, programmers acquainted with perl or C will find themselves in familiar territory with Salang.

Operators

Quotes and escapes

Single quotes ('), double quotes (") and backticks (`) can be used as quotes. Quotes are "smart"--you can use one type of quotes within another type of quotes, e.g. v = "someone's value". If you need to use the same types of quotes, you can use a backslash, e.g. v = 'someone\'s value'. Backslashes can be used to escape other characters, including dollars signs in values, to prevent them from being treated as variable evaluation operators.

The Configuration Hierarchy, and Configuration Nodes

Configuration files are on-disk representations of nodes. They are rooted in the LogAnalysisInfo directory, so the contents of LogAnalysisInfo is the top of the node hierarchy. Not all files are nodes, however, only these ones are:

Variables and other data are stored in the configuration hierarchy, a collection of data which exists on disk and partially in memory. The configuration hierarchy is a group of "nodes", each with a single value. Any node can also contain other nodes (subnodes). Each node has a type ("bool", "int", "float", "string", "table", "data", or "node"), and a value. Salang expressions can get values from the hierarchy (e.g. "internal.verbose" is an expression which gets the value of the "verbose" subnode of the "internal" node), or set them (e.g. "internal.verbose = 1;").

Configuration nodes are referenced using a dot-separated (.) sequence of names, which functions like a pathname. Configuration nodes are grouped hierchically either using subdirectories, or using .cfg files (cfg stands for "configuration group"). .cfg files have additional hierarchical groupings within them, specified by curly-bracketed groups. For instance, the node name "rewrite_rules" refers to the rewrite_rules directory of LogAnalysisInfo, and the node name "rewrite_rules.server_responses" refers to the server_responses.cfg file, in the rewrite_rules directory of LogAnalysisInfo. The server_responses.cfg file looks like this:

  server_responses = {
    100 = {
      regexp = "100"
      result = "Continue"
    }
    ...
    200 = {
      regexp = "200"
      result = "Successful Transfer"
    }
    ...
  } # server_response

(elipses [...] above indicate places where part of the file has been omitted for brevity), so there is a "100" node inside the server_responses node, and there is also a "200" node there. Therefore, the node name "rewrite_rules.server_responses.100" refers to the 100 node, and "rewrite_rules.server_responses.200" refers to the 200 node. Within the 100 node, there are two nodes, "regexp" and "result"; so "rewrite_rules.server_responses.100.regexp" refers to the regexp node in 100, and "rewrite_rules.server_responses.200.regexp" refers to the regexp node in 200.

Directories are treated equivalently to .cfg files in the configuration hierarchy. In both cases, they represent nodes, and can contain other nodes. In the case of directories, subnodes appear as subdirectories, or as .cfg files in the directory, or as .cfv files in the directory; in the case of .cfg files, subnodes appear within the file using curly-bracket syntax, as above.

As mentioned above, every node has a value. In the example above, the value of rewrite_rules.server_responses.200.regexp is "100", and the value of rewrite_rules.server_responses.100.result is "Continue". Even group nodes like rewrite_rules.server_responses.100, rewrite_rules.server_responses, and rewrite_rules can have values, but their values are typically empty (""); in practice, a node has either a value, or subnodes.

Nodes can be used like variables (they are variable). Nodes can be referred to by name, in log filters or other Salang code; for instance this:

  echo(rewrite_rules.server_responses.100.result)

will print the value "Continue" to the standard output stream. They can also be assigned values:

  rewrite_rules.server_responses.100.result = "Something Else"

Nodes are read from disk into memory automatically when they are accessed, and the in-memory version is used for the remainder of the process, and then discarded. Therefore, it is not generally necessary to read data from disk explicitly; instead, put it in a node and access it by name. In the example above, the value of rewrite_rules.server_responses.100.result will be temporarily changed to "Something Else"; only the in-memory version of rewrite_rules.server_responses.100.result will change, and the change will not be propagated to disk unless save_node() is used to write out the changes back out to disk.

.cvf files (cfv stands for "configuration value") are straight text files whose entire content is considered to be the string value of the node. For example, the file LogAnalysisInfo/somedir/somefile.cfv whose contents is "HELLO" could be referred to as somedir.somefile; the value of that node is "HELLO".

The node built-in type of Salang refers to a configuration node; it is a pointer to that node. You can get the value of the node using node_value(). You can also get a particular subnode using subnode_by_name(). You can check for the existence of a subnode using subnode_exists() (or use node_exists() with the full nodename as the parameter).

In addition to the LogAnalysisInfo directory, Salang also looks in other "search paths" to find nodes specified by name:

• If the node name begins with lang_, it will search for it in LogAnalysisInfo/language/{current} where {current} is the current language. For instance, if the current language is "english", then the node "lang_stats.menu.groups.date_time_group" refers to the date_time_group node, in the groups node, in the menu node, in the LogAnalysisInfo/languages/english/lang_stats.cfg.

• If a profile is active (e.g., was specified with -p on the command line), it will search for the node in the profile .cfg file. For instance, if Sawmill is called with -p myprofile, then the node "database.fields.date_time" refers to the date_time database field, and "database_fields.date_time.label" refers to the label fo the date_time database field.

• In log filters, if the node name is exactly the name of a log field, then the log field value is used. This is not actually a node, but functions as one for the purpose of reading and writing its value; you can refer to it like a normal Salang variable in most cases (e.g., "file_type = 'GIF'").

• If a local variable has been declared in Salang, it can be referred to by name. For instance, you can write:

      int i;
      i = 3;
      echo(i);

  to define and refer to a local variable node i. Local variables do not have full nodenames; they are not "rooted" in LogAnalysisInfo like most other nodes. They are free-floating in memory, and cannot be saved to disk; they can only be referred to by their short nodenames (e.g., "i").

• Inside subroutines, the subroutine parameters can be referred to by name. For instance, the following subroutine, which adds two numbers, refers to its parameters x and y directly:

    subroutine(add(int x, int y), (
      x + y
    ));

Subroutine parameters are similar to local variables (above) in that they are not rooted, and cannot be written to disk.

• The special variables $0, $1, $2, etc. refer to the matched subexpressions of the last call to matches_regular_expression().

Type data

Nodes of type data represent chunks of binary data in memory. They can be assigned to and from strings, if they do not contain null characters. They are currently used only in networking, to read data from a socket, or write it to a socket.

Report Filter Syntax

Filters used in reports take a special variant syntax that allows only certain operations. Subroutines (described below) are not allowed, and only database field names are allowed as variables. Only strings are allowed as constants. The <, >, <=, and => operators are permitted for the date_time fields and numerical fields only. The "within", "matches", and "matches_regexp" operators are permitted for any field. Expressions can be combined using "and", "or", and "not"; arbitrary parentheses are permitted to allow any combinations. No other syntax is permitted.

Log Filter Syntax

Log filters can use all syntax described on this page, and also support a few extra variables. Specifically, log filters can refer to log fields by name, so a reference to date_time in a log filter is a reference to the value of the date_time field in the log entry that is currently being processed. This can be used either to get or set values; e.g. "if (page eq '/index.html') then 'reject'" checks the current log entry's page field to see if it's "/index.html", and rejects the log entry if it is; and "page = '/index.html'" sets the page field of the current log entry to "/index.html".

Log filters can also use the special function current_log_line(), whose value is the entire current line of log data.

Types

Each configuration node has a type, which specifies how its data is stored internally. Possible types are:

• string: The data is an arbitrary string value like "hello" or "Bob".

• int: The data is an integer value like 15 or 49.

• float: The data is a floating point value like 1.4 or 18874.46567.

• bool: The data is a boolean value (true or false).

• node: The data is a reference (pointer) to a node in the configuration hierarchy.

Types are primarily useful for performance optimization. Salang is a weakly typed language, so it will allow any value to be assigned to any variable without warning or complaint; for instance assigning 134 to a string variable will result in "134" in the string variable, or assigning "134.5" to a float variable will result in the floating point value of 134.5 in the float variable. However, these types of conversions can be slow if they are performed many times, so if you know a variable is only going to hold and manipulate floating point values, you should use a float type rather than a string type.

The node type is particularly useful for performance optimization, since it prevents the need for node lookups; e.g. setting the value of a node explicitly with "a.b.c.d = 0;" requires a series of expensive lookups, as node "a" is looked up in the configuration root, and then "b" inside that, and then "c", and then "d"-- but if a node variable N already points to a.b.c.d, then "N = 0;" is a very fast operation that does the same thing. node variables are particularly useful for foreach loops, and functions like subnode_value(), where they can be used to iterate over all subnodes without requiring any expensive string-to-node lookups.

Statements

if A then B else C
This statement evaluates the A section; if the value of A is true, then the value of the entire statement is the B section; otherwise, the value of the statement is the C section.

subroutine(A(param1, param2, ..., paramN), B)
This statement defines a subroutine A with N parameters. The subroutine can be called with the statement "A(p1, p2, ..., pN)". The value of the subroutine call will be the value of B, with the parameters p1...pN plugged into the variables $param1..$paramN before B is evaluated. The value of a subroutine declaration is empty.

foreach I N B
This statement is used to iterator over the subnodes of a particular node in the Salang hierarchy. It iterates over all values of node N, setting I to the full nodepath each iteration, and then evaluating expression B. The value of this expression is the concatenation of the values of all the evaluations of B.

for (I; C; E)
This repeats an expression 0 or more times. The expression I is evaluated to initialize the loop. The expression C is evaluated, before each iteration, and if the result is false, then the iteration does not complete. If C is true, the E is evaluated. The value of this expression is the concatenation of the E values.

while (C) E;
This repeats an expression 0 or more times. The expression C is evaluated before each iteration, and if the result is true, then E is evaluated. This continues until C is false. The value of this expression is the concatenation of the E values.

next
This statement goes immediately to the next iteration of the immediately enclosing loop.

last
This statement immediately terminates execution of the immediately enclosing loop, and continues execution after the end of the loop.

Built-in subroutines

get_file_type_from_url(string U)
The value of this statement is the file extension of the URL U; e.g. get_file_type_from_url("http://something/file.gif") is "GIF".

get_log_field(string N)
Returns the value of the log field N, of the current log entry.

node_exists(string N)
Returns true if the node specified by the nodename N exists; false if the node does not exist.

node_name(node N)
Returns the name of the node N.

num_subnodes(node N)
Returns the number of subnodes of the node specified by nodename N.

subnode_by_number(node N, int I)
Returns the Ith subnode of node N.

subnode_by_name(node N, string M)
Returns the subnode of node N whose name is M. If there is no subnode by that name, it creates one.

subnode_exists(node N, string M)
Returns true if there is a subnode named M in node N, false otherwise.

set_subnode_value(node N, string M, anytype V)
Sets the subnode named M of node N to the value V. Creates the subnode if it does not exist.

node_value(node N)
Returns the value of node N.

set_node_value(node N, anytype V)
Sets the value of node N to V; no value is returned.

node_type(node N)
Returns the type of node N as a string (e.g. "int", "float", "bool", "string", or "node").

set_node_type(node N, string T)
Sets the type of node N to T ("int", "float", "bool", "string", or "node").

delete_node(node N)
Deletes the node specified by nodename N. If the node is a profile, this also deletes the profile file.

insert_node(node destnode, string sourcenode, int I)
Inserts the node specified by nodename sourcenode into the node specified by nodename destnode, so that sourcenode ends up as the Ith subnode of destnode (i.e. it inserts sourcenode into destnode at position I). sourcenode is removed from its current location, and moved to the new one, so for instance if sourcenode is a subnode of destnode before this is called, it will no longer be a subnode of destnode afterwards. I.e. it moves sourcenode, rather than copying it.

clone_node(node original_node, string clone_nodepath)
This makes a clone of the node original_node, and puts it at the node specified by clone_nodepath (which is created if it doesn't exist). If no second (clone_nodepath) parameter is specified, this returns the clone node. The original node is unchanged, and after completion, the clone is an exact replicate of the original, except the name (which is computed from clone_nodepath). This returns an empty value if clone_nodepath is specified, or the clone node clone_nodepath is not specified.

overlay_node(node node1, node node2)
This overlays node2 on node1. After it's done, node1 will have all the subitems it had before, plus any that were in node2. If there are subnodes in both, the one from node2 will replace the original one.

new_node()
This creates a new node, and returns it. The node is unrooted; it has no parent.

delete_node(node n)
This deletes the node n (frees the memory is uses). The node should no longer be used after this call.

rename_node(node thenode, string newname)
This renames the node thenode, so its name is newname.

node_parent(node thenode)
This returns the parent node of thenode, or NULL if thenode has no parent.

add_subnode(node parent, node subnode)
This add subnode as a subnode of parent.

set_language(string language)
This sets the current language to language, so future accesses to lang_stats, lang_admin, etc. in this process will use the language modules of the specified language (from LogAnalysisInfo/languages/language).

sort(node N, string M, bool E)
Sorts the subnodes of the node specified by nodename N. The subnodes are sorted in an order specified by method M. M is a string of the format "field:F,T,A", where F is a field name (the name of a subnode found in each subnode of N; use "value" to use the value of each subnode of N directly), T is "integer", "float", "alphabetical", or "chronological" (which determines the sort type), and A is "ascending" or "descending" (which determine the direction of sort). If E is true, then variable are expanded before sorting; if E is false, then the sort is done on the literal values, without variable expansion.

create_profile(string N)
This creates a new profile with name N. It pulls various variables from the node hierarchy to set up the profile, the nodes are set by the "create profile" interview.

format(anytype V, string T)
This formats the value V according to the format type T. T can be "integer", "page", "float", "two_digit_fixed", "bandwidth", "megabytes", "gigabytes," "bytes_per_second", "date_time", "hostname", "duration", "duration_compact", "duration_milliseconds", or "duration_microseconds". If the T starts with "strftime:", then the remainder of T will be used as a strftime()-style format specifying, and the value V will be considered an EPOC date/time value, and the result will be a strftime-formatted date/time. If the first character of T is a % character, the result will be formatted as a double-precision floating point number using printf-style formatting, e.g. '%.1f' will print the value in floating point format with one digit after the decimal place. If the first tho characters of T are a curly bracket and an equal, then the substring of T from the third character until the closing equal-curly will be considered a Salang expression, and evaluated to compute the formatted value; the variable "v" may be referenced from within this expression to get the value of V. If T is "date_time", V must be in the format "dd/mmm/yyyy hh:mm:ss" (any field may be substituted with underbars, e.g. "dd/mmm/yyyy hh:__:__", to represent a unit, an hour in this case). The format "dd/mmm/yyyy" is also supported as a synonym for "dd/mmm/yyyy __:__:__". If T is "epoc_to_date_time", V will be considered an EPOC format timestamp (seconds since 1970), and will be converted to a date/time in the format "dd/mmm/yyyy hh:mm:ss". If T is "epoc_to_date", V will be considered an EPOC format timestamp (seconds since 1970), and will be converted to a date in the format "dd/mmm/yyyy" (ignoring the seconds). If T is "epoc_to_time", V will be considered an EPOC format timestamp (seconds since 1970), and will be converted to a time in the format "hh:mm:ss" (ignoring the date). If T is "node_name", the value returned will be V, with all letters converted to lowercase, and all non-letter, non-digit characters converted to "_" (providing a string suitable for use as a conventional node name).

Some types of encryption are also supported through the format type T "!Encryption:algorithm:direction". When this is specified for T, the algorithm can be "Base64" (decryption only), and the direction can be "encrypt" or "decrypt".

image(string N, int W, int H)
The value of this subroutine is an HTML image tag which displays the image specified by filename N, with width W and height H. The image file should be in the temporary directory in CGI mode, or in the picts folder, which is in the WebServeRoot folder of the LogAnalysisInfo folder, in web server mode.

create_image(int width, int height)
This creates a new image canvas of the specified image and height, and returns the image ID for use in drawing functions. It returns the image ID for use in drawing functions.

allocate_image_color(string imageid, int red, int green, int blue)
This allocates a new color for use in drawing on image imageid (created by create_image()), with the specified red, green, and blue components (0 to 255). It returns the color value for use in drawing functions.

add_text_to_image(string imageid, string color, int x, int y, string text, string direction)
This draws text on image imageid (created by create_image()), with the specified color (created with allocate_image_color()) at the specified x and y position, in the direction specified by direction ("up" or "right"). The upper left corner of the text will be anchored at (x, y) when drawing horizontally; the lower left corner will be anchored there when drawing vertically.

add_line_to_image(string imageid, string color, int x1, int y1, int x2, int y2)
This draws a line on image imageid (created by create_image()), with the specified color (created with allocate_image_color()) from the point (x1, y1) to the point (x2, y2).

add_polygon_to_image(string imageid, string color, node points, bool filled)
This draws a polygon on image imageid (created by create_image()), with the specified color (created with allocate_image_color()). The vertices of the polygon are specified in order in the subnodes of points; each subnode must have and x and a y value specifying the location of that point. The polygon will be a single pixel border if filled is false, or filled if it is true.

add_rectangle_to_image(string imageid, string color, int xmin, int ymin, int xmax, int ymax, bool filled)
This draws a rectangle on image imageid (created by create_image()), with the specified color (created with allocate_image_color()) and minimum x/y dimensions. The rectangle will be a single pixel border if filled is false, or filled if it is true.

write_image_to_disk(string imageid, string pathname, string format)
This writes the image imageid (created by create_image()) to disk as an image file (GIF or PNG, as specified by format, whose value can be "GIF" or "PNG"), to the location specified by pathname.

int read_image_from_disk(string pathname)
This reads an image from disk, from the file specified by pathname. The format, which may be PNG or GIF, is inferred from the filename extension. This returns the image ID of the image read.

int get_image_width(string imageid)
This returns the width, in pixels, of the image imageid.

int get_image_height(string imageid)
This returns the height, in pixels, of the image imageid.

int get_image_pixel(string imageid, int x, int y)
This returns the pixel at position x, y, in the image imageid. The pixel is represented as a 32-bit integer: top 8 bits are 0; next 8 bits represent red; next 8 bit represent green, and next 8 bits represent blue.

int get_image_pixel_transparency(string imageid, int x, int y)
This returns the transparency of the pixel at position x, y, in the image imageid. The returned value is between 0 (transparent) and 255 (opaque).

img_src(string N)
The value of this subroutine is the value of an HTML src section of an image tag; i.e. it's intended to be used right after src= in an image tag. The image file should be in the temporary directory in CGI more, or in the picts folder, which is in the WebServeRoot folder of the LogAnalysisInfo folder, in web server mode.

fileref(string F)
This converts a local filename from the WebServerRoot directory to a URL that refers to that file. The filename should be in partial URL syntax, e.g. "picts/somefile.gif" to refer to the file somefile.gif in the picts directory of WebServerRoot. If necessary, it may copy the file (e.g. in CGI mode) to the temporary directory. All references to files in WebServerRoot should be done through this function, to ensure correct functioning in CGI mode.

save_changes()
This causes any changes made to the configuration hierarchy since the last save to be saved to the disk version of the hierarchy.

save_node(node N)
This saves node N to disk. For instance, if a N is "somenode.somechild.somevar", this will be saved to a file called somevar.cfg, in the somechild directory of the somenode directory of the LogAnalysisInfo directory. The format used is normal configuration format. This makes the node persistent, because any future access to that node (e.g. referring to somenode.somechild.somevar in an expression), even from a later process after the current one has exited, will automatically load the node value from disk. So by using save_node(), you can ensure that any changes made to that node will be a permanent part of the configuration hierarchy, and any values you have set will be available for use to all future processes and threads.

logout()
This logs the current user out (clearing the cookies that maintained their login), and takes them back to the login page (or the logout URL, if one is specified).

expand(string M)
This expands variables and expressions in M, and returns the expanded value. For instance, expand("$x") will return the value of the variable x, and expand("$x > $y") will return "12 > 10", if x is 12 and y is 10.

convert_escapes(string M)
This converts percent-sign escape sequences in M (e.g. converting %2520 to a space), and returns the converted value. For instance, convert_escapes("some%2520string") will return "some string".

set_active_profile(string profile_name)
This sets the active profile to profile_name. The specified profile will be searched when looking up variables; e.g. the variable database.fields.date_time.label (which would normally be invalid, if no profile were active) will be treated as a local nodename within the specified profile, and be resolved as though it were profiles.profile_name.database.fields.date_time.label.

debug_message(string M)
This converts variables and expressions in M, and displays the resulting value to the standard debug stream (i.e. to the console). This is useful for debugging Salang code.

echo(string M)
This outputs the string M (without any conversion) directly to the standard output stream. This is useful for debugging Salang code.

error(string M, bool skip_escaping)
This throws an error exception, with error message M. The second parameter, skip_escaping is optional, but if specified and true, will prevent the escaping of the error message for HTML, passing it through literally (if unspecified or false, escaping will occur). The error will be reported through the standard Sawmill error reporting mechanism.

node_as_string(node N, [bool show_types])
This converts the node N as a string value (as you would see it in a profile file). The value of this expression is the string value of N. The show_types parameter specifies whether types should be shown for each node; the default is false.

node string_to_node(string s)
This converts the string s to a node, and returns the node. This is similar to writing the string to a .cfg file, and reading it in as a node. For instance, string_to_node("x = { a = 1 b = 2 }") will return a node whose name is x, which has two subnodes a and b, with values 1 and 2.

autodetect_log_format(node log_source, string result, string id)
This autodetects the log format from the log source log_source, and puts the result in the node specified by result. id is an identifier which appears in the task info node for this process, allowing another process to tap into the progress for this process. This is used internally by the profile creation wizard.

get_directory_contents(string P, string R)
This gets the contents of the directory at pathname P, and puts the result in the node specified by R. R will contain a subnode for each file or subdirectory, and within each subnode there will be a name node listing the name of the file or directory and an is_directory node which is true for a directory or false for a file; there will also be a size node if it's a file, listing the file size in bytes. If is_directory is true, there will be an is_empty value indicating if the directory is empty, and a has_subdirectories option indicating if the directory has subdirectories.

get_file_info(string P, string R)
This gets information about the file or directory at pathname P, and puts the result in the node specified by R. R.exists will be true or false depending on whether P exists; R.parent will be the full pathname of the parent of P; R.type will be "file" or "directory" depending on whether P is a file or directory; R.filename will be the filename or directory name of P. R.modification_time will be the EPOC time of the last modification of P. R.size will be the size of P in bytes.

file_exists(string P)
The value of this is true or false, depending on whether the file or directory at pathname P exists.

string create_file_lock(string pathname)
This creates a file lock on the file whose pathname is pathname. This does not actually acquire the lock; to acquire the lock, use acquire_file_lock().

string acquire_file_lock(string file_lock)
This acquires a lock on the file specified by file_lock, which is a file locking object returned by create_file_lock(). This function will wait until the file is not locked, then it will atomically acquire the lock, and then it will return.

string release_file_lock(string file_lock)
This releases a lock on the file specified by file_lock, which is a file locking object returned by create_file_lock(). The lock must have been previously acquired with acquire_file_lock().

string check_file_lock(string pathname)
This checks if the file whose pathname is pathname is locked. If the file is locked, this returns true; otherwise, it returns false. This returns immediately; it does not block or wait for the lock to be released. To block until the lock is released, and then acquire it, use acquire_file_lock().

delete_file(string pathname)
This deletes the file whose location is specified by pathname.

move_file(string source_pathname, string destination_pathname)
This moves the file or directory whose location is specified by source_pathname to the location specified by destination_pathname.

copy_files(string source_pathname, string destination_pathname)
This copies the file or directory whose location is specified by source_pathname to the location specified by destination_pathname. If it is a directory and contains other files or subdirectories, all will be copied recursively.

delete_directory(string pathname)
This deletes the directory whose location is specified by pathname, including all subdirectories and files.

get_files_matching_log_source(node L, string R)
This gets a list of files matching the log source in node L, and puts the result in the node specified by R. R will contain a subnode for each matching file; the value of the subnode will be the filename. This is used by the interface to implement Show Matching Files.

length(string S)
The value of this expression is the length of the string S.

substr(string V, int S, int L)
The value of this expression is the substring of the string V, starting at index S and of length L. The L parameter is optional, and if it is omitted, the value of the expression is the substring of V starting at S and continuing to the end of V.

split(string s, string divider, string resultnode)
This splits the string s on the divider specified in divider, and puts the resulting sections into the node specified by resultnode. For instance, split("Hello,you,there", ",", "volatile.splitresult") will set volatile.splitresult.0 to "Hello", volatile.splitresult.1 to "you", and volatile.splitresult.2 to "there".

starts_with(string S, string T)
The value of this expression is true if the string S starts with the value of the string T.

ends_with(string S, string T)
The value of this expression is true if the string S ends with the value of the string T.

contains(string S, string T)
The value of this expression is true if the string S contains the value of the string T.

replace_all(string S, string T, string R)
The value of this expression is the value of S after all occurrences of T have been replaced with R.

replace_first(string S, string T, string R)
The value of this expression is the value of S after the first occurrence of T has been replaced with R. If T does not occur in S, the value of this expression is S.

replace_last(string S, string T, string R)
The value of this expression is the value of S after the last occurrence of T has been replaced with R. If T does not occur in S, the value of this expression is S.

lowercase(string S)
The value of this expression is the value of S after all uppercase letters have been converted to lowercase.

uppercase(string S)
The value of this expression is the value of S after all lowercase letters have been converted to uppercase.

convert_base(string value, int frombase, int tobase)
This converts the string value from the base frombase to the base tobase, i.e., it treats it as a integer in base frombase, and returns a string which represents an integer in base tobase. It returns the converted version of value. Currently, this only supports base 16 (hexadecimal) to base 10 conversion, and base 10 to base 16 conversion.

convert_charset(string value, string fromcharset, string tocharset)
This converts the string value from the charset fromcharset to the charset tocharset. It returns the converted version of value. Charset names are as documented for the GNU iconv conversion utility.

matches_regular_expression(string S, string R)
The value of this expression is true if the string S matches the regular expression R. If it matches, the variables $0, $1, $2, ... are set to the substrings of S which match the parenthesized subexpressions RE.

matches_wildcard_expression(string str, string wildcardexp)
The value of this expression is true if the string str matches the wildcard expression wildcardexp.

index(string S, string T, int start)
The value of this expression is the index (character position) of the substring T in the string S. The third parameter is optional; if it is specified, the search begins at index start. If T is not a substring of S, the value of this expression is -1.

last_index(string S, string T)
The value of this expression is the index (character position) of the final occurrence of substring T in the string S. If T is not a substring of S, the value of this expression is -1.

md5_digest(string S)
The value of this expression is the MD5 digest of the string S, as a 32-digit hexadecimal number.

get_license_info(node licensenode)
This looks at the Sawmill license key contained in licensenode, and extracts information about it, which returns it as a node. The subnodes of the result are type, valid, valid64, valid65, addon, expiration, profiles, and users.

node license_string_to_license_node(string licensestring)
This takes a license string, and returns a node version of the license, suitable for installing in licenses.cfg. If the license string is a sublicense (starts with "sublicense"), a sublicense node is returned, suitable for calling add_sublicense().

node add_sublicense(node sublicense, node license)
This add s a sublicense to an existing license. The parameter sublicense is the sublicense node returned by an earlier call to license_string_to_license_node(), with a "sublicense*" license string. The parameter license is the existing, installed license node (probably just licenses.license). The return value of add is a node; it is the license parameter node with the sublicense added. It can be checked for validity by looking at the "type" subnode, which will be "invalid" if the sublicense did not match the license. If it's valid, this return value can be installed directly in licenses.cfg.

node check_licensing()
This checks the licensing, and returns a node with licensing information. It is used internally for license validation.

authenticate(string username, string password, bool verify_only)
This attempts to authenticated a user username with password password. It verify_only is true, it only verifies the validity of the username and password, returning true if they are valid or false otherwise. If verify-only is false, it verifies the validity of the username and password, and if they are valid, logs them in.

string create_trial_license()
This creates and returns a 30-day trial license key. Only one trial key can be generated per installation; after the first time, it will return an empty string.

display_statistics_filters(node F, string D)
The value of this expression is HTML which describes the Filters in node F, for database field D. This is used internally by the statistics interface to display Filters.

query_one_db_item(node F, string R)
The queries the numerical database field totals for the Filters F, and puts the result in the node R.

query_db_for_report(node report, string result)
The queries the database for statistics report report, and puts the result in result. The format of result depends on the type of the report report. Also, if volatile.csv_export is true, this computes a CSV version of the query result, and puts it in volatile.csv_export_result. For log detail, if return_all_log_detail_rows is true, or if the table is sorted, this returns all rows of the table; otherwise it returns just the visible rows of log detail (for other tables, it already returns all rows).

query_db_calendar(string R)
This queries the database to compute calendar information (which days are in the database), and puts the result in R. R contains a numerical year node for each year, and within that a numerical month node for each month, and within that a numerical day node for each day in the database.

create_table(string name, node header)
This creates a new table object. The name of the new table is name, and the columns are described in header. Column subnode values include:

• database_field_name: the name of the database field for this column, if any.

• table_field_type: the type of the field, which is one of the following:

  • custom_itemnum: a field capable of holding arbitrary string values, with automatic normalization internaly to integers.
  • database_itemnum: a field containing the integer item number for a database field
  • int8: an 8-bit integer field.
  • int16: a 16-bit integer field.
  • int32: a 32-bit integer field.
  • int64: a 64-bit integer field.
  • int: an integer field using the maximum efficient number of bits for the platform.
  • float: a floating point field.
  • set: a field containing a set of integers.

The table created is a SSQL table, available for querying with SSQL.

load_table(string name)
This gets a table variable from a table which exists on disk (previously created with create_table() or query_db_for_view()). This returns the table variable.

unload_table(string table)
This frees memory usage, and closes files, used by the table object table, which was loaded by an earlier call to load_table().

node get_table_header(table t)
This returns the header node for the table t.

string get_table_name(table t)
This returns the name of the table t.

table_get_num_rows(table t)
This returns the number of rows in the table t.

table_get_cell_value(table t, int rownum, int colnum)
This returns the value in the cell at row number rownum and column number colnum in the table t.

table_get_cell_string_value(table t, int rownum, int colnum)
This returns the value in the cell at row number rownum and column number colnum in the table t, as a string. If it is an itemnum cell, it resolves the itemnum, returning its string value.

table_set_cell_value(table t, int rownum, int colnum, (int|float) value)
This sets the value in the cell at row number rownum and column number colnum in the table t, to the value value. For best performance, the type of value should match the type of the table column.

table_set_cell_string_value(table t, int rownum, int colnum, string value)
This sets the value in the cell at row number rownum and column number colnum in the table t, as a string. The cell must be an itemnum cell. It converts value to an itemnum, and puts that value in the table.

ssql_query(string query)
This performs a SSQL query. SSQL is a limited subset of SQL; see SSQL: Sawmill Structured Query Language (SQL). This returns the table result of the query, or the empty string ("") if the query does not return a result.

get_db_field_hierarchy(string dbfieldname, string nodename)
This get the hierarchy of the database field dbfieldname, and puts the result in nodename. The values and values of nodes in nodename match the itemnums of items for that database field, and the hierarchical structure of nodename exactly matches the structure of the field.

evaluate_expressions_in_table(table t, node query_info)
This iterates through all rows of table t, and executes the nodes specified by query_info on its rows, setting the rows to the values of the expressions. "query_info" must have a "header" subnode, and "header" must have a subnode for each column of the table, and may have an "expression" subnode for any of the subnodes of expressions. "query_info" must also have a number_of_days value, to be used as the number of days in the event that an expression refers to number_of_days. For any subnode of header which has an "expression" subnode, the expression will be run against each row, and the corresponding cell will be set to the result of the expression.

db_item_has_subitems(string D, string I)
The checks whether item I of database field D has subitems in the current database. It returns true if it does, and false if it isn't (i.e. if it's a bottom-level item).

database_sql_query(string query, bool temporary, bool update_tables)
This performs a SQL query of the current profile's database. This differs from ssql_query() in that ssql_query() always queries the internal SSQL database (which always exists, mostly for internal bookkeeping tables, even if a profile uses an external database), but this function queries the profiles main database. For instance, if you're using an Oracle database with the profile, then, this will query the Oracle database for the current profile, not the internal SSQL database. The result will be returned as a SSQL table. The table will be a temporary table (deleted when the process completes) if temporary is true. If update_tables is true, the query will also be scanned for table names, and all xref tables, hierarchy tables, and session tables found will be updated with the latest data in the main table, before the query is run.

database_table_exists(string table)
This checks the existence of a table in the current profile's database. It returns true if the table exists, or false if it does not.

ssql_query_upload_to_database(string query, string table_name)
This performs a SSQL query on the SSQL database (which is not the profile's database, unless the profile uses the internal SQL database). It uploads the result of the query to the profile's internal database, putting it in the table specified by table_name. This basically does the opposite of database_sql_query(); where database_sql_query() queries the profile's SQL database and writes it to an internal database table, this queries the internal SSQL database, and writes it to a table in the profile's SQL database. The two can be used together to download a table from the SQL database, edit it, and upload it.

database_item_to_itemnum(int dbfieldnum, string item)
This returns the internal item number of item item of database field dbfieldnum. If there is no item by that name, it returns 0.

database_itemnum_to_item(int dbfieldnum, int itemnum)
This returns the item value associated with the interal item number itemnum of database field dbfieldnum.

set_log_field(string N, string V)
The sets the value of the log field N of the current log entry to V.

include(node N)
This loads and processes the Salang code in the node specified by nodename N.

discard(anytype V)
The value of this expression is always empty (""). This is useful if you want to evaluate an expression V, but not use its value.

capitalize(string V)
This capitalizes the value V, using the capitalization rules in the language module.

pluralizes(string V)
This pluralizes the value V, using the pluralization rules in the language module.

char_to_ascii(string c)
This returns the integer ASCII code of the character c. c should be a one-character string.

ascii_to_char(int i)
This returns a one-character string containing the ASCII character whose code is i.

ip_address_to_int(string ipaddr)
This converts the IP address ipaddr to an integer (a 32-bit representation with 8 bits per octet).

dns_resolve_hostname_to_ip_address(string hostname)
This resolves hostname using DNS, and returns the IP address.

convert_field_map(string F, string M)
This converts the value of the log field F in the current log line we're processing, using the map M. M is a |-divided list of A->B mappings; e.g. if M is "1->cat|2->dog|3->ferret", then a field value "2" will be converted to "dog".

collect_fields_using_regexp(string R, string F)
This matches the current log line we're processing against the regular expression R, and if it matches, it extracts the parenthesized values in R and puts them in the fields specified by F. F is a comma-separated list of fields, and should include *KEY*, which specifies the key of the collected log entry to modify. E.g. if F is "page,*KEY*,date_time,host", the second parenthesized subexpression will be used as the key, and that key will specified which collected entry to modify; then the page, date_time, and host fields of that collected entry will be set to the first, third, and fourth parenthesized sections.

collect_listed_fields_using_regexp(string regexp, string divider, string separator, string field_names_map)
This matches the current log line we're processing against the regular expression regexp, and if it matches, it uses the first parenthesized section in regexp and uses that as the key, and uses the second parenthesized section and uses it as the name/values list. Them it uses that, and its other parameters, to do the same as collected_listed_fields().

collect_listed_fields(string key, string name_values_list, string divider, string separator, string field_names_map)

This extracts log field values from name_values_list, which is a list of name/values pairs, and puts them in the collected entry specified by key. Names and values are listed in name_value_list the format "name1separatorvalue1dividername2separatorvalue2dividername3separatorvalue3", e.g. pair are separate from each other by the value of divider, and each name is separate from its value by the value of separator. In addition, field_names_map can be used to convert field names; field_names_map is a pipe-separated (|-separated) list of values like "fieldname=newfieldname", and if any extracted field matches a fieldname value from field_names_map, it will be put in the newfieldname log field. If there is no map, or if nothing matches, then values will be put in the log field specified by the field name.

accept_collected_entry_using_regexp(string R, bool carryover)
This matches the current log line we're processing against the regular expression R, and if it matches, it extracts the first parenthesized value in R; using that value as a key field, it accepts the log entry corresponding to that key (as created by collect_fields_using_regexp()) into the database. If the carryover parameter is true, the keyed entry will remain in memory, and can be collected to, and accepted again; if it is false, the keyed entry will be removed from memory.

accept_collected_entry(string key, bool carryover)
This accepts accepts the log entry corresponding to the key key (as created by set_collected_field(), or collect_listed_fields(), or collect_fields_using_regexp(), or collect_listed_fields_using_regexp()) into the database. If the carryover parameter is true, the keyed entry will remain in memory, and can be collected to, and accepted again; if it is false, the keyed entry will be removed from memory.

set_collected_field(string key, string log_field_name, string set_value)
This sets the collected field log_field_name, in the collected log entry specified by key, to the value set_value.

get_collected_field(string key, string log_field_name)
This returns the value of the collected field log_field_name, in the collected log entry specified by key.

rekey_collected_entry(string F, string T)
This changes the key of the collected entry with key F so its key is T instead.

query_geoip(string ip, string querytype)
This queries the geographic database for the IP address ip. if querytype is "range", this returns a string of the format "ip1-ip2", where ip1 and ip2 are the beginning and end of the IP range in the database containing ip. If querytype is "country/region/city", this returns a string of the format "country/region/city", where country, region, and city are the names of the country, region and city of ip. If querytype is "ISP", this returns the ISP associated with ip, as looked up from the GeoIP ISP database (available for separate purchase from MaxMind). If querytype is "domain", this returns the domain associated with ip, as looked up from the GeoIP Domain database (available for separate purchase from MaxMind). If querytype is "organization", this returns the organization associated with ip, as looked up from the GeoIP Organization database (available for separate purchase from MaxMind).

get_file_type_from_url(string url)
This returns the file type (e.g., "GIF") from url. For instance, if url is "http://mysite.com/this.pdf", this will return "PDF".

get_worm_from_url(string url)
This returns the worm corresponding to a URL url, based on the rules in worms.cfg.

get_token()
This returns a semi-random string token.

generate_report_id(string P, string A)
This creates a new report ID, which can be used to generate a report with generate_report(). P is the profile name, and A is a list of any configuration options that need to be changed from the defaults in the report. The value of this function is the generated report ID.

generate_report(string P, string I)
This begins generates the report with id I, for profile P. Generation occurs in a separate task, so this returns immediately, while report generation continues in the background. Calls to the functions below can tap into the status of the report, and the final generated report. The value of this function is empty.

get_progress_info(string taskid, string resultnode)
This gets progress information for a running task with task ID taskid. It populates the node specified by resultnode with detailed progress information. If taskid is not a valid task or there is no progress available for that task, resultnode will have a subnode exists with value false. The value of this function is empty.

cached_report_exists(string P, string I)
This checks if the report from profile P with id I has been completely generated, and is now cached. The value of this function is true if the report is cached, and falsed if it is not cached (never generated or generation is in progress).

display_cached_report(string P, string I)
This displays the cached report from profile P with id I. This will fail if the report is not cached-- call cached_report_exists() to check. The value of this function is the complete HTML page of the report.

delete_cached_report(string P, string I)
This deletes a cached report from profile P with id I. Future calls to cached_report_exists() will be false until this is regenerated with generate_report(). The value of this expression is empty.

verify_database_server_connection(node server_info)
This verifies that a connection can be made to the database server specified by server_info. server_info contains subnodes type (mysql, odbc_mssql, or odbc_oracle), dsn, hostname, username, password, and database_name, which specify how to connect to the server. This function returns true if the connection succeeds, or flase otherwise.

get_database_info(string P, bool get_date_info)
This gets various information about the database for profile P, and returns it as a node. If get_date_info is true, this includes date range parameters earliest_date_time and latest_date_time. Including these parameters requires hierarchy tables and xref tables to be up to date, so it can be a very slow operation for a large database (if the tables are not up to date), so use false if the date range information is not required. The return node also includes a subnode date_info_requires_complex_query, which is true if a get_date_info=true would require a complex query; this can be used to determine if a call to get_date_info() to get date range information will be slow. The returned node should be deallocated with delete_node() after use.

build_database(string P, string R)
This builds the database for profile P. If the database build is occurring as part of an attempt to view a report, the report ID should go in R; otherwise, R should be empty ("").

update_database(string P, string R)
This updates the database for profile P. If the database build is occurring as part of an attempt to view a report, the report ID should go in R; otherwise, R should be empty ("").

erase_database(string profile)
This erases the database of the current profile, if there is one; otherwise, erases the database of the profile with name profile.

exec(string executable, node options, bool wait)
This runs the command-line program specified by executable, from the command line. If the executable option is an empty string, the main program executable is run (e.g. Sawmill runs itself from the command line). The options node contains the command line options, e.g. for each subnode of the options node, the value of that option (not the name, which is ignored) is a command line option. The wait option specified whether to wait for completion; if it is true, exec() will not return until the command is done; if it is false, exec() will return immediately, leaving the command running in the background. If the wait parameter is false, the return value is the PID of the process; if the wait parameter is true, the return value is the return value of the spawned process.

get_task_info()
This gets various information about currently active tasks, and returns it as a node. The returned node should be deallocated with delete_node() after use.

current_task_id(string current_task_id)
This returns the Task ID (PID) of the current task.

cancel_task(string task_id)
This cancels an active task with ID task_id. When this returns, the task has been cancelled.

run_task(node task_node)
This runs a task at the node specified by task_node. Typically, task_node would be a subnode of the "schedule" node, and this would runs that scheduled task immediately. This return a node with information about the return values of the actions in the task; each subnode corresponds to a task, and contains a retval subnode whose value is the return code of the tasks.

sleep_milliseconds(int M)
This delays for M milliseconds before evaluating the next expression.

save_session_changes()
This saves any changes to the configuration hierarchy to the sessions file, so they will carry over to future pages.

int connect(string hostname, int port, bool ssl)
Connect via TCP to the specified hostname and port; return the socket. If the optional ssl parameter is true, the socket uses SSL.

disconnect(int socket)
Disconnect from the specified socket.

read_from_socket(int socket, data d, int length)
Read length bytes from socket, and put them in d.

write_to_socket(int socket, data d, int length)
Write length bytes to socket, from the start of d.

open_file(pathname, mode)
This opens the file at pathname pathname, using the "C" style mode specifier "mode". Possible modes include "r" to open a file for reading at the beginning, "r+" to open a file for reading and writing at the beginning, "w" to create the file (deleting if it exists) for writing, "w+" to create the file (deleting if it exists) for reading and writing, "a" to open the file (creating if it does not exist) for writing at the end, "a+" to open the file for reading and writing (creating if it does not exists). This returns a stream handle which can be passed to write_string_to_file().

write_string_to_file(handle, string)
This writes a string to an open file. handle is the handle returned by open_file(); string is the string to write to the file.

read_line_from_file(handle)
This reads a line from an open file. handle is the handle returned by open_file(). This returns the line read, including end-of-line characters.

bool end_of_file(handle)
This returns true if handle is at the end of the file it refers to, or false if it is not. handle is the handle returned by open_file().

close_file(handle)
This closes a file opened with open_file(). handle is the handle returned by open_file().

write_file(P, S)
This writes string S to a file at pathname P. If the file does not exist, it is created; if it exists, it is overwritten. P is either a full pathname, or a pathname local to the Sawmill executable location, using / as pathname dividers; for instance, if your LogAnalysisInfo folder is in the same directory as the Sawmill binary (which is typical), then to read from a file test.html in WebServerRoot in LogAnalysisInfo, use "LogAnalysisInfo/WebServerRoot/test.html".

read_file(P)
This reads the contents of the file at pathname P. It returns the contents of the file as a string. P is either a full pathname, or a pathname local to the Sawmill executable location, using / as pathname dividers; for instance, if your LogAnalysisInfo folder is in the same directory as the Sawmill binary (which is typical), then to read from a file test.html in WebServerRoot in LogAnalysisInfo, use "LogAnalysisInfo/WebServerRoot/test.html".

ldap_initialize(string url)
This initializes a connection to the LDAP server at the specified URL. It returns a connection handle.

info = {
  label = "Test Label"
  number_of_steps = "2"
  current_step = "0"
  step = {
    0 = {
      operation = "0"
      label = "Step 0 Label"
      value = "0"
      max_value = "5"
    } # 0
    1 = {
      operation = "1"
      label = "Step 1 Label"
      value = "0"
      max_value = "5"
    } # 1
  } # step
} # info