{= include("docs.util"); start_docs_page(docs.technical_manual.page_titles.newsletters); =}

Sawmill Newsletter

May 15, 2010

Welcome to the Sawmill Newsletter!

You’re receiving this newsletter because during the downloading or purchase of Sawmill, you checked the box to join our mailing list. If you wish to be removed from this list, please send an email, with the subject line of “UNSUBSCRIBE” to newsletter@sawmill.net (please include the entire message, as the identifying information is at the bottom).

News

Sawmill 8.1.4 shipped on March 30, 2010. This is an bug-fix release--it fixes a number of bugs. This release is free to existing Sawmill 8 users. It is recommended for anyone who is experiencing problems with Sawmill 8.1.3 or earlier. Sawmill 8.1.3 has a known issue with some Windows installations. It requires the Microsoft Visual Studio 2008 redistributable package, but does not install it. Most Windows systems have this package already, but for those that don't, Sawmill will not run. If you are experiencing this problem, upgrade to 8.1.4, or install the redistributable package (x86 or x64). You can download Sawmill 8.1.4 from http://sawmill.net/download.html .

Sawmill 7 users can upgrade to Sawmill 8 for half of the license price; or if you have Premium Support, the upgrade is free. Major features of Sawmill 8 include support for Oracle and Microsoft SQL Server databases, real-time reporting, a completely redesigned web interface, better multi-processor and multi-core support, and role-based authentication control.

This issue of the Sawmill Newsletter describes how to create and graph a custom aggregating (numerical) field, by modifying the log format plug-in.

Get The Most Out Of Sawmill With Professional Services

Looking to get more out of your statistics from Sawmill? Running short on time, but need the information now to make critical business decisions? Our Professional Service Experts are available for just this situation and many others. We will assist in the initial installation of Sawmill using best practices; work with you to integrate and configure Sawmill to generate reports in the shortest possible time. We will tailor Sawmill to your environment, create a customized solution, be sensitive to your requirements and stay focused on what your business needs are. We will show you areas of Sawmill you may not even be aware of, demonstrating these methods will provide you with many streamlined methods to get you the information more quickly. Often you'll find that Sawmill's deep analysis can even provide you with information you've been after but never knew how to reach, or possibly never realized was readily available in reports. Sawmill is an extremely powerful tool for your business, and most users only exercise a fraction of this power. That's where our experts really can make the difference. Our Sawmill experts have many years of experience with Sawmill and with a large cross section of devices and business sectors. Our promise is to very quickly come up with a cost effective solution that fits your business, and greatly expand your ROI with only a few hours of fee based Sawmill Professional Services. For more information, a quote, or to speak directly with a Professional services expert contact consulting@flowerfire.com.

Tips & Techniques: Creating A Custom Aggregating Field By Modifying the Plug-in

Every profile contains at least one aggregating field (sometimes called a "numerical field")--these are the fields that appear in the Overview, and appear as the right columns in table reports. In a web server analysis, these are columns like hits, page views, visitors, bandwidth; streaming media profiles and other profiles include duration fields as well; and various other types of aggregating fields are defined depending on the type of the log data. Aggregating fields combine data from a collection of events, into a single numerical value. They might add the values of many events to get a single number (e.g., adding the number of bytes on each event to get a total bytes transferred, or adding the duration of many events to get total duration); or they might compute the number of unique values of a particular field (e.g., a visitors or "unique IPs") field, or they might compute a maximum or minimum value.

It is possible to add a new numerical field to a profile through the following steps:

Create a new Log Field (in Config -> Log Fields)
Create a new Log Filter to set the log field (in Config -> Log Filters)
Create a new Database Field based on the log field (in Config -> Database Fields)
Create a new Report Field based on the database field (in Config -> Database Fields)
Add the report field to one or more Report Elements in one or more Reports (in Config -> Reports)
Add the report field to appropriate Cross-reference Tables to speed up the reports it appears in (in Config -> Cross-reference Groups)

These steps have been described elsewhere (other newsletters). But there's an easier way, if you're willing to get into the log format plug-in with a text editor.

Log Format Plug-ins

The log_formats folder of the LogAnalysisInfo folder contains one file for each supported log format (currently, about 850 plug-ins are included with Sawmill). The first step is to identify which plug-in is the one you're using. The most reliable way of doing this is to look at the text of the profile CFG file, in the profiles folder of LogAnalysisInfo, for this line:

log_format = "plugin"

for instance, the line might be:

log_format = "apache_combined"

In this case, that means that the profile was created with the apache_combine plug-in, which is defined in the file apache_combined.cfg, in the log_formats folder.

It is sometimes preferable to make changes to the plug-in, rather than the profile. One major advantage of plug-in changes is that they affect all future profiles. But in the case of numerical fields, the biggest advantage is that the Create Profile Wizard automatically does the six steps above, for every numerical field in the plug-in.

First, Back Up The Plug-in

Modifying a plug-in is precise work, and a single character out of place can break the plug-in, making Sawmill unable to create a new profile until the plug-in is fixed. So start by backing up the plug-in file (or the whole log_formats folder), so you can easily return to an unmodified state if you introduce a syntax error into the plug-in that you can't easily find and fix.

Creating The Numerical Field

To add a numerical field to the plug-in, first find the section "numerical_fields" in the profile. For apache_combined, this looks like this:

database.numerical_fields = { hits = { label = "\$lang_stats.field_labels.hits" default = false requires_log_field = false type = "int" display_format_type = "integer" entries_field = true } # hits page_views = { label = "\$lang_stats.field_labels.page_views" default = true requires_log_field = false type = "int" display_format_type = "integer" } # page_views spiders = { label = "\$lang_stats.field_labels.spiders" default = true requires_log_field = false type = "int" display_format_type = "integer" } # spiders worms = { label = "\$lang_stats.field_labels.worms" default = false requires_log_field = false type = "int" display_format_type = "integer" } # worms errors = { label = "\$lang_stats.field_labels.errors" default = false requires_log_field = false type = "int" display_format_type = "integer" } # errors broken_links = { label = "\$lang_stats.field_labels.broken_links" default = true requires_log_field = false type = "int" display_format_type = "integer" } # broken_links screen_info_hits = { label = "\$lang_stats.field_labels.screen_info_hits" default = false requires_log_field = false type = "int" display_format_type = "integer" } # screen_info_hits visitors = { label = "\$lang_stats.field_labels.visitors" default = false requires_log_field = true log_field = "hostname" type = "unique" display_format_type = "integer" } # visitors size = { label = "\$lang_stats.field_labels.size" default = false requires_log_field = true log_field = "size" type = "float" display_format_type = "bandwidth" } # size } # database.numerical_fields

This is a fairly complex plug-in, with many specialized numerical fields, which makes it a good example--it contains several types ("int", "float", and "unique"), and several display types ("integer" and "bandwidth"--another common one is "duration_compact"). The important thing, though, is that you can add new nodes to this list, to add new numerical fields. For instance, suppose we want to track the number of accesses to a particular download URL. You could add this to the list:

database.numerical_fields = { ... downloads = { label = "Downloads" default = true requires_log_field = false type = "int" display_format_type = "integer" } # downloads ... } # database.numerical_fields

Now if you create a new profile, you'll have a Downloads option in the list of numerical fields in the Create Profile Wizard, and checking it will give you a Downloads column in all reports (and the Overview). But it will all be 0s until you describe what it means with a Log Filter:

Adding The Log Filter

To count downloads, we need to set the downloads field to 1 for every download (then the sum of that field across all events will be the number of downloads). That is done with a Log Filter. In the plug-in, the log filters are defined in a section like this:

# Log Filters log.filters = { simplify_referrer = { label = "\$lang_admin.log_filters.simplify_referrer_label" comment = "\$lang_admin.log_filters.simplify_referrer_comment" value = "if (referrer eq '-') then referrer = '(no referrer)' else if (matches_regular_expression(referrer, '^([^:]+://[^/]+/)')) then referrer = \$1 . '(omitted)'" } # simplify_referrer ... } # log.filter

To create your own log filter in the plug-in, add a new subnode to the log.filters node:

# Log Filters log.filters = { compute_downloads = { label = "Compute Downloads" comment = "This computes the downloads field" value = "if (page eq '/downloads/download.html') then downloads = 1"
} # compute_downloads simplify_referrer = { label = "\$lang_admin.log_filters.simplify_referrer_label" comment = "\$lang_admin.log_filters.simplify_referrer_comment" value = "if (referrer eq '-') then referrer = '(no referrer)' else if (matches_regular_expression(referrer, '^([^:]+://[^/]+/)')) then referrer = \$1 . '(omitted)'" } # simplify_referrer ... } # log.filter

The name of the node (compute_downloads) can be whatever you want, as can the label and the comment subnodes (which are used when displaying the log filter in the Log Filters editor). The value must be a Salang expression which does the work of the filter; in this case, it compares the "page" field to "/downloads/download.html" (change this to the path of the URL you want to track), and sets the downloads field to 1 if it matches. For examples of Salang log filters, look at the log_filter sections of the plug-ins in the log_formats folder of LogAnalysisInfo.

The Result

After making the plug-in modifications above, create a new profile, and be sure to select the log format for the plug-in you modified, and to leave Downloads checked, on the numerical fields page of the Create Profile Wizard. The resulting profile will show the Downloads field in the Overview, and every standard table report will include it as a column (and optionally a graph), as in this default Hour Of Day report, which includes a graph of Downloads by hour:

Hours Of Day Report, With Downloads Column And Graph

So, with two small modifications to the plug-in, we have added a new graphable field counting downloads, which will be included in every profile created for that format in the future.

The modified Apache Combined plug-in with these changes is attached to this newsletter.

Professional Services

This newsletter describes the procedure for creating a custom numerical column by modifying the log format plug-in. Many other types of plug-in modifications are also possible--anything that ca be done in Config can be done permanently for all future profiles, through a plug-in modification. If you need assistance with this or other Sawmill customization, or with any other Sawmill tasks, our Sawmill Experts can help. Contact sales@sawmill.net for more information.

[Article revision v1.0]
[ClientID: 43726]

{= end_docs_page() =}