Products | Features | Lite | Professional | Enterprise | Edition Matrix | Profile Calculator | System Requirements | Log Formats | Pricing

# SAWMILL VERSION HISTORY

ALL PLUG-INS

Sawmill has plug-ins to support the following log formats:

This is the version history for Sawmill 8. The Sawmill 7 version history is here and the Sawmill 6 version history is here.

Version 8.6.2, shipped May 22, 2013

Bugs fixed in version 8.6.2:

• [1279136] A Pages report, with a Page field as a pivot, gives an error like, "Can't find tableAlias=rep_*, fieldName= in table rep_*"

• [1283035] Clicking "Save as New Report" has no effect on the first click, when using Internet Explorer 8 (it works on the second click).

• [1283229] If a profile uses an MSSQL database, the Create Profile Wizard will not allow a space in the profile name.

• [1283766] Import through the web UI of a Sawmill 7 database with sessions may fail with an error like, "Attempt to read beyond end of {dbdirectory}\main\item s\day_of_week\offsets_by_num (fileSize=8000); attempted to read from 484293090593472512 to 484293090593472520"

• [1283834] Attempting to generate a pie chart, with sort set to chronological, causes a crash.

• [1284137] Adding a date/time timestamp field to a report other than Log Detail gives incorrect timestamps in 1970.

• [1284176] Profiles created in 8.6.0 may give an error, "Unknown variable 'lang_admin.snapons.geoip.comment' in expression" when displaying the Config -> Snapons page.

• [1284320] If a format has a numerical field named "requests," and if the field is unchecked during profile creation, a crash or an error like "Couldn't find node parameters in" will occur.

• [1284815] After adding a Field Ratio snapon, all table reports fail with an error, "Internal Error: Empty node name"

• [1284921] Sawmill's built-in HTTPS server supports SSLv2, an old and less secure version of the protocol. Switched to allowing only SSLv3.

• [1284973] Changed the "Turn on cross reference groups" option in the Create Profile Wizard so, if unchecked, it turns off all xrefs except the first one (date/time). Previously, it turned them all off, but that could lead to reporting timeouts when displaying the first report, on very large datasets.

• [1285043] Attaching the "Bounce Rate" snapon has no effect on reports.

• [1285059] If a profiles is created from a plug-ins which uses snapons, any require_field conditions in its log filters are ignored--log filters are not deleted properly if the fields they require are not present. This can cause errors during log parsing. For instance, Flash Media Server profiles created from a log file lacking c-client-id will give an error on build, "Unknown variable 'c_client_id' in expression".

• [1285137] In profiles which use distribute_format_line() to handle multiprocessor parsing of data with embedded format lines (e.g., Wowza), a small amount of log data *before* a format line may be parsed using the parsing rules specified in that format line, resulting in values being put into the wrong fields.

• [1285138] Opening a report in Gmail or Outlook does not show the legend colors from a pie or bar chart.

• [1285268] Fixed an issue where Wowza (and potentially Flash) log data would be reported with 0 duration when filtered. Also, improved memory usage of database filtering in Wowza and Flash import, and lifted restriction that Wowza and Flash data be processed chronologically and with a single processor.

• [1285273] Checking "Prevent Use Of Previous Passwords" in Admin/Preferences/Password is not saved and may cause a javascript error.

• [1285292] When the maximum number of licensed profiles is exceeded, and Reports is clicked for a profile, an ugly unformatted HTML error message is displayed.

• [1285370] If a database update is attempted against a database which has never been built, it fails with the error, "Cannot build database for {profile}-- someone is already writing to it" (it should instead build the database).

• [1286378] A "Send Report By Email" task, on a profile whose database is being updated, sends a report (or sometimes fails), rather than giving an error that the database is being written (as it should).

• [1286440] The custom action dump_main_table gives an error like, "Attempt to read beyond end of LogAnalysisInfo/Databases/{profilename}/main/items/date_time/offsets_by_num (fileSize=8000); attempted to read from 9916758672 to 9916758680

• [1287062] The filter "session start" still appears in the Report Filters menu, though this type of filter no longer applies to session analysis (and gives an error if you try to use it).

• [1287450] The date filter "last1week" selects 8 days (Sunday through Sunday) instead of 7 days (Sunday through Saturday).

• [1287520] Sending email in reports does not show the error message if an error occurs.

• [1287664] If a date range filter is used in a SQL database with a table prefix, it results in an error like, "Unable to Execute ODBC Query='select * from bottomleveldateitemnum'; diagnostics=ODBC error: rec1: SQLstate: S0002; msg=[Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'bottomleveldateitemnum'.;"

• [1287864] Display format type "megabytes" and "gigabytes" were implemented in Salang but not available in the GUI.

• [1287865] Reports show table bar graphs for very small numbers when they are expected to have zero length.

• [1287867] Report and report element descriptions have wrong margins.

• [1289203] Removed invalid reports editor "Show header bar" option in Reports Editor.

New features in 8.6.2:

• [1275810] Enhanced the "pattern" option for selecting profiles in the Scheduler (and the command line) so a profile matches if either its internal name (filename.cfg) matches, or its label (the way it appears in the Profiles list of the web UI) matches. Previously, only the internal name was checked.

• [1276977] The language (e.g., Chinese) can now be selected during the Startup Wizard.

• [1277418] Login plug-ins can now be enabled or disabled from the Preferences.

• [1277966] All passwords in profiles and Preferences are now encrypted when saved to disk.

• [1280145] Add a new command-line action, copy_node_from_another_profile, which copies a node from one profile to another. This can be used, for instance, to copy a custom report from one profile to another.

• [1282080] Restored the "show all rows" option in table reports (this was removed in version 8.6.0 as part of a UI redesign).

• [1282781] Restored the check_main_table_integrity and check_itemnum_integrity command line actions, which can be used to do database integrity checking, and some repair.

• [1282917] Added a new "skip file expression" option which provides a way to create a rule for skipping files in the log source based on a Salang expression. This allows for advanced skipping algorithms, for instance, skipping files where the date in the filename is older than 7 days.

• [1282927] Added Media Reports snapon to SHOUTcast format.

• [1283423] Added support for LUNA Insight Media Manager Service log format.

• [1283596] Enhanced Wowza log format plug-in, so it shows stream names both hierarchically and non-hierarchically, in separate reports.

• [1283619] Changed Wowza log format plug-in to use the new "play duration" snapon to compute its play duration field. This allows play duration to be computed more precisely in cases where log data is processed with multiple processors, or multiple log files processed out of chronological order, which previously would have resulted in small discrepancies in the reported duration.

• [1283946] Improved the robustness of database updates, by performing some parts of the update off to the side of the main table, and moving them into place only if the update completes without an error. This eliminates MUINT_MAX errors which can occur from database corruption resulting from an earlier failed update.

• [1284000] Renamed the "allow empty log source" option to "Warn on log source errors", since it has gradually acquired many other conditions other than just an empty log source. Also, added logging of these warnings to TaskLog, and to console output. They are also displayed in the Progress display.

• [1284018] Added a new option, "Skip File Expression," which allows files in the log source to be skipped based on a custom Salang expression. This allows for advanced skipping, for instance, skipping any files older than 30 days, according to the timestamp embedded in their filename.

• [1284105] Added support for WebLogic W3C format logs.

• [1284136] Added support for formatting of a date/time timestamp report field as "date" or "time", showing only the date or time of the timestamp.

• [1284208] Added Media Reports snapon to Microsoft Media Server format.

• [1284236] Added integration support for Vision Reporter, a mobile reporting app ( http://www.visionreporter.com/ ).

• [1284358] Added support for Nginx log data with a customer log_format string.

• [1284837] Added new "Date" and "Time" display formats for timestsamp report fields, to display just the date or time portion, for instance to have separate Date and Time columns in Log Detail.

• [1284963] Added Media Reports to the Akamai Streaming format.

• [1285140] Added support for Courier POP3/IMAP log format.

• [1285207] Improved detection of BlackBerry devices, for the Operating Systems and Web Browsers reports.

• [1285429] Added support for columns of type "text" in ODBC log sources in MSSQL databases.

• [1285687] Added a new Miscellaneous -> Active Filters Info page to Reports, where the command-line and "expression" versions of the current report filters can be seen.

• [1286514] Added support for VARCHAR() fields in the result of database_sql_query().

• [1287194] Added support for MDaeomon Routing log format.

Version 8.6.1, shipped February 14, 2013

Bugs fixed in version 8.6.1:

• [1278183] If an LDAP password contains a apostrophe, it causes an error on login like, ""Sawmill Alert Syntax error: Unknown operator in expression."

• [1278669] When the option Report Options -> Numbers & Text -> "Use base 10 for byte displays" is changed, the changes does affect for previously-generated (cached) reports.

• [1280548] When building with multiple processors on 64-bit Windows, if an integer field value has a value more than about 2 billion, and it will be truncated to about 2 billion, even if the database field is specified as 64-bit.

• [1280717] Reports filtered on a date range query the main table (resulting in slow report generation), even if a cross-reference table exists able to optimize the query.

• [1281429] IIS profiles do not detect web browsers properly; in particular, they identify Internet Explorer as "Unknown Mozilla".

• [1281632] The values are invisible in table cells in standard PDF reports.

• [1281718] If, after making graph changes in the Reports Editor, you click "Save Changes", and then go to Manage fields and click OK, the graph changes are eliminated.

• [1281797] The Scheduler gives an error, "Unknown variable 'lang_admin.snapons.device_type.report_label'", when there are profiles created in 8.5.9, which use the Web Server Package.

• [1281861] Large table reports (e.g., a Hostnames report with a million rows) use a very large amount of memory, and if large enough, eventually fail with an error like, 'Unable to allocate 469762048 bytes of memory; maximum memory is 2242822425, but 2098194845 is already used, and no further memory can be freed. Allocation attempted at fstring.cpp:225, description: "fstring buffer"'

• [1281968] Temporary tables with names starting with blitmp and join_maptable, are never removed from the database; large numbers of these can collect over time, until the database is rebuilt.

• [1282110] When running a "remove database data" operation against a profile which uses a MySQL 4 database as a bank end, an error can occur like, "SQL query failed: 'delete from main_table using main_table x where (main_table.date_time < '2012-01-06 21:59:37')' error=Unknown table 'main_table' in MULTI DELETE at ../src/expire_database.cpp:212"

• [1282133] View an Overview report for a profile with no data in its database, gives an error: "Attempt to read beyond end of LogAnalysisInfo/Databases//main/Tables//data.tbl (fileSize=0); attempted to read from 0 to 48"

• [1282143] Enhanced support for NetScreen, to extract virus information from a slight variant of the format, and to simplify the "message" field to improve performance and reduce database size.

• [1283027] The "gear" and "save report" icons in report elements, do not render in the correct places when viewing reports in Internet Explorer 7.

• [1283039] When clicking "Save as new report" in Reports, a JavaScript error can occur, causing the window not to load. (Clicking again will load the window).

• [1283040] There was a "pie chart" option for chronological graphs, in the "customize report" window in Reports, even though chronological data cannot be graphed as a pie. This caused an error when it was selected. This option has been removed.

New features in 8.6.1:

• [1277718] Modified the "Device Type" reports a bit, to show a Mobile Devices report listing only mobile devices, instead of a general Device Types report (the Device Categories report remains unchanged).

• [1280054] Improved progress reporting while scanning a recursive log source, to show the directory being scanned.

• [1280267] Added support for UTM 80-E Firewall.

• [1280489] Improved database build performance slightly by removing the unnecessary xref on the session_sequence_number field.

• [1281737] Switched Apache Custom plug-in to use the Web Server Package, giving it the advanced functionality of the Web Server Package snapon, including a dashboard and advanced reporting of agent and referrer information.

• [1281888] Added a Dashboard to the Media Reports snapon; added the Media Reports snapon to Flash Media Server.

• [1282340] Added automatic conversion of incorrect but commonly attempted report syntax "fieldname = 'abc'" to the correct "fieldname within 'abc'"

• [1282389] Added a Dashboard to all formats which use the Gateway Reports snapon (which is most major gateway log formats).

• [1283929] Slightly enhanced the previously-seen-data skipping algorithm, so if a log file is zero length or inaccessible, and "skip files by pathname" is enabled, the file will be ignored, and not saved as "seen," so if it gets data later or becomes accessible, it will be processed at that point. This is specifically important for IIS logs, where the latest log (the log being written) is not accessible for reading, and should be retried until it becomes available.

Version 8.6.0, shipped December 12, 2012

Bugs fixed in version 8.6.0:

• [1263664] The built-in web server fails to load pages or files, sporadically, on non-Windows systems.

• [1272644] When importing a v7 profile without a database to 8.5, session information (reports and fields) is not carried over to the new profile.

• [1272731] "Remove Database Data" operations did not correctly check for existing database-writing processes, potentially causing corruption of a database if they begin during a database update or rebuild.

• [1274531] When building a database from Juniper MFC 12 (W3C) data without suppress_cs_range or suppress_etag fields, an error occurs, reporting the absence of these fields.

• [1274718] Network actions like create_user, which shouldn't require -p, generate an error if p is not specified, "Unknown variable 'internal.profile_name' in expression"

• [1274963] After attaching a "Report Field Ratio" snapon, reports give an error, "Unknown database field '{fieldname}' in v.query_result.header"

• [1275020] Report filters on numerical fields using >= give incorrect results (often, filtering out nothing).

• [1275153] If a version 8.1 profile has a "unique" database field of Type "string", conversion to 8.5 will result in a version 8.5 profile which fails on database build with an error like, "Internal: Attempt to find main table column number from database field 29 [visitors], but there is no such column in main_table"

• [1275902] If a database field is non-hierarchical, and there is a cross-reference group of it which is hierarchical, the corresponding report will show an extra blank line counting all events.

• [1276172] If the web server is running more than 30 days straight, tasks may begin to fail immediately after starting (this is due to MasterProcessLock files being prematurely deleted).

• [1276578] Attaching the Geographic Location Information from Config -> Snapons gives an error, "Snapon attempted to add database field 'location', which already exists"

• [1276952] Improved detection of libcrypto during "configure" on Linux, to handle systems with limited versions of the library.

• [1276987] The "Other Rows" line in tables is incorrect, containing one of the rows which is visible.

• [1277076] The "omit parenthesized items" item is not saved when using the Customize link in Reports for pivot tables.

• [1277141] Session information is lost when importing a Sawmill 7 profile through the web interface.

• [1277448] When using an SFTP log source, entering "/" for the pathname does not show the files in /. (workaround: use "/*").

• [1277702] Generating a report where one of the report elements has a label containing a double-quote, gives an error like, "Unexpected = in group node (v.progress.step.0.abc def ))".

• [1278066] The delete_user action (or network action) gives an error, "Syntax error: Unknown variable 'profile_name' in expression."

• [1278917] The command-line progress display often displays less than 100% at the end of a successfully completed action.

• [1279444] A database build can crash if there are no xref and no indices and no database filters.

New features in 8.6.0:

• [1272161] itemnum tables are no longer indexed, when the corresponding database field is set to not be indexed.

• [1273614] Separately implemented country/region/city support as a snapon, for better modularity; this also makes it possible to have more than one Geo analysis in a single profile (based on different IP fields).

• [1274335] Greatly improved the performance of filtered Log Detail reports, by eliminating the calculation and display of the total available rows. This decreases the time from 169 seconds to 7 seconds in one 180-million-line dataset.

• [1274617] Created a new snapon, Create Default Xref Groups, which at the time it is attached, creates default xref groups for all database fields (each xref group having that field, date/time, and all aggregating fields), and all reports (each xref group having all fields in the report). This is similar to what has happened automatically at profile creation, in earlier versions, but it is now possible to reset the xrefs to optimal configuration after adding or removing fields or reports.

• [1274618] Log Detail is now ordered automatically with the timestamp at the left, followed by the non-aggregating fields in the order they appear in the Report Fields, followed by the aggregating fields in the order they appear in Report Fields. In previous versions, all fields were in Report Fields order, which can give undesirable orderings, especially when some fields are created with snapons.

• [1275211] Added error message display when JavaScript is disabled.

• [1275333] Added a white line between adjacent slices in 2D pie charts, for better contrast.

• [1275344] Enhanced LDAP login plug-in so user roles and profile permissions can be managed through the Sawmill web interface, and will not be overwritten each time by the LDAP login.

• [1277169] Added support for McAfee Web Gateway log format.

• [1277726] The Countries/Regions/Cities reports have been somewhat restructured, when they are created from a snapon (as is currently the case for Apache logs, Common Access Log Format, and IIS logs; more will follow). The reports no longer use a hierarchy, but instead use custom fields with custom formatting, which give a cleaner and easier-to-read appearance to the names of the regions and cities. Some city and region categories have also been consolidated.

• [1277785] Added support for Sonicwall NSA (Network Security Appliance) log format.

• [1277837] Added support for Smoothwall Network Guardian and Advanced Firewall log format.

• [1278135] Added support for IceCast Playlist log format.

• [1278289] Added a snapon to report Service Name, e.g., "HTTP" computed from port 80, protocol TCP.

• [1278764] Improved progress reporting to show "scanning log source" as a separate stage after "erasing database."

• [1278851] Added support for Websense log format.

• [1278870] Enhanced support for Microsoft DHCP log format, to handle non-syslog version, and missing field values in some fields.

• [1279121] Added the option to build all database field hierarchies from the command line with "-a rdh" by omitting the -fn parameter.

• [1280817] Added Dashboard functionality. This is a collection of features—side-by-side report elements that flow and wrap to maximize visible data, simpler and smaller versions of reports and graphs, and a number of other report element options—which can be used to implement simple "dashboard" style reports with many small graphs or tables in a two-dimensional layout. Used this functionality in the Web Server Package snapon to implement a Dashboard for Apache Combined, IIS, and Common Access log formats (more to come).

• [1280820] Enhanced reporting of Web Browser information, for plug-ins that use Web Server Package (currently, IIS, Apache Combined, and Common Access). The new report shows browser name, major version, and full browser version in three separate report elements in one report.

• [1280821] Cleaned up and improved the appearance of reports and graphs in a variety of small ways.

Version 8.5.9, shipped August 5, 2012

Bugs fixed in version 8.5.9:

• [1267884] When building a database larger than about 4GB, on 32-bit Windows, an error will occur, like, "Internal: Error in PagingCachingBuffer [LogAnalysisInfo\\Databases\\{profile}\\main\\Tables\\main_table\\data.tbl]: position=4295016444, but pi.endOffset=49152"

• [1268490] When building or updating a database, an error can occur, "Internal: SQLQueryEditorTable::PageForward() called, but queryDone=true"

• [1268944] Table Filter expressions on the Log Detail report, are not saved to the profile, if edited in Config.

• [1271373] If a report has a report date filter and one or more report elements had also a report element date filter, extra empty bars can appear in date bar graphs.

• [1272026] If a database build is cancelled or failed, attempts to access the reports will fail with an error like, "Unable to read file LogAnalysisInfo/Databases/{profile}/main/Tables/xref30/header.cfg", instead of rebuilding the database as they should.

• [1272083] If a database name contains a space, it causes an error when creating a MySQL profile.

• [1272199] If a profile has two xrefs with the same fields, and one of them is hierarchical and the other is not, the Overview may over-report the number of events.

• [1272374] If a report query requires a full scan of the main table, for a profile created with Sawmill 8.5.8, "event" field values will be truncated to values between -127 and +128. (Workaround: change the "Single value integer bits" parameter for the events database field to 64, and rebuild the database).

• [1272530] When a number_of_rows option is specified for a report in a plug-in, it has no effect on the created profile.

• [1272638] When a version 7 profile is imported into version 8.5.8 and then converted, the name of the "Year" report is missing from the report menu.

• [1272768] The main table indices are dropped during database update, for profiles with complex database filters which edit the main table (e.g., Wowza, and also any profile with Sessions), resulting in a slower report (as one index is built) the next time a filter is used on a report which cannot be delivered by cross-references.

• [1272782] When building a MySQL database, temporary memory usage can grow without bound, eventually (for a large dataset) causing an error stating that no further memory could be freed. The console also will show messages like, "#### WARNING: PROBABLE MEMORY LEAK; WE NOW HAVE 100 TEMP STRING POOLS".

• [1272924] While creating a profile with the tomcat_pattern plug-in (Apache Tomcat (using Access Log Valve pattern)), an error occurs, "Couldn't find node tomcat_pattern in language.english.lang_stats.field_labels"

• [1272927] If there is no data in the log source on rebuild, reports will fail with an error (instead of reporting all 0s) like, "Unable to read file LogAnalysisInfo/Databases/tomcat/main/Tables/xref1/header.cfg (Operation timed out)"

• [1272988] When creating a profile from either of the non-W3C IronPort S-Series Access Log formats, an error occurs, "Error in profile_setup add_custom_report_element(), the report element type 'session_pages' is not supported."

• [1273000] If an FTP log source uses a relative pathname like "stats/logs", it will not find the log files, and will generate an error when creating the profile or building the database.

• [1273312] Wowza reports may have the field values shifted (the value from the 25th field of the log data appearing in the 24th field, for instance), if there is a c_user_agent field.

• [1273403] Fixed a bug where Flash Media Server profiles from log data without a cs-uri-stem field fail to build with an error, "Syntax error: Unknown variable 'cs_uri_stem' in expression"

• [1273763] When the cs-stream-bytes or sc-stream-bytes snapon instances are attached to a Wowza profile, on a 32-bit system, the resulting fields are only 32-bit, and will overflow into negative numbers for values > 4GB. (Workaround: change the database field integer bits to 64 for those fields)

New features in 8.5.9:

• [1270711] Added a Custom tab to the Date Picker in Reports, to allow direct entry of date filter expressions like "last 3 months", "Jan/2012-Mar/2013", etc.

• [1271248] Added a new Device Type report to the main Apache plug-in and the IIS W3C web server plug-in. This report will be migrated to other web server plug-ins in the future.

• [1271352] Added support for vsftpd log format.

• [1272385] Added support for Juniper Media Flow Controller Access (W3C) Log Format.

• [1272450] Improved performance of reports run against the main table (i.e., without a cross-reference), where all non-aggregating fields are "flat" (i.e., all items are directly below the root in the subitems hierarchy). Most simple fields are like that, so this improves the performance of most non-xreffed reports. Performance is vastly improved over Sawmill 8.5.8 (10x-20x or more), but since Sawmill 8.5.8 had a serious performance regression for this type of reports, performance improvement is less impressive vs. 8.5.7 and earlier (maybe 1.5x).

• [1273050] Added support for DansGuardian logged through a syslog server.

• [1273057] Added support for McAfee Email Gateway log format.

• [1273788] There is now an option to omit the total row (subtotal row) beneath each subtable in a Pivot Table.

Version 8.5.8, shipped June 27, 2012

Bugs fixed in version 8.5.8:

• [1271315] The uninstaller does not properly shut down existing Sawmill.exe processes on Windows. This can cause a "version mismatch" error when upgrading from 8.5.7 to later versions. The workaround is to manually kill both Sawmill.exe processes (one will automatically restart if you kill it, but will exit if you kill the other), and rerun the installer.

• [1271762] Profiles are created with skip_previously_seen_data_on_update option set to false by default, which causes previously-seen data to be re-added on updates, in a default profile.

New features in 8.5.8:

Version 8.5.7, shipped June 12, 2012

Bugs fixed in version 8.5.7:

• [1258482] Certain large datasets can crash during the database filtering step of a database build.

• [1261954] Conversion of a Sawmill 7 database fails with an empty error; conversion of a Sawmill 7 profile seems to succeed, but gives an error on viewing reports.

• [1264206] When using SFTP to process files compressed with bzip2, an error can occur like, "SSH connection failed: read_packet(): Packet len too high(1608634376 5fe1d008)"

• [1264277] If a custom report field uses another report field to calculate its value, and that report field is not a visible column in the report, it will give an error like, "Unknown column 'accesses' in cell_by_name()."

• [1264861] Fixed a bug where conversion of a MSSQL Sawmill 8.1 profile to 8.5 fails with an error like, "Unable to Execute ODBC Query='create table main_table_plus_dfc select loadorder, db_filters_computed, ... from main_table'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near the keyword 'select'.;"

• [1265145] A "remove database data" operation on a MySQL 5.5 database, gives an error like, "SQL query failed: 'delete from main_table using main_table x where (x.date_time < '2012-03-22 02:15:01')' error=Unknown table 'main_table' in MULTI DELETE"

• [1265527] Reports can fail to generate, with an error, "Unknown variable 'lang_stats.general.' in expression."

• [1266068] Reports sorted by "string" columns can crash in some circumstances.

• [1266187] CSV export from the Scheduler, when "all rows" is specified, exports only one row.

• [1266371] When using multiple SFTP log sources, database updates can skip an entire log source, if some data in the previous log source has already been imported.

• [1266544] When viewing an unfiltered Log Detail report, paging forward changes the line numbers, but not the data displayed.

• [1266546] In FTP or SFTP log sources, if the pathname ends with a /, no files will be selected.

• [1267210] When an unqualified hostname is used in a URL to access the Sawmill web interface, the error message displayed states that there was error while trying to display an error. There should be an error; but it should state that hostnames must be fully qualified.

• [1267261] The new report field option "Skip Escaping" is on by default, which can cause an error, "Syntax error: Unknown operator in expression," while displaying reports containing literal s and other special characters. • [1267732] Attaching the "Gateway Reports" from the Snapons page gives an error, "Unknown variable 'lang_admin.snapons.gateway_reports.parameters.have_client_ip_field.form_element_label' in expression" • [1267856] Building the database of a profile using Nortel ACD format causes an error, "#### Internal: Attempt to find main table column number from database field 19 [average_tsf], but there is no such column in main_table" • [1268317] Per-user report filters have no effect. • [1268763] If an 8.1 profile has a "unique" database field pointing to a database field which doesn't exist, the converted 8.5 profile will give an error like, "Unknown database field 'cs_cookie' as source field of database field 'visitors'". • [1268972] If all log sources are disabled, a database build can crash. • [1269054] Large MySQL database builds, and other operations, can use large amounts of memory (more than permitted by the Preferences). • [1270484] When using an external SQL database, if all xref and indices are turned off for a profile, the main table will not be populated (or only 1000 lines of it). New features in 8.5.7: • [321927] Added support for Kerio Connect 7 logs. Previous versions supported only Keri Mail Server through versions 6.5. The new version also reports separately on SMTP, HTTP, and WebDAV events, reports usernames when available, and reports SSL status. • [1261075] Added rotation of the tagging server log file, and moved it to a subdirectory LogAnalysisInfo/logs/tagging. • [1261253] Added support for conditions in snapon parameters, so when attaching a snapon, certain parameters can appear or disappear depending on the values of other parameters. Used this to implement a better Aggregating Field snapon, which prompts for the source field when the aggregation operator is "unique", or prompts for a log filter or database filter expression otherwise. • [1262435] Added an "index granularity" option for database fields, which specifies the precision of the indexing of database fields in the main table. Previous versions had a default of 0, resulting in highly precise indices; the new default is 1000, which provides only general regional indexing of the table. With the new value, indices are 10x smaller or more, and index builds are as much as 5x faster. However, filtered reports can be as much as 50% slower, if a cross-reference table is not available to provide the report. • [1263556] Changed internal database deletion so it deletes only the "main" subdirectory of the database directory, rather than the entire database directory. This prevents an occasional issue which could cause data loss if the database directory was set to an existing directory with other data (like the log source directory). • [1265556] Reduced database disk usage by about 50% in most cases, by using smaller integers when reasonable. • [1266416] Enhanced Wowza analysis to include new Media Usage reports for a simple top-level view of usage. Also added referrer tracing with search engine and search phrases analysis. Improved grouping of reports menu, and removed a few extraneous fields for better performance. • [1266466] Improved performance of command-line index building (-a rdi) by building all indices in a single pass through the main table (which is how it works during a database build already). This can make the index builds 10x faster or more, when rebuilding all indices from the command line. • [1268454] Added support for Cisco NetFlow logs, created with "nfdump -o long" • [1268470] Implemented Mail Server Reports snapon, with general reports appropriate to mail servers: Sender Domains, Recipient Domains, Senders, Recipients, Recipients by Sender. Added this snapon to the Postfix plug-in. • [1269005] Added support for Akamai HTTP Streaming Log Format. • [1269200] Added a built-in Salang function, dns_resolve_hostname_to_ip_address(), for resolving hostnames to IP addresses. Version 8.5.6, shipped March 23, 2012 Bugs fixed in version 8.5.6: • [1250051] When parsing very large datasets (billions of lines) with multiple processors, an error can occur during log parsing, "Invalid format of PARSED response from parsing server." • [1256945] Sawmill 8.1 profiles which include "session entrances" or "session exits" as report element columns do not convert properly; the resulting profile will give an error like, '"The report field "ssession_entrances" does not exist in columns of report element "year"' in the Report Editor. • [1257518] On Windows, if a LogAnalysisInfo has been moved using LogAnalysisInfoDirLoc, the upgrade installer installs the latest version in the default location, rather than in the version specified by LogAnalysisInfoDirLoc. • [1257840] If there is only one row in the database (typically, because there is only one line in the log data), viewing reports after a rebuild will give an error, "#### Unable to read file LogAnalysisInfo/Databases/apacheextended/main/Tables/xref0/header.cfg (Operation timed out)" • [1258384] When mapping a drive or share through the Network Disks section of the File Browser, the disk does not immediately appear after being mapped. • [1258619] After about 2 billion lines processed, the progress display shows the number of line processed as a negative number • [1258808] If a report field is added by a snapon at profile creation time, and its display format is not "integer," it is overridden by profile creation to be an "integer" display field. • [1258809] On Windows, if a custom report field displays as "integer," and contains a value larger than about 2 billion, it will display as about 2 billion in HTML reports. • [1259019] If a syslog-required field has a database field called "duration," the Log Fields Editor shows a blank field label. • [1259034] After a Remove Database Data operation, the Calendar report still shows links for days that have been removed. • [1259120] The "concurrent connections" snapon (used to do concurrent connection analysis for several media server plug-ins, including Microsoft Media Server, can overcount concurrent connections in some cases, as it fails to register the end of certain connections. These errors tend to accumulate as more lines are processed, so the number of concurrent connections will incorrectly slow a gradual upward slope. • [1259396] Fixed a bug where, if using a SQL table prefix or suffix, a Date Range filter would fail with an error like, "SQL query failed: 'select * from bottomleveldateitemnum' error=Table '.bottomleveldateitemnum' doesn't exist" • [1260066] If the "Log field separator" is a tab, the Config -> Log Processing -> Format page will show an error if you make a change to it, "No value. Please define a value." • [1260232] Sending mail to a qpsmtpd server, using username/password authentication, gives an error, "503 AUTH not defined for HELO." • [1260425] When a version 8.1 profile has a "session begin" or "session end" column in a report, and it is converted to 8.5 format, the Report Editor gives an error like, 'The report field "ssession_begin" does not exist in columns of report element "year".' • [1260484] Viewing Log Detail with a filter which discards all events gives an error like, "Attempt to read beyond end of LogAnalysisInfo/Databases/{profile}/main/Tables/rep_9a7ada7d8bbfe1830f4751a689e58504/data.tbl (fileSize=0); attempted to read from 0 to 280" • [1260610] Fixed a bug where the Log4j parser did not properly handle %d dates without a curly-bracket format, in PatternLayout (and rejected all entries). • [1261320] When using an Oracle database, the Log Detail report gives an error like, "Unable to Execute ODBC Query='select x.broken_link, x.date_time, x.day_of_week, x.hour_of_day, x.s_ip, x.cs_method, x.cs_uri_stem, x.cs_uri_stem, x.file_type, x.screen_dimensions, x.screen_depth, x.worm, x.s_port, x.cs_username, x.c_ip, x.domain_description, x.location, x.organization, x.domain, x.isp, x.web_browser, x.operating_system, x.spider, x.sc_status, x.sc_substatus, x.sc_win32_status, x.hits, x.page_views, x.time_taken, x.time_taken, x.session_entrances, x.session_duration, x.bounces from main_table xrownum between 1 AND 50'; diagnostics=ODBC error: rec1: SQLstate: S1000; msg=[Oracle][ODBC][Ora]ORA-00933: SQL command not properly ended ;" • [1261498] Flash Media Server profiles do not report concurrent connections. • [1262050] When using an FTP or SFTP log source on Windows, if the log file or pathname contains a colon (:), it gives an error like, "Unable to create folder LogAnalysisInfo\TempLogs\1330129826\7768\directory\\subdir:with:colon" • [1262299] When using "-a pv" to display database fields summaries from the command line, max/min fields based on date_time display values from 1970. • [1262381] Temporary tables with names like xref0_update are not properly delete when the database is built, making the database about 25% larger than it needs to be. • [1262595] If a relative date filter is used which selects just one day, on a Days report with a graph, it will give an error, "#### Unknown variable 'lang_stats.months_short.e_t' in expression". If it is used on a Months report with a graph, it will give an error, "#### Couldn't find node 0 in volatile.temp_month" • [1262794] Enhanced (and in some cases fixed) memory management, to better limit memory usage. The previous version generally kept its memory usage under control (under the specified limit), but certain types of memory usage were not counted toward the limit, which could result in substantially higher memory usage than allowed. • [1262847] The Create Profile Wizard does not prompt for the Access Log Valve pattern, when creating a profile using the format, "Tomcat (using Access Log Valve pattern)" • [1263906] If LogAnalysisInfo is relocated using LogAnalysisInfoDirLoc on Windows, the service will not properly shut down Sawmill.exe processes when it stops. • [1264170] Create a new report element "Hour of day" with "Display: Graphs". On tab Graph Options change the graph type to "Line graphs". On tab Graphs change sort by to "Hour of day", keep sort direction "Ascending". Click save changes and view the report. The report will show the graph with sort_direction "descending" although the report element editor indicates "ascending". • [1264241] When using MySQL or MSSQL as the back-end database server, if the database already exists and is already a Sawmill database, and if the profile uses the sessions snapon, an error will occur on profile creation, "Duplicate column name 'session_id'". New features in 8.5.6: • [1102604] Added support for arbitrary (almost) Log4J parsing, through support for most PatternLayout values. • [1256999] Improved reporting performance has been improved for large reports with "omit parenthesized items" turned on (an example of this is a standard web server "search engines" report). In one example (200 million line dataset), reporting performance increased 24x. • [1258492] Added support for Clavister SG log format. • [1258816] Added support for a variant of GroupWise Post Office Agent Log Format, which logs Net Id. • [1259024] Added a new snapon, "Top Level Domain" which creates a "top level domain" field, and populates it with a log filter, using the list of known top- and -second-level domains to convert a URL to a reasonable top domain name, e.g., "abc.xyz.com" becomes "xyz.com" and "abc.xyz.co.de" becomes "xyz.co.de". • [1259026] Added a new snapon, "Gateway Reports" which creates a category of four simple reports for gateway decides, for HR purposes: Users Summary, Categories Summary, Domains summary (using the new Domains field snapon), and Usage Detail. The Summary reports include pie charts, and the Usage Detail reports shows category, user, site, start and end time, and duration. This this snapon is attached by default to Palo Alto Integrated, and Squid (without category); other formats will follow. • [1259220] Greatly improved performance of report generation, for reports containing fields using custom expressions, including "average" fields (e.g., page views per session). Reports using this kind of field, and containing millions of rows, generate as much as 100x faster now. (Reports with few rows are not much affected). • [1259630] Improved performance of database builds of the internal database, especially multiprocessor builds and profiles without database filters (e.g., profiles without session analysis, or other snapon functionality which creates a database filter). Performance improvements vary by profile, but may be 40%-100% faster than previously, on a multiprocessor system. • [1259704] Added back support for the rebuild_cross_reference_tables action, which allows all xrefs, or any single xref (with -crt N) to be rebuilt from the command line, without rebuilding all the rest of database filters, or indices. • [1259821] Added support for numerical reporting of the content_bytes (%B) field, in Apache Custom format strings. • [1259842] Improved cleanup of Sawmill's "recycling bin" (LogAnalysisInfo\TemporaryFiles\DeleteMe), by immediately deleting everything put in it (simultaneously with whatever else is going on), rather than waiting for the next cleanup cycle. This can make a huge different in the amount of disk space temporarily used during certain operations, especially 8.1 profile conversion. • [1260326] Add a new clean_up_database action (e.g. sawmill -p {profile} -a cud), which drops all temporary tables from the database (except those whose parent process is still running). This also now occurs automatically at the beginning of any database update, or "remove database data." This eliminates the clutter which sometimes results when reports or other processes terminate abnormally, and fail to clean up their temporary tables. • [1260480] Added functionality to include multiple log filter initializations, and multiple log filter finalizations, in a single profile (as subnodes of log.filter_initialization and log.filter_finalization). They are run in order. Old-style initializations and finalizations (expressions directly on log.filter_initialization, etc.) are still supported. Added a new snapon operation to add a filter initialization or finalization to a profile. Together, these features allow snapons to add independent filter initializations and finalizations to a profile. • [1260482] Implemented the "Advanced Example: Rejecting spiders based on JS and /robots.txt access" log filter example as a snapon, for much easier implementation in a profile. • [1261082] Added a new action, rebuild_database_filters, which rebuilds all database filters. • [1262382] Enhanced Wowza analysis to use the new concurrency snapon to track concurrent streams, instead of the older "session" style analysis. • [1264537] Added support for Microsoft Forefront log format. Version 8.5.5, shipped January 17, 2012 Bugs fixed in version 8.5.5: • [948006] Removing data from an Oracle database using a filter gives an error like, "#### Unable to Execute ODBC Query='delete from main_table x where not (filtertmp_3932_0.itemnum IS NULL)'; diagnostics=ODBC error: rec1: SQLstate: S0022; msg=[Oracle][ODBC][Ora]ORA-00904: "FILTERTMP_3932_0"."ITEMNUM": invalid identifier" • [1244697] Duration fields are incorrect for Wowza Media Server profiles created with Sawmill 8.5.3 or 8.5.4. • [1246049] Reports, especially for a database which hasn't been rebuilt for a long time, can fail with an error like, "Attempt to read beyond end of LogAnalysisInfo/Databases/{profile}/main/Tables/_select_result_75103_1/sets/visitors/header.dat (fileSize=0); attempted to read from 0 to 64." • [1256046] When using a MS SQL database with Palo Alto log data, the Sessions Overview gives an error like, "select count(distinct x.user), sum(x.page_views), max(x.date_time), min(x.date_time), count(distinct x.session_id), sum(x.session_duration) from main_table x where 1=1'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near the keyword 'user" • [1256170] The "Maximum caching buffer memory full load" field is empty, in the Config UI. • [1256189] If there is a database field which has a different source than itself (e.g., a unique field like "visitors" which counts another field), database export will fail with an error like, "Can't find tableAlias=, fieldName=visitors in table main_table." • [1256296] Non-root-admin users cannot change their password; the "Save Changes" button has no effect. • [1256368] A license installation with sublicense only allows the number of profiles specified by the main license. • [1256450] Report fields which compute an average value of a field (x/y), show 0. • [1256501] When using a MS SQL database with a 64-bit integer field, aggregated numbers larger than about 2 billion can cause an overflow error which terminates the build. • [1256506] The delete_database_field action does not delete columns from report table (and report tables themselves) which are derived from the field, e.g., the "city" column and table are not deleted when the "location" field is deleted. • [1256517] An unfiltered Log Detail report takes a long time to generate (scans the whole main table, instead of just the visible rows). • [1256528] Some temporary tables are not removed from internal databases, especially during filtered report generation, resulting in a large number of unnecessary database tables (and unnecessary files on disk) after a long period with many reports generated. • [1256952] Profiles created from the Juniper Networks Secure Access 4000/6000 plug-in give an error when reports are displayed: 'Unknown configuration group "session_paths" in node "profiles.in.statistics.reports"' • [1257001] Fixed an issue with the performance of some queries, which made reports slow with large datasets. • [1257002] Fixed a bug which could cause a crash on database build, if there were more cross-reference groups than database fields. • [1257516] If a Sawmill 8.1 profile has a custom "session user" field (session visitor ID), and that field has no corresponding report field, the profile when converted to 8.5 will fail to generate the Individual Sessions report with the error, 'The report field {customfield} does not exist in columns of report element "individual_sessions".' • [1257752] PIX logs with high hit-cnt values can take a very long time to process, or even fail with an error like, "Unexpected response from SPS server: PARSED 43969360 289230 0 0" New features in 8.5.5: • [916304] Added support for Amazon Cloudfront Streaming logs, including a database filter which emulates Amazon's own "bytes transferred" calculation. • [1052024] Enhanced the Coradiant TrueSight log format support, to handle variable field lists in the header, to report on all known numerical fields, and to categorize all known non-numerical field reports. • [1218318] Added a new "Execute command line" action to the Scheduler, to run an arbitrary command line. • [1256677] Added a performance warning to the Progress display, when a particularly complex normalized database field is detected during database building. • [1257003] Added an xref for each report, by default (with no date range info). This can significantly improve the performance of unfiltered top-level reports, and especially the Single-Page Summary. • [1257026] Improved the selection of the xref table used for a query, so the smallest (fastest) match is chosen, rather than the first match. This improves the performance of some unfiltered, or date-filtered, reports. Version 8.5.4, shipped December 13, 2011 Bugs fixed in version 8.5.4: • [1243631] Generating a Sessions Overview for a dataset with no session events gives an error, "Internal error: mapping 'LogAnalysisInfo/Databases/{profile}/main/Tables/session_users_stage1/sets/sessions' read-only, but its lists.dat (LogAnalysisInfo/Databases/{profile}/main/Tables/session_users_stage1/sets/sessions/lists.dat) does not exist" • [1246830] Converting a v8 profile whose name contains a dot, gives an error, 'Unknown configuration group "options" in node "profiles.{profilename}.database"' • [1247484] If no -er (ending_row) is specified for a get_report action, it defaults to 0, which shows just one row (it should really show ten). • [1252329] On Window, if an index (or certain other database files) exceeds 2GB, it can grow without bound, causing the disk to fill and the database build to fail. This typically happens with large datasets, of 200 million lines or more, but it can happen with certain smaller ones. • [1256100] For certain (uncommon) large datasets, on Windows, a database build can crash during database filtering. • [1256165] Filtering reports, especially on single days, sometimes gives an error like, "Attempt to read beyond end of LogAnalysisInfo\Databases\{profile}\main\Tables\filtertmp_5676_1\data.tbl (fileSize=4); attempted to read from 882531391810568340 to 882531391810568344." New features in 8.5.4: Version 8.5.3, shipped November 21, 2011 Bugs fixed in version 8.5.3: • [1189781] Bug reports always go to support@sawmill.net, even if Support Email is set to something different in lang_stats or Preferences. • [1192538] When using real-time reporting with multiprocessor parsing and the internal database, the reports do not show the latest data. • [1201875] Setting the thousands divider, or the decimal divider, in the user's settings, as no effect in reports. • [1217367] Adding a new license to an already licensed installation gives an error, "Unknown operator in expression." • [1218812] Filtered reports which cannot use xrefs, do not take advantage of database indices, resulting in report generation time linear with the size of the database. This makes some reports, especially those filtered on an item with few associated events, much slower than they could be. With this enhancement/fix, these reports are now 30x-50x faster for a 10 million row dataset, and will be proportionally even faster (or rather, were proportionally slower), for larger datasets. • [1239331] The Log Source window shows "undefined" for HTTP log sources created there. • [1246243] Certainly types of filtered reports would generate an error like, "#### Attempt to read beyond end of LogAnalysisInfo/Databases//main/Tables/filtertmp_2091_1/data.tbl (fileSize=8); attempted to read from 18446744073709551592 to 18446744073709551600" New features in 8.5.3: • [1191655] Support has been added for the Mikrotik Web Proxy log format. Version 8.5.2, shipped October 31, 2011 Bugs fixed in version 8.5.2: • [1005835] Missing parenthesis causes an error when using the Microsoft Windows 7/2008 Eventlog via Syslog plug-in. • [1140085] Pages ending with "{default}" are not correctly converted/translated to "(default page)" for W3C data. • [1151672] Some images, including graphs, are broken in CGI-mode reports. • [1156230] For very large or complex datasets, on machines without a lot of memory, database builds can hang during the "Running database filters" step. • [1160752] When a session times out, a popup window appears with an technical message about the session timeout. • [1160874] An error (otherwise harmless) is written to TaskLog any time the login page is displayed: "Unknown variable 'volatile.session_id' in expression." • [1161680] Temporary files in the folder LogAnalysisInfo\Locks are never removed. • [1166248] Conversion of 8.1 profiles fails if they have a session_id or session_duration field unrelated to the main session analysis, and if they also have a main session analysis. E.g., Flash profiles are like this. The conversion fails with the error, "Snapon attempted to add database field 'session_id', which already exists." • [1166466] The header for the default "X by Y" reports doesn't include the name of the first field. • [1167221] Clicking "Lookup Pages" in the "Paths Through A Page" report gives an error, "Unknown variable 'v.fp.report_name' in expression." • [1174513] If a log format plug-in contains both a database field called "session_id" and a session analysis, profile creation fails with an error, "Snapon attempted to add database field 'session_id', which already exists." • [1176829] In certain unusual cases (with the report field node contains a database_field subnode which is empty), upgrading an 8.1 profile to 8.5 will fail with an error, "Internal Error: Empty node name." • [1177407] Zooming on a table item, and zooming to Log Detail, when using an Oracle database, gives an error like, "Unable to Execute ODBC Query='select x.date_time, x.day_of_week, x.hour_of_day, x.hit_type, x.page, x.file_type, x.worm, x.screen_dimensions, x.screen_depth, x.hostname, x.domain_description, x.location, x.organization, x.isp, x.domain, x.referrer, x.referrer_description, x.search_engine, x.search_phrase, x.web_browser, x.operating_system, x.spider, x.server_domain, x.authenticated_user, x.server_response, x.hits, x.page_views, x.spiders, x.worms, x.errors, x.broken_links, x.screen_info_hits, x."size", x.session_entrances, x.session_duration from main_table x left join filtertmp_3176_0 on x.hit_type = filtertmp_3176_0.itemnum where not (filtertmp_3176_0.itemnum IS NULL) WHERE rownum between 1 AND 50'; diagnostics=ODBC error: rec1: SQLstate: S1000; msg=[Oracle][ODBC][Ora]ORA-00933: SQL command not properly ended" • [1186397] The command line filters, -df and -f, have no effect when used with the get_report action. • [1186960] LDAP authentication via the LDAP login plug-in doesn't work; even with Enterprise licensing it gives an error, 'This version of Sawmill is configured to use "login plug-ins" but this feature is not supported by the current license. Please contact your system administrator to remove the login plug-ins or to use a different license.' • [1192563] Log Detail reports, especially unfiltered Log Detail reports on large datasets, are much slower and more expensive (in disk space and memory) than they should be (due to a bug in sorting). • [1193475] Integer fields with values larger than about 2 billion, may be sorted incorrectly in report tables. • [1196107] Exported individual sessions show session begin and session end in epoch. • [1204456] Building a database with MS SQL and BULK INSERT gives an error like, "Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 5, column 37 (session_id)." • [1205209] Clicking on Change Trial Mode, Admin or About hides the main page. Clicking on Support or Help opens the respective page in a new window and the current window. New features in 8.5.2: • [989079] Restored Message ID report to Barracuda Spam Firewall plug-in. This is to help correlate messages. • [1149070] Added RBAC privileges for access to network actions, so non-root-administrative user credentials can now be supplied to execute network actions. • [1171241] Added detection and reporting of Android user-agents in the Operating Systems report. • [1183330] Added support for IDP events from JunOS 11.1 with IDP to the Juniper SRX240/SRX3400 plug-in. Some field names are also improved. Version 8.5.1, shipped September 6, 2011 Bugs fixed in version 8.5.1: • [1007266] Various counts in the Postfix Mail Server or Brightmail Gateway plug-in, such as "Messages blocked" were being counted for each recipient, inflating the count. • [1115018] The Sessions snapon is always installed with a timeout of 1800, even when the log format plug-in has a different timeout. Also, the Sessions snapon does not recognize "logout" events. • [1117001] When using MS SQL as the back-end database, and using bulk_insert as the insert mode, and using a wildcard report filter, report generation gives an error like, "BULK INSERT filtertmp_5852_0 FROM 'C:\LoadDataDirectory\results.aec.gov.au.2011.to.2012\sql_load_data_5852_seq0.tsv" • [1134950] On Mac OS, the Sawmill logo and and Use Sawmill window graphic are out of date. • [1136138] Network action accesses give an error, "Syntax error: Unknown variable 'action_fail' in expression" • [1137776] The "upgrade these 8.1 profiles to 8.5" text does not appear, when using Internet Explorer 6. • [1139624] A bug report submission can fail on 32-bit systems, given an error like, "Unable to allocate 4294958111 bytes of memory; maximum memory is 939524096, but 3654011 is already used, and no further memory can be freed." • [1144553] If a profile has session information, and the database directory is non-default, profile conversion will fail with an error, "Can't find tableAlias=, fieldName=ssession_id in table main_table," or "Unknown database field "ssessions" in cross-reference group." • [1149031] Fixed a bug where conversion of an 8.1 database to 8.5 would fail silently if the profile had no session information, eventually causing an error like, "Unable to read file LogAnalysisInfo/Databases//main/Tables/xref0/header.cfg." • [1156064] The get_report action does not include the main (non-aggregating) column in the XML result. • [1159985] For the log format "Aventail Client/Server Access," profile creation fails with the error, 'Error in profile_setup create_report(), the report element type "sessions_overview" is not supported.' • [1164125] In Sawmill Lite click on Log Source, Database Info, Update Database or Build Database in Config. Sawmill shows the "No permission to view this page ... " alert. • [1164128] In IE6 try to open the Config Option Database Info, Update Database, Build Database, Log Parsing Filters or Database Filters. They show a blank page indicating a javascript error. • [1164135] In IE6 go to Config Report Options and click the Config Options link. The navigation bar shows form elements (list boxes) from the underlying form. • [1164140] In IE6 open Config Log Parsing Filters or Database Filters. The textarea element aligns to the right. • [1164142] In IE6 or IE8 open Config Log Filters and click on individual log filters, then view a different page. Sawmill shows an alert about unsaved log filters changes although no changes have been made. • [1165249] Fixed a bug which could cause sporadic crashes of the subordinate web server process (resulting in images and other support files failing to load in the web interface). • [1165798] Go to Admin/User Settings and enter a new password, mistype the second password and click Save Changes. The changes are not saved and no error message is shown about the mistyped password. • [1201868] Fixed a bug where PDF reports were invalid, if they were generated in a language which used commas as decimal separators. New features in 8.5.1: • [1135664] Added a Subjects report and parsing of the email subject from lines with Subject: and subject= to the Postfix Mail Server plug-in. Version 8.5.0, shipped July 28, 2011 Bugs fixed in version 8.5.0: • [1084636] Asking for password to be emailed gives an error, "Invalid sender in send_email()." • [1093134] Database builds can crash if some xrefs are disabled. • [1094316] The "+" icons in the Session Paths report have no effect. • [1097200] The Pages/Directories report isn't hierarchically zoomable. • [1101001] Queries on real-time databases give an error, "Unable to read file LogAnalysisInfo/Databases/wri/main/Tables/xref0/header.cfg (Operation timed out)." • [1113948] A Flash Media Server profile created with x-duration unchecked in the New Profile Wizard, gives an error on build, "Syntax error: Unknown variable 'x_duration' in expression" • [1114148] Updating a database for a Flash Media Server profile, or any other profile with a "node" type database filter variable, may give an error like, "Couldn't find node 4616027 in untitleds." • [1114349] Network action access to get_report gives an error, "Couldn't find node header in untitleds." • [1114357] The Admin menu appears obscured by the Config menu, when the Config menu is pulled down. • [1123802] The Individual Sessions report contains enormous bogus session IDs in some rows. • [1129556] Viewing reports after attaching a "Particular File Access" snapon gives an error like, 'the database field node "accesses_on__robots_txt" refers to the log field node "" but this log field does not exist.' • [1130487] Clicking "Customize Report In Config" doesn't open the report--it opens the Report Editor, but not the report itself. • [1153681] Large integer values (e.g., byte numbers larger than 2GB) can be truncate to less than 2GB, or be negative, when reporting on logs on a 32-bit system. New features in 8.5.0: Version 8.5b5, shipped May 27, 2011 Bugs fixed in version 8.5b5: • [952934] Progress display shows "Generating report" during snapon attachment. • [1064630] Attaching the "Particular file access" snapon gives an unexpanded variable value "lang_admin.snapons.particular_file_access.parameters.pathname.parameter_value" as the default value in the "Pathname of file" field.

• [1069711] Attaching double_hits snapon gives the error: "Expected ADD or DROP in ALTER TABLE query, found 'double_hits'"

• [1069746] Progress displays on Windows during database filtering contain odd characters.

• [1070815] Parsing of Flash Media Server logs (or any other profile with many "float" type aggregating fields) on multiprocessor Windows systems is about 5x slower than it ought to be.

• [1074140] When attempting to attach the "Unique Values" snapon to a Flash profile, an error occurs, "Unknown variable 'lang_admin.snapons.unique_values.unique_field_name.ratio_field.parameter_value' in expression."

• [1075902] When creating a profile with a Shoutcast w3c log, after clicking "Finish" this error is displayed: "Snapon attempted to add database field 'session_id', which already exists"

• [1075940] Byte numbers are wrong (much too high) for Flash profiles.

New features in 8.5b5:

Version 8.5b4, shipped May 10, 2011

Bugs fixed in version 8.5b4:

• [991021] Viewing reports of a v8 profile converted to new format, gives, "Can't find tableAlias=, fieldName= in table xref0."

• [1058009] Visitors, and other "unique" fields, show zeros in some reports.

• [1062806] The Overview shows 0 for visitors, or other unique fields, for some (larger?) datasets.

• [1063606] Attempting to attach the "Particular File Access" snapon immediately gives an error, "Unknown variable 'lang_admin.snapons.particular_file_access.parameters.page_field.label' in expression".

New features in 8.5b4:

Version 8.5b3, shipped April 24, 2011

Bugs fixed in version 8.5b3:

• [981742] Building the database for a PIX profile gives an error, 'Unknown index colum "user" in main_table.'

• [986528] For larger datasets, unique numbers (e.g., visitors) may be too low, or zero, in the Overview.

• [988303] The Log Processing page in Config is empty, for some profiles.

New features in 8.5b3:

• [861054] Language, Thousands divider, and Decimal divider, are all now per-user options, and editable in Admin -> Settings.

• [1028838] Enhanced the Create Profile Wizard to allow plug-ins to prompt for snapon parameters at profile creation time.

Version 8.1.10, shipped September 23, 2011

Bugs fixed in version 8.1.10:

• [1059897] If a multiprocessor database build is started from the web interface, and if it fails with an error, the parsing server processes will not exit when the main process does.

• [1066255] When using the Salang connect()/disconnect() network features (e.g., in get_title_by_http), sockets are not closed when they disconnect, resulting in a file handle leak and eventually an error like, "Unable to create socket on port 80 of hostname xyz."

• [1069978] Enhanced the previously-seen-data skipping algorithm. If there was a file in the log source which contained just the header, and a second file in the log source, which started with the header, but contained additional information, data from the second file could be re-processed on update. This happened because the second file was assumed to be the first file with additional data, because it started with the same data as the first file. The problem was resolved by extending the checksummed range of the second file when it was first encountered, to include the additional data.

• [1088802] Convert a version 7 profile to version 8. Open the profile in Congfig/Reports. This causes a javascript error due empty graphs nodes in the report element.

• [1089110] Convert a version 7 profile to version 8. Open the profile in Congfig/Log Processing. This causes a javascript error due missing log_processing.output nodes.

• [1095674] Create a new profile, as log source select MS SQL Database (ODBC). Click Next. This will result in the error: Unknown variable 'v.fp.page_token' in expression This error has been reported by Michael via email with subject: Unknown variable 'v.fp.page_token' revisited

• [1099313] Enhanced the previously-seen-data skipping algorithm. If there was a file in the log source which contained just the header, and a second file in the log source, which started with the header, but contained additional information, data from the second file could be re-processed on update. This is a separate enhancement from the other, similar one made in this version. In this second case, data would be re-added because two existing checksums matched the second file when it was reprocessed: the first-file checksum (header only) and the second-file checksum (entire file, as extended by the first enhancement), and Sawmill would choose between them based on which one it had seen first, which could cause it to choose the first checksum, and from that infer that the second file was the first file, but with new data, which it would then process. This problem was resolved by always choosing the larger checksum (the one checksumming more data), rather than the earlier checksum as before; this causes it to choose the second-file checksum (extended), which causes it to infer that there is no new data in the second file.

• [1111615] The dump_main_table action gives an error like, "Invalid integer size 7235437916109933165 in LocalFileTable::GetIntCell()"

• [1115440] Viewing Config->Log Filters caused error with ASSP Anti-spam SMTP Proxy plug-in.

New features in 8.1.10:

• [1007263] Improve the functionality of the Cisco VPN Concentrator plug-in and added session reports.

• [1018138] The get_report action (or network action) has been greatly enhanced so it can get not just Overview reports, but also any other table report; and its XML output is much more complete now, including additional metadata about field labels, units, etc.

• [1064350] Improved autodection in the Juniper SRX240 and SRX3400 JunOS RT_FLOW plug-in.

• [1100086] Added support for the XML log format variant for IAS in a new plug-in called "Microsoft IAS (XML)".

Version 8.1.9, shipped May 6, 2011

Bugs fixed in version 8.1.9:

• [981415] City names with non-ascii characters appear garbled in the Countries report.

• [992699] Database update gives an error, "Unexpected = in group node (saved_collected_entries.*)" for certain unusual log data.

• [994193] IronPort C reports do not properly report bounced emails.

• [997621] The fields threatid, category, severity and direction are not being set in the Palo Alto Networks Firewall (Integrated Threat & Traffic) and Palo Alto Networks Firewall (Threat) plug-ins.

• [998378] If a syslog format is selected first in the Create Profile Wizard, the Next button has no effect.

• [1001540] The Database Performance Options page does not appear in the Create Profile Wizard when the licensing tier is Professional.

• [1004324] In IIS profiles, the "average time taken" field cannot be edited in the Database Fields Editor; it shows "Select aggregation method" as the aggregation method, and the proper value ("average") is not available.

• [1004509] Palo Alto Firewall profiles ignore events in certain Kiwi syslog installations.

• [1005040] In "-f v" debug output, curly brackets in matched regular expressions appear with an extra backslash before them.

• [1007653] A date filter like "1week" will select the wrong portion of the week, if the current date is different from the end of the log data.

• [1007730] If a custom delimiter is specified in Config -> Report Options -> CSV Export, visiting that page will give an error, 'setFormValue(), element with id "report_options:csv_delimiter:cutom" does not exist.'

• [1007731] When \t is used as a CSV export delimiter, it appears literally rather than as a tab.

• [1012167] When generating PDF from a profile which uses an HTML page header or footer with a long paragraph of text, the text does not wrap to multiple lines.

• [1012304] In PDF reports, page footers can appear at the wrong location vertically, either directly on top of the table, or too far below it.

• [1012312] Non-English characters (like ü) appear garbled in PDF output

• [1012652] When using MS SQL as a back-end database, zooming on an item containing a backslash (\) will give zero results.

• [1015345] On UNIX systems, a carefully constructed URL can cause arbitrary commands (command lines) to be executed on the server.

• [1015350] A security vulnerability exists allowing a cross-site scripting attack through the use of a carefully crafted URL containing JavaScript code, and triggering an "encountered an error while reporting an error" condition in the web server.

• [1015360] A malicious attacker can potentially perform Sawmill operations, if a user is logged into Sawmill while visiting a web site controlled by the attacker, and if the attacked knows the URL of the Sawmill installation.

• [1015369] If a network action is repeatedly called via HTTPS, with username and password missing from the URL or empty, it can result in a server crash or hang.

• [1016655] Labels are misaligned in very long-term date line graphs.

• [1016742] If custom network actions have been executed against the Sawmill server (uncommon), the WebServerAccessLog file contains the administrator password in plain text, a potential security vulnerability.

• [1017398] PDF generation can crash for reports containing a pie chart.

• [1021981] When logging in directly with a URL using lun/lpw, if already logged in as another user, it does not log in as the new user, but stays logged in as the old.

• [1032410] The average-per-day column of the Sessions Overview does not show an average for several rows which should have one.

• [1033422] PDF export of a report fails, if there is a pie chart and one of the items in the legend is literally "<>"

• [1038143] Switched cookie-setting method to JavaScript, to work around a bug (security feature?) in recent versions of Opera and Safari, which prevented login.

• [1048242] Improved the wording of the Session Contains filter description.

• [1060937] Filtering the Sessions Overview on a session field (like Session Pages) gives an error, "Attempt to join sessions to sessions_join on 's.loadorder = x.loadorder', but there must be one column from each of the two tables in the ON condition, and there doesn't seem to be."

New features in 8.1.9:

• [565794] Improved the counting of delayed, deferred and sent messages in the Sendmail plug-in. There is a Max Delay numerical column. There is also some support for Milter reporting.

• [1008462] Sessions reports have been restored to the Palo Alto Networks Firewall THREAT and Integrated plug-ins. Sessions are only for THREAT entries in the integrated plug-in.

• [1017423] Added support for Basic HTTP authentication of network actions, so the username/password used to authenticate the action is no longer visible in the URL.

• [1032057] Added new lines to the full log format list, with device name first, so devices can be found more easily by name.

• [1032860] Added a new option in RBAC to hide Help links.

Version 8.1.8, shipped January 17, 2011

Bugs fixed in version 8.1.8:

• [946520] Selecting Apache Custom in the log formats list gives a blank page, instead of prompting for the format string.

• [947994] Expiring (removing) data from an profile with a back-end Oracle database gives an error, "SQL command not properly ended"

• [955556] Indices can become corrupt on database update, causing the database to be much larger than it needs to be, and the update to take much longer than it should.

• [960979] When expiring all rows of a database, an error can occur, "Internal: Attempt to write header for read-only table unique_loadorder."

• [963660] Duplicating the first action in the scheduler inserts it first, but the display shows it second.

• [966350] In Config -> More Options -> Miscellaneous, the name of the DNS Lookup appears initially as "Support & Action Email."

• [966655] Relative date filters like "yesterday" offset the time zone in the wrong direction on Windows, sometimes resulting in the wrong day being displayed.

• [966926] Some PDF reports give an error when displayed with Windows Acrobat Reader, "An error exists on this page."

• [975496] Multiple copies of the same item, differing only by case (e.g., /Dir1 and /dir1) may appear in tables, even when the field is case insensitive.

• [975637] The "Distributed processing/Parsing server distribution method" options appear in Lite and Professional modes, though they have no effect except in Enterprise.

• [984449] If a profile is created by a particular user, then than user is deleted, and later a new user with the same name is created, the profile is shown to have been created by the new user.

New features in 8.1.8:

• [957993] Support for SHOUTcast log format versions 1.6, 1.8 and 1.9 are now in a single plug-in. The name is "SHOUTcast Media Server / DNAS (Distributed Network Audio Server)". A Players report has been added to this plug-in. The W3C version has also changed. The Web Browsers report is not called Players, and the Spiders report is gone from the Vistor System report group. The plug-in name is now "SHOUTcast Media Server / DNAS (Distributed Network Audio Server) (W3C)".

Version 8.1.7, shipped October 11, 2010

Bugs fixed in version 8.1.7:

• [871194] Documentation does not display in the user's selected language--it displays in the Preference language, regardless of user settings.

• [889872] Fixed a bug which, when MP query splitting was turned on, could cause an error like, "Lock disappeared on Task-85306-Lock, but SubqueryDone_85015_3 still does not exist; subquery process must have crashed!"

• [892452] Profiles which handle their own header parsing using filter_preprocessor, re-add data on a database update, if one of the log files is nothing but a header.

• [896386] An XSS vulnerability exists, potentially allowing an attacker to execute arbitrary JavaScript code on another user's system.

• [913690] Using a date filter on the Single-Page Summary gives an error like, 'Unknown configuration group "date_filter_info" in node "sessions_cache.1b41dc18092a36480b171fb9508386ec.profiles.access.report_ jobs.69931523589d08bcfe8b7dc9990d403c.report_elements.0" (030FADDC)'

• [914013] The profiles list in the User editor is not sorted.

• [915629] Removing data from a MS SQL database by date can give an error like, "The multi-part identifier dategetmp_X_Y.itemnum" could not be found.

• [923130] Profiles including a user-agent field can crash during database build (but usually don't).

• [927348] If there is an extra closing parenthesis in a log filter, at the top level, the remainder of the log filter will be quietly ignored.

• [927438] Viewing reports of Nortel ACD gives an error: 'The database field node "average_tsf" refers to the log field node "average_tsf" but this log field does not exist. It is recommended to manually correct the "average_tsf" database field node.'

• [927768] Log filter action "C" is labeled "CH"

• [933940] Turning on DNS lookup in a profile created with 8.1.6 gives an error on build, "Couldn't find node hash_table_expansion_factor in profiles..database.tuning"

• [934837] Command-line database updates do not correctly detect a running database build, and proceed with the update, potentially corrupting the database.

• [936430] Running Sawmill in CGI mode, with the directory containing Sawmill not writable by Sawmill, gives an error, "Can't open lock file Lock (Permission denied)."

New features in 8.1.7:

• [911484] Functionality was improved in the three Palo Alto Networks Firewall plug-ins, Threat, Traffic and Integrated Threat & Traffic. The use of the "time generated" timestamp as the Date/Time field has been restored. The Integrated plug-in is now fully in sync with the others. Specifically it now has all of the numeric fields, Bytes, Packets, etc, that are in the Traffic plug-in. Support for some variations in the log format were also added.

Version 8.1.6, shipped September 6, 2010

Bugs fixed in version 8.1.6:

• [841842] Creating and building the database of a profile with a real-time log source on Windows sometimes results in an error, "Unable to read file LogAnalysisInfo\Databases\sawmill_realtime\main\Tables\bottomleveldatebo ttomlevelitem\header.cfg (Broken link)"

• [851291] Creating a profile with Microsoft Media Server log data, and unchecking "session events" gives an error, "Syntax error: Unknown variable 'session_events' in expression" on database build.

• [851829] Added local time zone support for the date filter string option; defaulted to local time zone for UI date filters.

• [860107] When using the "auto" date format, years past 2030 are considered corrupt (which is a problem for Thai log data, since Thai years are around 2553).

• [862673] This was an escaping issue with control characters for all filter expressions added in Config (per profile, per report or per report element filter expression) or entered via the command line.

• [868898] The Salang function current_log_pathname() returns the wrong pathname, for the first file of a multi-file log source, when using multiprocessor log processing.

• [869075] Field values containing ASCII code 26 (control-Z) cause an error on report generation like "Unterminated quote in LogAnalysisInfo\profiles_cache\\raw_report_elements\39f3bc8e94a923143b0ef079f5dc4805.cfg at line 1282."

• [870672] After importing a profile from v7 to v8, the Database Fields page of Config may show "Select log field" as the log field, or may not display at all.

• [877311] Using a "session start" filter on the Individual Sessions report, causes an error like, "Invalid integer size 63 in LocalFileTable::GetIntCell()."

• [880490] Building Sawmill from the encrypted source code, on Fedora Core 13, gives an error, "error: invalid conversion from ‘const SSL_METHOD*’ to ‘SSL_METHOD*"

• [883332] Build a database with Oracle back-end, with a xref table with four or more non-aggregating fields, gives an error when indexing the xref table similar to: "create index x19_3xbx19_13xbx19_17xbx19_18xb on x19l0_0_0_0ux19 (bottomleveldate, cs_username, sc_filter_result, sc_filter_category)'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Oracle][ODBC][Ora]ORA-00972: identifier is too long"

• [884348] When using an Oracle database with the Oracle client driver, if a string field has a value longer than 200 characters, but shorter than 255 characters, it will generate an error "value too long."

• [885674] When using the internal database, main table indices can become larger and larger (more than expected) as database updates occur.

• [886062] Building a database with an external SQL database back-end, and with a SQL prefix, results in indices with names do not include the prefix, potentially causing index name collisions on Oracle, if using multiple databases in the same server.

• [887931] If a database field name is too long, and an external SQL database is being used, report generation will fail with an error like, "Unable to Execute ODBC Query='select x.bottomleveldate, count(x.bottomleveldate) from zxref0 x inner join zbottomleveldatebottomlevelitem b on x.bottomleveldate = b.bottomlevelitem group by x.bottomleveldate'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Oracle][ODBC][Ora]ORA-00972: identifier is too long." This is more common on Oracle, where any hierarchical database field longer than 15 characters will cause this error.

• [888885] Many plug-ins, including Tomcat, show duplicate Day of Week and Hour of Day reports, both in the "Date and time" group and at the top level of the report menu.

• [890225] The built-in function convert_charset() deallocates memory improperly, resulting in crashes or incorrect results in some cases where it is used from a custom log filter.

• [891575] The default "web browser" is reported as "Netscape Navigator" for unknown "Mozilla" user agents (changed it to "Unknown Mozilla").

• [894120] When a report is filtered with a filter which excludes all rows, it generates an error like, "Unable to read 304 bytes from LogAnalysisInfo/Databases//main/Tables/_select_result_21095_0/data.tbl.moved; got only 0 bytes (No such file or directory)"

• [894684] When using a SQL table prefix with non-internal database, database export fails with an error like "Table 'profile.main_table' doesn't exist"

• [896378] A non-administrator can access Setup Wizard with a specially formatted URL.

• [896381] Non-administrators can create/delete user accounts with a specially formatted URL.

• [896383] By changing local JavaScript variables, a malicious non-administrator can gain access to some sections of the user interface restricted to administrators.

• [906782] Import of exported database into an Oracle database gives an error like, "#### Unable to prepare ODBC query: ODBC error: rec1: SQLstate: S0002; msg=[Oracle][ODBC][Ora]ORA-00942: table or view does not exist."

• [911657] On Oracle databases using Unicode charset, database builds with fields with long string values can give an error like, 'ORA-12899: value too large for column "SYSTEM"."FILE_TYPEITEMNUM"."FILE_TYPE" (actual: 378, maximum: 255).'

• [914898] The Duplicate Profile popup windows displays a box above it containing the word "false"

New features in 8.1.6:

Version 8.1.5, shipped June 2, 2010

Bugs fixed in version 8.1.5:

• [797770] Update a database after removing data can cause an error like, "Short read in SingleWindowCachingBuffer file LogAnalysisInfo/TemporaryFiles/Task24630/ris_main_table_x_loadorder_le_1014599626/data.tbl; tried to read 1 bytes from position 1014599626, but only got 0 bytes."

• [798195] Creating a profile using the WarFTPalt plug-in causes a Sawmill alert, 'Unknown configuration group "destination_ip" in node "profiles.profilename.log.fields"'

• [798479] In the Session Paths report the pages are not in chronological order.

• [806410] Bug fixed where percentages were wrong when "Use overview for totals" was set for reports, and "Show total value" was set for columns of type Unique.

• [808447] If Users/Roles (RBAC) has "view only" changes made to permissions, it now no longer displays a "confirm message" asking the user to save changes before leaving the page. The toolbar message is also improved to be "Your grants for this page are limited to view and changes cannot be saved."

• [814810] When an incorrect password is entered for an SFTP log source, the error message is "unknown keyboard-interactive response: 1".

• [816641] Fixed bug which could cause 0's in the Overview, after removing data, and updating with new data. This could also cause an error about reading past the end of a temporary file.

• [818357] Using an FTP log source to certain versions of Pure-FTPd server can result in an error like, "Unexpected response from FTP server: 150 4.977 seconds (measured here), 0.50 Mbytes per second."

• [823230] Bug fixed where Internet Explorer 8 shows skewed calendar months.

• [839414] Filtering on a single second gives an error like "No date applied. The date filter '7apr1998_165306'" is invalid.'"

• [842123] Export of a database from the command line (with -a ed) generates empty itemnum tables.

• [843395] Command-line export (-a ed) of a MySQL database gives an error, "No database selected"

• [844142] Building an Oracle database with some versions of the Oracle Client driver gives an error, "Internal Error: Attempt to get SQL_INTEGER value from a SQL_FLOAT"

New features in 8.1.5:

• [800148] Added support for Microsoft Exchange Server log data, logged through a syslog server.

• [814809] Extended the command-line authentication protocol so the script can print *ROLE*:N to indicate that the user logging in is part of role N (N is the internal role node name).

• [816854] Added a "Automatically direct to reports upon login if user accesses only one profile" to the Users editor.

Version 8.1.4, shipped March 30, 2010

Bugs fixed in version 8.1.4:

• [767223] When using multiple log sources, the progress displays count of bytes processed restarts at zero for each log source, instead of showing the total cumulative bytes processed.

• [770870] The built-in Salang function convert_charset() allocates memory equal to the string to be converted, and does not deallocate it. This can result in high memory usage over long builds, if log filters use this function.

• [785338] After doing a "remove database data" operation, the Overview hangs for a long time with no progress display.

• [785650] The week_of_year field uses 0 for the first week of the year, instead of 1 as documented.

• [790229] The "config" file of a database is not updated when xrefs are rebuilt using the "-a rcrt" command-line option.

• [791207] When using the command-line authentication script, the script is called for every click, rather than just once per login session.

• [793905] After creating a report using "Generic W3C Web Server" as the format, reports give an error, "Unknown variable 'volatile.new_profile_name' in expression"

• [794485] On MacOS 10.6, the "run at startup" option does not start Sawmill successfully.

• [795215] Multiprocessor builds of profiles using the global date regular expressions option, can incorrectly reject some log entries.

• [796244] Fixed problem where files with Kiwi ISO syslog headers autodetect as "Kiwi (mm-dd-yyyy dates)" as well as "Kiwi (ISO/Sawmill)". The ISO files have the year first (yyyy-mm-dd), so selecting the wrong syslog completely prevents the logs from parsing. Files with the year last in the date are detected as both "Kiwi (mm-dd-yyyy dates)" and "Kiwi (mm-dd-yyyy dates)" so that the correct date can be selected. Now files with the year first are only detected as "Kiwi (ISO/Sawmill)".

• [798584] The SHOUTcast W3C Log Format plug-in did not parse correctly after encountering a second W3C field header in the same file. This bug was introduced in Sawmill 8.1.3 in a change to the plug-in that worked around for a logging problem in SHOUTcast 1.9.8.

• [798843] Log Detail gives an error, if the number of rows in the filter set is less than the number of rows displayed in the table, like: "Unable to read 392 bytes from LogAnalysisInfo\Databases\profile\main\Tables\_select_result_1344_ 0\data.tbl.moved; got only 0 bytes".

• [799951] The built-in Salang function collect_listed_fields() does not work properly when the divider or separate parameters are not constants.

• [804591] When using command line authentication, logging in as root administrator causes an error like: Unknown configuration group "user_grants" in node "sessions_cache.479a3ff1bcb1ea9ba79e4f2113018196.session_info" (02705354).

• [806853] Profile patterns entered in the Scheduler were are treated as patterns unless they are preceded by "pattern:".

• [806929] The Windows installer does not properly install the Microsoft 2008 redistributable packages, which are required for Sawmill to start on some older versions of Windows.

New features in 8.1.4:

• [686796] Support for a new Windows Event Log variant has been added. It is comma separated and has a m/d/yyyy date format. The fields are Level, Date, Time, Source, Event ID, Task Category and Message. The plug-in was tested using files from Vista logs with 24 hour times and Windows Server 2008 logs with AM/PM times.

• [781872] Support has been added to the FortiGate Traffic Log Format plug-in for a format variation that is comma instead of space separated.

• [799782] Added support for a format variant of the Astaro SMTP Proxy Log Format which has a single space after between the To email address and the "Reason". Other variants have a field there whose value is ignored. The result of not allowing this was rejection of most To lines (=> lines).

Version 8.1.3, shipped February 12, 2010

Bugs fixed in version 8.1.3:

• [742760] When the language in the Preferences does not match the language for the current user, reports can contain a mix of languages.

• [744621] Database build, using an Oracle database, with a database field whose internal name exceeds 30 characters, fails with an error "ORA-00972: identifier is too long"

• [752760] If all lines of log data are rejected during log processing, the resulting database, which correctly shows zeros for all values, incorrectly shows "01/Jan/1970, 1 day (entire date range)" as the date range in the Reports page.

• [752760] The date range at the top of Reports shows 1970, when there are no entries in the database.

• [758276] When the Log Detail is filtered, then paged, the second page data matches that of the first page.

• [759965] Building a database with more than 64 main table indices (e.g., a default profile with more than 64 non-aggregating field) gives an error: Too many keys specified; max 64 keys allowed.

• [763994] Apache Custom format strings containing literal \" values do not parse properly.

• [765835] The Concurrent Session number is computed from the unfiltered session data--report filters have no effect on it.

• [767731] The "+" operator does not match properly in regular expression log sources when using Show Matching Files in Config->Log Sources.

• [767843] The Windows installer does not install the VC++ 2008 redistributable libraries, which can cause errors when trying to start Sawmill, if the libraries are not already installed.

• [772469] SFTP log sources give an error "Unknown SSH server prompt 'Password: ' -- only 'Password:' is supported" on certain SFTP servers (specifically, SUSE Linux Enterprise Server 10).

• [775640] Building a database with Microsoft SQL Server, in a profile with a session analysis, where the name of the session page field is "file" or some other reserved SQL keyword, gives an error like, "Incorrect syntax near the keyword 'file'."

• [776369] When password expiration is configured in Preferences, an error occurs when the password expires: "Couldn't find node login_plugins in".

• [776937] The Session Paths report can cycle back on itself, making sessions infinitely deep, when using the web UI.

New features in 8.1.3:

• [704687] Merged changes from Sawmill 7 that were never replicated in Sawmill 8. This adds support for a new format variant and extracts some additional fields.

• [717071] Added support for an all caps string instead of a hex number after the url in the Barracuda Spyware Firewall / Web Filter Log Format plug-in. This field is ignored in all cases because it's purpose is unknown.

• [718244] Reporting of pass throughs (BP events) was added to the Cisco Wide Area Application Services (WAAS) TCP Proxy log format plug-in for WAAS 4.1.x. Counts of connection starts, pass throughs and two types of connection ends were added to the numeric fields.

• [748047] Implemented support in the SHOUTcast W3C Log Format plug-in for a logging problem in SHOUTcast 1.9.8 which causes numeric values Play Duration, Server-to-client bytes and Average Bandwidth. The W3C log format specifies that if there is no value in a field, it must be replaced with a dash, but SHOUTcast logs sometimes have a completely empty cs-user-agent (player) field which caused the remaining fields to be shifted and the value of sc-bytes to end up in the x-duration field. The modified plug-in compensates for this problem. This is an important concern for Soundexchange RIAA reporting.

Version 8.1.2, shipped December 10, 2009

Bugs fixed in version 8.1.2:

• [629538] When using MYSQL 5.1, database data removal fails with an error, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'as x using main_table as x where (x.date_time < 'YYYY-MM-DD HH:MM:SS')' at line 1"

• [634747] Changed the way play duration is distinguished from non-play duration in the Wowza Media Server Pro plug-in. This fixes a bug in 8.1.1 where the stream duration is incorrect when there are pauses or seeks. Also restored full tracking of x_duration in the stream_duration field and added the play_duration and pause_duration numeric fields. Added check for no publishing start time because we it is possible to have no c_client_id values on unpublish events.

• [676495] Reports show incorrect results, when zooming on multiple days in the Days report, and simultaneously zooming on multiple items in some other report.

• [680285] Profile creation fails, when using Aventail Client Server log format, with an error, "Could not find main_report_field_name in log format report description 'connect_tunnel_sessions_overview'".

• [686802] When using the Microsoft ODBC for Oracle driver, Sawmill gets an error on database build: Internal Error: Attempt to get SQL_DOUBLE value from a SQL_DECIMAL ODBC table column: 0. Note: with this bug fix, this error no longer occurs; however, other errors may occur because Sawmill requires an ODBC 3 driver, and the Microsoft ODBC Driver for Oracle does not support ODBC 3 (it supports only ODBC 2). Therefore, do not use this driver with Sawmill; instead use the Oracle Client driver (provided by Oracle), or another third-party ODBC driver for Oracle.

• [686819] Building a database in a profile with session information, using an Oracle back-end database with the Oracle Client driver, gives an error, "Internal Error: currentBufferColumn=8 in UploadChunk() for sessions_update."

• [686901] If two reports are run simultaneously against a real-time database, the second report can "collide" with the resuming database build, causing various errors.

• [688118] Database xref tables, in the internal database, are much larger on disk that they need to be, for some datasets.

• [691995] Internal database indices skip a row when being updated. Among other possible effects, this can cause one day of data to be missing from the reports, and the Date Picker.

• [692073] The order of log sources in Config -> Log Source is not permanently saved when they are manually reordered.

• [693811] The name of the plug-in the Generic W3C Log Format has changed to Generic W3C Web Server Log Format to indicate the basic kinds of fields it expects to find. The way fields are created has been changed overcome a bug where the creation of some derived fields with functions caused an error because the database fields did not exist.

• [699372] A Microsoft SQL Server log source give the error "Invalid log source. Numeric valid out of range" when one of the selected columns is a BIGINT, and one of the values is larger than about 2 billion.

• [703189] Use of "create_many_profiles" gives an error: "Syntax error: Unknown operator in expression" due to a syntax error at the keyword "string".

• [707419] HTTPS does not work on Windows (it starts an HTTP server instead).

• [707422] Database updates using an Oracle database server give an error, "ORA-00001: unique constraint (STAT.LOADORDER) violated"

• [710643] Double-quoted W3C fields values which with with a literal double-quote (escaped as two double quotes), e.g. appearing literally in the log data as """"hello""" to indicate a value of a double-quoted value hello, are treated as an empty value.

• [713843] Importing the database of a Sawmill 7 profile using MySQL, and changing the database name during import, gives an error, "Table '.locationitemnum' doesn't exist at ../src/run.cpp:2768"

• [717967] Database updates can crash during the index build/merge step, for very large datasets.

• [725411] The Overview report in the Single-page Summary sometimes shows incorrect average bandwidth values.

• [737620] ISA CSV log data is very slow to process, due to a problem normalizing the highly unique filter_info field.

• [741574] With IronPort C-Series logs, relayed SBRS events are reported as "rejected."

• [743313] Filtered date/time reports can display graphs from previous cached filtered versions of the report, instead of the correct graph.

New features in 8.1.2:

• [573418] Added basic support for keyboard-interactive sshd authentication; this allows SFTP access to default FreeBSD systems, and others using keyboard-interactive with a "Password:" prompt.

• [731045] Added support for Windows 7 detection in user-agent string.

Version 8.1.1, shipped September 8, 2009

Bugs fixed in version 8.1.1:

• [659410] Creating a profile with Mirapoint SMTP Log Format gives an error, 'Unknown configuration group "date_time" in node "profiles..log.fields"'.

• [662875] When using Professional trial mode, Save Report To Menu gives an error, "Checksum does not match for file 'templates/statistics/save_as_new_report/get_report_dat.cfv'. Your licensing does not permit editing of template files, but it looks like that file has been edited. Please upgrade to Enterprise licensing if you need to edit templates, or remove the edits from that file to continue with your current licensing."

• [662959] Updating a database when the database has not yet been build gives an error, "Can't find tableAlias=main_table, fieldName=bottomleveldate in table main_table_join_sessions_join".

• [665590] Use of an ODBC log source on a Microsoft SQL Server with a BIGINT field, results in an error, "Invalid log source Unknown ODBC type; 5"

• [665606] Cross-reference builds done with the internal database, with large datasets, sometimes give an error like, "Internal: list 99 in IntegerLists .... ends with -3412465, which is the start of a range"

• [671050] In some circumstances, dates which have data are not clickable in the Date Picker.

• [671133] If a scheduled task is running when a second scheduled task is scheduled to run, the second task does not start, in some cases.

• [671684] A session contains filter, using wildcards, on a page which does not exist, gives an error, "Internal Error: Empty node name."

• [671949] The global_date_filename_regular_expression has no effect in multiprocessor database builds.

• [679482] Fixed a bug which could cause an error on database build or update, similar to "#### Internal: listitem=338931424, which is larger than the array cache size (308152)."

• [680155] Database builds may crash at the end of log reading, if DNS lookup is turned on, and parsing servers are being used for multiprocessor building.

• [680181] Numeric comparisons in report filters give incorrect results.

• [680671] Running a multiprocessor "split" build (using Sawmill to split data into multiple per-profile log datasets) can result in an error "mismatched brackets" in the info.cfg file.

• [681790] Removing data from a database without session information gives an error, "Can't delete file LogAnalysisInfo/Databases//main/session_table_info.cfg".

New features in 8.1.1:

• [606985] Added a plug-in to report on OpenFire IM logs. These are XML logs and the lines can be extremely long. Testing was done by introducing newlines between closing and opening packet tags.

• [661926] Added new support for AIX CPU Utilization log format.

• [667394] Added support for Communigate Pro 5.2 to the Communigate Pro Log Format plug-in. Improved tracking of multiple recipients. Eliminated counting a message twice when ACCOUNT/delivered and DEQUEUER/LOCAL/delivered are both in the log. Added basic support for DEQUEUER/LIST/relayed. Eliminated underused operation field and added an action field just for delivery type. Added Queue ID field.

Version 8.1.0, shipped September 8, 2009

Bugs fixed in version 8.1.0:

• [567010] Switching between "single color" and "numerical field color" in Config/Report Options/Graph Color Schemes sometimes has no immediate effect in the report output; if there is a cached report with the old setting, it is still used.

• [591276] Many of the Database Tuning options have no effect, and should be removed from the web UI.

• [597332] Reports show no rows when using a comparison filter (e.g., >=, <=, <, >) on a field which is the same as the main field of the table, but when the table is not showing bottom-level items. In particular, this happens when using a date range filter on the Years or Months report.

• [603582] A version 7 profile put in the version 8 profiles directory (instead of being imported properly) gives a cryptic error: 'Unknown configuration group "server" in node "profiles.my_v7_profile_1.database.options"' instead of a better error like "you can't put a v7 profile in your v8 profiles directory."

• [605703] Zyxel Firewall Welf profiles are missing some of their Security reports.

• [613325] Parsing regular expressions extract log fields only as far as the first non-matching ()? section, if one or more of the ()? sections does not match.

• [613884] Running "remove database data" for an internal-database profile, from the Scheduler, gives an error, "Error parsing SSQL query; expected ')'; found ''.

• [614397] Shoutcast W3C profiles can give an error: Unknown configuration group "session_event_type" in node "profiles..log.fields"

• [617076] If the Log Detail report has been customized to include a non-timestamp date/time field, it will generate an error when exported to CSV, e.g. "Attempt to read beyond end of LogAnalysisInfo\Databases\\main\items\date_time\offsets_by_num (fileSize=4000); attempted to read from 669806112 to 669806116."

• [619197] The Overview report shows incorrect numbers (approximately double the correct values), when filtered with a NOT filter on a single value.

• [619738] Internal database index builds are extremely slow for certain very large datasets.

• [621923] Exported Log Detail contains one less row than the number shown in the UI table.

• [622939] Documentation search gives an error, 'Unknown configuration group "options" in node "database"', in installations which have been upgraded from pre-8.0.9 installations.

• [623023] The encrypted source compilation fails with an error about unknown symbols, when ODBC is enabled, and ODBC driver manager headers and libraries are installed.

• [623070] Sawmill 7 profiles with session information give the error, "Unknown database field "ssession_entrances" in cross-reference group," on database build or report generation, when converted to Sawmill 8 profiles.

• [623118] Unique numerical fields (like "visitors") show 0 or a sum in the Total row by default, instead of the intended "-".

• [629646] In CGI mode on Windows, when using a temporary directory/URL, CGI export fails with an error: Unknown fileref file 'csv_export\username\profile\reportname.csv'

• [630805] Fixed a bug which could cause database corruption when using a database created and updated with Sawmill 8.0.8 or earlier, and updating it with Sawmill 8.0.9. Possible error messages include, "Attempt to read beyond end of LogAnalysisInfo\Databases\\main\Tables\xref\sets\\lists.dat (fileSize=); attempted to read from to " and "Unable to allocate 4294967292 bytes of memory ()" (or similar large number).

• [633524] Cisco Netflow profiles give an error, "Can't find tableAlias=, fieldName=start_time" when the database is built.

• [635081] SFTP support is not enabled in the binary distribution for x86 Linux (ES/AS 5).

• [636792] The field_length parameter to database fields, which specifies the maximum length of the database field value in characters, does not appear in the web interface.

• [637463] If a database update finds no new lines in the log data, future updates will not update their cross-reference tables correctly. This causes the newest data to be missing from the date range, Calendar, Database Info, unfiltered Overview, and other unfiltered top-level reports.

• [638943] Creating a profile using the border_manager or iisweb_with_syslog plug-in gives an error, "Unknown variable 'util.parse_w3c' in expression".

• [649962] If the profile name is too long when using an Oracle database, the database build fails with an error about the length of an index name.

• [649978] Regular expression report filters on an Oracle profile give an error about "regexp binary."

• [650092] Oracle Database builds use hundreds of cursors, and do not clean them up properly, which can cause the build to fail with a "too many cursors" error.

• [656549] Columns do not line up properly in emailed table-with-subtable reports--numerical columns are one column to the left of their header.

• [660518] Removing database data, then viewing a report which requires an index, sometimes gives an error like, "Internal: found leftrownum 1303 in index LogAnalysisInfo/Databases/ae/main/Tables/main_table/indices/mt_hit_type, but the table only has 1301 rows"

• [663015] A "remove database data" operation on an internal database does not properly track what data is new when updating xref tables, resulting in some data being missing from top-level reports.

New features in 8.1.0:

• [580203] Added an option to duplicate a profile from the web interface.

• [580204] Added an option to rename a profile from the web interface.

• [604421] Enhanced documentation search to show a match when any word in the phrase matches, instead of requiring the whole phrase to match.

• [610256] Added a new custom action, convert_version_7_user, to convert users from a Sawmill 7 installation using the command line.

• [637041] Added support for HTTPS in Sawmill's built-in web server.

• [637044] Added a network API, for running custom actions by accessing the Sawmill server using a specially formatted URL.

• [637549] Added new support, or enhanced existing support, for the following log formats: AIX messages, Citrix NetScaler, Clearswift MIMEsweeper, Juniper Netscreen, Kingdon Firewall, Marshal8e6 Content Appliance, Open WebMail, Watchguard Firebox XTM.

• [643222] Added a plug-in to support the Pure FTP (Syslog) Log Format. The syslog format is the typical Unix daemon format and is different enough from the standalone version of the format that a separate plug-in is required.

• [655308] Added support for two types of events from Citrix Netscaler 9.0 to the Citrix Netscaler Log Format plug-in. Also increased the number of types of 8.x events that are supported.

Version 8.0.9, shipped June 23, 2009

Bugs fixed in version 8.0.9:

• [532124] Filtered reports show higher numbers than unfiltered reports, when using "is NOT" filters.

• [532124] The description of report filters sometimes contains a "not" even when it isn't a "not" filter, if there is also another filter which is a "not" filter.

• [545749] Indices exceeding 4GB are truncated on 32-bit platforms, resulting in error like "Internal: itemnum=7081689, which is larger than the array cache size (7081664)" when generating reports from large datasets.

• [554819] Multiprocessor database builds create zombie processes on UNIX systems, which last for the duration of the build; on systems with very few processes available, this can cause errors during build, including, "Error spawning process for multi-threaded SSQL query split."

• [584458] The sort_by (-sb) and sort_direction (-sd) command line options for "-a ect" are not documented.

• [586149] Profiles with duplicate labels are allowed by the Create Profile Wizard

• [590678] Log detail report is very slow, when using with a large internal database and no filters.

• [591432] The update.pl script in the Extras folder does not copy the system.cfg file, so installations updated with that script rerun the installation process.

• [591629] Scheduled tasks with a single profile, and multiple actions, run all actions simultaneously, instead of running them in sequence.

• [592187] Database indices, and unique-tracking lists, can become corrupted on 32-bit systems, when processing large datasets (typically, any dataset which would require an index > 2GB, though it could happen as low as 600MB). This can give various errors about "array caches" or "ranges."

• [592295] When using file-by-file distributed parsing, and "skip previously seen files on update," the list of previously-seen files is not recorded properly, resulting in some files being re-processed on update.

• [595619] Imported v7 profiles show integers in the sum row of "start time" or "end time" columns, instead of showing "-".

• [596194] When filtered, the Sessions Overview sometimes shows a negative number for "sessions for one-time users", and shows a few other incorrect values.

• [596597] Database fields with large aggregated values (more than about 10 billion) incorrectly show very small values (< 10) in the Overview.

• [596753] The current_log_pathname() function returns the wrong pathname for some log lines, when using multiprocessor database builds.

• [597095] Removing database data with a date range in a MS SQL database gives an error, "Invalid column name."

• [597455] Database indices can become corrupt on update, when using the internal database. This can cause errors when displaying filtered reports, or errors during future database updates. Possible error messages include "Unable to allocate N bytes of memory (Reallocating array cache)", "Internal: list N in IntegerLists X ends with -M, which is the start of a range", and possible others.

• [597949] In real-time profiles, reports hang at 33% during the initial database build, if the build is using parsing servers (multiple processors), or if the file-by-file option is turned on.

• [597953] Date filter expressions containing capital letters give an error, "Date filter not valid."

• [598038] Log format plug-ins which use "collected listed", where field values are quoted (e.g., Fortigate), ignore the values immediately after quoted fields.

• [601338] Cross-reference tables of imported Sawmill 7 profiles do not contain numerical session fields, resulting in slow report generation.

• [601365] Regular expression or wildcard filters on the "hour of day" or "day of week" field are displayed incorrectly in the yellow filter description at the top of reports, as "corrupt date/time".

• [601810] Database updates applied to databases imported from Sawmill 7, or to databases using non-internal database servers, do not propagate the new data to the xref tables in some cases, resulting in reports which do not have the latest data.

New features in 8.0.9:

• [526083] In the Juniper/Netscreen Secure Access Log Format plug-in, split existing numeric fields into four distinct fields for Connection Duration, Web Access Duration, Planned Meeting Duration and Meeting Attendee Duration.

• [569464] Added support for a new format variant of the NcFTP Xfer log. For now, the new fields at the end of each line are being ignored.

• [584392] Added support for overriding the port of SFTP, by using "hostname:port" in the hostname field.

• [606010] Added support to the Citrix Netscaler Log Format plug-in for a log format variant with an integer before the second colon (:). The integer is currently not in a report or numeric column because we have no information on what it is for.

Version 8.0.8, shipped May 20, 2009

Bugs fixed in version 8.0.8:

• [527528] The "Lookup pages" link in "Paths through a page" gives an error, "Syntax error: Unknown variable 'xyz' in expression," with certain datasets.

• [538929] The return address option for Action emails in the profile is not editable from the web interface.

• [543954] Database fields with identical labels (but different internal names) cannot be distinguished in the web UI.

• [548862] Building a database from Quicktime Streaming Server log data which has no cs-uri-stem field gives an error, "Can't find tableAlias=main_table, fieldName=cs_uri_stem in table main_table".

• [549277] Information about the extra MaxMind databases (Organization, ISP, and Domain) is missing from the documentation.

• [550333] Salang truncates fractional floating point numbers to 6 decimal points, when converting them to strings, and also unnecessarily converts parameters to strings and back when calling subroutines. This could cause graphs percentages to incorrectly show 0%.

• [550413] PDF report generation from the Scheduler on Windows gives an error if the pathname contains "\r"

• [552621] CSV files sometimes parse incorrectly when using multiprocessor parsing--headers are not distributed properly to all processes, resulting in some log lines being ignored.

• [553307] ODBC log sources do not support TINYINT fields.

• [554929] The Help link in reports replaces the current report with the documentation (instead of just opening a window with the documentation in it).

• [555013] The Advanced licensing tier is still mentioned in the trial mode page, and the trial mode menu.

• [555297] Charset conversion does not occur on CSV export from the command line

• [555602] The "remove data" action does not complain when the -df option is used; it does not support the -df option, so it should throw an error message if -df is attempted.

• [558333] The Calendar report fails or crashes, if there is no data (no accepted entries) in the database.

• [560172] The Create Profile Wizard hangs at the database page (won't click past it), if the log format has only one numerical field.

• [560264] Report generation can hang when using the internal database, if a report contains more than one column displaying the same "unique" field.

• [560324] Verbose filter (-v f) output causes an error ("unknown value") when used with replace_last().

• [560565] Database date range is reported incorrectly when all xref tables are disabled.

• [560818] Creating a profile using Cisco NetFlow (flow-export) format gives an error: Unknown configuration group "location" in node "profiles.*.log.fields"

• [560832] Emails sent by Sawmill from Windows systems have an incorrect SMTP Date header (off by one hour).

• [561105] The .= operator does not work on log fields.

• [563263] Several report filter examples involving date ranges, in the documentation, contain syntax or usage errors.

• [564442] Customizations to spiders.cfg are overwritten when upgrading

• [565897] The date_filter option "1day" selects the most recent day from now, rather than selecting the most recent day of the log data.

• [569311] Building a database from Firewall-1 NG (text export) Log Format gives an error, "Unknown variable 'elapsed' in expression," if the exported data does not contain an "elapsed" field.

• [569542] Import of Sawmill 7 database fails with the error "Syntax error in line ... of file ...; no '=' in itemnum record" when one of the items in the database contains a literal line break character.

• [569615] The CFG newsletter documentation refers to "1" when it should refer to "$1". • [573522] The DSN parameter is not saved by the Config section of the profile. • [576315] The options in "Admin > Preferences > Support & Action Email" show no "i" icon documentation. New features in 8.0.8: • [531179] Added tracking of the "action type" field in Barracuda Spyware Firewall / Web Filter Log Format. • [532973] Created a plug-in to support the Palo Alto Networks Firewall Traffic Log Format. • [550936] Improved the performance of database building for large Postfix datasets, and probably other large datasets, by defaulting to keep itemnums in memory, and to use rand_hash as the itemnums hashing function when none is specified. • [554267] Added documentation of the custom actions (-a command line options) in LogAnalysisInfo/actions, so the -a section of the option documentation. • [555124] Improved progress for index builds, to show the number of indices built, and the total number to build. • [565553] Added support to the Firewall-1 (fw log -ftn export) Log Format plug-in for log variant with no date in the log file, but log files containing a date in this format _yyyy-mm-dd_, as in this example: o_2008-01-28_000000.log. • [570348] Improved performance of filtered reports using indices on the main table (which is generally any filtered report other than those using date/time filtering). • [574676] Improved performance of loading large normalization tables--this improves the performance of hierarchy table builds, and probably some reports also, especially when fields have a huge number of unique values. • [574677] Improved progress display during build of hierarchy tables. • [576873] Added a new custom action create_user (-a cu) for creating or updating users from the command line. • [579290] Improved performance of database updates with external SQL database server, by enhancing itemnum upload to upload only new itemnums on database update, instead of re-uploading the whole list for every update. Version 8.0.7, shipped April 17, 2009 Bugs fixed in version 8.0.7: • [520725] Common Access Log profiles with large datasets use a huge amount of memory on database builds. • [528835] The parser adds "{default}" to the end of any log field value which ends in the hierarchy divider specified by its log field, even if the field is not hierarchical. I.e., {default} or "(default page)" sometimes appears unexpectedly in reports of non-hierarchical, non-page fields. • [529667] Some strings in the web interface are in English, even when non-English translations are used. • [533349] Ironport C Series reports do not show recipients when message rewriting occurs in the log data. • [534986] Session ID reports appear as integers in the Individual Sessions report, on profiles where the session ID is computed by log filters, instead of appearing as the computed value. • [535446] The IPC folder of LogAnalysisInfo collects many files with names HTTPRequest*, which linger for hours; these could be cleaned up much sooner. • [535576] Building a database for a MS SQL profile will give an error like, "drop the index 'main_table.mt_field_name', because it does not exist in the system catalog," if another profile, or a profile using different prefix/suffix, has used that same database to build a Sawmill database in the past. • [536085] Firewall-1 NG (text export) Log Format format does not support hh:mm:ss format for the "elapsed" field. • [536108] Database builds started from Config -> Database Info immediately display a "build completed" page, before the build is actually completed, when a profile is "real time." • [536525] PDF report generation sometimes crashes on x64 Windows. • [538883] Filters created in the Filters window using "is NOT item name" appear as "is item name" in the filter description of the report. • [539107] During a multiprocessor database build, the IPC folder of LogAnalysisInfo collects several files with names like ParsingServerPort_*.done, which linger for days; these could be cleaned up much sooner. • [539683] PDF export fails from the scheduler when using a drive letter in the pathname, with an error like "Can't create directory c: (File exists)". • [539696] Session duration shows 0 for profiles without a session ID field, and with "maximum session duration" set to 0. • [539814] Individual Sessions report of imported Sawmill 7 profile gives error, "Unable to read file LogAnalysisInfo/Databases/mms_pub_imported/main/Tables/ssession_idsubitem/header.cfg". • [540871] Deleting a database field does not delete it from all cross-references tables. • [541655] Schedules configured to "update all profiles" run all updates simultaneously, rather than sequentially. • [541975] The "Creating Many Profile in a Batch" topic is missing from the FAQ. • [543436] Some command line operations give the rather cryptic error message "Couldn't find node licenses in" when there is no license installed. • [543560] When using Firefox, page formatting is messed up if a report is generated, the server is stopped, the Admin page is accessed, the server is started, and the Admin page is accessed again. • [543657] Profiles with a field named "level" give an error "invalid identifier" when used with Oracle. • [543851] An internal string management issue could cause crashes during database builds or updates, when using a SQL database server. • [544036] The "Session ID" profile option is not customizable in the web UI. • [544295] Database updates can give an error, "Unable to allocate X bytes of memory (allocating preconversion buffer)" where X is a huge number. • [545134] Running the Windows installer to upgrade overwrites the preferences.cfg and default_profile.cfg files from the previous installation. • [546690] The label of the report element filter field has a typo: "epxression" • [548666] Log data with pseudo-W3C headers starting with "# Date\tTime" (like some Exchange 2000 logs) does not parse, resulting in empty reports. • [550414] Exporting PDF tables containing certain data can give an error, "Unknown HTML entity" • [553375] Improved performance of building indices, for the internal database, for large datasets. • [553713] Memory usage can be very high on profiles using the internal database, with very complex fields (fields with many unique values). New features in 8.0.7: • [538958] Added a -pp ("path page") command-line option for specifying the focus page for command-line export of the Paths Through A Page report. • [539823] Added better detection of mobile web browsers. • [541059] Enhanced Firewall-1 (fw log -ftn export) Log Format to handle logs with lines starting with dates; added some new fields. • [545835] Added support for Linksys VPN Router log format • [546006] Added the option to disable cross-reference tables individually, for better control over database build and report performance. • [548920] Enhanced support for IPtraf logs, to support a variant with single-digit days. • [550396] Improved performance of report generation in most cases. This is partly due to an increase in the default value of the "maximum paging caching buffer memory usage", and partly due to other optimizations. Profiles will benefit most when setting "maximum paging caching buffer memory usage" to the new default value of 64MB. In one example (a large pivot table), report generation speed increased from 13 minutes to 2 minutes; memory usage increased by less than 2x. • [553685] Improved progress reporting for cross-reference builds and index builds, to show more granular progress while building each xref table or index (especially when using the internal database). Version 8.0.6, shipped March 19, 2009 Bugs fixed in version 8.0.6: • [532899] Corrected typo that was breaking sessioning (sessions_id -> session_id) and restored session_id to database.fields. Renamed the field clips to successful_accesses. Removed c_rate from numeric fields because it is a value relative to 1 which means the standard rate. c_rate is now the Visitor Systems->Client Rate report instead of the Client Rate column. • [533310] Date filter strings of the form last1d, last1m, last3m, etc (with single-letter units) generate an error, "The date filter is not in valid date filter syntax." • [534897] The Paths Through A Page report uses much more memory than it should (several GB). • When using a prefix or suffix with MS SQL or Oracle or MySQL, the expiration query fails with a "no such table" error. • The "file by file" option for log processing is not editable in the web UI. • Database build performance is slow in some cases due to splitting processing across N+1 threads (when N is the number of processors or cores) instead of the more efficient N. • Log parsing uses arbitrarily large amounts of memory when reading log data from corrupt log data or log data with extremely long lines. • Command-line authentication login fails with error, "No Permission You don't have grants to view this page or profile. Please contact your system administrator for more details." • Profiles using filter_finalization give an error on database build, "Internal: sourceFileNode=NULL during ConfigNode::ParseInfixStatements()" • Database build generates an error, "Error in writing ODBC table main_table: ODBC error: rec1: SQLstate: 22003; msg=[Microsoft][ODBC SQL Server Driver]Numeric value out of range;" when importing data into MS SQL which has negative integers in it. • Browsing to a UNC pathname like \\pub\pub\logs, on Windows, shows nothing in the right panel of the File Browser. • Viewing reports sometimes gives an error, "Attempt to get node number 0 from node v.temp_x_ticks." • Browse button gives the directory above the entered pathname, if the pathname ends with a slash. • Global page headers and footers do not appear in reports. • The "Automatically update database when older than" option is not saved properly in the UI. • Database builds could sometimes give an error, "Unable to read contents of directory LogAnalysisInfo/TemporaryFiles/DeleteMe_... (No such file or directory)" • Tasks scheduled to run at the same time run in sequence instead. • Temporary files are not cleaned up quickly enough, resulting in large numbers of DeleteMe files in the TemporaryFiles directory. • Memory usage is very high when zooming to large reports. • Date/time graphs with one-minute granularity show no data. • "Default date filter" (global per-profile date range filter) has no effect. • Imported Sawmill 7 profiles sometimes give error, "Couldn't find node 0 in language.english.lang_stats.weekdays_short." • Removing all cross-reference groups gives an error, "Couldn't find node xrefs in v.fp." • The Paths Through A Page report takes a very long time, and a huge amount of disk space. • The date filter expression "last1month" (and similar ones) give an error about incorrect date range format. • A database removal operation on a MS SQL database gives an error, "Unable to Execute ODBC Query='delete from main_table x using main_table x where 1=1'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Microsoft][ODBC SQL Server Driver][SQL Server]Line 1: Incorrect syntax near 'x'.;" • Months report of imported Sawmill 7 profile, where the date/time report had been edited in Sawmill 7, gives an error, "Unable to read file LogAnalysisInfo\Databases\\main\Tables\rows_...o\header.cfg (No such file or directory)" • Microsoft Media Server plug-in gives wrong results for the c-rate field. • Generating a report with a customer date range filter string gives an error "Couldn't find node in." • Reports give an error, "Couldn't find node modification_time in v.lang_stats_file_info," when English language is not installed. • Reports show wrong numbers for fields containing single field values greater than 2 billion, when using multiprocessor database builds on a 32-bit system. • The "Process subdirectories" checkbox erroneously states that it is for "local folders only." • ODBC log sources do not support LONGVARCHAR (a.k.a. TEXT) or BIGINT fields. • Microsoft Media Server profiles show 0 for session duration. • W3C log data (like IIS log data) is very slow to parse, when using multiprocessor database building. • Date filter strings of the format "last1d", "last1m", and others using single-letter units, give an error, "The date filter is invalid." • The Profile menu in an action in the Scheduler is not alphabetized. • The "automatically update when older than" option sometimes has no effect. • The "configure" script does not complain when there is no C++ compiler, when building Sawmill from encrypted source, which causes the compilation to fail later when it attempts to compile the C++ files. New features in 8.0.6: • [486806] Added new plug-in to support the Palo Alto Networks Firewall Traffic log format. • A new option has been added for doing LOAD DATA imports into local MySQL database servers, which is faster than the default LOAD DATA LOCAL INFILE, and works on servers where local infile is disabled, but requires the MySQL server to be on the Sawmill server. • A new variant of Unix Daemon Messages log format is now supported. • The name of the Sawmill executable has been changed to just "sawmill" on non-Windows platforms, instead of having the version number in the name. • Sawmill now determines the amount of physical RAM on the system, and when splitting queries across multiple threads with "auto," it ensures that each thread has at least 2GB of RAM. • A new option, "maximum read block size," controls how much memory can be allocated to holding a single line of log data. Version 8.0.5, shipped February 26, 2009 Bugs fixed in version 8.0.5: • Fixed a bug which could cause an error when searching the documentation in Windows. • Fixed a bug which could cause an error on builds with MS SQL databases, if there was a database field called "rule." • Fixed a bug which could cause database corruption if the main table of the database was more than 4GB, with 32-bit versions of Sawmill. This generally happened with datasets more than about 30 million lines. It could cause a variety of symptoms, including out-of-memory errors while building indices, crashes during reporting, and incorrect numbers in the reports. • Fixed a bug which could cause an error "Parsing server returned itemnums list for unknown database field 'fieldname'" when building two databases simultaneously on Windows. • Fixed a bug which could cause an error "Unable to read file ... sessions_join" when no valid log entries were found in the dataset. • Moved the MySQL DLL to the Sawmill installation directory, to reduce conflicts with PHP and other programs which install different versions of the MySQL DLL. • Added support for MySQL on the x86 Linux ES4 platform, where is was disabled. • Restored the "create many profiles" script (now in LogAnalysisInfo/util), which was missing earlier versions of Sawmill 8. • Fixed a bug where multiprocessor log processing would fail with a parsing server error, if the "Server hostname" was not empty in Preferences. • Fixed a bug where lines of log data could be split incorrectly, for log formats where quoted values can span multiple lines, if the lines spanned the boundary between log reading blocks. This could result in slightly low numbers. • Fixed a bug in Shoutcast 1.8 processing, which could cause a regular expression error on parsing. • Fixed a bug which could cause incorrect sorting of rows in tables on 32-bit systems, if they contained large floating-point numbers (more than about 4 billion). • Fixed a bug which caused the graph tick marks and labels to be in the wrong locations, when rendered by Firefox. • Fixed a bug which could cause an error if all log filters were deleted. • Fixed a bug where the Create Profile Wizard incorrectly validated the pathname when regular expressions were used, causing a "no such pathname" error in some cases where the pathname was correct. • Added the "maximum paging caching buffer memory usage" option to the web UI. • Removed some database optimization options which no longer have any effect in Sawmill 8. New features in 8.0.5: • Enhanced support for Smarter Mail log format to handle dates containing dots. • Enhanced support for Wowza log format to handle large negative x-duration values (due to logging or server bugs). • Enhanced support for Limelight log format, to handle spaces in #Fields lines, in addition to tabs. • Added the "integer bits" option for database fields to the web GUI. • Improved handling of v7-to-v8 database conversion in the case where the database directory was overridden--it now prompts for a new directory for the Sawmill 8 database directory, so it won't overwrite the Sawmill 7 one. Version 8.0.4, shipped February 13, 2009 Bugs fixed in version 8.0.4: • Fixed a bug which could cause progress reporting to halt partway through long database builds, on 32-bit Windows. • Fixed a bug in the Intermapper Event Log Format plug-in that caused an error if Events was not selected as a numeric field during profile creation. • Fixed a bug where reporting would hang in CGI mode, if a "temporary directory" was set. • Added "always include bottom-level items" to the database fields editor. • Fixed a bug which could cause an error when processing Blue Coat W3C logs, if the log data contained both a cs-uri-path and a cs-uri-stem field. • Fixed an error, "Attempt to join main_table to filtertmp_5652_1 on 's.session_id = filtertmp_5652_1.itemnum', but there must be one column from each of the two tables in the ON condition, and there doesn't seem to be", which could occur when zooming in on a particular session in DIndividual Sessions. • Fixed a bug which would cause an error if a template was edited in an Advanced installation, though template editing is permitted by Advanced licensing. • Fixed a bug which could cause an overflow when reporting processing time in ISA 2004 log data. • Fixed/improved cleanup of temporary files in LogAnalysisInfo. • Fixed a bug where PDF generation would fail if the report contained a pie chart with a legend. • Fixed a bug where "-a rdi" (rebuilding indices from the command line) gave an error when used with the internal database. • Corrected references to SawmillCL.exe, which is now called Sawmill.exe (in version 8). • Fixed a bug where the Cities report showed States instead, when reporting on W3C log data (like QTSS). • Fixed a bug where the Logout link did not work on the first click, in Firefox. • Fixed a bug where the Support Email Address in Config -> Miscellaneous -> Support & Action Email, did not save. • Fixed a bug where the Config -> Reports page would not display when logged in as a Manager. • Fixed a bug which could cause crashes when using the page tagging server on Windows. • Fixed a bug which could cause 0s in reports, when clicking single date item while displaying a date report at a different level (e.g., clicking a month, while displaying the Days report). • Fixed a bug which could cause an error when there was a database field named "count". • Fixed a bug where long values could overlap the columns to the right of them, in HTML tables. • Fixed a bug where database updates reported themselves as "Rebuild Database" in the Database Info page. • Fixed a bug where the command-line progress display did not show the number of lines processed. • Fixed a bug where the Sessions Overview report did not change when filters were applied. • Fixed a bug which could cause some log data not to parse, if the "sessions visitor ID" field was empty. • Added the End User License Agreement file to the distribution directory (it was in the installer, but not the installation). • Fixed a bug which could cause an overflow when displaying time-taken for Blue Coat W3C logs, on a 32-bit system. • Fixed a bug which would cause an error when filtering a report, if there was a SQL table prefix or suffix specified in the profile. • Fixed a bug which would cause 0 reports when using a regular expression filter on a non-itemnum field (like hour of day). • Fixed a bug which could cause an error, "Unknown variable 'lang_stats.field_labels.time_stamp' in expression", when reporting on Common Access log data. • Fixed a problem with the formatting of FAQ pages. • Fixed a bug in iMail parsing which caused an error: 'Unknown configuration group "from" in node "profiles.qmail.log.fields".' New features in 8.0.4: • Modified the Intermapper Event Log Format plug-in to find the year in the log file name it contains the pattern LogYYYYMMDD. • Added an option to RBAC to disable password changing. • Extended Sidewinder Firewall reporting to show countries, regions, and cities for each IP. • Added suport for Squid format logging to Unix Syslog, with a Squid timestamp. • Added support for a new variant of Mirapoint SMTP log format. • Improved the performance of some internal database operations used for indices and uniques tracking. • Improved the performance of some internal database queries involving the sessions table. Version 8.0.3, shipped January 23, 2009 Bugs fixed in version 8.0.3: • Fixed a bug which could cause database builds to terminate unexpectedly. This occurred with ASA log data, but could theoretically happen with any log data. • Fixed an issue with the copyright in the Czech translation. • Fixed/enhanced the TaskLog file to suppress the execute_sql_query, which were numerous and mostly useless. • Added documentation for the "Remove Reloads From Sessions" option. • Corrected the MacOS Install.txt file to remove incorrect upgrade instructions. • Fixed a bug which would cause an error when zooming on a session user, and zooming to the Session Pages report. • Fixed a bug which could cause an error, "Error in get_progress_state.cfv" when displaying the Overview. This bug was caused by a failure during the database build, so profiles showing this behavior will need to have their databases rebuilt. • Fixed a bug which could cause database builds to hang, when built from Config -> Database Info, if an error occurred while the database was being deleted. • Fixed a bug with the v7-to-v8 profile converter, which would cause errors when viewing reports, if the v7 profile contained references to database fields which were not translated in Sawmill 8. • Fixed a bug which could cause an error during database build, if the database directory was on a different drive or partition from the LogAnalysisInfo (installation) folder. • Fixed a bug where the current_log_pathname() function did not work when using parsing servers (multithreaded database builds). • Fixed a bug which would cause an error, "Syntax error: Expected ) in expression; found '" when creating a profile from BlueCoat W3C data. • Fixed a bug which could cause an error when relocating the database directory. • Fixed a bug which could cause sporadic (though harmless) crashes of the web server. • Fixed a bug which could cause an error, "Unknown configuration group 'cs_uri_stem_bottom_level_items'" when viewing reports for a Microsoft Media Server profile. New features in 8.0.3: • Improved the Windows installer to omit the minor "rcN" from the version number, to simplify the Sawmill version number display, and make it clear that these is a production releases (successful release candidates), not pre-production releases (unsuccessful release candidates). • Improved documentation of "session events" and "sessions." • Added section to documentation, "Adjusting date/time values for Daylight Savings Time". • Enhanced the Limelight plug-in to handle arbitrary W3C headers, allowing variable field layout. • Enhanced the Squid plug-in to handle "action" field values containing spaces (like "TCP MISS"). Version 8.0.2, shipped January 16, 2009 Bugs fixed in version 8.0.2: • Fixed a bug where the "automatically update if older than" option was not saved in Config. • Fixed a bug which caused an error when creating an ISA profile: 'Unknown configuration group "filterinfo" in node "profiles.isa_logs.log.fields".' • Fixed a bug where some hierarchical reports would show no results when drilled into. • Fixed a bug where direct login by URL would sometimes give an error. • Fixed a bug which would give the error 'column "" does not exist' when analyzing Bluecoat log data. • Fixed a bug which could cause an error when using a literal backslash in a custom report expression. • Fixed a bug which could cause an error while analyzing Nortel ACD log data: 'Unknown configuration group "avg_agent_time_busy" in node "profiles.test.log.fields.' • Fixed a bug which could cause an error when clicking Scheduler after importing v7 profiles with field names containing numbers. • Fixed a bug where full-month filters on the Overview gave zero results. • Fixed errors compiling the encrypted source on SuSe 11, and other recent operating systems. • Fixed a bug where the progress display for one profile could show in another profile's Reports display, if there were two simultaneous profiles in use. • Fixed a bug where x86 Windows was reported as x64, and vice versa, in automated bug reports. • Fixed several places where images were broken in CGI mode. • Fixed a bug where very long field values could generate an error, when using ODBC. • Fixed a bug which could cause an error when analyzing Sidewinder log data with Microsoft SQL Server. • Fixed a bug where Log Detail report was empty when using ODBC databases. • Added support for not splitting queries at all, in the web interface. • Fixed a bug where some URLs used HTTP, even when running Sawmill in CGI mode under an HTTPS server. • Fixed a bug where parsing servers could linger after the process that spawned them was gone, in the event of a parsing error. • Fixed a bug which caused an error, 'Internal Error: Empty node name,' when setting a session field to "(None)". • Fixed a bug which caused an error when generating the Top Malware report in Ironport S-Series logs, 'Couldn't find node display_format_type in sessions_cache.xyz.profiles.profilename.extended_profile_dat.' • Fixed a bug which could cause an error like, 'Unable to read file LogAnalysisInfo\Databases\profile\main\Tables\rows_1_6248o\header.cfg' on location reports, in imported v7 profiles. • Fixed a bug which could cause sporadic crashes of the web server process. • Added the update_xrefs_on_update option to the web interface. • Fixed a bug which caused the Help link to sometimes have no effect. • Fixed a bug where pivot tables would sometimes show NULL rows. • Fixed a bug which could cause errors when using a SQL prefix or suffix. • Fixed a bug which could cause an error, "Unknown variable 'lang_stats.field_labels.ms_ras_client_name' in expression." • Fixed a bug which could cause an error when analyzing Sidewinder logs with MS SQL. • Fixed a bug where Config/Reports did not prompt to save changes. • Converted some hard-coded strings to language module strings, for internationalization. • Fixed a bug which would cause the error, "Couldn't find node next_pages in v.query_result.data" when viewing a Session Paths report with no data. • Fixed a bug which prevented error messages from being viewed when not logged in. • Fixed several broken images in the documentation, in CGI mode. • Fixed a bug where non-root-admin users could not see the version number of Sawmill in the web UI. • Fixed a bug which could cause the error, "Unable to read...\main\Tables\sessions_join\header.cfg (Day of week)." • Added the missing language module variable lang_admin.log_filters.simplify_playerid_label. • Fixed a bug which would cause error when rebuilding a database, if it was stored a different drive or partition from the LogAnalysisInfo folder. • Fixed a bug which would cause database build errors when the log data contained extremely large or infinite numbers. • Restored the missing FAQ about emailed report and Outlook 2003. • Fixed the FAQ about resetting the admin password to describe the new method. • Fixed a bug where the SPARC Solaris 9 distribution did not have the necessary libraries included. • Fixed a bug where Config -> Log Source did not support multiple log sources in Professional tier. • Fixed a bug where once a trial license expired, it would not accept another license key. • Restore the Support page from Sawmill 7, to Sawmill 8. • Restored the HTML comment describing the filters, from Sawmill 7, to Sawmill 8. New features in 8.0.2: • Updated the Czech, German, and Polish translations. • Enhanced Radware DefensePro plug-in to handle ip:port variation. • Enhanced Flash Media Server plug-in to support analysis when there is no cs_uri_stem field. • Implemented a custom action, reset_root_admin, for resetting the root administrator password from the command line. • Added more documentation about real-time log importing. Version 8.0.1, shipped December 24, 2008 Bugs fixed in version 8.0.1: • Fixed a bug where once a trial license expired, Sawmill would not accept a new license. • Fixed a bug where a Sawmill 7 license would be called "invalid" instead of being reported as a valid older license, no longer valid for Sawmill 8. • Fixed the upgrade instructions in the README of some platforms, which were still describing the Sawmill 7 upgrade method. • Fixed a bug where Sawmill 8 would not install its service, if Sawmill 7 was already installed, and would uninstall the Sawmil 7 service when Sawmill 8 was uninstalled. • Fixed a bug in the Create Profile wizard, which would cause an error if the Pathname was not literally a valid existing pathname, even if it contained wildcards which should have matched valid pathnames. This almost always caused wildcard or regular expression log sources to fail on profile creation. • Fixed a bug where wildcard and regular expression report filters did not work (generated an error when attempted) with Microsoft SQL Server profiles. New features in 8.0.1: • Added a separate chapter to the documentation about real-time importing. • Added a Support link to the Admin page. • Added the Salang expression of the current filter to the HTML of the report, as an HTML comment (useful for creating command-line or scheduled filters, or for debugging). • Chopped off the "rcN" part of the version number in the web interface, to make it look better. • Switched the default port of the web server to 8988. This makes it much simpler to run Sawmill 7 and Sawmill 8 together on the same system. • Enhanced support for ISA W3C format, to handle a significant variant (2007) • Added support for Unicode with Microsoft SQL Server as back-end database, so non-ASCII log data can be imported and stored in MS SQL, and queried from outside Sawmill. Version 8.0.0, shipped December 12, 2008 Bugs fixed in version 8.0.0: New features in 8.0.0: • Changed the GUI concept from html frames to single pages. • Added report fields for more flexibility and fine tuning of report elements and table data. • Added a simplified date_filter syntax (i.e: 2m, last2m, etc.) for date/time filtering. • Added a new caching system which caches various report components and database data independently. • Added RBAC (Role Based Access Control) • Added support for sequential actions per schedule in scheduler. • Added a "Run Now" button in the Scheduler, to run any task immediately. • Added log fields editor • Added database fields editor • Added session fields editor • Added report fields editor • Added new field wizard (which allows to create a log field, database field and report field at once) • Added URL support to view reports by URL definition by defining: profile name, report name, date filter, filter expression, filter comment • Added support to view any hierarchical depth of a hierarchical database field as non-hierarchical report. This allows i.e.: to view a months report or a region or city report. (These reports have been added by default) • Added the calendar as optional report. • Added support to dynamically create a pivot table within the reports GUI • Added support for different sort field and sort direction of the drill down field. • Added support to drill down data on a table with multiple string fields. • Added a new date picker which combines single date, date range and relative date selection. • Changed the zoom concept in that zoom automatically adds the zoomed item to filters. • Added support to zoom to multiple items at once. • Added support to save filter items as filter group • Improved the filters editor. • Added support to email a report within the reports GUI • Added support to define the row numbers and aggregation rows when exporting a table within the reports GUI • Added min and max aggregation rows to tables. • Added a row_visibility_expression per report element. This expression allows to show/hide table rows by an expression, i.e. show only rows where page_views > 300 and page_views < 1500. • Added support for a default date filter per profile. • Added support for a date filter per report or per report element. • Improved the Customize Report Element form/options. • Added table column info support. • Added table row selection support (to mark a row in yellow color). • Added support for 3D pie charts • Added support for antialiased PNG graphs • Added support for use of MS SQL or Oracle databases as back-end databases (where processed log data is stored, and reports generated from). • Added support for use of MS SQL, Oracle, or MySQL database as log sources • Implemented multiple scalability improvements, to allow Sawmill to process log data with less memory usage. These largely lift the restrictions of log processing on 32-bit systems, so any amount of log data can be processed without exceeding the address space of a 32-bit system. • Added real-time reporting. Reports can be generated while data is being imported, and will be up-to-the-moment, based on the latest imported data. Data can be streamed continually into the database, without any reporting downtime. • Implemented major SQL performance improvements, especially in the building of cross-reference tables, and in the performance of database update. • Enhanced the internal database to support SQL syntax, for universal SQL querying of any Sawmill database. • Added support for reading log data from a SFTP server • Added directory recursion on (S)FTP servers, so a log source can point to a directory, and all subdirectories will be processed. • Added filtering of reports on numerical fields; e.g., show all events where bytes > 1000 • Implemented multiprocessor splitting of report queries: report calculations are split across multiple processors for better performance. • Enhanced multiprocessor and multi-system log processing. Log process is now done more efficiently on a single system, and does not use the disk as much; and it is also possible to split log processing across multiple "parsing servers" on the same network. • Added database import and export. • Added support for user-created actions (-a options), with fully customizable parameters and behavior. Version 8.0.0b8, shipped December 06, 2008 Bugs fixed in version 8.0.0b8: • Fixed a bug which would cause an error when creating a BlueCoat W3C profile. • Fixed a bug where "start time" and "end time" were integers in imported MySQL profiles, rather than bring formatted timestamps. • Fixed a bug where W3C log data (and other log data with header lines) would not be imported properly (some lines would be dropped) when using multiprocessor log parsing. • Fixed a bug where the "index" attribute of database fields (whether to index the database field) was not editable in the web interface. • Fixed a bug where the database name was required in Config -> Database -> Server, when it should have been optional. • Fixed a bug where the web interface did not have an option to *not* use distributed parsing. • Changed xref tables to be hierarchical by default, which results in faster report generation, especially for large datasets, but somewhat slower xref builds on import. • Improved performance of the "session pages" report, and other reports which use indexed joins internally. • Fixed a bug where the "use overview for totals" option had no effect. • Fixed a bug where the "use overview for totals" option was not editable from the web interface. New features in 8.0.0b8: Version 8.0.0b7, shipped November 30, 2008 Bugs fixed in version 8.0.0b7: • Fixed a bug where the Reset/Collapse All button did not display properly in the Session Paths report. • Added the "User Overview For Totals" option to the Config interface. • Added editing of the Index option to the Database Fields editor (to specify which database fields should be indexed). • Fixed multiple bugs in reporting, which caused error when generating reports with a SQL database table prefix or suffix. • Fixed a bug where Windows pathnames were shown with slashes doubled, in error messages. • Fixed a bug where entering a directory pathname ending with a slash, then clicking Browse, would browse the parent of the directory, rather than the directory itself. • Fixed a bug where the Report Menu would not be displayed, when viewing a report with no elements. • Fixed a bug with Internet Explorer 7, where the Date Picker did not display properly. • Fixed a bug which caused an error when viewing reports for profiles with a database fields called "from" (or other SQL keywords), e.g., Postfix. • Improved performance of Squid log processing. • Fixed a bug where the expiration date always appeared as Feb 4, 2008, in the Licensing page. • Added links to documentation examples, from the Report Filters editor. • Fixed a bug where Config -> Database -> Server required a database name to be entered, even though it should default to the profile name when empty. • Fixed a bug where the Date picker showed a red "undefined" as the first day of the week. • Fixed a bug where the ODBC form required a username to be entered, even of the DSN had one embedded. • Removed "log processing threads" from the web interface; it is deprecated. • Fixed a bug where some windows did not have title bars, in CGI mode. • Fixed a bug where CGI mode would fail when running queries, if they were large enough to be split across multiple processors. New features in 8.0.0b7: Version 8.0.0b6, shipped November 29, 2008 Bugs fixed in version 8.0.0b6: • Fixed a bug where the Windows install included an unnecessary DLL in the installation directory. • Fixed a bug where the Calendar report was not enabled on imported v7 profiles. • Fixed a bug which could cause an "Unknown hash function" error when building some types of profiles, including Microsoft Media Server. • Fixed a bug which could cause progress reporting errors, if there was error during a database build or report generation. • Fixed a bug which could cause a crash during autodetection, if there was an error in autodection (instead of reporting the error). • Fixed a bug which could cause random results in the Overview, if the filter set contained no rows, and the database type was Microsoft SQL Server. • Fixed a bug where Microsoft SQL Server databases were not updated properly during a database update; in particular, the itemnum (normalization) tables were not updated. • Fixed a bug which could cause an error when zooming on session fields, and zooming to session reports. • Fixed a bug where MySQL databases were not imported successfully from Sawmill 7 profiles. • Improved autodetection of Instagate log format, so it doesn't detect Apache Combined as Instagate. New features in 8.0.0b6: Version 8.0.0b5, shipped November 27, 2008 Bugs fixed in version 8.0.0b5: • Fixed a bug where relative date filters (e.g., "last 4 months") could cause an error. • Fixed a bug which could cause an error when displaying pivot tables with no data in them. • Fixed a bug which caused the Overview to contain random numbers, when there was no data in the filter set. • Fixed a bug which could cause incorrect numbers in the Sessions Overview, when there were no session events in the database. • Improved the performance of the Sessions Overview report. • Fixed a bug which could cause a very high "elapsed time" value (e.g., 39 years) to appear in the progress display at the beginning of some tasks. • Fixed a bug where MS SQL database updates did not automatically update the xref tables. • Fixed a bug where generating a PDF report could cause an error about "width:-8". • Fixed a bug where individual characters could be skipped during database updates. New features in 8.0.0b5: • Brought the plug-ins and their associated language information up-to-date with the latest Sawmill 7. Version 8.0.0b4, shipped November 25, 2008 Bugs fixed in version 8.0.0b4: • Fixed problems with real-time profiles which prevented successive reports from showing the latest information. • Fixed several cosmetic issues with the bug report text. • Fixed a bug where FTP profiles could not process the log data, if there was no leading / in the pathname. • Fixed an issue which could cause errors during database builds, on some 32-bit systems (especially, Solaris). • Fixed a bug where the Use Sawmill icon was not properly removed on uninstall. • Fixed and improved the progress display for database builds. • Fixed a bug with v7-to-v8 profile conversion, which could cause an error when clicking Config -> Database. • Fixed a bug with the conversion of v7 MySQL databases to v8. • Fixed a bug where Sawmill did not see files in an FTP log source, in some circumstances (especially, when accessing a Microsoft FTP server). • Fixed a bug which could cause an error when viewing the Single-page summary with a session field filter. New features in 8.0.0b4: • Improved performance of filtered reports. Version 8.0.0b3, shipped November 23, 2008 Bugs fixed in version 8.0.0b3: • Fixed an uninstaller bug where the Sawmill 8 icon remained on the desktop after uninstall. • Fixed/improved the progress display for database builds, so it includes all steps, and shows better descriptions of each step. • Fixed a bug with v7-to-v8 profile converter, which did not set up the database tuning options properly, resulting in an error when viewing the Database section of Config. • Fixed conversion of v7 MySQL database in the import wizard. • Fixed a bug where Sawmill could not see files on a Microsoft FTP server. • Fixed a bug which could cause an error when zooming on session fields, and displaying reports generated without cross-reference tables. • Fixed a bug where indices were completely rebuilt after database updates; they are now properly incrementally updated from the new new data. • Fixed a bug which could cause a "duplicate key" error when viewing reports from a MS SQL database. • Added support for input of "node" licenses through the web interface. • Fixed bug where the database build would fail if no entries were accepted. • Fixed error which could occur when building from Microsoft Media Server logs. • Fixed a bug where CSV export in CGI mode had a broken link for the CSV file. • Fixed a bug which could cause an error when building a database from Ironport S-Series logs. • Fixed a bug where references to non-existent template pages would give an error "no node 'templates' in 'templates'". • Fixed the timestamp of emailed reports sent from Windows. New features in 8.0.0b3: • Improved performance of a common type of query on in internal database. • Reduced memory required by xref builds and other queries. Version 8.0.0b2, shipped November 19, 2008 Bugs fixed in version 8.0.0b2: • Fixed bug which would cause a database build to abort with an error, if the -v f option was used, and a log field value contained a$.

• Added ODBC driver manager libraries used by the x64 Linux (ES5) version, to eliminate a "libodbc.so not found" error.

• Fixed a bug where "start time" and "end time" were formatted and named incorrectly in imported MySQL profiles.

• Fixed a bug where some images were broken in CGI mode.

• Fixed a bug where command-line authentication did not restrict profiles or permissions properly.

• Fixed a bug where a "garbage" line appeared at the top of printer friendly pages.

• Fixed a bug where the Database Info page would generate an error if the database no longer existed.

• Fixed a bug which would cause an lang_stats error when reporting on PIX logs.

• Fixed a bug which would cause an error when using "log processing threads" > 0 in Advanced tier.

• Fixed a bug which could cause an error with Log Detail, for certain datasets.

• Fixed a bug where CGI mode did not display reports.

• Fixed an installer issue where a necessary DLL (libeay32.dll) was not installed properly on 64-bit Windows.

• Fixed a bug which would cause an error when filtering on items containing a backslash.

• Fixed a formatting problem of the date in the licensing page.

• Greatly improved performance of filtered reports which must query the main table.

• Fixed a bug where the Cross Reference Groups editor showed "undefined" next to all numerical fields.

• Fixed a bug where the Session Exits field did not have a default "Session field" value in the report fields editor.

• Fixed a bug where the charset could not be changed in the Log Processing options.

• Fixed a bug where password were not masked when entering SQL database information.

• Fixed a bug where the Send Email window in the Scheduler incorrectly asked for an "output directory."

• Fixed a bug where the Import Wizard did not report the error message, in the case of an import failure.

• Fixed a bug where the text "Saving..." appeared below (or above) reports, and messed up the zoom formatting somewhat.

New features in 8.0.0b2:

Version 8.0.0b1, shipped November 1, 2008

Bugs fixed in version 8.0.0b1:

New features in 8.0.0b1:

• Changed the GUI concept from html frames to single pages.

• Added report fields for more flexibility and fine tuning of report elements and table data.

• Added a simplified date_filter syntax (i.e: 2m, last2m, etc.) for date/time filtering.

• Added a new caching system which caches various report components and database data independently.

• Added RBAC (Role Based Access Control)

• Added support for sequential actions per schedule in scheduler.

• Added a "Run Now" button in the Scheduler, to run any task immediately.

• Added new field wizard (which allows to create a log field, database field and report field at once)

• Added URL support to view reports by URL definition by defining: profile name, report name, date filter, filter expression, filter comment

• Added support to view any hierarchical depth of a hierarchical database field as non-hierarchical report. This allows i.e.: to view a months report or a region or city report. (These reports have been added by default)

• Added the calendar as optional report.

• Added support to dynamically create a pivot table within the reports GUI

• Added support for different sort field and sort direction of the drill down field.

• Added support to drill down data on a table with multiple string fields.

• Added a new date picker which combines single date, date range and relative date selection.

• Changed the zoom concept in that zoom automatically adds the zoomed item to filters.

• Added support to zoom to multiple items at once.

• Added support to save filter items as filter group

• Improved the filters editor.

• Added support to email a report within the reports GUI

• Added support to define the row numbers and aggregation rows when exporting a table within the reports GUI

• Added min and max aggregation rows to tables.

• Added a row_visibility_expression per report element. This expression allows to show/hide table rows by an expression, i.e. show only rows where page_views > 300 and page_views < 1500.

• Added support for a default date filter per profile.

• Added support for a date filter per report or per report element.

• Improved the Customize Report Element form/options.

• Added table column info support.

• Added table row selection support (to mark a row in yellow color).

• Added support for 3D pie charts

• Added support for antialiased PNG graphs

• Added support for use of MS SQL or Oracle databases as back-end databases (where processed log data is stored, and reports generated from).

• Added support for use of MS SQL, Oracle, or MySQL database as log sources

• Implemented multiple scalability improvements, to allow Sawmill to process log data with less memory usage. These largely lift the restrictions of log processing on 32-bit systems, so any amount of log data can be processed without exceeding the address space of a 32-bit system.

• Added real-time reporting. Reports can be generated while data is being imported, and will be up-to-the-moment, based on the latest imported data. Data can be streamed continually into the database, without any reporting downtime.

• Implemented major SQL performance improvements, especially in the building of cross-reference tables, and in the performance of database update.

• Enhanced the internal database to support SQL syntax, for universal SQL querying of any Sawmill database.

• Added directory recursion on (S)FTP servers, so a log source can point to a directory, and all subdirectories will be processed.

• Added filtering of reports on numerical fields; e.g., show all events where bytes > 1000

• Implemented multiprocessor splitting of report queries: report calculations are split across multiple processors for better performance.

• Enhanced multiprocessor and multi-system log processing. Log process is now done more efficiently on a single system, and does not use the disk as much; and it is also possible to split log processing across multiple "parsing servers" on the same network.

• Added database import and export.

• Added support for user-created actions (-a options), with fully customizable parameters and behavior.