.

WHAT'S NEW IN SAWMILL 8.8?

Fully Sharded Databases For Better Performance

Sawmill 8.8 now support splitting databases into independent shards, for higher performance on multiprocessor systems (Enterprise licensing is required for multiprocessor usage of Sawmill).

Previous versions would parse log data in parallel across multiple processes, but database filtering (including building cross-reference tables, and building main table indexes), a major portion of database build time, used only one process. Sawmill now runs database filters in parallel across all processors, with each processor responsible for a separate shard of the database. This can give much faster performance on multiprocessor systems.

Report generation is also done fully in parallel, with each processor computing one shard of the report, and the results then merged; for some types of queries (especially those querying the entire main table), this can be much faster on multiprocessor systems.

Here are some performance numbers of 8.8, building a 3 billion line Apache web log dataset (full detail, all cross-references and indexes enabled):

GeoIP 2 Database

Sawmill 8.8 now uses the MaxMind GeoIP 2 Database, for better geolocation (detection of country, region, and city).

Improved Password Security

Sawmill 8.8 uses enhanced password storage methods to combat password-guessing attacks (e.g. rainbow tables).

Key-based SFTP authentication

Sawmill 8.8 now support key-based authentication for downloading log data from SFTP servers. Previous versions supported only password authentication.

Oracle improvements

Sawmill 8.8 supports SQL*LOADER upload of data to a remote Oracle database, and supports conversion of time zone (or other timestamp conversions) when loading data into Oracle with SQL*LOADER.

Deep inspection of multiprocessor progress

A new action print_progress provides deep inspection of progress of sharded database operations, including progress information for each shard.