Newsletters
 |
Sawmill Newsletter March 15, 2010 |
Welcome to the Sawmill Newsletter!
You’re receiving this newsletter because during the downloading or purchase of Sawmill, you checked the box to join our mailing list. If you wish to be removed from this list, please send an email, with the subject line of “UNSUBSCRIBE” to newsletter@sawmill.net (please include the entire message, as the identifying information is at the bottom).
News
Sawmill 8.1.3 shipped on February 12, 2010. This is an bug-fix release--it fixes a number of bugs. This release is free to existing Sawmill 8 users. It is recommended for anyone who is experiencing problems with Sawmill 8.1.2 or earlier. Sawmill 8.1.2 has a known issue with some Windows installations. It requires the Microsoft Visual Studio 2008 redistributable package, but does not install it. Most Windows systems have this package already, but for those that don't, Sawmill will not run. If you are experiencing this problem, upgrade to 8.1.3, or install the redistributable package (x86 or x64). You can download Sawmill 8.1.3 from http://sawmill.net/download.html .
Sawmill 7 users can upgrade to Sawmill 8 for half of the license price; or if you have Premium Support, the upgrade is free. Major features of Sawmill 8 include support for Oracle and Microsoft SQL Server databases, real-time reporting, a completely redesigned web interface, better multi-processor and multi-core support, and role-based authentication control.
This issue of the Sawmill Newsletter describes the process of deleting database fields to simplify the database.
Get The Most Out Of Sawmill With Professional Services
Looking to get more out of your statistics from Sawmill? Running short on time, but need the information now to make critical business decisions? Our Professional Service Experts are available for just this situation and many others. We will assist in the initial installation of Sawmill using best practices; work with you to integrate and configure Sawmill to generate reports in the shortest possible time. We will tailor Sawmill to your environment, create a customized solution, be sensitive to your requirements and stay focused on what your business needs are. We will show you areas of Sawmill you may not even be aware of, demonstrating these methods will provide you with many streamlined methods to get you the information more quickly. Often you'll find that Sawmill's deep analysis can even provide you with information you've been after but never knew how to reach, or possibly never realized was readily available in reports. Sawmill is an extremely powerful tool for your business, and most users only exercise a fraction of this power. That's where our experts really can make the difference. Our Sawmill experts have many years of experience with Sawmill and with a large cross section of devices and business sectors. Our promise is to very quickly come up with a cost effective solution that fits your business, and greatly expand your ROI with only a few hours of fee based Sawmill Professional Services. For more information, a quote, or to speak directly with a Professional services expert contact consulting@flowerfire.com.
Tips & Techniques: Deleting Database Fields
Most log formats contain dozens of fields in the data: each event may include timestamps, source and destination IPs or addresses, usernames, pages or filenames, and a myriad of less essential parameters like server responses, error codes, MIME types, bytes transferred (sometimes in multiple directions), packets lost, transmission quality, codecs, versions of software and hardware, etc. A few formats have hundreds of fields.
In most cases, only a handful of these fields are useful. In a proxy server analysis, for instance, you might want to know the source IP, username, destination URL, timestamp, and a of couple others. But exactly which fields are useful depends on the application, and the installation, and the user, so Sawmill's default behavior is to report them all, to ensure that the fields you want, and the reports you want, are there in the default profile. The remainder of the fields are still in the database, however, and they take time to populate into the main table of the database, time to normalize, time to index, and time to cross-reference; they also use disk space for all of these purposes.
To optimize the size and performance of your database, you can remove unused database fields. This is a somewhat involved process, as many other parts of a profile depend on each database field, but following the steps below will remove the field, and all its dependencies.
Step 1: Determine If It Is An Aggregating Field
Several of the steps are different for an "aggregating" field vs. a "non-aggregating field," so first you need to determine which it is. Aggregating fields, sometimes called "numerical" fields, are fields like "hits" or "bytes" or "page views", which can be totaled in some way, often by adding them together, but also by maxima, minima, or computing number of uniques as in a "unique client IPs" or "visitors" field. If it makes sense to ask, "what is the total X?", then X is an aggregating field. Non-aggregating fields are all the others, things like timestamps, filenames, IP addresses, or anything else that can't be totaled. Non-aggregating fields are the main fields of reports (e.g., the "File Types") report; aggregating fields are columns of reports (e.g., the Visitors columns of a report).
Step 2: Remove The Field From Reports
In Config->Reports, either:
- For non-aggregating fields, delete all report elements, and the reports that contain them, and the report menu items referring to those reports, which contain the field. For instance, if you're deleting the File Type database field, delete the File Types report. Also, delete the File Types report element of the Single-Page Summary. Then, remove the field from Log Detail, and any other reports it appears in as an extra column (but default, this does not usually happen for reports other than Log Detail). Remove it by using the Manage Fields link in the report element--just unchecking it isn't enough. Finally, delete the report menu item that points to the File Types report.
- For aggregating fields, remove the field from all reports where it appears as a column. Remove it by using the Manage Fields link in the report element--just unchecking it isn't enough.
This works just like Step 2, but you remove it from Config -> Cross-Reference Groups instead. If it's a non-aggregating field, you'll typically remove its main cross-reference group, and remove it from any other cross-reference groups it's in; if it's an aggregating field, you'll remove it from all cross-reference groups it's in.
Step 4: Delete The Report Field
In Config->Report Fields, delete any report fields that use the database field. There will usually be only one, but if there are several, delete them all.
Step 5: Delete The Database Field
Now you can finally delete the database field itself, in Config -> Database Fields.
Step 6: (optional) Delete The Log Field And Log Filter
Steps 1-5 are enough to save the database space, but if time is being spent parsing and filtering the field, you can speed up the build a bit by deleting any log filter that sets the field, in Config -> Log Filters, and then deleting the log field itself, in Config -> Log Fields.
Advanced Topic: Creating a Custom Action To Delete a Database Field
As a demonstration of custom actions, and to provide a convenient way to do all the steps above, we have created a custom action, delete_database_field, for this newsletter. Custom actions are implemented with CFG files in the Actions folder of LogAnalysisInfo, so to install this custom action, just drop the attachment into the Actions folder. Custom actions are invoked from the command line, in this case like this:
sawmill -p profile -a ddf -dfn dbfieldname
e.g.:
sawmill -p my_profile -a ddf -dfn file_type
The attached action automatically performs all steps 1-5 above (step 6 still needs to be done manually, if desired), deleting the database field, any report fields that depend on it, any report elements that use it, and reports that contain only such report elements, and any report menu items that point to such reports. It also removes the database field from all xrefs, and deletes any xrefs that are useless after deleting the field.
This custom action is a particularly sophisticated example of profile manipulation using a custom action (at 240+ lines, it is twice as long as any of the other custom actions currently installed). Any programmer can create custom actions, by writing a file like the one attached.
This custom action will be a standard part of Sawmill 8.1.4 and later.
Professional Services
This newsletter describes the procedure for deleting database fields, and all dependencies. If you need assistance with this or other profile customization, or with creating custom actions, or with any other Sawmill tasks, our Sawmill Experts can help. Contact sales@sawmill.net for more information.
[Article revision v1.0]
[ClientID: 43726]