{= include("docs.util"); start_docs_page(docs.technical_manual.page_titles.newsletters); =}

Sawmill Newsletter

  October 15, 2008



Welcome to the Sawmill Newsletter!

 You’re receiving this newsletter because during the downloading or purchase of Sawmill, you checked the box to join our mailing list. If you wish to be removed from this list, please send an email, with the subject line of “UNSUBSCRIBE” to newsletter@sawmill.net .


News

Sawmill 7.2.15 shipped on May 16, 2008. This is a minor "bug fix" release, and it is free to existing Sawmill 7 users.  It is recommended for anyone who is experiencing problems with Sawmill 7.2.14 or earlier. You can download it from http://sawmill.net/download.html .

Sawmill 8 is near the end of "alpha" stage. That means that we have completed development of all features, have fixed most major bugs, and Sawmill 8 will ship (8.0.0) when all remaining known bugs are fixed. Major features of Sawmill 8 include support for Oracle and Microsoft SQL Server databases, real-time reporting, a completely redesigned web interface, better multi-processor and multi-core support, and role-based authentication control. A "beta" version will be publicly available when all major bugs are fixed, probably in the next few weeks. Watch sawmill.net for the beta announcement!

This issue of the Sawmill Newsletter describes using Sawmill to convert complex log data to simple delimited text, using the process_logs command-line action.


Get The Most Out Of Sawmill With Professional Services

Looking to get more out of your statistics from Sawmill? Running short on time, but need the information now to make critical business decisions? Our Professional Service Experts are available for just this situation and many others. We will assist in the initial installation of Sawmill using best practices; work with you to integrate and configure Sawmill to generate reports in the shortest possible time. We will tailor Sawmill to your environment, create a customized solution, be sensitive to your requirements and stay focused on what your business needs are. We will show you areas of Sawmill you may not even be aware of, demonstrating these methods will provide you with many streamlined methods to get you the information more quickly. Often you'll find that Sawmill's deep analysis can even provide you with information you've been after but never knew how to reach, or possibly never realized was readily available in reports. Sawmill is an extremely powerful tool for your business, and most users only exercise a fraction of this power. That's where our experts really can make the difference. Our Sawmill experts have many years of experience with Sawmill and with a large cross section of devices and business sectors. Our promise is to very quickly come up with a cost effective solution that fits your business, and greatly expand your ROI with only a few hours of fee based Sawmill Professional Services. For more information, a quote, or to speak directly with a Professional services expert contact consulting@flowerfire.com.


Tips & Techniques: Converting Log Data With process_logs

Typically Sawmill is used to parse log data into a database, and then generate reports from that database with Sawmill's web reporting interface. But Sawmill can also be used as a stand-alone parser, or log converter. This is useful for parsing complicated log formats, like mail logs, which typically have many lines per event, and where the lines must be parsed intelligently, in a particular order, and then re-integrated into a single event. This sort of parsing is difficult, but Sawmill already knows how to do it for all common log formats. So if you're writing a script to import log data into a database, or to do alerting based on log data, or to doing anything else with log data, you can use Sawmill as the first stage of your script, to do the parsing and send the parsed data on to you.

This type of conversion is typically done from a script, or from the command line. For instance, you might run the following command on a profile created from an Apache log file:


  \$ sawmill -p my_profile -a pl -v 0 > out.csv

The -p option specifies the internal profile name; the "-a pl" is short for "-a process_logs", which specifies the action; "-v 0" tells Sawmill to generate no output other than the process_logs output. If the input (Apache) log data is this:


140.177.203.25 - - [07/Apr/1998:16:53:06 -0500] "GET / HTTP/1.0" 200 734 "-" "Mozilla/4.04 [en] (X11; I; SunOS 5.6 sun4u)"
140.177.203.25 - - [07/Apr/1998:16:53:06 -0500] "GET /sawmill/picts/TopBanner.gif HTTP/1.0" 200 4573 "http://asooo/" "Mozilla/4.04 [en] (X11; I; SunOS 5.6 sun4u)"
192.17.19.148 - - [07/Apr/1998:21:32:04 -0500] "GET / HTTP/1.0" 200 734 "http://developer.javasoft.com/developer/earlyAccess/java3d-features.html" "Mozilla/4.04 (Macintosh; I; PPC, Nav)"
192.17.19.148 - - [07/Apr/1998:21:32:12 -0500] "GET /sawmill/picts/TopBanner.gif HTTP/1.0" 200 4573 "http://asooo.wolfram.com/" "Mozilla/4.04 (Macintosh; I; PPC, Nav)"
192.17.19.148 - - [07/Apr/1998:21:32:15 -0500] "GET /flowerfire/ HTTP/1.0" 404 154 "http://asooo.wolfram.com/" "Mozilla/4.04 (Macintosh; I; PPC, Nav)"
192.17.19.148 - - [07/Apr/1998:21:32:18 -0500] "GET / HTTP/1.0" 200 734 "http://asooo.wolfram.com/flowerfire/" "Mozilla/4.04 (Macintosh; I; PPC, Nav)"
192.17.19.148 - - [07/Apr/1998:21:32:21 -0500] "GET /sawmill/ HTTP/1.0" 200 3344 "http://asooo.wolfram.com/" "Mozilla/4.04 (Macintosh; I; PPC, Nav)"
192.17.19.148 - - [07/Apr/1998:21:32:26 -0500] "GET /sawmill/picts/title.gif HTTP/1.0" 200 48771 "http://asooo.wolfram.com/sawmill/" "Mozilla/4.04 (Macintosh; I; PPC, Nav)"
192.17.19.148 - - [07/Apr/1998:21:32:29 -0500] "GET /sawmill/samples.html HTTP/1.0" 200 8996 "http://asooo.wolfram.com/sawmill/" "Mozilla/4.04 (Macintosh; I; PPC, Nav)"
192.17.19.148 - - [07/Apr/1998:21:32:30 -0500] "GET /sawmill/picts/header.gif HTTP/1.0" 200 12371 "http://asooo.wolfram.com/sawmill/samples.html" "Mozilla/4.04 (Macintosh; I; PPC, Nav)"

then the result in out.csv will be this, a traditional CSV file with a comma-separated header listing the names of the fields, and one line per event with comma-separated values:

  date_time,day_of_week,hour_of_day,hit_type,page,file_type,worm,screen_dimensions,screen_depth,hostname,domain_description,location,organization,isp,domain,referrer,referrer_description,search_engine,search_phrase,web_browser,operating_system,spider,server_domain,authenticated_user,server_response,hits,page_views,spiders,worms,errors,broken_links,screen_info_hits,visitors,size
07/Apr/1998 16:53:06,3,16,page view,/{default},(no type),(not a worm),(-) x (-),(-),140.177.203.25,IP Address,United States/IL/Champaign,Wolfram Research,(unknown ISP),(unknown domain),(no referrer),(no referrer),(no search engine),(no search phrase),Netscape Navigator/4.04 ,SunOS,(not a spider),-,(not authenticated),200,1,1,(empty),(empty),(empty),(empty),(empty),140.177.203.25,734
07/Apr/1998 16:53:06,3,16,hit,/sawmill/picts/(nonpage),GIF,(not a worm),(-) x (-),(-),140.177.203.25,IP Address,United States/IL/Champaign,Wolfram Research,(unknown ISP),(unknown domain),http://asooo/(omitted),Unknown,(no search engine),(no search phrase),Netscape Navigator/4.04 ,SunOS,(not a spider),-,(not authenticated),200,1,(empty),(empty),(empty),(empty),(empty),(empty),140.177.203.25,4573
07/Apr/1998 21:32:04,3,21,page view,/{default},(no type),(not a worm),(-) x (-),(-),192.17.19.148,IP Address,United States/IL/Urbana,University of Illinois,(unknown ISP),(unknown domain),http://developer.javasoft.com/(omitted),Commercial (com),(no search engine),(no search phrase),Netscape Navigator/4.04 ,Macintosh,(not a spider),-,(not authenticated),200,1,1,(empty),(empty),(empty),(empty),(empty),192.17.19.148,734
07/Apr/1998 21:32:12,3,21,hit,/sawmill/picts/(nonpage),GIF,(not a worm),(-) x (-),(-),192.17.19.148,IP Address,United States/IL/Urbana,University of Illinois,(unknown ISP),(unknown domain),http://asooo.wolfram.com/(omitted),Commercial (com),(no search engine),(no search phrase),Netscape Navigator/4.04 ,Macintosh,(not a spider),-,(not authenticated),200,1,(empty),(empty),(empty),(empty),(empty),(empty),192.17.19.148,4573
07/Apr/1998 21:32:15,3,21,broken link,/flowerfire/(nonpage),(no type),(not a worm),(-) x (-),(-),192.17.19.148,IP Address,United States/IL/Urbana,University of Illinois,(unknown ISP),(unknown domain),http://asooo.wolfram.com/(omitted),Commercial (com),(no search engine),(no search phrase),Netscape Navigator/4.04 ,Macintosh,(not a spider),-,(not authenticated),404,1,(empty),(empty),(empty),1,1,(empty),192.17.19.148,154
07/Apr/1998 21:32:18,3,21,page view,/{default},(no type),(not a worm),(-) x (-),(-),192.17.19.148,IP Address,United States/IL/Urbana,University of Illinois,(unknown ISP),(unknown domain),http://asooo.wolfram.com/(omitted),Commercial (com),(no search engine),(no search phrase),Netscape Navigator/4.04 ,Macintosh,(not a spider),-,(not authenticated),200,1,1,(empty),(empty),(empty),(empty),(empty),192.17.19.148,734
07/Apr/1998 21:32:21,3,21,page view,/sawmill/{default},(no type),(not a worm),(-) x (-),(-),192.17.19.148,IP Address,United States/IL/Urbana,University of Illinois,(unknown ISP),(unknown domain),http://asooo.wolfram.com/(omitted),Commercial (com),(no search engine),(no search phrase),Netscape Navigator/4.04 ,Macintosh,(not a spider),-,(not authenticated),200,1,1,(empty),(empty),(empty),(empty),(empty),192.17.19.148,3344
07/Apr/1998 21:32:26,3,21,hit,/sawmill/picts/(nonpage),GIF,(not a worm),(-) x (-),(-),192.17.19.148,IP Address,United States/IL/Urbana,University of Illinois,(unknown ISP),(unknown domain),http://asooo.wolfram.com/(omitted),Commercial (com),(no search engine),(no search phrase),Netscape Navigator/4.04 ,Macintosh,(not a spider),-,(not authenticated),200,1,(empty),(empty),(empty),(empty),(empty),(empty),192.17.19.148,48771
07/Apr/1998 21:32:29,3,21,page view,/sawmill/samples.html,HTML,(not a worm),(-) x (-),(-),192.17.19.148,IP Address,United States/IL/Urbana,University of Illinois,(unknown ISP),(unknown domain),http://asooo.wolfram.com/(omitted),Commercial (com),(no search engine),(no search phrase),Netscape Navigator/4.04 ,Macintosh,(not a spider),-,(not authenticated),200,1,1,(empty),(empty),(empty),(empty),(empty),192.17.19.148,8996

This can be used as input to a script. It's easier to parse than the original log data, and also has some extra information, like geographic locations; and Sawmill's own log filtering functionality can be used to include additional columns (database fields), populated programatically. For instance, it could use BCP or another program to import the comma-separated data into a SQL database for later querying.


Advance Topic: Customizing The Output of process_logs

The default output of process_logs is good for many purposes, but if you have a script which expects a particular format of input, you may need to use some of Sawmill's additional options to customize the output. Specifically,
Note: all four options above are (will be) available in Sawmill 7.2.16 and later. If you need them immediately, contact support for a pre-release download.


Conclusion

This newsletter describes the process_logs command-line action, which has general utility in creating environments where Sawmill acts as a log parser or converter, and in later stages in the process operate on the logs that Sawmill has parsed, converted, and simplified. If you would like assistance in setting up this sort of environment, you can also use Sawmill Professional Services. Contact sales@sawmill.net for more information.


[Article revision v1.0]
[ClientID: 43726]

{= end_docs_page() =}