FAQ: Referrer, Agent, and Error Logs

How do I see referrer (referring URL, search engines, and search terms), agent (browser and OS), or error statistics?

Short Answer

Use "extended" or "combined" log format to see referrer and agent information, or analyze the log files with a separate profile. For error logs, analyze them with a separate profile.

Long Answer

Different log formats contain different types of information. All major web log formats include page, date/time, and browsing host information, but not all contain referrer and agent information. If your log format does not include referrer or agent information, Sawmill will not include that information in its database. The easiest way to get referrer or agent information is to change your web server's log format to an "extended," or "combined" format, which includes referrer and agent information; then Sawmill will automatically include referrer and agent information in the database and in the statistics.

If it's not possible to change your log format, and you have a separate referrer log (often called referer_log), then you can analyze that log directly with Sawmill. Just point Sawmill at the log, and Sawmill should recognize it as a referrer log. Sawmill will show statistics with referrer and page information. Host and date/time information are not available in a standard referrer log, so referrer and page is all Sawmill can extract. By using an extended or combined log format, you will be able to do more powerful queries, for instance to determine the referrers in the most recent week.

Similarly, if you can't configure your server to use extended or combined, but you have a separate agent log, you can analyze it with Sawmill by creating a separate profile that analyzes the agent (web browser and operating system) information only. Since an agent log contains only agent information, you won't be able to cross-reference the agent information with page, date/time, host, referrer, or anything else; to do that, you'll need an extended or combined format log.

To analyze error information, you'll need an error log (often called error_log). Just point Sawmill at your error log when you create the profile. Since the error log contains only error messages, you won't be able to cross-reference the errors against page, date/time, or any other fields; if you need cross-referencing of errors, you may be able to get what you need by cross-referencing the "server response" field of your normal web log to the fields you need cross-referenced; then apply "404" as a filter on the server response field and you'll see only those web site hits that generated 404 (file not found) errors.