FAQ: Discarding hits from spiders

How can I throw away all the spider hits, so I only see statistics on non-spider hits?

Short Answer

Use a Log Filter to reject all hits from spiders (and worms).

Long Answer

Create a new Filter to reject all hits from spiders. The easiest way to create log filters is in the Log Filter Editor, in the Log Filters section of the Config. To get to the Log Filters editor, click Show Config in the Profiles list (or click Config in the reports), then click Log Data down the left, then click Log Filters. To create the filter:

• Select "Log Filters" from menu.
• Click "New Log Filter."
• Click "New Condition."
• Select "Spider" from the Log Field menu.
• Select "is NOT equal." from the Operation menu
• Enter "(not a spider)" in the Value field.
• Click OK.
• Click New Action.
• Choose "Reject log entry" from the Action menu.
• Click OK.
• Click the button at the left with two downward arrows, to move the new log filter to the bottom of the list. The filter need to be run after the Spider field is computer, which for some log formats occurs in a Log Filter.
• You can create a name for this filter (like "Reject Spiders Filter") and a comment (using Edit Comment) if you choose.
• This filter must be positioned below the "Compute user-agent derived fields" filter.

You can also use the Advanced Expression Syntax option from the Filter Type drop down list (on the Filter tab), and type in this filter expression into the value field:

if (spider ne "(not a spider)") then "reject";

For log format which use the User Agent Analysis snapon to compute spider values, the spider field is called uaa_spider, and the expression above must be changed to refer to that field name instead.

Then rebuild your database, and all hits from spiders will be discarded.

For more details on Filters see Using Log Filters.