Sawmill

DOWNLOAD
SAWMILL 8.1.8
free trial!!
Home Products Downloads Purchase Support About About
Sawmill Sawmill

SAWMILLFORUM

Sawmill Discussion Forum

Subject: "filtering out domains"     Previous Topic | Next Topic
Printer-friendly copy    
Conferences Pre-Sales Topic #824
Reading Topic #824
jhambrick
Member since Feb-7-11
5 posts
Feb-07-11, 07:57 AM (PDT)
Click to EMail jhambrick Click to send private message to jhambrick Click to view user profileClick to add this user to your buddy list  
"filtering out domains"
 
   I am thinking of evaluating you software, having had some experience with it a few years ago. But I only know enough to be dangerous. I am currently hosting on 1and1.com under a shared plan. The way 1&1 handles their logs is, all web site data are "logged" into the same file, but I only want to get the stats of only one domain. So I assume I can extract the profile of this one site using filters, right? But can I do it? I have looked at the documentation and yes I can figure it out eventually, but it takes time, will I spend my 30 days to evaluate it trying to figure it out? So what kind of help can I expect? Is there a mother lode of samples of filters I can download? I am not big on waiting for a response for 2 days if I can figure it out myself, and i cant afford (at least not now, just starting off) to pay for premium support. The only limitation I can see is myself. So if I spent my 30 days "figuring" it out I am not giving it a fair evaluational. And I have "lost" 30 days and you don't sell anything. Looking for a win win.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top

 
Conferences | Topics | Previous Topic | Next Topic
dgilmoreadmin
Member since Nov-18-04
3646 posts
Feb-08-11, 02:02 PM (PDT)
Click to EMail dgilmore Click to send private message to dgilmore Click to view user profileClick to add this user to your buddy list Click to send message via AOL IM  
1. "RE: filtering out domains"
In response to message #0
 
Hi-

Fairly easy to only parse the "domain" you want out of the log files assuming there's something in the log file that identifies what the domain is or by an IP address. There's what we call "log filters" in the config section of a profile. These log filters provide a very easy way to exclude or include info into a Sawmill database.

The basic process would be-->

Download your log files somewhere local.
Install Sawmill & quickly configure the trial.
Create a profile using the Sawmill's profile creation wizard and point Sawmill to the directory where the logs have been stored.
Complete the profile creation wizard by clicking on the "view config" step at the end vs. "process and view reports"
Click on the log filter tab.
Create a new log filter and set up a condition like:
"If my-web-domain.com does not match domain" then action is reject.

This is not the action "code" or filter, but you'd basically be setting up a comparison that excludes/rejects lines if they are not lines for your domain based on either the URL, or the domain field and depending upon the actual log data this varies.

If you know the format of the log files I can try to point you more specifically to the actual log filter. You can also paste a sample of the log data and we can try to tell you what the condition/comparison would be to reject all but your domain from inclusion in your database.

David
Sawmill Product Support Team
support@flowerfire.com


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
jhambrick
Member since Feb-7-11
5 posts
Feb-08-11, 07:54 PM (PDT)
Click to EMail jhambrick Click to send private message to jhambrick Click to view user profileClick to add this user to your buddy list  
2. "RE: filtering out domains"
In response to message #1
 
   Wow - now that is what I call support. So far so good. I will go ahead and set up the trial sawmill and take a look at the instructions you sent. Thanks David


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
jhambrick
Member since Feb-7-11
5 posts
Feb-10-11, 10:53 AM (PDT)
Click to EMail jhambrick Click to send private message to jhambrick Click to view user profileClick to add this user to your buddy list  
3. "RE: filtering out domains"
In response to message #2
 
   The raw log files are in Apache format:
CustomLog /var/log/httpd/access_log "%{Host}i %h %l %u %t \"%r\" %s %b %v \"%{Referer}i\" \"%{User-agent}i\"

Sample of log;\:


76.30.60.79 - - <10/Feb/2011:00:00:30 -0500> "GET /images/c21_logo_blk_back%20copy.jpg HTTP/1.1" 200 49801 www.orlandorealestateflorida.com "http://www.orlandorealestateflorida.com/famous/famous.htm"; "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.11) Gecko/20100701 Firefox/3.5.11" "-"
76.30.60.79 - - <10/Feb/2011:00:00:35 -0500> "GET /favicon.ico HTTP/1.1" 200 3068 www.orlandorealestateflorida.com "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.11) Gecko/20100701 Firefox/3.5.11" "-"
76.30.60.79 - - <10/Feb/2011:00:00:38 -0500> "GET /favicon.ico HTTP/1.1" 206 3068 www.orlandorealestateflorida.com "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.11) Gecko/20100701 Firefox/3.5.11" "-"

It is little hard for me to understand but it seems the site I looking to filter in (get the stats for this site only is the 1st www.orlandorealestateflorida.com that shows up after the code and bites delivered (HTTP/1.1" 200 3068) and not the http://www.orlandorealestateflorida.com/famous/famous.htm

If that is the case how would I filter this site out from the others. the others so as to get me started.

Thanks
John


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
dgilmoreadmin
Member since Nov-18-04
3646 posts
Feb-11-11, 10:46 AM (PDT)
Click to EMail dgilmore Click to send private message to dgilmore Click to view user profileClick to add this user to your buddy list Click to send message via AOL IM  
4. "RE: filtering out domains"
In response to message #3
 
Hi-

The %v is the virtual domain, which is www.your-domain.com and Sawmill would parse that.

In general you can use the following as a guide to understand the Apache log format directive:

http://httpd.apache.org/docs/2.0/mod/mod_log_config.html#formats

As a note, the log file you have pasted does not look to be generated by that custom format string you posted. Are you sure this is the log format string?

David
Sawmill Product Support Team
support@flowerfire.com


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
jhambrick
Member since Feb-7-11
5 posts
Feb-22-11, 05:38 PM (PDT)
Click to EMail jhambrick Click to send private message to jhambrick Click to view user profileClick to add this user to your buddy list  
5. "RE: filtering out domains"
In response to message #4
 
   I got this back from 1and1.

Thank you for contacting us.

As we have checked, the format in our Frequently Asked Questions page is correct. These were tested before it was posted in our website and these are indeed correct.

If you have any further questions please do not hesitate to contact us.

--
Sincerely,
Kyle Del Mar
Technical Support
1&1 Internet

I have noticed that 1and1 support is lacking the ability comprehend English, so this is where we are at now.


In the mean time I looked at the language the filters use and it is Greek to me. By that I mean It looks like on has to learn a whole new language to use it. That is not so bad because I will bet you can make this thing sing if you know what you are doing. The problem is that I don't want to learn a new language for filtering, I just don't have the time.

all I want is to start with a few filters and I should be able to figure it out from there, So how much support are we going to get, just starting off with out having to pay for support... ie can you give us the code for my simple simple request, without me learning the code and then reinvent the wheel, again

I also asked (but did not receive an answer) if you maintained a file of popular filter request that we could look at and modify for our needs.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
jhambrick
Member since Feb-7-11
5 posts
Feb-22-11, 05:48 PM (PDT)
Click to EMail jhambrick Click to send private message to jhambrick Click to view user profileClick to add this user to your buddy list  
6. "RE: filtering out domains"
In response to message #5
 
   BTW is there some sort of very fast tutorial that one can get through and skims the system to pick the code relevant to my project.

I have , i think a simple request- I only want to get results of one domain name mixed in with several other domains. How?????


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
dgilmoreadmin
Member since Nov-18-04
3646 posts
Feb-22-11, 06:30 PM (PDT)
Click to EMail dgilmore Click to send private message to dgilmore Click to view user profileClick to add this user to your buddy list Click to send message via AOL IM  
7. "RE: filtering out domains"
In response to message #6
 
LAST EDITED ON Feb-22-11 AT 06:40 PM (PDT)
 
Hi-

We can definitely give you some info on how to fulfill your request, however, I'm a bit confused at where things stand now. I read the thread, can you let me know what you'd like to do, or is your request the same as you've posted?

Is it possible to send in a log sample (compressed), and we can take a look at it? In general what you posted does look like it's got referrer, however, I'm not seeing the virtual domain.

David
Sawmill Product Support Team
support@flowerfire.com


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top

Conferences | Topics | Previous Topic | Next Topic
© 2011 Flowerfire | Copyright | Privacy Policy | License Agreement | Terms of Use | Contact | Feedback | About
Sawmill Software
Sawmill Software
Back to Sawmill Home