FAQ: Using a Command-line Log Source

Can Sawmill use scp, or sftp, or ssh, or https, to download log data? Can it uncompress tar, or arc, or sea, or hqx, etc.?

Short Answer

Not directly, but you can do it by using a command-line log source to run a command line, script, or program that does whatever is necessary to fetch the data, and prints it to Sawmill.

Long Answer

Sawmill supports many different methods of acquiring log data, including direct access to local files, and FTP or HTTP access to remote files; it can also decompress the major compression formats on the fly, including zip, gzip, and bzip2. If you need to use a different method to fetch the log data, like scp, sftp, or ssh, or if you need to read the log data from a database, or if you need to uncompress, decode, or decrypt a format that is not directly supported by Sawmill, you can do it using a command-line log source.

Command-line log sources are very simple in concept. You give Sawmill a command line; it runs the command line whenever it needs to get the log data; the command, script or program you specify "prints: the log data (i.e. generates it to stdout, the standard command line output stream), and Sawmill reads the output of the command to get the log data. The provides you with unlimited flexibility in how you feed your data to Sawmill.

For instance, suppose Sawmill didn't support gzip for at (it does). Then you could use the following (UNIX) command log source: /bin/gunzip -c /logs/mylog.gz. Since the -c flag tells gunzip to dump the output to stdout, Sawmill will read the log data directly from this command, without needing to use its built-in gunzipper. More usefully, any decompression utility with a similar flag can be used to allow Sawmill to read any compressed, archived, or encrypted log directly, even if it doesn't know anything about the format.

Even if you don't have a program that will dump the data to stdout, you can still use this approach by writing a tiny script. Consider the following (UNIX) shell script which scp'd files from a remote server and feeds them to Sawmill:

  scp user@host:/logs/mylog.txt /tmp/templog
  cat /tmp/templog
  rm /tmp/templog

This script copies a log file from a remote machine (securely, using scp), prints it to stdout using "cat", and deletes it when it's done. The same script with slight modifications, could copy multiple files, or use a different method than scp to fetch the files (like sftp).

A simpler (and better) example which does the same thing is this command:

  scp -qC user@host:/logs/mylog.txt > /dev/stdout

This explicitly scps the files to stdout, which sends them straight into Sawmill without the intermediate step of being stored on the disk or deleted. Since it's just one line, there's no need to use a script at all; this single line can be the command for the log source.