MonitorWare Knowledge Base

We have set up rsyslog on a Fedora 11 virtual machine, and are currently using it to monitor our Firebox x1000. We get a lot of logs, so we have it rotating to file using the system date, on a daily basis. We implemented phpLogCon as well, and it's awesome to be able to view our logs and search them. The problem I'm seeing is with performance. Searches are taking quite a while, but not knowing a baseline performance standard is probably my biggest issue. We also plan to start using rsyslog to monitor our Cisco switches and other internal equipment at our primary site, and probably roll out similar rsyslog servers at our other locations as well. What I'm driving at here is, would it be better performance to write the logs to mysql vs. flat file? The flat file logs are averaging anything between 50 - 100 MB daily. What I was thinking was to syslog each device to a separate DB, and set up multiple sources, one for each device.

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

I was thinking resources may have been an issue, so I upgraded the VM from 512MB to 4GB RAM, and added a 2nd processor. After doing so, when performing a search, I run top on the machine and httpd is the only thing getting hit, @ around 10% CPU only, not much in the way of memory either. Going to try syslogging to mysql to see if there's a big difference.

Hi.
Not sure exactly how much data you are storing, nor how exactly phpLogCon works (does it get logs from MySQL or flat files ? or both ?).

Basically, storing logs in a DB is great for low volumes.
Things you want to consider are the following :
- are the columns you are searching indexed ?
- can the entire index hold in ram ? Try tweaking MySQL.

The performance bottleneck when handling logs will (almost) always be IO. Be it IO from reading the MySQL table, or IO from reading files. This explains the 10% CPU use. If you have indexes, and MySQL uses those indexes, it should be slightly faster than files. However, reading files will almost always be faster than a full mysql table scan (Table scans will happen when there is no index, the index can't be used, etc. Any query containing 'LIKE' will cause a table scan.).

A great way to limit this IO bottleneck is compression. Log data compresses tenfolds with gzip, and reads will be 5 to 10 times faster. Using tools like zgrep make your life easy afterwards. Bzip2 compresses better, but is way too slow.

side-note: the recent devel of v4 can write log files in zip format (no need to zip later, save of i/o time)