MonitorWare Knowledge Base

I have been testing Rsyslog 3.19 & phpLogCon 2.5.6 and things have been going smoothly, so we expanded the test to receive messages from our firewalls and a couple of our Tier 1 servers. I quickly noticed that messages from the localhost were no longer making it to phplogcon. I have attributed this to the extremely high amount of data from the firewalls and so scaled back what information we send from the firewalls, however I have been running a tail -f /var/log/secure on the centralized rsyslog server and I consistantly find mesages that are being logged but not written to the DB and therefore not appearing in phplogcon. One of the biggest reasons for looking at phplogcon & rsyslog is to be able to monitor (in as close to real-time as possible) who is logged into these various servers? Any suggestions for how to setup a centralized rsyslog server to handle large amounts of traffic reliably?

• Custom phpLogCon configuration?
• Maintenance Contract?
• Installation support?

Can you please post your rsyslog.conf. Actually, a primary design goal for rsyslog is reliability, so this concerns us much. For the time being, I think a filter goes into the way. If not, we probably need to enable debug logging for a short while.

Thanks,
Rainer

There is not a filter applied at the moment, I have noticed how the filter appears in the URL and I am looking at the index.php. I have actually set the page to refresh every 5 seconds and load 50 records so that I could ensure that it wasn't be missed by the refresh.

Rsyslog.conf is as follows:

#rsyslog v3 config file

# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

#### MODULES ####
$ModLoad ommail #Work in progress, this module is not active yet.
$ActionMailSMTPServer 10.0.0.100
$ActionMailSMTPPort 25
$ActionMailFrom root
$ActionMailTo root
$template mailSubject,"Notification: %server%"
$template mailBody,"RSYSLOG Alert\r\nmsg=%msg%"
$ActionMailSubject mailSubject
$ActionExecOnlyOnceEveryInterval 60

$ModLoad ommysql.so # provides support for MySQL
$ModLoad imudp.so
$UDPServerAddress 10.0.0.100
$UDPServerRun 514

$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by rklogd)
$ModLoad immark.so # provides --MARK-- message capability
# Provides TCP syslog reception
$ModLoad imtcp.so
$InputTCPServerRun 514

$WorkDirectory /rsyslog/work # default location for work (spool) files

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName dbq # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

#*.* :ommysql:10.0.0.100,db,un,pw

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

*.* :ommysql:127.0.0.1,db,un,pw
*.* :ommail:;mailBody

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/spppl/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
#
### end of the forwarding rule ###

Example:

We have a cron job that utilizes stunnel to the localhost, the stunnel connection and the cron job are correctly written to the logs, however only the cron job appears in phplogcon.

[p_leech@tx-testbox ~]$ sudo tail -f /var/log/secure
Password:
Sep 23 10:05:01 linksys stunnel[2105]: as-database connected from 127.0.0.1:46603
Sep 23 10:10:02 linksys stunnel[2105]: as-database connected from 127.0.0.1:46612
Sep 23 10:15:02 linksys stunnel[2105]: as-database connected from 127.0.0.1:46621
Sep 23 10:20:02 linksys stunnel[2105]: as-database connected from 127.0.0.1:46630
Sep 23 10:25:02 linksys stunnel[2105]: as-database connected from 127.0.0.1:46639
Sep 23 10:30:02 linksys stunnel[2105]: as-database connected from 127.0.0.1:46648
Sep 23 10:35:02 linksys stunnel[2105]: as-database connected from 127.0.0.1:46657
Sep 23 10:40:02 linksys stunnel[2105]: as-database connected from 127.0.0.1:46666
Sep 23 10:45:02 linksys stunnel[2105]: as-database connected from 127.0.0.1:46675
Sep 23 10:47:41 tx-testbox sudo: p_leech : TTY=pts/0 ; PWD=/home/p_leech ; USER=root ; COMMAND=/usr/bin/tail -f /var/log/secure

[p_leech@tx-testbox ~]$ sudo tail -f /var/log/cron
Sep 23 10:01:01 tx-snort2 crond[14567]: (root) CMD (run-parts /etc/cron.hourly)
Sep 23 10:05:01 linksys crond[3956]: (root) CMD (/usr/local/bin/monitor-servers)
Sep 23 10:10:02 linksys crond[4011]: (root) CMD (/usr/local/bin/monitor-servers)
Sep 23 10:15:01 linksys crond[4067]: (root) CMD (/usr/local/bin/monitor-servers)
Sep 23 10:20:01 linksys crond[4121]: (root) CMD (/usr/local/bin/monitor-servers)
Sep 23 10:25:01 linksys crond[4178]: (root) CMD (/usr/local/bin/monitor-servers)
Sep 23 10:30:01 linksys crond[4233]: (root) CMD (/usr/local/bin/monitor-servers)
Sep 23 10:35:01 linksys crond[4289]: (root) CMD (/usr/local/bin/monitor-servers)
Sep 23 10:40:01 linksys crond[4343]: (root) CMD (/usr/local/bin/monitor-servers)
Sep 23 10:45:01 linksys crond[4400]: (root) CMD (/usr/local/bin/monitor-servers)

I guess your /var/log/messages test was not valid Why?

you have

Code: Select all

$ActionExecOnlyOnceEveryInterval 60
... (no actions in between)...
*.info;mail.none;authpriv.none;cron.none /var/log/messages


That means /var/log/messages is only written once each minute In rsyslog, action modifiers modify the next action -whatever and whereever it may be.

Rainer

mmhh.. that doesn't explain what is in your last post. But anyhow, I'd start by restructuring the rsyslog.conf. There are many things that most probably do not modify the action that they are meant for.

See for example here: http://git.adiscon.com/?p=rsyslog.git;a ... 3c;hb=HEAD line 39 to 49 define a *single* action.

Rainer

That means /var/log/messages is only written once each minute ;) In rsyslog, action modifiers modify the next action -whatever and whereever it may be.

That explains some of the missing logging. Thanks!

mmhh.. that doesn't explain what is in your last post. But anyhow, I'd start by restructuring the rsyslog.conf.

I have restructured the rsyslog.conf and I think that may have corrected the problem...i'll be monitoring it for the rest of the day and will post back tomorrow with success of failure.

Thanks for the help!

if you like, you can post the new config and I'll review it for you - just in case...

#rsyslog v3 config file

# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

$ModLoad immark # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # kernel logging (formerly provided by rklogd)

#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/spppl/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
#
### end of the forwarding rule ###


#### MODULES ####
#$ModLoad ommail //Any advice on getting this to work would be appreciated...I installed from an RPM and not sure how activate ommail
#$ActionMailSMTPServer 10.0.0.100
#$ActionMailSMTPPort 25
#$ActionMailFrom root
#$ActionMailTo root
#$template mailSubject,"Notification: %server%"
#$template mailBody,"RSYSLOG Alert\r\nmsg=%msg%"
#$ActionMailSubject mailSubject
#$ActionExecOnlyOnceEveryInterval 60

$ModLoad ommysql.so # provides support for MySQL
$ModLoad imudp.so
$UDPServerAddress 10.0.0.100
$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp.so
#$InputTCPServerRun 514

$WorkDirectory /rsyslog/work # default location for work (spool) files
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName dbq # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure

*.* :ommysql:127.0.0.1,db,un,pw
#*.* :ommail:;mailBody

The config looks great. On the ommail issue: what exactly is it? There is nothing needed to activate it specifically. Do you see an error message when you startup rsyslogd (you may want to do an error checking run, please see http://www.rsyslog.com/doc-troubleshoot.html (under "configuration problems").

What I guess is that the RPM simply does not provide the ommail plugin...

Rainer

First off, thanks for the assistance yesterday, the problem with the 'missing' event entries is resolved.

As for the ommail issue, I would like to take advantage of the email alerting capability of rsyslog, but since I did not compile from source I was not sure how to enable it. I looked at the troubleshooting documentation, however I'm running 3.19. If the only way to enable email notification is to compile from source then that's what I'll need to do. If so I will open a post in the rsyslog forum for assistance.

Again thanks for the help getting phplogcon working!

I am not sure if you actually need to install from source. It depends on what the package offers. Most packagers have also created subpackages, so there may be a rsyslog-ommail (or whatever name) package available. Anyhow, compiling from source should be relatively painless.