MonitorWare Knowledge Base

Hi, I've had rsyslog running for a few months now and everything seemed to be normal except for a netstream error that appeared every hour or so (i posted about this is another thread, but so far have been unable to find anything relating to the error in the logs). My setup is as such:

1 rsyslog server
Centos 5.3
Rsyslog 3.20.6
logging to mysql
also running phplogcon 2.6.2

3 rsyslog clients on LAN
Centos 5.3
rsyslogd 3.20.4
logging via TLS over TCP

1 rsyslog client remote
Centos 5.3
rsyslog 3.20.4
logging via TLS over TCP

The issue is that rsyslog ramps up to full memory usage on the server machine until it starts swapping like crazy and then eventually killing the rsyslog process. I've tried setting the stack limit lower and buffering the input to the mysql database but it doesn't seem to be doing anything. The only error messages I get are the OOM messages when I do a dmesg.

I've also tried upgrading to the devel versions of rsyslog like 4.3.0 but I don't have gnutls version 2. Centos 5's rpm is 1.4.something.

Any ideas or things I'm missing?

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

I forgot, here is the server conf file:


$AllowedSender UDP, 127.0.0.1, 192.168.1.0/24

$ModLoad immark.so # provides --MARK-- message capability
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # kernel logging (formerly provided by rklogd)
$ModLoad ommysql.so

*.* >127.0.0.1,*****,*****,****

# UDP Syslog Server:
#$ModLoad imudp.so # provides UDP syslog reception
#$UDPServerRun 514 # start a UDP syslog server at standard port 514

# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /etc/rsyslog/ca.pem
$DefaultNetstreamDriverCertFile /etc//rsyslog/cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog/key.pem

$ModLoad imtcp.so # load listener

$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated
$InputTCPServerRun 10514 # start up listener at port 10514

$WorkDirectory /etc/rsyslog # default location for work (spool) files
$MainMsgQueueFileName mainq # set file name, also enables disk mode

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName dbq # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure

Okay, upon further review. It appears that without phplogcon running, rsyslog acts just fine. Is there a reason why Apache/phplogcon would cause rsyslog to eat up the memory? And I ask this because it definitely was rsyslog that was consuming the RAM. Apache was using a bunch, but like rsyslog.

If I had to guess, which I have to based on the information in the post, phplogcon was eating up too many resources on the backend database, causing rsyslog to buffer more than it should because of a low insert rate into the database.

Check out what the database is doing with apache/phplogcon is running to see what's going on.

Thanks, it makes sense. I can tell it's definitely phplogcon.

I'll post over there to put the thread in the right spot.