rsyslog-7.4.4: using imklog+ompgsql causes 100% CPU usage

This is the place for developers to discuss bugs, new features and everything else about code changes.

Google Ads


rsyslog-7.4.4: using imklog+ompgsql causes 100% CPU usage

Postby chrisk » Sun Mar 02, 2014 7:03 pm

Hi,

With rsyslog-7.4.4 on Debian, using a combination of imklog and ompgsql results in a 100% CPU load as soon as something gets written to the kernel log. Furthermore, the following message is continuously spammed to kern.*:

Code: Select all
 rsyslogd: imklog: error reading kernel log - shutting down: Bad file descriptor


I can successfully reproduce this problem, on different systems, by running rsyslogd with the following minimal config and then triggering a write to the kernel log, for example by plugging in a USB stick:

Code: Select all
$ModLoad imuxsock
$ModLoad imklog

*.*;kern.none   /tmp/kern.none.log
# "Bad file descriptor" messages get spammed to this
kern.*          /tmp/kern.log

$ModLoad ompgsql
*.* :ompgsql:HOST,USER,DB,PASS


I don't know why it should make a difference, but in any case: the Postgres database I'm connecting to with ompgsql is connecting to is a remote one (TCP).

Can anyone reproduce this on a non-Debian system?

Christian
chrisk
New
 
Posts: 3
Joined: Sun Mar 02, 2014 6:54 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: rsyslog-7.4.4: using imklog+ompgsql causes 100% CPU usag

Postby dlang » Mon Mar 03, 2014 5:48 am

I don't think the ompgsql has anything to do with the problem.

I do know that the process to get data from the kernel changed 'recently' and I have run into problems before reading from the imklog input.

I'd suggest first just commenting out the imklog and see if we are still getting the kernel logs or not.

If not, my guess is that this is a permission problem (one of those things that varies from distro to distro), try launching rsyslog in debug mode (-d -n) and see if there are any messages about klog during startup. If nothing else, this should identify what is being opened so that we can check the permissions of it.

are you using SELinux or AppArmor by any chance?

David Lang
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog-7.4.4: using imklog+ompgsql causes 100% CPU usag

Postby chrisk » Mon Mar 03, 2014 6:59 pm

Hi David,

thanks for your feedback.

dlang wrote:I don't think the ompgsql has anything to do with the problem.

I do know that the process to get data from the kernel changed 'recently' and I have run into problems before reading from the imklog input.


I've tried this out and the problem really seems to be imklog+ompgsql. If I drop either one, rsyslog behaves normally.

dlang wrote:I'd suggest first just commenting out the imklog and see if we are still getting the kernel logs or not.


Nope, didn't receive anything.

dlang wrote:If not, my guess is that this is a permission problem (one of those things that varies from distro to distro)


That was my first thought as well (especially given the history between the kernel and imklog you alluded to), but with the minimal config from my previous post, rsyslogd is running as root.

dlang wrote:try launching rsyslog in debug mode (-d -n) and see if there are any messages about klog during startup. If nothing else, this should identify what is being opened so that we can check the permissions of it.


Now here's something interesting: running with -d -n caused the problem to disappear. When running only with -d, it is still present, however. So, it appears to be the combination of imklog + ompgsql + auto-backgrounding.

dlang wrote:are you using SELinux or AppArmor by any chance?


No, I am not.

Christian
chrisk
New
 
Posts: 3
Joined: Sun Mar 02, 2014 6:54 pm

Re: rsyslog-7.4.4: using imklog+ompgsql causes 100% CPU usag

Postby dlang » Mon Mar 03, 2014 8:21 pm

Ok, the fact that ompsql seems to be required, and running in debug mode avoids the problem says to me that there is some sort of race condition here. This is getting beyond the scope of what I can deal with and we'll need to have Rainer take a look at this.

Before we take up his time, can you check if the problem is still there in 7.4.6? There are enough fixes in there that it may have solved this issue.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog-7.4.4: using imklog+ompgsql causes 100% CPU usag

Postby chrisk » Mon Mar 03, 2014 9:24 pm

Oh, looky here:
rsyslog-7.6.0/Changelog wrote:ompgsql bugfix: improper handling of auto-backgrounding mode. If rsyslog was set to auto-background itself (default code behaviour, but many distros now turn it off for good reason), ompgsql could not properly connect. This could even lead to a segfault. The core reason was that a PG session handle was kept open over a fork, something that is explicitely forbidden in the PG API. Thanks to Alain for the analysis and the patch.


Version 7.6.0 is not visible from http://www.rsyslog.com/change-log/ at the moment, it's only reachable from the Downloads page. I only noticed this after you mentioned 7.4.6 despite 7.4.10 already being available, so I assumed you meant 7.6.0.

I haven't verified this yet -- there are new build dependencies not yet present in Debian -- so I'll wait for the maintainer of the Debian package to update it. But that bugfix sounds pretty much spot on.

Thanks for your help, David!
chrisk
New
 
Posts: 3
Joined: Sun Mar 02, 2014 6:54 pm

Google Ads



Return to Developer's Corner

Who is online

Users browsing this forum: No registered users and 0 guests

cron