MonitorWare Knowledge Base

Twice I've had rsyslogd randomly stop on Ubuntu 6.10 servers with no explanatory (or even informatory) message logged.

I know that's not much information, but that's what I've got to go on. There are no cronjobs scheduled to run at the time that it stopped, and no system disruption or other correlating event. There are no core files lying around my filesystem.

Any ideas on troubleshooting this? I could run rsyslogd in debug mode, piping all output to a file - but only twice (on two different servers) in the last 2 months or so makes that impractical.

-HKS

• Pulling out your Hair?
• Wasting Time and Money?
• Deadline Approaching?

Oh, that's hard to debug. Can you post (or mail) your configs? I am asking for them because I'd look for what features are used and how. Maybe I can spot something that is not yet used on a broad basis and thus a good candidate...

Rainer

I know, sorry for the pain-in-the-ass problem report.

As I pasted in my config below, I realized that the working directory doesn't actually exist. I've created it on all the relevant servers, but I don't know whether that will help.

Thanks for any help.
-HKS

Code: Select all

# /usr/local/etc/rsyslog.conf
#
# Client configuration file for rsyslogd, enhanced syslog.


$ModLoad immark.so              # --MARK-- message capability
$ModLoad imuxsock.so            # support for local system logging
$ModLoad imklog.so              # kernel logging support
#$ModLoad omrelp.so             # RELP output support


# ----------------------------------------
# Templates
#  must be defined before they're used
# ----------------------------------------


# ----------------------------------------
# Global Settings
# ----------------------------------------
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$ActionForwardDefaultTemplate RSYSLOG_ForwardFormat

# Dump these noisy logs since there's nothing we can do about them
# /var/log/samba/log.winbindd still has them
:msg, contains, "nsswitch/winbindd_group.c:winbindd_getgrent"           ~
:msg, contains, "could not lookup domain group"                         ~


# ----------------------------------------
# Forwarding
#  everything to <munged> except mark messages
# ----------------------------------------
$WorkDirectory /var/spool/rsyslog/work/         # where to place spool files, if necessary
$ActionQueueType LinkedList                     # start up a separate queue for this action
$ActionQueueFileName srvrfwd                    # set file name and implicitly enable disk mode
$ActionResumeRetryCount -1                      # infinite retries on failure
$ActionQueueSaveOnShutdown on                   # save in-memory data to disk upon shutdown
:msg, !isequal, "-- MARK --"                     @<munged>:514


# ----------------------------------------
# Console alerts
# ----------------------------------------
*.err;kern.warning;auth.notice;mail.crit        /dev/console
*.emerg                                         *

# ----------------------------------------
# Local storage
# ----------------------------------------
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit    /var/log/messages
auth.info;authpriv.info                         /var/log/auth.log
cron.*                                          /var/log/cron.log
daemon.*                                        /var/log/daemon.log
ftp.info                                        /var/log/xferlog
kern.*                                          /var/log/kern.log
mail.info                                       -/var/log/maillog
security.*                                      /var/log/security.log
*.=debug                                        /var/log/debug.log


Trying to keeping up with things. Did the problem re-occur?

Rainer

It has not. My entire environment is in flux right now (one of the reasons I've not been around as much), so for now let's just assume that it was random events unrelated to rsyslogd code. Things *should* settle down in the next 4-6 weeks, and I'll let you know if it recurs.

Thanks for the followup.

-HKS

And naturally, I discover that it stopped a few days ago on one of my servers. This is an Ubuntu 6.10 box, running rsyslogd 3.18.1. Following is some output in /var/log/messages:

Code: Select all

*** glibc detected *** /usr/local/sbin/rsyslogd: double free or corruption (out): 0xb4e7c730 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6[0xb7ebd8bd]
/lib/tls/i686/cmov/libc.so.6(__libc_free+0x84)[0xb7ebda44]
/usr/local/sbin/rsyslogd(msgDestruct+0x40)[0x8067370]
/usr/local/sbin/rsyslogd(actionCallAction+0xbc)[0x80716ac]
/usr/local/sbin/rsyslogd[0x8050578]
/usr/local/sbin/rsyslogd(llExecFunc+0x50)[0x8070930]
/usr/local/sbin/rsyslogd[0x8050207]
/usr/local/sbin/rsyslogd[0x8061326]
/usr/local/sbin/rsyslogd(wtiWorker+0x24f)[0x805f1ef]
/usr/local/sbin/rsyslogd[0x805e5b4]
/lib/tls/i686/cmov/libpthread.so.0[0xb7f9e504]
/lib/tls/i686/cmov/libc.so.6(__clone+0x5e)[0xb7f2451e]
======= Memory map: ========
08048000-0807c000 r-xp 00000000 08:01 182524     /usr/local/sbin/rsyslogd
0807c000-0807f000 rw-p 00033000 08:01 182524     /usr/local/sbin/rsyslogd
0807f000-08255000 rw-p 0807f000 00:00 0          [heap]
b4b00000-b4b48000 rw-p b4b00000 00:00 0
b4b48000-b4c00000 ---p b4b48000 00:00 0
b4c00000-b4c21000 rw-p b4c00000 00:00 0
b4c21000-b4d00000 ---p b4c21000 00:00 0
b4d00000-b4e00000 rw-p b4d00000 00:00 0
b4e00000-b4e81000 rw-p b4e00000 00:00 0
b4e81000-b4f00000 ---p b4e81000 00:00 0
b4f00000-b5000000 rw-p b4f00000 00:00 0
b5000000-b5100000 rw-p b5000000 00:00 0
b5100000-b5200000 rw-p b5100000 00:00 0
b5200000-b5300000 rw-p b5200000 00:00 0
b5300000-b5400000 rw-p b5300000 00:00 0
b5400000-b5500000 rw-p b5400000 00:00 0
b5500000-b5600000 rw-p b5500000 00:00 0
b5630000-b563a000 r-xp 00000000 08:01 274192     /lib/libgcc_s.so.1
b563a000-b563b000 rw-p 00009000 08:01 274192     /lib/libgcc_s.so.1
b563f000-b5640000 ---p b563f000 00:00 0
b5640000-b5e40000 rw-p b5640000 00:00 0
b5e40000-b5e41000 ---p b5e40000 00:00 0
b5e41000-b6641000 rw-p b5e41000 00:00 0
b6641000-b6642000 ---p b6641000 00:00 0
b6642000-b6e42000 rw-p b6642000 00:00 0
b6e42000-b6e43000 ---p b6e42000 00:00 0
b6e43000-b7643000 rw-p b6e43000 00:00 0
b7643000-b7644000 ---p b7643000 00:00 0
b7644000-b7e44000 rw-p b7644000 00:00 0
b7e44000-b7e49000 r-xp 00000000 08:01 182832     /usr/local/lib/rsyslog/imklog.so
b7e49000-b7e4a000 rw-p 00004000 08:01 182832     /usr/local/lib/rsyslog/imklog.so
b7e4a000-b7e4b000 rw-p b7e4a000 00:00 0
b7e4b000-b7e4d000 r-xp 00000000 08:01 182527     /usr/local/lib/rsyslog/imuxsock.so
b7e4d000-b7e4e000 rw-p 00001000 08:01 182527     /usr/local/lib/rsyslog/imuxsock.so
b7e4e000-b7e4f000 r-xp 00000000 08:01 182525     /usr/local/lib/rsyslog/immark.so
b7e4f000-b7e50000 rw-p 00000000 08:01 182525     /usr/local/lib/rsyslog/immark.so
b7e51000-b7e55000 r-xp 00000000 08:01 182483     /usr/local/lib/rsyslog/lmnet.so
b7e55000-b7e56000 rw-p 00003000 08:01 182483     /usr/local/lib/rsyslog/lmnet.so
b7e56000-b7e57000 rw-p b7e56000 00:00 0
b7e57000-b7f84000 r-xp 00000000 08:01 274200     /lib/tls/i686/cmov/libc-2.4.so
b7f84000-b7f86000 r--p 0012c000 08:01 274200     /lib/tls/i686/cmov/libc-2.4.so
b7f86000-b7f88000 rw-p 0012e000 08:01 274200     /lib/tls/i686/cmov/libc-2.4.so
b7f88000-b7f8b000 rw-p b7f88000 00:00 0
b7f8b000-b7f92000 r-xp 00000000 08:01 274216     /lib/tls/i686/cmov/librt-2.4.so
b7f92000-b7f94000 rw-p 00006000 08:01 274216     /lib/tls/i686/cmov/librt-2.4.so
b7f94000-b7f95000 rw-p b7f94000 00:00 0
b7f95000-b7f97000 r-xp 00000000 08:01 274203     /lib/tls/i686/cmov/libdl-2.4.so
b7f97000-b7f99000 rw-p 00001000 08:01 274203     /lib/tls/i686/cmov/libdl-2.4.so
b7f99000-b7fa8000 r-xp 00000000 08:01 274214     /lib/tls/i686/cmov/libpthread-2.4.so
b7fa8000-b7faa000 rw-p 0000f000 08:01 274214     /lib/tls/i686/cmov/libpthread-2.4.so
b7faa000-b7fac000 rw-p b7faa000 00:00 0
b7fac000-b7fbf000 r-xp 00000000 08:01 131828     /usr/lib/libz.so.1.2.3
b7fbf000-b7fc0000 rw-p 00012000 08:01 131828     /usr/lib/libz.so.1.2.3
b7fc0000-b7fc1000 rw-p b7fc0000 00:00 0
b7fc1000-b7fc3000 r-xp 00000000 08:01 182487     /usr/local/lib/rsyslog/lmtcpclt.so
b7fc3000-b7fc4000 rw-p 00001000 08:01 182487     /usr/local/lib/rsyslog/lmtcpclt.so
b7fc4000-b7fc6000 rw-p b7fc4000 00:00 0
b7fc6000-b7fdf000 r-xp 00000000 08:01 276053     /lib/ld-2.4.so
b7fdf000-b7fe1000 rw-p 00018000 08:01 276053     /lib/ld-2.4.so
bfcfa000-bfd10000 rw-p bfcfa000 00:00 0          [stack]
ffffe000-fffff000 ---p 00000000 00:00 0          [vdso]


Let me know what I can do to help track this down, and thanks!

-HKS

Popping my head up out of my hole to add that this is still occurring on 3.18.5. I'm only seeing it on a single box (Ubuntu 6.10), and here is the little information I've got out of /var/log/messages:

Code: Select all

*** glibc detected *** /usr/local/sbin/rsyslogd: double free or corruption (out): 0xb5562958 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6[0xb7ea28bd]
/lib/tls/i686/cmov/libc.so.6(__libc_free+0x84)[0xb7ea2a44]
/usr/local/sbin/rsyslogd(msgDestruct+0x40)[0x8067380]
/usr/local/sbin/rsyslogd(actionCallAction+0xbc)[0x807173c]
/usr/local/sbin/rsyslogd[0x8050588]
/usr/local/sbin/rsyslogd(llExecFunc+0x50)[0x80709b0]
/usr/local/sbin/rsyslogd[0x8050217]
/usr/local/sbin/rsyslogd[0x8061346]
/usr/local/sbin/rsyslogd(wtiWorker+0x24f)[0x805f20f]
/usr/local/sbin/rsyslogd[0x805e5d4]
/lib/tls/i686/cmov/libpthread.so.0[0xb7f83504]
/lib/tls/i686/cmov/libc.so.6(__clone+0x5e)[0xb7f0951e]
======= Memory map: ========
08048000-0807c000 r-xp 00000000 08:01 178595     /usr/local/sbin/rsyslogd
0807c000-0807f000 rw-p 00033000 08:01 178595     /usr/local/sbin/rsyslogd
0807f000-080d0000 rw-p 0807f000 00:00 0          [heap]
b5300000-b536f000 rw-p b5300000 00:00 0
b536f000-b5400000 ---p b536f000 00:00 0
b5500000-b556b000 rw-p b5500000 00:00 0
b556b000-b5600000 ---p b556b000 00:00 0
b5615000-b561f000 r-xp 00000000 08:01 274192     /lib/libgcc_s.so.1
b561f000-b5620000 rw-p 00009000 08:01 274192     /lib/libgcc_s.so.1
b5625000-b5626000 ---p b5625000 00:00 0
b5626000-b5e26000 rw-p b5626000 00:00 0
b5e26000-b5e27000 ---p b5e26000 00:00 0
b5e27000-b6627000 rw-p b5e27000 00:00 0
b6627000-b6628000 ---p b6627000 00:00 0
b6628000-b6e28000 rw-p b6628000 00:00 0
b6e28000-b6e29000 ---p b6e28000 00:00 0
b6e29000-b7629000 rw-p b6e29000 00:00 0
b7629000-b762a000 ---p b7629000 00:00 0
b762a000-b7e2a000 rw-p b762a000 00:00 0
b7e2a000-b7e2f000 r-xp 00000000 08:01 246464     /usr/local/lib/rsyslog/imklog.so
b7e2f000-b7e30000 rw-p 00004000 08:01 246464     /usr/local/lib/rsyslog/imklog.so
b7e30000-b7e31000 rw-p b7e30000 00:00 0
b7e31000-b7e33000 r-xp 00000000 08:01 246411     /usr/local/lib/rsyslog/imuxsock.so
b7e33000-b7e34000 rw-p 00001000 08:01 246411     /usr/local/lib/rsyslog/imuxsock.so
b7e34000-b7e35000 r-xp 00000000 08:01 246409     /usr/local/lib/rsyslog/immark.so
b7e35000-b7e36000 rw-p 00000000 08:01 246409     /usr/local/lib/rsyslog/immark.so
b7e37000-b7e38000 rw-p b7e37000 00:00 0
b7e38000-b7e3a000 r-xp 00000000 08:01 246405     /usr/local/lib/rsyslog/lmtcpclt.so
b7e3a000-b7e3b000 rw-p 00001000 08:01 246405     /usr/local/lib/rsyslog/lmtcpclt.so
b7e3b000-b7e3c000 rw-p b7e3b000 00:00 0
b7e3c000-b7f69000 r-xp 00000000 08:01 274200     /lib/tls/i686/cmov/libc-2.4.so
b7f69000-b7f6b000 r--p 0012c000 08:01 274200     /lib/tls/i686/cmov/libc-2.4.so
b7f6b000-b7f6d000 rw-p 0012e000 08:01 274200     /lib/tls/i686/cmov/libc-2.4.so
b7f6d000-b7f70000 rw-p b7f6d000 00:00 0
b7f70000-b7f77000 r-xp 00000000 08:01 274216     /lib/tls/i686/cmov/librt-2.4.so
b7f77000-b7f79000 rw-p 00006000 08:01 274216     /lib/tls/i686/cmov/librt-2.4.so
b7f79000-b7f7a000 rw-p b7f79000 00:00 0
b7f7a000-b7f7c000 r-xp 00000000 08:01 274203     /lib/tls/i686/cmov/libdl-2.4.so
b7f7c000-b7f7e000 rw-p 00001000 08:01 274203     /lib/tls/i686/cmov/libdl-2.4.so
b7f7e000-b7f8d000 r-xp 00000000 08:01 274214     /lib/tls/i686/cmov/libpthread-2.4.so
b7f8d000-b7f8f000 rw-p 0000f000 08:01 274214     /lib/tls/i686/cmov/libpthread-2.4.so
b7f8f000-b7f91000 rw-p b7f8f000 00:00 0
b7f91000-b7fa4000 r-xp 00000000 08:01 131828     /usr/lib/libz.so.1.2.3
b7fa4000-b7fa5000 rw-p 00012000 08:01 131828     /usr/lib/libz.so.1.2.3
b7fa5000-b7fa9000 r-xp 00000000 08:01 246400     /usr/local/lib/rsyslog/lmnet.so
b7fa9000-b7faa000 rw-p 00003000 08:01 246400     /usr/local/lib/rsyslog/lmnet.so
b7faa000-b7fac000 rw-p b7faa000 00:00 0
b7fac000-b7fc5000 r-xp 00000000 08:01 276053     /lib/ld-2.4.so
b7fc5000-b7fc7000 rw-p 00018000 08:01 276053     /lib/ld-2.4.so
bfd68000-bfd7e000 rw-p bfd68000 00:00 0          [stack]
ffffe000-fffff000 ---p 00000000 00:00 0          [vdso]


I can't find any core dumps. There are no cronjobs running around the time that it died, no log rotations.

FWIW, this could just be a broken box. I'm anxious to get moved away from Ubuntu anyway, so it's not a huge problem. But worth noting.

Back to my hole now.

-HKS