MonitorWare Knowledge Base

hello -- i'm an NFS developer and consequently rely very heavily on syslog for printk debugging output.

i am absolutely baffled by something: i have 2 clusters, both set up from scratch with Fedora 9 installs.
on one of them, all kernel messages appear in dmesg and in /var/log/messages.
on the other, for whatever reason, my kernel logging goes only to dmesg; given that i need to coordinate debugging between userland and kernel components, i need all the logging in one file.

both clusters have the stock fedora 9 /etc/rsyslog.conf. this is really baffling me. at the very least, shouldn't a single simple rule like:
$ModLoad imklog.so
*.* <TAB> /var/log/messages
.. just punt and send everything there?

i can cat /proc/kmsg and see the stuff that's going into dmesg and i don't understand what could be going on. frankly, given that i'd lived with syslog-ng for so long, i tried using yum to uninstall rsyslog and immediately got hung up on dependency hell (over 500 packages would also be erased, and yum apparently can't be forced = lame), so it appears that trying to replace rsyslog is going to hose everything.

that said, however, i don't have any allegiance to any logger, syslog-ng or rsyslog or whatever -- i just need my kernel and daemon messages going to the same place.

can you recommend anything that i might try to fix this? i need only the most basic of setups.

thanks kindly,

david

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

The config you have quoted should bring the result you mention. However, I remember there were some fixes in regard to kernel logging. Could you pull the latest v3-stable version and install from source? I think there are good chances this will solve the problem.

thank you for the suggestion -- I know I wasn't able to provide any real useful specifics on the problem.

I downloaded the latest stable rsyslog and installed it -- to no avail. I tried all kinds of little tweaks to the conf file, verified that rsyslogd had /proc/kmsg open and that /proc/kmsg contained the relevant data.

It wasn't until I gave up in disgust and rebooted that things started working again. That kind of thing gives me a lot of concern -- I'll be watching in the coming days, but my other cluster nodes which've not been rebooted are still nonresponsive. These aren't junk machines or flaky kernels or failing hard drives or anything, so I remain confused and disconcerted.

If I get any more useful data, I'll let you know.

thanks again,

d
.

So this smells like a bug... Could you get me a debug log? Info here:

http://www.rsyslog.com/doc-troubleshoot.html

Rainer

thank you -- once again, i appreciate your help. fortunately, i think i have some more-useful data this time.

the short story is that i'm working with pNFS and GFS2, debugging a mount-time issue. i'm getting a lock-related warning printk, and it's thereafter that rsyslogd appears to stop logging further kernel messages -- so i wonder if something in the formatting there is an issue. the printk looks like the following in dmesg (full debug log will follow):

Code: Select all

dlm: connecting to 4
dlm: got connection from 4

================================================
[ BUG: lock held when returning to user space! ]
------------------------------------------------
clvmd/2182 is leaving the kernel with locks still held!
1 lock held by clvmd/2182:
#0:  (&ls->ls_in_recovery){....}, at: [<f8b1dbd1>] dlm_new_lockspace+0x5d0/0x78f [dlm]

that "dlm: got connection from 4" line, however, is the last that rsyslogd appends to /var/log/messages. I'm attempting to attach the log to this post... and that seems not to be working, so instead it's online at:
http://www.citi.umich.edu/u/richterd/rs ... rn-logging

please let me know if i can give you any other info. i really need lock debugging on in general, but for now i'm going to try to disable that printk and see if that's the culprit. thanks again,

david.

ooh, funky -- yup, if i disable the kernel config that triggers that BUG(), i can mount and at least for the time being everything appears copacetic.

so, perhaps there actually is a problem relating to some part of the formatting: the blank line, the line of all equals-signs, or something in that regard.

thanks, d.

sorry, I have been all day bound by a threading issue. I'll have a look at the formatting asap...

no worries -- I'm doing some development before going back to testing. once I do, I'll probably have more examples.

ah, one thing that maybe I should've mentioned more clearly last time: one weird thing I saw that I think I briefly mentioned before is that stopping and restarting rsyslogd didn't cause the kernel logging to reappear -- I had to reboot, which strikes me as really odd. perhaps I should do a static build and see if somehow the shared libraries are involved.... but, on the face of it, that sounds kind of far-fetched to me. <shrug> still, I don't have a better answer right now but I'll let you know if I come up with more.

thanks,
david.