MonitorWare Knowledge Base

Please post your server and client config. I am not sure if you disk-enabled the queue - if not, and you ran out of memory (I think 1000 msgs is the default for action queues) messages need to be discarded because there is no more space to store them.

Rainer

• Pulling out your Hair?
• Wasting Time and Money?
• Deadline Approaching?

server config (i've cut out some irrelevant rules):

###############################################################
###############################################################
#rsyslog v3 config file

# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

#### MODULES ####

$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by rklogd)
#$ModLoad immark.so # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerAddress 0.0.0.0
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp.so
$InputTCPServerRun 514

#Provides RELP syslog reception
$ModLoad imrelp # needs to be done just once
$InputRELPServerRun 20514

#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

$template DYNmessages,"/data/temp/log/tcp/HOSTS/%HOSTNAME%/%$year%/%$month%/%$day%/messages"

#### RULES ####

###Send local logs to nfs###

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd1 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
if $inputname != 'imudp' \
and $syslogseverity-text != 'debug' \
and ( $syslogfacility-text != 'mail' and $syslogfacility-text != 'authpriv' and $syslogfacility-text != 'cron' ) \
then ?DYNmessages
# ### end forwarding rule ###

###############################################################
###############################################################


client config:
###############################################################
###############################################################

#rsyslog v3 config file

# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

#### MODULES ####

$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by rklogd)
$ModLoad omrelp # provides support for remote logging via relp protocol
#$ModLoad immark.so # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp.so
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp.so
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#### RULES ####

# remote logging

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

*.info;*.crit;authpriv.*;kern.*;mail.none;cron.none mrelp:lxserv140:20514
#*.info;*.crit;authpriv.*;kern.*;mail.none;cron.none @@lxserv140:514

# ### end forwarding rule ###

###############################################################
###############################################################

mhh... looks good. Can you provide me a debug log showing an extended period of failure? Maybe the simple patch did not provide enough glue to work in all cases (I am thinking about the retries).

ok, i've started a debug and will get back to you with the output.

i checked my logs again and since i store the logs locally as well as send them to log server, i can see that the problem has something to do with the log rotation. when i remounted nfs i got all the logs from the messages-file, but nothing from the messages.1-file which was logrotated saturday morning. in conjunction with the logrotate the rsyslog daemon was restarted. i will do some restarts of rsyslog in my new debug test to see if i get all messages.

i did a new debug (check your mail!) and even though it was not over any longer time it may be useful.
i followed these steps:
1. unmounted nfs
2. started rsyslog in debug mode
3. logged some descriptive stuff
4. stopped rsyslog and started it again in debug mode
5. logged some more stuff
6. mounted nfs, with rsyslog still running.

and i actually lost all messages prior to step 4, so the stop/start of rsyslog seems to mess with my buffering somehow.

I would really appreciate if you could give it another try. That happens when you do such things "in between". The fix was incomplete (actually screamingly incomplete if you just bother to read a few comments ). Add this:

http://git.adiscon.com/?p=rsyslog.git;a ... d0c76b29ab

I have to admit I still have not done a full analysis, but the analysis is at least much better than last time. Plus,I now ran a limited test (against a filled-up local disk, not NFS due to setup effort required).

Rainer

i have rebuilt it with the new patch but the result is the same, see debug output on mail.

ok, begins to make sense. I'll see that I can dig even a bit deeper tomorrow, I am now confident we can tackle this beast

hmmm, I was a bit too optimistic yesterday. I was misleaded by some debug statements. On closer look, it actually looks like everything was written... So.. can you provide me what actually was written? I think I'll enhance the debug output, so that we know more precisely what's going on.

Would it be OK that you temporarily use v4? That would save me a lot of time in this process...

Rainer

good news and bad news: I have found a bug inside the code, and been able to fix that (not yet committed). However, I tried with the smb redirector (don't have nfs at hand) and it acks the writes, but does not ensure data is actually put onto the remote site. So there probably is no way to make sure we really have the data. Maybe the situation is better with NFS.

below some excerpts from my twitter stream:
# i have lots of garbage inside the log when I reconnect the network... looks like cifs driver can not really handle this situation1 minute ago from twhirl

# it is interesting to see how the smb driver continues to accept data (at a very slow rate) while the network is off....9 minutes ago from twhirl

# #rsyslog: issue is more complicated than I thought - probably a bug in dynafile creation processabout 1 hour ago from twhirl

# ok, think I got a bug. FD is not set to indicated "closed" after actual close call - can lead to endless loopabout 2 hours ago from twhirl

# as soon as I enter a new message, the missing content *is* writtenabout 2 hours ago from twhirl

# after disconnect, nothing is written...about 2 hours ago from twhirl

# very interesting... I get successful returns from write() to the network file - with plugged cable, lazy write, I guess...about 2 hours ago from twhirl

# #rsyslog: OS buffering plays a big role in network-file retries - on the initial tries I do not see any error code at all! (w/o cable!!)about 2 hours ago from twhirl

The fixes are now committed to git, but it looks like there is no equally easy backport to v3. So I would really appreciate if you could try it (v4) with nfs. Note that I today looked at the dynafile code, only (your use case). Static filenames may require an extra tweak, will look at this tomorrow).

Rainer

i will try the v4 version tomorrow, given time, and get back with results.

I have done some more review, bug fixing and enhancements. You should use the most recent version from the omfile-errHandler branch. I'll probably merge this into devel pretty soon.

Rainer

im sorry, but my build fails at the 'autoreconf -fvi'. maybe im doing it wrong, or maybe im using the wrong versions (RHEL5).

can you post some details of what fails?