MonitorWare Knowledge Base

Here's tjhe config. It's not much to look at, but I thought it should work.

Code: Select all

# Global directives
$ModLoad imuxsock.so
$WorkDirectory /var/spool/rsyslog
$ActionQueueType LinkedList
$ActionQueueFileName MainMsg
$ActionQueueMaxFileSize 100m
$ActionQueueDiscardSeverity 6
$ActionQueueSaveOnShutdown on
$ActionResumeInterval 1
$ActionResumeRetryCount -1

*.info;mail.none;authpriv.none;cron.none                @@sdlogs01.vmix.com


• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

Hi,

the config looks fine. I've gone back to the posted debug output. Unfortunately, it is a much too short log file. Can you please re-run the test and post the complete debug output. I need the first messages (when the connection fails) and then the messages when the remote end comes up again. I think this will be at least around 1,000 lines of debug output. You can also mail it to rgerhards@gmail.com if you prefer.

Rainer

Thanks. I've emailed you the information.

Dan.

Hi Dan,

I don't know what's wrong, but I did not receive any email. Please re-send (as a side-note, I noticed a lot of failing services yesterday, maybe there is something going on in Internet routing. Is a major link down?).

Rainer

Sorry Rainer. I've been away from the forums for quite a while. I just resent the email. Thanks.

Dan.

Hi Dan,

I got and reviewed the log. I also have to admit that Iprobably misundertood your question a bit (but my memory is not really clear on that). The bottom line is that there is some message loss, no way around. Interestingly, I am currently discussing this with another implementer. You may want to have a look here:

http://blog.gerhards.net/2008/05/why-yo ... e-tcp.html

That said, the best approach would be to RELP, but you can't because v2 does not support it. Martin, the other implementor (on NetBSD) suggested a work-around for this very issue. It does not work 100% reliably, but I think it is a better work-around that what I currently use. I will see if I can find time to implement it (depends on a couple of other things, next weeks is *very* busy).

That work-around will, in many cases, guard against the loss of a single message if the the remote server was terminated and restarted. It very much depends on timing, and so it is not a total cure. It also doesn't guard against network failures.

I suggest to keep watching the mailing list for updates. If I can, I will also update this thread and it would be great if you could try out a new version.

Thanks,
Rainer

rgerhards wrote:Hi Dan,

I got and reviewed the log. I also have to admit that Iprobably misundertood your question a bit (but my memory is not really clear on that). The bottom line is that there is some message loss, no way around. Interestingly, I am currently discussing this with another implementer. You may want to have a look here:

http://blog.gerhards.net/2008/05/why-yo ... e-tcp.html

That said, the best approach would be to RELP...

Hi, I'm having the same problem - losing the first message post rsyslog restart over a TCP connection to a remote server, running v3.18.4 on both sides.
I've tried using DA queues on both the sender and receiver in all possible configurations, and as you say - it's still not reliable.
I also tried RELP and found that it actually lost more messages during an rsyslog restart.

At the time I tried RELP I was unaware of the different queueing models and so didn't try them in conjunction with RELP. Will using a DA queue on both sender and receiver help with the RELP reliability? Are DA queues actually a required configuration to get the reliable transmission?

Thanks in advance,
Jim

Hi Jim,

yes, you need to use a DA queue, because otherwise a rsyslogd restart can not persist the messages. Without that, you obviously loose everything that could not be transmitted. You also need to configure the queue to persist messages when rsyslogd shuts down. This is not done by default.

For the plain TCP problem, there is no cure, but I think the latest beta build has an improved algorithm which further limits the message loss potential.

HTH
Rainer

rgerhards wrote:Hi Jim,

yes, you need to use a DA queue, because otherwise a rsyslogd restart can not persist the messages. Without that, you obviously loose everything that could not be transmitted. You also need to configure the queue to persist messages when rsyslogd shuts down. This is not done by default.

HTH
Rainer

Thanks Rainer,

I've tried the following configurations with RELP - with a utility constantly logging on the remote machine, one message per second, three messages are lost every time I kill -HUP the receiving rsyslog daemon. Have I got something wrong / missed something in the configurations?

I also tried queue type Disk instead of LinkedList, but that seemed to randomly fail to restart logging remote messages on the server following a kill -HUP

Cheers,
Jim

Server Configuration

Code: Select all

# rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.

$ModLoad imrelp # RELP
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # kernel logging (formerly provided by rklogd)

$InputRELPServerRun 5014

# Allow TABs through
$EscapeControlCharactersOnReceive off

$WorkDirectory /var/log/rsyslogspool
$MainMsgQueueDiscardSeverity 8
$MainMsgQueueFileName msgbuf
$MainMsgQueueCheckpointInterval 1
$MainMsgQueueSaveOnShutdown on
$MainMsgQueueType LinkedList

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
kern.*                                                 /dev/console

# only this host
:hostname,isequal,"beaver" /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                 -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*  /var/log/boot.log

$ActionQueueFileName msgRcv
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueCheckpointInterval 1
$ActionQueueType LinkedList
$ActionResumeRetryCount -1
$ActionResumeInterval 1

$template MessagesFile,"/var/log/remote/%HOSTNAME%/messages"
*.info;mail.none;authpriv.none;cron.none;local4.none;local5.none   ?MessagesFile

$template HeartbeatLogFile,"/var/log/remote/%HOSTNAME%/ha-log"
$template HeartbeatDebugFile,"/var/log/remote/%HOSTNAME%/ha-debug"
local5.info      ?HeartbeatLogFile
local5.debug   ?HeartbeatDebugFile


Remote Configuration

Code: Select all

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                     /dev/console
$ModLoad imuxsock
$ModLoad omrelp

# Allow TABs to be transmitted
$EscapeControlCharactersOnReceive off

$WorkDirectory /var/log/rsyslogspool
$MainMsgQueueDiscardSeverity 8
$MainMsgQueueFileName msgbuf
$MainMsgQueueCheckpointInterval 1
$MainMsgQueueSaveOnShutdown on
$MainMsgQueueType LinkedList

$ActionQueueFileName msgSend
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueCheckpointInterval 1
$ActionQueueType LinkedList
$ActionResumeRetryCount -1
$ActionResumeInterval 1


# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;local3.none;local4.none;local5.none;mail.none;authpriv.none;cron.none /var/log/messages
& :omrelp:10.11.66.253:5014;RSYSLOG_ForwardFormat

# The authpriv file has restricted access.
authpriv.*                  /var/log/secure

# Log all the mail messages in one place.
mail.*                     -/var/log/maillog


# Log cron stuff
cron.*                     /var/log/cron

# Everybody gets emergency messages
*.emerg                     *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                  /var/log/spooler

# Save boot messages also to boot.log
local7.*                     /var/log/boot.log
local5.info                     /var/log/ha-log
local5.debug                  /var/log/ha-debug

local5.debug;local5.info         :omrelp:10.11.66.253:5014;RSYSLOG_ForwardFormat