rsyslog: does rsyslog support in-oder log messages delivery

General discussions here

Moderator: rgerhards

Google Ads


rsyslog: does rsyslog support in-oder log messages delivery

Postby vijaysahu.iitdelhi » Wed Dec 23, 2015 7:54 am

I want to use rsyslog to forward my logs to remote rsyslog server. If remote server is down, then it will cache the logs at client host and when remote server is back then it should first transfer the cached log and then transfer the live (real time logs).
Is it supported in rsyslog ? if yes, what is the rsyslog.conf ?
vijaysahu.iitdelhi
New
 
Posts: 2
Joined: Wed Dec 23, 2015 5:45 am

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby dlang » Wed Dec 23, 2015 8:28 pm

no, between the problem of queues, multiple threads working at once, logs being relayed through servers that may go down and then come back later delivering messages, the fact that the network does not guarantee in-order delivery of packets, and a few other cases, rsyslog does not guarantee that logs will be delivered in the exact order they were received. back in the 2.x days, rsyslog went to a lot of effort to try and deliver all the logs in-order, but then we realized that it just wasn't possible, so we relaxed the restrictions and gained a LOT of speed.

In the case of logs sent to a disk queue file, rsyslog used to deliver the old logs before new ones, but we found that this was enough slower than delivering new logs first (along with race conditions in deciding if new logs needed to be sent to the disk queue or not) that this was changed to send new logs first.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby vijaysahu.iitdelhi » Thu Dec 24, 2015 6:17 am

Hi David,

Thanks for your replay and detailed explanations with some sort of history.


Thanks,
Vijay
vijaysahu.iitdelhi
New
 
Posts: 2
Joined: Wed Dec 23, 2015 5:45 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby dlang » Thu Dec 24, 2015 7:55 am

as soon as you have any parallel processing (multiple threads, failover machines, multiple network connectons, etc) you have no real ability to keep everything exactly in order. The best you could possibly do is tag each message with a sequence number early on and sort them later (but even there, how do you order things that arrive via different inputs at the same time)

Even the simplest traditional syslog could end up with logs re-ordered due to different UDP packets taking different paths through the network.

Trying to keep things in order requires global locks, one thread waiting for another that may not finish because it's trying to deliver to a destination that's down, etc.

So we decided to focus on speed, because if you are fast enough, you can use a single thread instead of multiple threads for the same throughput, and the logs will be more in order (even though you aren't guaranteeing that they will be in order). It's under very high load or failure conditions that rsyslog will end up getting them out of order. Spilling to disk and then sending the old logs is one of the more common failure conditions, and I think it's unfortunate that it turns out to be so much more efficient to keep sending the newest logs and send the old logs as you can, but that is the case.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby IgorS » Thu Mar 16, 2017 1:11 pm

What about receiving order, does same principles apply?

For example, i have rsyslog installed on my machine with default configuration. Only one output to file is created.

If from another machine i do the following:
cat <path_to_file_1GB> | nc <rsyslog host> 514
Or, i just create a TCP socket (in python for example,one thread only), connect to (rsyslog_host, 514), read the 1GB file and send line-by-line.

I see that syslog save all messages to destination file, no message is lost but order is mixed.

If from the same machine where rsyslog is installed i execute:
cat <path_to_file_1GB> | nc <localhost host> 514

then messages are ordered.

This behavior is reproducable and i'm wondering what is the issue here. TCP should guarantee order as well, right?

I'm aware of all refactoring that was done (http://www.gerhards.net/download/LinuxK ... syslog.pdf) but what confused me was this line from this post (http://blog.gerhards.net/2013/06/rsyslo ... ction.html)

By default, both queues are set to one worker maximum. The reason is that this is sufficient for many systems and it can not lead to message reordering. If multiple workers are concurrently active, messages will obviously be reordered, as the order now, among others, depends on thread scheduling order.

Is post outdated or i'm confused, there is a good chance latter is true since i don't have a lot of experience with rsyslog :)

Is there any way to configure rsyslog to preserve message order that it receives?

Igor.
IgorS
New
 
Posts: 8
Joined: Thu Mar 16, 2017 10:32 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby dlang » Thu Mar 16, 2017 1:29 pm

Inside rsyslog, there are a couple of things that can cause messages to be processed out of order

1. if you have disk-assisted queues and the memory queue overflows and messages get written to disk, rsyslog will process new messages all in memory and the messages that got written to disk will end up out of order.

2. if you have multiple worker threads, each worker will grab up to batchsize messages and process all of those messages before grabbing another batch. This means that worker 1 may grab messages 1-100 while worker 2 grabs messages 101-200, assuming equal processing speed, that means that these two batches will end up intermingled

3. if you are using imfile, rsyslog will read a batch of messages from each file in round-robin fashion, this will keep the messages from each file in the same order, but if you are looking at timestamps and don't realize they have different source files, it will look like they are being reordered.

does this clarify things?
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby IgorS » Thu Mar 16, 2017 2:25 pm

Yes, tnx, these explanations makes sense, but i'm not sure if they are relevant for my configuration. If i understand correctly disk-queues are not enabled by default and by default there is one worker thread, correct?

My configuration looks like this(/etc/rsyslog.d/my_configuration.conf):

$template rawmsgonly,"%rawmsg%\n"

# Templates
template(name="log_for_syslog_one_on_port_514" type="string" string="/data/destination.log")

# Rule sets
ruleset(name="rule_for_port_514") {
?log_for_syslog_one_on_port_514;rawmsgonly
stop
}
# Input bindings
input(type="imtcp" port="514" ruleset="rule_for_port_514")


Maybe, there is one worker thread but additional worker threads are spawned and things can get mixed up (http://www.rsyslog.com/doc/v8-stable/co ... read-pools)

Igor.
IgorS
New
 
Posts: 8
Joined: Thu Mar 16, 2017 10:32 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby dlang » Thu Mar 16, 2017 2:37 pm

the configuration you are showing is only a subset of your full config. The messages arriving on port 514 in your config are still going into the main queue, which means that other things in the main config can affect these logs.

add a queue to this ruleset and you will have it completely isolated from the main ruleset and whatever configuration is there.

Then see if you still can get it to reorder these messages.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby IgorS » Thu Mar 16, 2017 2:52 pm

Even if i comment loading of imtcp input in /etc/rsyslog.conf like this?

#input(type="imtcp" port="514")

Main configuration is nothing unusual i guess:

#### MODULES ####
module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
module(load="imklog") # provides kernel logging support (previously done by rklogd)
module(load="imudp") # needs to be done just once

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")

#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

#### RULES ####
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg
IgorS
New
 
Posts: 8
Joined: Thu Mar 16, 2017 10:32 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby dlang » Thu Mar 16, 2017 2:58 pm

yes, you would still be using the main queue and sharing it with other logs if you don't define a queue on the ruleset that you tie to the port 514 input

Any input goes to the main queue by default, you have not changed that default for that input.

I would have to do some digging to check all the defaults for the main queue, but it's defaults are different in many ways from other queues.

In any case, you should only be seeing additional workers being spawned if you are under very high load.

I guess I've lost the exact point of the question, are you trying to guarantee that there will never be any reordering of logs? If so, just define your queue and define that it will only have a max of 1 worker. If that's not what you are asking, try restating the question.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby IgorS » Thu Mar 16, 2017 3:19 pm

Thanks for clarification.

What i was trying to understand is the following: i have 2 machines(CentOS); let's call them machine A and machine B. rsyslog is started on machine B
rsyslog is staying idle most of the time, this is development environment.

[Scenario 1] : If i execute on machine A: cat <file_of_1GB> | nc <machine B> 514 ----messages arrives, no message is lost but order is mixed
[Scenario 2] : If i execute on machine B: cat <file_of_1GB> | nc <localhost B> 514 ----messages arrives, no message is lost, order is OK

<file_of_1GB> has ~2 million lines.

This behavior is reproducible. I got same results when i ran this test several times.

Upon your clarification, i guess all messages pass through main queue, additional workers will be spawned and order can get mixed up. But why order is OK (at least in several tests i did) in [Scenario 2] ? I think rate of enqueue-ing messages in this case is faster comparing to [Scenario 1] so i would expect more workers in this case....

If you think its worth it i can try to run it in debug mode and check any interesting messages in the log.


Thanks, Igor.
IgorS
New
 
Posts: 8
Joined: Thu Mar 16, 2017 10:32 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby dlang » Thu Mar 16, 2017 3:33 pm

there are enough timing differences between the two that it's hard to evaluate. We'd need debug logs (witch themselves alter timeing) to try and see what's different.

Please explicitly configure the queue for the ruleset and see if you still have the same behavior.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby IgorS » Thu Mar 16, 2017 5:05 pm

I configure explicit queue (in-memory disk-assisted) with queue.workerthreads="1" and it works, order is preserved. Thanks.

Just for my understanding, this means that now i'm by-passing main queue, right? Can you point me to some article/documentation where i can see the defaults for main queue?
I'm trying to understand what will be the differences if i just decide to switch to explicitly configured queue.

Thanks a lot. Igor.
IgorS
New
 
Posts: 8
Joined: Thu Mar 16, 2017 10:32 am

Re: rsyslog: does rsyslog support in-oder log messages deliv

Postby IgorS » Fri Mar 17, 2017 8:41 am

I think i found in the debug log:

5946.660883503:main thread : main Q: starting queue
5946.660889422:main thread : main Q: is NOT disk-assisted
5946.660891746:main thread : main Q: params: type 0, enq-only 0, disk assisted 0, spoolDir '', maxFileSz 1048576, maxQSize 100000, lqsize 0, pqsize 0, child 0, full delay 97000, light delay 70000, deq batch size 256, high wtrmrk 80000, low wtrmrk 20000, discardmrk 98000, max wrkr 2, min msgs f. wrkr 40000
5946.660896415:main thread : main Q:Reg: finalizing construction of worker thread pool (numworkerThreads 2)


numworkerThreads is 2, that explain why sometimes messages are re-ordered.
I also verified in the log that under heavy burst, additional worker is started making total number of current working threads = 2.

Igor.
IgorS
New
 
Posts: 8
Joined: Thu Mar 16, 2017 10:32 am

Google Ads



Return to General

Who is online

Users browsing this forum: No registered users and 1 guest

cron