Direct vs LinkedList|performance issue( ompipe and omfile)

This is the place for you, if you got rsyslog up and running but wonder how to make it do what you want.

Moderator: rgerhards

Google Ads


Direct vs LinkedList|performance issue( ompipe and omfile)

Postby printul77700 » Thu Nov 02, 2017 12:02 pm

Hello, it might seem that it is discussed the issue everywhere but it might the issue to be in details so I dare to post this.
If it harms I ll také the blame, thanks in advance

Rsyslog stalled/frozen on sender and/or source which produce a total server stalling/slowness this starts from an issue of course and continues/relates more or less to :http://kb.monitorware.com/post24885.html#p24885] ,which I think is a good post,with good answers that touches some issues I have these days as well, but I still can not see the solution.

I have many doubts about rsyslog itself -even after reading through the documentation tens of times and having some infrastructure for it around, but I will focus on those related to this post/current issue I saw/have.

Maybe some piece of config can help before the question;let's say now I have it something like this, but I had it even with direct on both rulesets and also direct on both of them; I am just putting it here for the sake of having some ideas , don't take it as absolute complete configuration, as I am not sure which way is actually best for less stress on server and client while not loosing events, but in the end loosing some events prooves to be more desirable than stalling everything.


Code: Select all
# configure receivers
input(type="imptcp" port="514" Ruleset="syslog_tcp514")
input(type="imudp" port="514" Ruleset="syslog")

ruleset(name="syslog"
  queue.type="LinkedList"
  queue.FileName="receiver.queue"
  queue.MaxDiskSpace="10G"
  queue.HighWatermark="90000"
  queue.SaveOnShutdown="on"
  queue.Size="150000"
  queue.maxfilesize="100M"
  queue.workerThreads="4"
) {

call x
call y
stop
}

ruleset(name="syslog_tcp514"
  queue.type="LinkedList"
  queue.workerThreads="4"
  queue.FileName="receiver.tcp514.queue"
  queue.MaxDiskSpace="10G"
  queue.HighWatermark="90000"
  queue.SaveOnShutdown="on"
  queue.Size="150000"
  queue.maxfilesize="100M"
) {

call z
call w
stop
}



#config of called ruleset

ruleset(name="x"
  queue.type="Direct"
) {
        if $fromhost-ip == [
          ip1,
          ip2
        ] then {




                   action(type="ompipe"
                pipe="/name_of_pipe"
                name="action_syslog_pipe_bluecoat_proxy"
                queue.type="Direct"
                template="RSYSLOG_TraditionalFileFormat"
                )



stop

        }
}
~

#forwarding to remote server again:

ruleset(name="forwarder"
  queue.type="LinkedList"
  queue.workerThreads="4"
  queue.FileName="queue"
  queue.MaxDiskSpace="30G"
  queue.HighWatermark="100000"
  queue.SaveOnShutdown="on"
  queue.Size="360000"
  queue.maxfilesize="100M"

) {


  if ($rawmsg contains "string") then {

    action(type="omfwd"
     
      Target="IP"
      Port="PORT"
      Protocol="tcp"
#      ZipLevel="9"
#      compression.mode="stream:always"
      Template="Raw"
      queue.workerThreads="2"
      queue.type="LinkedList"
      queue.FileName="queue"
      queue.MaxDiskSpace="30G"
      queue.HighWatermark="100000"
      queue.SaveOnShutdown="on"
      queue.Size="120000"
      queue.maxfilesize="100M"
    action.resumeRetryCount="-1"
     )


    action(type="omfwd"
      Target="IP"
      Port="PORT"
      Protocol="tcp"
      Template="Raw"
      queue.workerThreads="2"
      queue.type="LinkedList"
      queue.FileName="queue"
      queue.MaxDiskSpace="30G"
      queue.HighWatermark="100000"
      queue.SaveOnShutdown="on"
      queue.Size="120000"
      queue.maxfilesize="100M"

      action.resumeRetryCount="-1"
    )

  }
  stop
}

# Configure the local listener
input(type="imptcp" Port="10514" Ruleset="forwarder" )
input(type="imudp" Port="10514" Ruleset="forwarder" )



So the word direct:
a)direct action queues and
b)direct ruleset queues. What is suppose to happen when messages are supposed to go towards unreachable destinations that are behind/after( chronologically speaking ) these direct queues ? As explained, direct means no queue itself but it says something about "return code (success/failure)" ;does this mean that the sender or previous thread will be notified if the events fails to be sent ?

For example: we have direct queue in the ruleset ( the one where the listener sends the messages- not sure if the messages go before this ruleset through another "main" queue or these rulesets after listener are the first ones in the whole ecosystem") + we have direct queue in front of action.But we can not fulfill the action because pipe is full/destination (tcp) not reachable etc. -what is next ?

*

the thread/worker of action's direct queue will inform the thread/worker of ruleset's direct queue ( both on same server system) which in turn will inform the local OS/ or maybe the client/sender rsyslog/client/sender OS that events can not be written and everything will end up with local and remote servers being stalled/frozen ? If that is true(above) then configuring either both action and ruleset queues as LinkedList or only the ruleset queue as LinkedList will solve the issue? ( maybe by forcing even timeoutenqueue to 0 )

**

Or to put it in a different format - how does direct and LinkedList differ in terms of communication with the client/sender/producer and which one will put less stress on it/ which one is more configurable to give the sender ( even if it is a remote one or a previous worker on rsyslog on same system) easy times?

Always I have the impression I understood or I am close to understand the queues in rsyslog but always reality slaps me. For example I first thought that setting everything to direct will put no stress on system, message is coming and it is sent and never checked, if it makes it makes, but if does not make it as there is no queue it is lost.I am not sure anymore this is the case. In turn I thought LinkedList or other type of queues will indeed keep the message in the queues and depending on some configuration detail ( timeoutenqueue) will keep the message in the previous queue and so put some stress on the system.

***

We can of course extend the question in relation to pipes/files and ompipe/omfile behavior, after all here can be the last bump and if here there is an issue it can just trigger a domino back to the remote sender going all the way back through all the queues that have been configured on the local system. I think that while we have pipes as destination and ompipe configured in front of them there is no feedback from OS towards rsyslog if the action succeeded ( saying from empiric observations );while having pipes as destination and using omfile seems to have a lot of feedback , queues starts to fill really fast as the pipe seems to feel really quick( let's suppose the pipe is not read at all, just for simplicity)- seems not to be the case of pipes+ompipe where pipes accepts far more messages.

I am a little bit worried for putting here two topics ( queues + ompipe/omfile ) but maybe seeing the whole context can help people with experience to recommend or explain a little bit more the behavior of rsyslog

Thanks a lot , help here is really needed tese days! otherwise really good info around, really good , but I think I still miss these details on server side, while some options on the client have been explained on kb

PS: I might post this there -http://kb.monitorware.com - as well as I am not sure where an answer has the bigger chances to pop up :)
Attachments
rsyslog_config_example.PNG
rsyslog_config_example.PNG (77.72 KiB) Viewed 424 times
printul77700
New
 
Posts: 3
Joined: Fri Jan 08, 2016 12:57 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Google Ads


Return to Configuration

Who is online

Users browsing this forum: No registered users and 1 guest

cron