Information: Forum is in read-only mode
For details and other support options see https://www.adiscon.com/news/support-forum-set-to-read-only-mode/

Client queues and hanging syslog reciever

This is the place for you, if you got rsyslog up and running but wonder how to make it do what you want.

Moderator: alorbach

Google Ads


Client queues and hanging syslog reciever

Postby BjarneB » Fri Feb 16, 2018 10:00 am

Hi, new guy here.

We are having some issues with syslog messages not getting all the way trough from client to server.

We have two arcsight recievers behind a loadbalancer and they sometimes run out of resources. The connection is not closed from the serversite, but just "hangs". The result is that the messages on all the clients get stuck on the clients and are not forewarded to arcsight.

To remedy that, I would like rsyslog to close the connection and reopen. The loadbalancer will then route to the working one.

##### BEGIN of queue def #### Do NOT separate from END
# Log everything to central
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName ToRemote # unique name prefix for spool files
$ActionQueueMaxDiskSpace 128m # 128m space limit (could also use "50m")
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionSendTCPRebindInterval 500 # after 500 messages, close and reopen connection
$ActionResumeRetryCount -1 # infinite retries if host is down
*.* @@syslog:1999
##### END of queue def #### Do NOT separate from BEGIN

We are using RHEL7, rsyslog 7.4

Now the questions are:

The ActionSendTCPRebindInterval, how does that work?
When will the connection to the remote server be closed and reopened?
After 500 messages has been sent to rsyslogd OR after rsyslogd has sent 500 messages?
As far as I can see, it is the latter. If that is the case, are there anyway to change that?
Right now we have to restart rsyslog on all 200 clients after one arcsight sever stops responding. I would like an automatic recovery.


About the queues
Are there any way to see how far in the queue processing rsyslog has come?
I have several severes where the are queue files several days old, it would be nice to see if these messages where actually sent. I would assume ActionQuename.qi would tell me, but I don't know how to read it.

For example, on one sever there is 111 queue files in /var/lib/rsyslog and about 40 of them are old messages.

# cat *.qi

<OPB:1:qqueue:1:
+iQueueSize:2:6:121940:
+tVars.disk.sizeOnDisk:2:8:63927061:
>End
.
<Obj:1:strm:1:
+iCurrFNum:2:2:96:
+pszFName:1:8:ToRemote:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:0:
+sType:2:1:1:
+tOperationsMode:2:1:2:
+tOpenMode:2:3:384:
+iCurrOffs:2:6:993707:
+inode:2:1:0:
>End
.
<Obj:1:strm:1:
+iCurrFNum:2:2:36:
+pszFName:1:8:ToRemote:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:1:
+sType:2:1:1:
+tOperationsMode:2:1:1:
+tOpenMode:2:3:384:
+iCurrOffs:2:5:15675:
+inode:2:7:8390392:
>End
.

The latest queue file is named ToRemote.00000146, created half an hour ago and the oldest is named ToRemote.00000036 created february 1st.
ToRemote.qi is old too.

-rw-------. 1 root root 1048933 Feb 1 19:47 ToRemote.00000036
-rw-------. 1 root root 93806 Feb 16 09:02 ToRemote.00000146

-rw-------. 1 root root 506 Feb 9 15:40 ToRemote.qi

Any input is most welcome.
BjarneB
New
 
Posts: 1
Joined: Fri Feb 16, 2018 8:57 am

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Google Ads


Return to Configuration

Who is online

Users browsing this forum: No registered users and 1 guest

cron