Information: Forum is in read-only mode
For details and other support options see https://www.adiscon.com/news/support-forum-set-to-read-only-mode/

Disk Queues Deleted on Shutdown

Diskussions related to the development of PhpLogCon

Moderator: alorbach

Google Ads


Disk Queues Deleted on Shutdown

Postby wevans1 » Tue Aug 19, 2014 3:51 pm

I am trying to set up a disk queue to save forwarded messages when the server is down. I had it working on an old release (5.8). I am trying to move to 8.4 and it does not seem to work.

My configuration says to use a "disk" queue and also saveOnShutdown is set to "on".

Code: Select all
local0.*                                        action(type="omfwd" protocol="tcp"
                                                        target="rhel2" port="5544"
                                                        streamDriver="gtls" streamDriverMode="1"
                                                        streamDriverAuthMode="anon"
                                                        queue.type="disk"
                                                        queue.fileName="fwdRule1"
                                                        queue.spoolDirectory="/opt/rsyslog"
                                                        queue.saveonshutdown="on"
                                                        queue.size="1000000"
                                                        queue.maxDiskSpace="1g")


(I originally started with a LinkedList, but changed to disk just to debug)

I see the queue file created while rsyslogd is running, but when I shutdown rsyslogd, the queue gets deleted (seems to ignore the saveonshutdown parameter). Here is the rsyslogd debug output:

8595.273009272:main thread : strm 0x7fb179c59480: file 6(fwdRule1) closing
8595.273018760:main thread : strm 0x7fb179c59480: file 6(fwdRule1) flush, buflen 0 (no need to flush)
8595.273033543:main thread : strm 0x7fb179c59700: file 8(fwdRule1) closing
8595.273049000:main thread : strm 0x7fb179c59980: file -1(fwdRule1) closing
8595.273061331:main thread : strmCloseFile: deleting '/opt/rsyslog/fwdRule1.00000001'

Do I have a bad configuration? I admit I don't know the new style configuration too well.
Any help will be appreciated.
wevans1
New
 
Posts: 4
Joined: Tue Aug 19, 2014 3:37 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: Disk Queues Deleted on Shutdown

Postby rgerhards » Tue Aug 19, 2014 3:56 pm

sorry if that sounds like a dumb question, but... did you make sure the remote server is unreachable?

Rainer
rgerhards
Site Admin
 
Posts: 3807
Joined: Thu Feb 13, 2003 11:57 am

Re: Disk Queues Deleted on Shutdown

Postby wevans1 » Tue Aug 19, 2014 4:28 pm

Yes, the remote server is definitely down. I send a few logs while the server was up, then shut down the server and tried to send a few more logs. I can see on the server only those logs that were sent while it is up. When the server was shut down, the client shows the following error:

GNUTLS ERROR: Error in the push function.

And the debug log has this when the client first detects the connection is lost:
1465.493736286:action 9 queue:Reg/w0: rhel2
1465.493745147:action 9 queue:Reg/w0: rhel2:5544/tcp
1465.493756532:action 9 queue:Reg/w0: omfwd: add 64 bytes to send buffer (curr offs 0)
1465.493767152:action 9 queue:Reg/w0: omfwd: endTransaction, offsSndBuf 64, iRet -2121
1465.493896986:action 9 queue:Reg/w0: unexpected GnuTLS error -53 in nsd_gtls.c:1571
1465.493946696:action 9 queue:Reg/w0: TCPSendBuf error -2078, destruct TCP Connection!

It then tries to re-establish connection but fails:
1465.506111138:action 9 queue:Reg/w0: TCPSendInit FAILED with -2027.

I never see a .qi file and when the client is shutdown, the original queue file is deleted as I mentioned in my original post.
wevans1
New
 
Posts: 4
Joined: Tue Aug 19, 2014 3:37 pm

Re: Disk Queues Deleted on Shutdown

Postby wevans1 » Tue Aug 19, 2014 7:22 pm

Update.

I was able to get it to work using the old style syntax in rsyslog.conf:

Code: Select all
$WorkDirectory /opt/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
local0.*                                                @@rhel2:5544


When I configure it this way, the queue file stays after shutdown and a .qi file is created.

However, I switched to the new style because I want to take advantage of encrypted queues and I don't know how to set the cry provider and key using the old style. Is this even possible?
wevans1
New
 
Posts: 4
Joined: Tue Aug 19, 2014 3:37 pm

Re: Disk Queues Deleted on Shutdown

Postby wevans1 » Wed Aug 20, 2014 7:29 pm

I was able to get the disk queue to be saved by adding values for the following parameters:
Code: Select all
action.resumeRetryCount="-1"
name="mesgForwarder"


Once I added those additional parameters to the omfwd action, the files remained on disk after shutting down ryslog.
Now I have a new problem...

I want the disk files to be encrypted. I added the following parameters to the omfwd action:
Code: Select all
queue.cry.provider="gcry"
queue.cry.key="1234567890123456"


It works great. The file is encrypted on disk. However, if I change the queue.type from "disk" to "LinkedList", then the file is *not* encrypted after shutdown, but rather, it is in clear text. From the debug file, it shows the writing of the queue at the end:
Code: Select all
6334.678150149:logstashforwarder queue:DAwpool/w0: strmPhysWrite, stream 0x7f1c93859480, len 432
6334.678290064:logstashforwarder queue:DAwpool/w0: file '/opt/rsyslog/logstashqueue.00000001' opened as #3 with mode 384
6334.678309213:logstashforwarder queue:DAwpool/w0: strm 0x7f1c93859480: opened file '/opt/rsyslog/logstashqueue.00000001' for WRITE as 3
6334.678335176:logstashforwarder queue:DAwpool/w0: strm 0x7f1c93859480: file 3 write wrote 432 bytes
6334.678347143:logstashforwarder queue:DAwpool/w0: logstashforwarder queue[DA]: write wrote 432 octets to disk, queue disk size now 432 octets, EnqOnly:1

From the code in stream.c, it looks like it should call the Encrypt function of the crypto provider, but it doesn't appear to do this (is pThis->cryprov not set for some reason?)
Code: Select all
DBGPRINTF("strmPhysWrite, stream %p, len %u\n", pThis, (unsigned)lenBuf);
if(pThis->fd == -1)
        CHKiRet(strmOpenFile(pThis));

/* here we place our crypto interface */
if(pThis->cryprov != NULL) {
        pThis->cryprov->Encrypt(pThis->cryprovFileData, pBuf, &lenBuf);
}
/* end crypto */

iWritten = lenBuf;
CHKiRet(doWriteCall(pThis, pBuf, &iWritten));


I'll see if I can debug this more from my end. Any help will be appreciated.
wevans1
New
 
Posts: 4
Joined: Tue Aug 19, 2014 3:37 pm

Google Ads



Return to Developer's Corner

Who is online

Users browsing this forum: No registered users and 1 guest

cron