File Input Module - already processed lines get logged again

This is the place for you, if you got rsyslog up and running but wonder how to make it do what you want.

Moderator: rgerhards

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Wed Aug 13, 2008 4:24 pm

excellent - the traces seem still to be in transit, but I just got a segfault! :)
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Professional Services Information

  • Custom written rsyslog.conf?
  • Maintenance Contract?
  • Installation support?

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Wed Aug 13, 2008 5:04 pm

Just FYI: this looks like a bad bug. It segfaults somewhere deep inside phtreads, so I assume that I have overwritten some memory - but valgrind (memory debugger) does not show anything... I am trying to get hold of the cause, will post more when I have it (most probably tomorrow).

Rainer
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Re: File Input Module - already processed lines get logged again

Postby lperr on Wed Aug 13, 2008 5:20 pm

I'll keep checking this thread for sure!

In case you haven't received the traces yet I've posted them at http://filebin.ca/cmsvxp/latest-traces.zip.

Let me know if I can assist any further, I'll be glad to test the new code when it's ready.
lperr
Advanced
 
Posts: 26
Joined: Wed Aug 06, 2008 12:00 pm

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Thu Aug 14, 2008 9:58 am

OK, I finally found the cause - under some circumstances I am not cleaning up a cancel handler. Now I need to find a proper cure ;)

Rainer
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Thu Aug 14, 2008 10:17 am

I still wonder why it previously worked ;) I come to the conclusion that it was either consistent luck or something has been changed in recent pthreads lib which helps expose the bug :)

Anyhow... The fix wasn't that much of a problem after knowing what went wrong. Could you please try out this version:

http://download.rsyslog.com/rsyslog/rsy ... st4.tar.gz

In my lab, I can not repro the problem any more. Also, the state file is written again ;) The bug I found also clearly describes the segfault, so I think it is a good fix. You can see here what was changed:

http://git.adiscon.com/?p=rsyslog.git;a ... 25794ade32

Please let me know if the new version works for you.

Thanks,
Rainer
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Re: File Input Module - already processed lines get logged again

Postby lperr on Thu Aug 14, 2008 11:54 am

I'll be sure to test it thoroughly and will let you know later.
lperr
Advanced
 
Posts: 26
Joined: Wed Aug 06, 2008 12:00 pm

Re: File Input Module - already processed lines get logged again

Postby lperr on Thu Aug 14, 2008 3:11 pm

I suppose that while doing your magic you left something out :)

When RSyslog starts I get a few errors that should be caused by this:
Code: Select all
could not load module '/opt/apps/rsyslog-3.18.3-Test4/lib/rsyslog/imfile.so', dlopen: /opt/apps/rsyslog-3.18.3-Test4/lib/rsyslog/imfile.so: cannot open shared object file: No such file or directory


I checked and sure enough imfile.so isn't created when compiling.
I configured with
Code: Select all
--enable-imfile
and indeed I get
Code: Select all
file input module enabled:                yes


Still the library can't be found. Just in case I did
Code: Select all
find / -name imfile.so
but no luck.

I hope this is easy to sort out :)
lperr
Advanced
 
Posts: 26
Joined: Wed Aug 06, 2008 12:00 pm

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Thu Aug 14, 2008 3:28 pm

umm... it built on my system and I didn't change the build system :? Can you please post the output of make (after a make clean).

Thanks,
Rainer
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Re: File Input Module - already processed lines get logged again

Postby lperr on Thu Aug 14, 2008 3:47 pm

Doing make after a make clean gives this output http://filebin.ca/zmdcgm/output.zip.

However you might be interested in these lines that get printed after doing make > /opt/output.txt:
Code: Select all
tcps_sess.c: In function 'tcps_sessDebugPrint':
tcps_sess.c:107: warning: unused parameter 'pThis'
tcpsrv.c: In function 'tcpsrvDebugPrint':
tcpsrv.c:654: warning: unused parameter 'pThis'
ksym_mod.c: In function 'LookupModuleSymbol':
ksym_mod.c:456: warning: comparison between signed and unsigned
ksym_mod.c:457: warning: comparison between signed and unsigned
ksym_mod.c:458: warning: comparison between signed and unsigned
imfile.c: In function 'pollFile':
imfile.c:208: error: expected expression before '}' token
make[2]: *** [imfile_la-imfile.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2


I suppose that the last few lines might be revealing. Let me know if there's more I can do.
lperr
Advanced
 
Posts: 26
Joined: Wed Aug 06, 2008 12:00 pm

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Thu Aug 14, 2008 3:49 pm

let me check, looks like a last-minute error...
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Thu Aug 14, 2008 3:57 pm

Mmm... can you please post imfile.c. I think this is an issue with the (dumb) pthread_cleanup_pop implementation, which POSIX defines as a macro. The syntax works for me, but different versions of compiler/library may interpret it differently. I feared such an issue this morning and was glad it wasn't... But now lets hope something else is wrong.

Rainer
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Thu Aug 14, 2008 4:07 pm

Well... I changed the code a bit to care for the macro side-effects. Please give this version a try:

http://download.rsyslog.com/rsyslog/rsy ... st5.tar.gz
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Re: File Input Module - already processed lines get logged again

Postby lperr on Thu Aug 14, 2008 4:16 pm

imfile.c : http://filebin.ca/qfoftt/imfile.c

I'll try the new code right now.
lperr
Advanced
 
Posts: 26
Joined: Wed Aug 06, 2008 12:00 pm

Re: File Input Module - already processed lines get logged again

Postby lperr on Thu Aug 14, 2008 4:27 pm

No luck yet, hope these help.

make output: http://filebin.ca/raqcry/output.zip

console output:
Code: Select all
tcps_sess.c: In function 'tcps_sessDebugPrint':
tcps_sess.c:107: warning: unused parameter 'pThis'
tcpsrv.c: In function 'tcpsrvDebugPrint':
tcpsrv.c:654: warning: unused parameter 'pThis'
ksym_mod.c: In function 'LookupModuleSymbol':
ksym_mod.c:456: warning: comparison between signed and unsigned
ksym_mod.c:457: warning: comparison between signed and unsigned
ksym_mod.c:458: warning: comparison between signed and unsigned
imfile.c: In function 'pollFile':
imfile.c:210: error: label at end of compound statement
make[2]: *** [imfile_la-imfile.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
lperr
Advanced
 
Posts: 26
Joined: Wed Aug 06, 2008 12:00 pm

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Thu Aug 14, 2008 6:11 pm

Oh, I hate these macros...

Please try http://download.rsyslog.com/rsyslog/rsy ... st6.tar.gz
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Google Ads


PreviousNext

Return to Configuration

Who is online

Users browsing this forum: No registered users and 0 guests

cron