File Input Module - already processed lines get logged again

This is the place for you, if you got rsyslog up and running but wonder how to make it do what you want.

Moderator: rgerhards

Re: File Input Module - already processed lines get logged again

Postby lperr on Thu Aug 14, 2008 6:32 pm

This compiled fine.
I only get this on the console, the previous errors are gone:
Code: Select all
tcps_sess.c: In function 'tcps_sessDebugPrint':
tcps_sess.c:107: warning: unused parameter 'pThis'
tcpsrv.c: In function 'tcpsrvDebugPrint':
tcpsrv.c:654: warning: unused parameter 'pThis'
ksym_mod.c: In function 'LookupModuleSymbol':
ksym_mod.c:456: warning: comparison between signed and unsigned
ksym_mod.c:457: warning: comparison between signed and unsigned
ksym_mod.c:458: warning: comparison between signed and unsigned


make output seems fine since it ends with:
Code: Select all
make[2]: Leaving directory `/usr/src/rsyslog-3.18.3-Test6/plugins/imfile'
make[1]: Leaving directory `/usr/src/rsyslog-3.18.3-Test6'


And imfile.so is where it should be.

I'll test it later and update this thread.
lperr
Advanced
 
Posts: 26
Joined: Wed Aug 06, 2008 12:00 pm

Professional Services Information

  • Custom written rsyslog.conf?
  • Maintenance Contract?
  • Installation support?

Re: File Input Module - already processed lines get logged again

Postby lperr on Thu Aug 14, 2008 9:40 pm

I'm pleased to inform you that now everything seems to work fine.

State files are written when stopping RSyslog and they are cleared when it restarts.
Permissions are correctly set.

This applies to the MainMsg queue which I configured as a disk-assisted one and to the 3 files I'm monitoring with imfile.

Now imfile correctly logs lines only once, no duplicates whatsoever.

Moreover I also tested the functionality that allows not to lose data when files are rotated.
This works fine when log.file gets rotated into log.file.1 and *also* when it gets rotated to something like log.file-20080814.
I didn't expect this to happen even with a date suffix and I'm glad that the feature "just works".

There are two things that came to my mind but I didn't check what happens in these cases since they aren't relevant for my environment.
- What happens when log.file gets rotated more than one time between RSyslog checks? For example at the first poll there is only log.file and at the second there are log.file, log.file.1 and log.file.2
- What happens if log.file gets rotated and compressed into log.file.1.gz?

Another thing that I noticed is this:
- RSyslog is monitoring a certain file, say log.txt
- Remove log.txt directives from rsyslog.conf
- Stop and restart the daemon
- The state file for log.txt isn't cleared from the working directory

I suppose this is no bug and that is simply how RSyslog handles such an eventuality.

I'll make sure to closely follow RSyslog development waiting for a release with these latest fixes and the "date-subseconds" property option, a very useful feature indeed.

Thanks again for the quick fixes!
lperr
Advanced
 
Posts: 26
Joined: Wed Aug 06, 2008 12:00 pm

Re: File Input Module - already processed lines get logged again

Postby rgerhards on Fri Aug 15, 2008 4:42 pm

lperr wrote:I'm pleased to inform you that now everything seems to work fine.

State files are written when stopping RSyslog and they are cleared when it restarts.
Permissions are correctly set.

This applies to the MainMsg queue which I configured as a disk-assisted one and to the 3 files I'm monitoring with imfile.

Now imfile correctly logs lines only once, no duplicates whatsoever.

Moreover I also tested the functionality that allows not to lose data when files are rotated.
This works fine when log.file gets rotated into log.file.1 and *also* when it gets rotated to something like log.file-20080814.
I didn't expect this to happen even with a date suffix and I'm glad that the feature "just works".


:)

lperr wrote:There are two things that came to my mind but I didn't check what happens in these cases since they aren't relevant for my environment.
- What happens when log.file gets rotated more than one time between RSyslog checks? For example at the first poll there is only log.file and at the second there are log.file, log.file.1 and log.file.2
- What happens if log.file gets rotated and compressed into log.file.1.gz?


There is no logic to handle multiple rollovers. At a reasonable polling interval and rotation period, that should be no problem. .gz files are not (yet ;)) processed. That's because we never process "old" files - we just keep the original file descriptor open, so that the rotator can not delete if before we had a chance to pull the data (there is no other way to make -reliably- sure you get all data).

lperr wrote:Another thing that I noticed is this:
- RSyslog is monitoring a certain file, say log.txt
- Remove log.txt directives from rsyslog.conf
- Stop and restart the daemon
- The state file for log.txt isn't cleared from the working directory

I suppose this is no bug and that is simply how RSyslog handles such an eventuality.


simply: yes ;)

lperr wrote:I'll make sure to closely follow RSyslog development waiting for a release with these latest fixes and the "date-subseconds" property option, a very useful feature indeed.

Thanks again for the quick fixes!


Expect a release on Monday. I am waiting to see if the testbed brings up any other issues :) However, the -Test5 is the official one without some cosmetic things.

Rainer
User avatar
rgerhards
Site Admin
 
Posts: 1780
Joined: Thu Feb 13, 2003 11:57 am

Google Ads


Previous

Return to Configuration

Who is online

Users browsing this forum: No registered users and 0 guests

cron