MonitorWare Knowledge Base

Ok, this is from my database template I am using, so here is the template:

$template db,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag, delay, dsn) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%', '%msg:R,ERE,1,FIELD:delay=([0-9]+\.[0-9])--end%', '%msg:R,ERE,1,FIELD:dsn=([0-9]+\.[0-9]+\.[0-9])--end%')",sql

As you can see, I am using reg-exp to pull digits for two fields in my DB, here is what it looks like in the db.

*************************** 1018. row ***************************
ID: 1018
CustomerID: NULL
ReceivedAt: 2008-11-10 15:40:02
DeviceReportedTime: 2008-11-10 15:40:02
Facility: 2
Priority: 6
FromHost: staging
Message: 53F631500C5: to=<doug@bronto.com>, relay=aspmx.l.google.com[209.85.163.27]:25, delay=1.5, delays=0.16/0.06/0.43/0.82, dsn=2.0.0, status=sent (250 2.0.0 OK 1226349602 z26si15368276ele.7)
NTSeverity: NULL
Importance: NULL
EventSource: NULL
EventUser: NULL
EventCategory: NULL
EventID: NULL
EventBinaryData: NULL
MaxAvailable: NULL
CurrUsage: NULL
MinUsage: NULL
MaxUsage: NULL
InfoUnitID: 1
SysLogTag: brontostaging/smtp[25260]:
EventLogType: NULL
GenericFileName: NULL
SystemID: NULL
dsn: 2.0.0
delay: 1.5
*************************** 1019. row ***************************
ID: 1019
CustomerID: NULL
ReceivedAt: 2008-11-10 15:40:02
DeviceReportedTime: 2008-11-10 15:40:02
Facility: 2
Priority: 6
FromHost: staging
Message: 53F631500C5: removed
NTSeverity: NULL
Importance: NULL
EventSource: NULL
EventUser: NULL
EventCategory: NULL
EventID: NULL
EventBinaryData: NULL
MaxAvailable: NULL
CurrUsage: NULL
MinUsage: NULL
MaxUsage: NULL
InfoUnitID: 1
SysLogTag: brontostaging/qmgr[24531]:
EventLogType: NULL
GenericFileName: NULL
SystemID: NULL
dsn: 53F631500
delay: 53F631500
1019 rows in set (0.01 sec)

mysql>

As you can see in the first part of the query, row 1018, the dsn and delay fields are properly populated, but in the second part of the query, it is putting part of the message id in dsn and delay, which I don't think rsyslog should be doing, but I very well could be wrong on that. Again, here is a complete postfix maillog I am using to test against.

Nov 10 15:40:05 staging brontostaging/pickup[22709]: 53F631500C5: uid=602 from=<doug>
Nov 10 15:40:05 staging brontostaging/cleanup[25256]: 53F631500C5: message-id=<20081110204005.53F631500C5@email.brown-house.net>
Nov 10 15:40:05 staging brontostaging/qmgr[24531]: 53F631500C5: from=<doug@email.brown-house.net>, size=336, nrcpt=1 (queue active)
Nov 10 15:40:06 staging brontostaging/smtp[25260]: 53F631500C5: to=<doug@bronto.com>, relay=aspmx.l.google.com[209.85.163.27]:25, delay=1.5, delays=0.16/0.06/0.43/0.82, dsn=2.0.0, status=sent (250 2.0.0 OK 1226349602 z26si15368276ele.7)
Nov 10 15:40:06 staging brontostaging/qmgr[24531]: 53F631500C5: removed

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

Doesn't seem to be regex or config related (as far as I can tell), so I'm going to have to step out of this one. The answer's in the code, which I can't read.

-HKS

Ok, thank you very much for all your help.

hkspvt wrote:Rainer, can you clarify the difference between "submatch" and "match-number" in the property replacer documentation? That has me a bit confused.

This refers to what the regexp api returns. A match is a location where the regex successfully returns. Take a very simple sample. Sequence is "aaa-bbb-ccc", you search for "-". There are two matches. The regexp api returns initially with position 4 and if rsyslog is configured to look for match 2, I simply re-issue the request starting at position 4, so the string then searched is "bbb-ccc". The regexp api again returns at position 2 and as this was the second time the api was called, this is match 2. The submatch is what the regexp api finds within the returned match, that is within the parenthesis.

Does this clarify?

I now checked the actual code. The string "53F631500C5: removed" does not return a match. The property replacer is configured to use the field value (FIELD) as return value if there is no match. So, in theory, "53F631500C5: removed" should be returned. In my environment, this is also what is written to the file. May it be that co-incidentely your database column is too small so that the ": removed" part gets truncated?

Here is what I did, you may want to try to reproduce it, to see if you get the same results:

rsyslog.conf contains:

Code: Select all

$template db,"re: '%msg:R,ERE,1,FIELD:dsn=([0-9]+\.[0-9]+\.[0-9])--end%', msg: '%msg%'\n"
*.* -/path/to/some/file;db


I then used logger to generate the message:

Code: Select all

logger 53F631500C5: removed


My output file contains this:

Code: Select all

re: ' 53F631500C5: removed', msg: ' 53F631500C5: removed'


Given the configuration, I think this is the correct result. May it be that you wanted to field to be empty if there is no match? This could be done via

Code: Select all

$template db,"re: '%msg:R,ERE,1,BLANK:dsn=([0-9]+\.[0-9]+\.[0-9])--end%', msg: '%msg%'\n"
*.* -/path/to/some/file;db


Please provide feedback, so that I can continue to troubleshoot the problem

Thanks,
Rainer

I have now also updated the online regex checker/generator. It now handles the various "nomatch-Modes", which were probably the source of confusion here. While I worked on it, I also saw one bug that lead to invalid submatch specification. Finally, I have improved the documentation on what the nomatch mode means (link from the regex cheker).

I have also enhanced the code for a new ZERO mode, in which 0 will be returned if there is no match. This can potentially be useful for numerical values. This will be available as part of the next beta (soon). I've done this directly to the v3-stable release, as it was a very, very minimal change. So the v3-stable will see it with the next iteration (in roughly 2..4 weeks time).

Feedback is appreciated.

Rainer

rgerhards wrote:This refers to what the regexp api returns. A match is a location where the regex successfully returns. Take a very simple sample. Sequence is "aaa-bbb-ccc", you search for "-". There are two matches. The regexp api returns initially with position 4 and if rsyslog is configured to look for match 2, I simply re-issue the request starting at position 4, so the string then searched is "bbb-ccc". The regexp api again returns at position 2 and as this was the second time the api was called, this is match 2. The submatch is what the regexp api finds within the returned match, that is within the parenthesis.

Does this clarify?

Yep, that's perfect. Thanks.

-HKS