Add sequence number to message - global variables

This is the place for developers to discuss bugs, new features and everything else about code changes.

Google Ads


Add sequence number to message - global variables

Postby semicontinuity » Tue Jun 16, 2015 9:48 am

I was looking for possibility to add some kind of sequence number for every message to be sent to elasticsearch.
The problem is, often messages are logged quite fast, and have the same timestamp, for instance

2015-06-16 08:25:51,203 DEBUG main/Logger: Writing data..
2015-06-16 08:25:51,203 DEBUG main/Logger: Writtten data
2015-06-16 08:25:52,123 DEBUG main/Logger: Writing more data..

if timestamp is the same, the messages order is lost in elastic.

I was thinking that it's possible to add another field 'sequence', so that messages will be ordered by timestamp and sequence number. It could be forever-increasing number, but it ther might be problems - where to keep the number between rsyslog restarts, etc., so ideal would be the number in the group of messages having the same timestamp, starting from 0 and increasing until new timestamp is seen, so the following would be sent to elastic:

{ts="2015-06-16 08:25:51,203" n=0 level="DEBUG" .. message="Writing data.."}
{ts="2015-06-16 08:25:51,203" n=1 level="DEBUG" .. message="Writtten data"}
{ts="2015-06-16 08:25:52,123" n=0 level="DEBUG" .. message="Writing more data.."}

It seems, that it's not possible with RainerScript - i need some sort of 'global' variable to hold the sequence number and the last seen timestamp.

So, the question is - is it possible to add support for 'global' variables?
semicontinuity
New
 
Posts: 2
Joined: Tue Jun 16, 2015 9:32 am

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: Add sequence number to message - global variables

Postby teifler » Tue Jun 16, 2015 11:59 am

Hi,

I think the modul mmsequence could be the solution for your problem. Below you can find the link to the documentation that you can have a look ;)
http://www.rsyslog.com/doc/master/confi ... uence.html


Tim
teifler
Frequent Poster
 
Posts: 56
Joined: Mon Sep 05, 2011 9:49 am

Re: Add sequence number to message - global variables

Postby dlang » Tue Jun 16, 2015 8:35 pm

There are global variables in rainerscript, $\ defines global variables that will persist from message to message

What version of rsyslog are you using?

If elasticsearch looses messages that have the same timestamp, that's a major problem with ES. Can you please point at where this is documented?
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: Add sequence number to message - global variables

Postby dlang » Tue Jun 16, 2015 8:36 pm

it's probably not reasonable to try and have the counter be unique for each timestamp (what happens when a second server sends a lot message with a time in the past for example), but you can do a continually incrementing number.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Google Ads



Return to Developer's Corner

Who is online

Users browsing this forum: No registered users and 0 guests

cron