What Message Format do we use?

Diskussions related to the development of PhpLogCon

Google Ads


What Message Format do we use?

Postby alorbach » Thu Feb 14, 2008 2:31 pm

As we will have a line between the input modules, and the displaying code part - the question is how do we generalize the format?

Or in easier words, what will be returned from the input modules, regardless if the input is a flat file or mysql?

In my opinion we have 2 options.
A) Each syslog message in raw format as ONE string, All Syslog messages in a one big Array.
B) Each syslog message already parsed and splitted into a string array. Example:

Code: Select all
$syslogmsg[SYSLOG_ID] = Uniquie ID if available
$syslogmsg[SYSLOG_DATE] = Date of the Syslogmessage
$syslogmsg[SYSLOG_FACILITY] = Syslog Facility
$syslogmsg[SYSLOG_PRIORITY] = Syslog Priority
$syslogmsg[SYSLOG_HOST] = Syslog Source
$syslogmsg[SYSLOG_SYSLOGTAG] = Syslog Tag Value if available
$syslogmsg[SYSLOG_MESSAGE] = Syslog message
$syslogmsg[SYSLOG_MESSAGETYPE] = Type, like "SYSLOG, EVENTREPORTER etc etc"
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Postby rgerhards » Thu Feb 14, 2008 2:35 pm

I'd definitely go for option B). This implies that the object model must support

logStream --> parser --> consumer

(btw: how do we call the consumer, the App? ;)).

To get started, I'd say that we simply implement a very limited set (probably just a type value and the raw message string). It could also be considered that the parser is called via UPC. This would enable the re-use of rsyslog parsers (though at the cost of performance - but that may be worth it).

Do we agree on these thoughts? Please discuss ;)

Rainer
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Postby rgerhards » Thu Feb 14, 2008 2:36 pm

ummm.. object model view was a bit too simplistic, I think. Actually more like this

logstream --> consumer <--> parser

hard to do here in ascii art. If we take that route, we can create uml.

Rainer
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Postby alorbach » Thu Feb 14, 2008 2:43 pm

Ok ... so I have some defines made in functions_constants.php which look like this:

Code: Select all
define('SYSLOG_ID', 0);
define('SYSLOG_DATE', 1);
define('SYSLOG_FACILITY', 2);
define('SYSLOG_PRIORITY', 3);
define('SYSLOG_HOST', 4);
define('SYSLOG_SYSLOGTAG', 5);
define('SYSLOG_MESSAGE', 6);
define('SYSLOG_MESSAGETYPE', 7);


I will use this defines as array index for the fields I need to display in the frontend. I will use a fixed test array of data during meantime for testing and developing the frontend view.

I think this can be easily adapted to real data later.
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Postby rgerhards » Thu Feb 14, 2008 2:49 pm

It's clean, but I think it would be better to work on a scheme similar to rsyslog. Rsyslog already has fields defined in its property replacer:

http://www.rsyslog.com/doc-property_replacer.html

However, they are ugly ;) - but known by users.

In rsyslog, I am about to implement a new way of accessing properties. This may be a good time to clean up things (even though I am hesitant to do that, given currently existing knowledge with the current property set - which, remeber, is also based on general MonitorWare property sets).

What do you think?
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Postby alorbach » Thu Feb 14, 2008 3:01 pm

Well you can clearly see that the properties list has grown in time. Tbh. I don't see an advantage to have all this properties stored in the array which contains the data I need to display.
Things like the property "syslogseverity-text" or "PRI-text" can be done generated in the frontend anytime.

But I would append some of the properties to our data array here which might be useful like:
Code: Select all
PROTOCOL-VERSION   
STRUCTURED-DATA   
APP-NAME   
PROCID   
MSGID   
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Postby rgerhards » Thu Feb 14, 2008 3:03 pm

No, the frontend should NOT generate anything by itself. That violates encapsulation and could bring us in big (extensibility) trouble when things change (and they do over time). Both from a general MontiorWare and a rsyslog POV, I think it has more than paid to have a single place where all the properties are taken care of.

I don't see any big problem if we use the -text properties instead of generating them.
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Postby rgerhards » Thu Feb 14, 2008 3:05 pm

Some (limited) UML now at http://www.phplogcon.com/jpgs/
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Postby alorbach » Thu Feb 14, 2008 3:11 pm

Hrm I don't think there will be any new Syslog Priorities or Syslog Facilities any time soon ;) - so I don't see a problem regarding the extensibility.

However we are most likely going to need this textual representations anyway, so it doesn't matter where it's done.

But I wouldn't generate all these properties, some are simple not needed at all and this would cause unnecessary overhead ^^
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Postby rgerhards » Thu Feb 14, 2008 3:19 pm

Hehe wrong bet - new facilities are a hot topic in IETF discussion ;)

The unnecessary properties is a very good point. A clean soltution is to tell the logStream class which properties the application would like to see. This can simply be passed in when the stream is opened. Thus, we both have a clean interface AND good performance. Any objections?
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Postby rgerhards » Thu Feb 14, 2008 3:21 pm

oh... and if I may add: Even though the app may like to see certain properties, there can be no guarantee the logstream class can actually deliver them. For example, if we process plain text files from /var/log, there neither is a facility nor a severity. It's bad, but that's the way it is...
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Postby alorbach » Thu Feb 14, 2008 3:25 pm

Sounds good to me, so I tell the upstream class by using a simple string array, which properties I need. Then the logstream can fill only the properties needed.
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Postby rgerhards » Thu Feb 14, 2008 3:26 pm

exactly. I'll ask Michael to modify the class in this respect. Our first interface change - nice to see it in that phase ;)
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Postby alorbach » Thu Feb 14, 2008 3:31 pm

I am going to change my defines to this:

Code: Select all
define('SYSLOG_DATE', 'timereported');
define('SYSLOG_FACILITY', 'syslogfacility');
define('SYSLOG_FACILITY_TEXT', 'syslogfacility-text');
define('SYSLOG_PRIORITY', 'syslogseverity');
define('SYSLOG_PRIORITY_TEXT','syslogseverity');
define('SYSLOG_HOST', 'FROMHOST');
define('SYSLOG_SYSLOGTAG', 'syslogtag');
define('SYSLOG_MESSAGE', 'msg');
define('SYSLOG_MESSAGETYPE', 'IUT');


Left is what I will use in the frontend code for better readability, and right the real names of the array indexes.
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Postby rgerhards » Thu Feb 14, 2008 3:40 pm

what is the left-hand side used for?
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Google Ads


Next

Return to Developer's Corner

Who is online

Users browsing this forum: No registered users and 0 guests

cron