Message parsers in Perl or Python?

This is the place for developers to discuss bugs, new features and everything else about code changes.

Google Ads


Message parsers in Perl or Python?

Postby orekdm » Sat Nov 02, 2013 12:13 am

Greetings,

I've done some searching and read through the Message Parsers page, but haven't found anything that suggests it's possible to utilize scripting languages for parsers. Is there a little known method to accommodate this, or is it possibly a feature on the roadmap?

I have many different custom log formats to normalize (much beyond the headers) and I would prefer to manage that all within the rsyslog context.

Thanks,

Keith
orekdm
New
 
Posts: 5
Joined: Fri Oct 04, 2013 7:47 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: Message parsers in Perl or Python?

Postby rgerhards » Mon Nov 04, 2013 10:03 am

sorry, this is not possible. A prime reason is that it is too slow.
rgerhards
Site Admin
 
Posts: 3807
Joined: Thu Feb 13, 2003 11:57 am

Re: Message parsers in Perl or Python?

Postby orekdm » Thu Nov 21, 2013 10:53 pm

Thank you for responding, I had just now noticed that you did so. While I understand the stance that the core message parsers need to be designed for speed, there are too many unique syslog message styles to not have dedicated parsers for the payload. What is the recommended best practice for forwarding specific message types to other parsers?

For instance, should I use a regex to match a specific pattern and have a template route that to a specific directory or FIFO? Or both? Also, how does this play with output modules for databases and hdfs, without more granular payload parsing, wouldn't I be restricted to text matching operations at query time or after? I would like to avoid that.

Keith
orekdm
New
 
Posts: 5
Joined: Fri Oct 04, 2013 7:47 pm

Re: Message parsers in Perl or Python?

Postby orekdm » Fri Feb 07, 2014 7:38 pm

Rainer, I just wanted to circle back and thank you for the wonderful announcement about external-plugins. Christmas came late this year, but this present is the best! I will be upgrading and testing this functionality with a perl plugin for NetScreen firewall parsing starting today.

Thanks again,

Keith
orekdm
New
 
Posts: 5
Joined: Fri Oct 04, 2013 7:47 pm

Re: Message parsers in Perl or Python?

Postby dlang » Sat Feb 08, 2014 3:03 am

we have other thoughts in mind that will allow for other module types in other languages, but that will be more work.

David Lang
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: Message parsers in Perl or Python?

Postby rgerhards » Sat Feb 08, 2014 3:12 pm

Hi Keith,

thanks for the feedback. As you have seen, this is a pretty new project. We have just started with the output modules, but will work toward all kinds of plugins. So you can't write the parser right at this moment, HOWEVER, this will become available hopefully within the very next weeks. Please let's keep connected, I would really like to have a practical use case when I craft that functionality.

Rainer
rgerhards
Site Admin
 
Posts: 3807
Joined: Thu Feb 13, 2003 11:57 am

Google Ads



Return to Developer's Corner

Who is online

Users browsing this forum: No registered users and 1 guest

cron