Page 1 of 1

Message parsers in Perl or Python?

PostPosted: Sat Nov 02, 2013 12:13 am
by orekdm
Greetings,

I've done some searching and read through the Message Parsers page, but haven't found anything that suggests it's possible to utilize scripting languages for parsers. Is there a little known method to accommodate this, or is it possibly a feature on the roadmap?

I have many different custom log formats to normalize (much beyond the headers) and I would prefer to manage that all within the rsyslog context.

Thanks,

Keith

Re: Message parsers in Perl or Python?

PostPosted: Mon Nov 04, 2013 10:03 am
by rgerhards
sorry, this is not possible. A prime reason is that it is too slow.

Re: Message parsers in Perl or Python?

PostPosted: Thu Nov 21, 2013 10:53 pm
by orekdm
Thank you for responding, I had just now noticed that you did so. While I understand the stance that the core message parsers need to be designed for speed, there are too many unique syslog message styles to not have dedicated parsers for the payload. What is the recommended best practice for forwarding specific message types to other parsers?

For instance, should I use a regex to match a specific pattern and have a template route that to a specific directory or FIFO? Or both? Also, how does this play with output modules for databases and hdfs, without more granular payload parsing, wouldn't I be restricted to text matching operations at query time or after? I would like to avoid that.

Keith

Re: Message parsers in Perl or Python?

PostPosted: Fri Feb 07, 2014 7:38 pm
by orekdm
Rainer, I just wanted to circle back and thank you for the wonderful announcement about external-plugins. Christmas came late this year, but this present is the best! I will be upgrading and testing this functionality with a perl plugin for NetScreen firewall parsing starting today.

Thanks again,

Keith

Re: Message parsers in Perl or Python?

PostPosted: Sat Feb 08, 2014 3:03 am
by dlang
we have other thoughts in mind that will allow for other module types in other languages, but that will be more work.

David Lang

Re: Message parsers in Perl or Python?

PostPosted: Sat Feb 08, 2014 3:12 pm
by rgerhards
Hi Keith,

thanks for the feedback. As you have seen, this is a pretty new project. We have just started with the output modules, but will work toward all kinds of plugins. So you can't write the parser right at this moment, HOWEVER, this will become available hopefully within the very next weeks. Please let's keep connected, I would really like to have a practical use case when I craft that functionality.

Rainer