MonitorWare Knowledge Base

Hi!

I wanted to announce that I'm interested in creating an output module for Postgres.

I've looked an ommysql.c and it doesn't seem to be that much of work. The only thing I'm unsure about: How to integrate that with the current config file format?

Use < instead of > as with mysql? *G*

Mentioning mySQL: As you, Rainer, said on the ML that you might be interested in a separate maintainer for ommysql: A friend of mine might be interested because rsyslog could be useful to his company and he likes mySQL very much (as opposed to me, as I don't like mySQL).

Regards,
sur5r

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

Hi,

this is *excellent* news!

... and you immediately hit the spot The current output module interface is missing some things, most notably in the config char area. But that must not be a show stopper and I would *really* like to have a postgres output module.

I am sure we can discuss on how to do that best with the current interface, and you already offered a solution. Let me get a little through my notes -- I barely remember I had written down an idea for a good interim solution.

And, yes, I am also interested in someone else looking at MySQL - this is not my core area of expertise and I'd prefer to have things done by experts

There are two things that must be considered:

a) stability of module interface
The current interface is somewhat unstable. I think I need to change the interface definition, so new versions of rsyslog would most probably force some work on your side. On the other hand, it's kinda cool to help define the interface

b) no immediate change
I'd like not to change the current packaging within the next month or so. The reason is that Fedora has selected it for F8, and I don't like to introduce changes very late in the process (until absolutely necessary). So I'd like to keep the ommysql for the time being but would of course immediately applaud another project that provides postgres!

Oh, yes, and even though this is absolutely off-topic: I will make a dream of my come true and visit a space shuttle launch. That eats up a "little bit" of my time, but I really couldn't resists when I had the chance - sorry for that ...

Keep the thoughts coming!
Rainer

rgerhards wrote:I am sure we can discuss on how to do that best with the current interface, and you already offered a solution. Let me get a little through my notes -- I barely remember I had written down an idea for a good interim solution.

Ok, as soon as a way for integrating postgres into the config files is found I will start. It feels like this is something to be done during the better part of a weekend, including some basic testing.

rgerhards wrote:And, yes, I am also interested in someone else looking at MySQL - this is not my core area of expertise and I'd prefer to have things done by experts

Ok, I'll ask him tonight.

rgerhards wrote:a) stability of module interface
The current interface is somewhat unstable. I think I need to change the interface definition, so new versions of rsyslog would most probably force some work on your side. On the other hand, it's kinda cool to help define the interface

I haven't had a look at the om-interface in detail yet.

rgerhards wrote:b) no immediate change
I'd like not to change the current packaging within the next month or so. The reason is that Fedora has selected it for F8, and I don't like to introduce changes very late in the process (until absolutely necessary). So I'd like to keep the ommysql for the time being but would of course immediately applaud another project that provides postgres!

So you want to keep it the way it is in current CVS (plugins/*) or the way it was before?
For the packaging I would definitely like to see ommysql and ompgsql being packaged with the main rsyslog package. IMHO it doesn't make sense to separate this as it simply creates the risk of having incompatible versions together, and, as packagers pointed out on the ML, make packaging more laborious.

rgerhards wrote:Oh, yes, and even though this is absolutely off-topic: I will make a dream of my come true and visit a space shuttle launch. That eats up a "little bit" of my time, but I really couldn't resists when I had the chance - sorry for that ...

Have fun then

Sorry it took my some time to get back to you. I have looked at my notes yesterday evening and modified the ommysql module this morning so that you have a template.

The idea is that a single character, as it is now, is not sufficient to handle all possible "action call cases". I think that was your root question, too. In the long to medium term, it is also not good practice to let the module itself decide it a config line is for it or not. Just think about an evil module, which may steal away other module's config lines (which may include passwords or such). So the basic idea is to move this module selection process up into the rsyslog core.

To do it immediately, this requires too many changes (especially given the fact that I will be away soon and we are still hunting for a bug). However, what we can do is make the modules mimic that behavior - and this is also quite easy to do.

The most important thing is that we do no longer relay on single characters. To distinguish old style from new style, however, I need to one last time introduce a special character. That is ":". If an action selector starts with it, the next characters sequence uniquely identifies the module. The module name must be terminated by another ":", whereafter the actual module parameters follow. In short: the syntax is ":modname:modparams".

In the actual case of the ommysql, the syntax is starting with 1.19.10:

*.* mmysql:dbserver,dbname,uid,pwd;format

Similarly, an ompostgres module could be called with

*.* mpostgres:dbserver,dbname,uid,pwd;format

[assuming that the parameter part is similar - but that's up to the module decision]

In order to not break existing configuration files, the ">" syntax must still be supported for ommysql. For new modules, usage of the single character syntax is not permitted (well, maybe its better to say not recommended ...).

The module should strip the :modname: part right at the beginning of its processing. The reason is that over time rsyslog itself will evaluate this part of the action selector and rsyslog will strip the module designation and only after that will call the module with the param part. So the module will, at some time in the future, only see that part. Of course, it requires changes to the module once this interface change happens. If the module is today programmed to drop the :modname: selector, this change will simply be removing that code (and may be a few tweaks). Thus it is highly recommended to do it that way.

I have made the relevant code change to ommysql. You can see the diff at:

http://rsyslog.cvs.sourceforge.net/rsys ... 1.2&r2=1.3

As you can see, that version is currently in cvs. It will be part of 1.19.10, which I will release early this week. I think this solution is a good one: it can be applied quickly, but is also something that is clean and can stay for quite a while.

Regarding whether or not to include third-party modules into the main tarball, I agree that for the time being it is probably the best to do so. Once the interface has really matured (and hopefully many more plugins are available), that will probably change. But we can think about it then ...

Regarding the directory structure, the new source distribution tarball has received sufficient review and should be fine both from the developers, users and packagers point of view. So I want to keep it the way that each plugin resides in the ./plugins directory, within its specific subdirectory and has its own makefile (which is referenced by the main makefile). As I said, over time this will change for practical reasons, but this time is not yet.

I hope this message, though late, is useful. Feedback is appreciated. If I have overlooked something in regard to the interface, let's discuss this, it is now a great time to do so.

Rainer

rgerhards wrote:Sorry it took my some time to get back to you. I have looked at my notes yesterday evening and modified the ommysql module this morning so that you have a template.

Thanks.

rgerhards wrote:The idea is that a single character, as it is now, is not sufficient to handle all possible "action call cases". I think that was your root question, too. In the long to medium term, it is also not good practice to let the module itself decide it a config line is for it or not. Just think about an evil module, which may steal away other module's config lines (which may include passwords or such). So the basic idea is to move this module selection process up into the rsyslog core.

I completeley agree.

rgerhards wrote:The most important thing is that we do no longer relay on single characters. To distinguish old style from new style, however, I need to one last time introduce a special character. That is ":". If an action selector starts with it, the next characters sequence uniquely identifies the module. The module name must be terminated by another ":", whereafter the actual module parameters follow. In short: the syntax is ":modname:modparams".

In the actual case of the ommysql, the syntax is starting with 1.19.10:

*.* mmysql:dbserver,dbname,uid,pwd;format

Similarly, an ompostgres module could be called with

*.* mpostgres:dbserver,dbname,uid,pwd;format

Ok, so I will start ompgsql (or ompostgres, which name do you prefer?) this way.

rgerhards wrote:I hope this message, though late, is useful. Feedback is appreciated. If I have overlooked something in regard to the interface, let's discuss this, it is now a great time to do so.

Of course it is useful
I will have a more detailed look on the module interface. If anything comes to my mind, I will post it here.

sur5r

sur5r wrote:Ok, so I will start ompgsql (or ompostgres, which name do you prefer?) this way.

You got it: ompgsql sounds much better

Rainer

Some kind of short progress report:

* Almost done
* ommysql.c:263 (eat indicator sequence) misses a "-1" or should be changed to strlen(), because sizeof includes the trailing \0. It ate the first byte of the hostname *g*
* StdDBFmt does not work for postgres because of different quoting. maybe OMs should bring their own default?

Regards,
sur5r

Something else I forgot in my previous post:

There is a function that formats a date for mySQL, but it's outside of ommysql. Maybe OMs should be able to bring their own special-data formatting functions?

sur5r

Hi,

I am just back from my space shuttle launch viewing trip I am catching up with what has happened in the mean time...

Regarding the formatting function, I have plans to include customer functions (also loadable). That would fit nicely. For the time being, we could add a formatting option - I'd not like to change the output module interface to cover a functionality that will be there via some other way in the not so distant future. Just let me know what you need.

And - thanks for all your work. I am looking forward to the postgres module! Maybe you can also share part of your experience using the interface, that would definitely be useful for me.

Rainer

Rainer

Hi,

i finally made some progress and it seems that i've got it working so far.

rgerhards wrote:Regarding the formatting function, I have plans to include customer functions (also loadable). That would fit nicely. For the time being, we could add a formatting option - I'd not like to change the output module interface to cover a functionality that will be there via some other way in the not so distant future. Just let me know what you need.

I've found a solution for this for the moment, please tell me whether it's ok the way I did it.

My patch (which I consider alpha quality ATM as I didn't had the time to test it thoroughly yet) does the following:

* add plugins/ompgsql/*
* extend configure.ac and Makefile.am
* msg.h
** struct msg_t: add pszTIMESTAMP_PgSQL, add pszRcvdAt_PgSQL, fix typo
* msg.c
** getTimeGenerated(): add handling of tplFmtPgSQLDate
** getTimeReported(): add handling of tplFmtPgSQLDate
* syslogd.h:
** add definition of formatTimestampToPgSQL()
* syslogd.c:
** rename StdDBFmt to StdMySQLFmt (maybe StdDBFmt should stay for some time)
** add formatTimestampToPgSQL()
** mainThread(): include StdPgSQLFmt, change StdDBFmt to StdMySQLFmt
* template.h:
** add tplFmtPgSQLDate to tplFormatTypes
* template.c:
** doOptions(): date-pgsql -> tplFmtPgSQLDate
** tplPrintList(): add tplFmtPgSQLDate to switch()
* plugins/ommysql/ommysql.c
** parseSelectorAct: change StdDbFmt to StdMySQLFmt
* a small try to update the manpages a bit

rgerhards wrote:And - thanks for all your work. I am looking forward to the postgres module! Maybe you can also share part of your experience using the interface, that would definitely be useful for me.

Now, for the experience (in no particular order):

* the BEGIN/CODESTART/END macros from module-template.h feel strange at first, but the advantage is obvious. (I've read your paper on enhancing portability w/header files)
* data formatting functions should (if they are OM-specific) be implemented in the OMs.
* the length of the output buffer for the date formatting functions has to be specified three times, something that can easily be overlooked. (of course I did... )
* the current msg_t might be a problem if externally developed OMs should be added

I will send the patch to the mailing list because the forum doesn't seem to support attachments.

Regards,
sur5r

This sounds like excellent news! Congrats. I am looking forward to the patch and I will reply in-depth when I have seen it. One thing, though: We can not change any of the existing options as that would break backward compatibility. But that shouldn't be much of a problem...

Rainer

Ok, I finally managed to post the patch to the list. I've made it a bit more conservative than announced above.
Namely, I didn't change the name of StdDBFmt and didn't change the manpages.
It's maybe better anyway, to make the patch only change the portions that it absolutely needs to change.

I'm looking forward to your reply.

Regards,
sur5r

Hi,

I would like to add to this thread that the postgres SQL support is now part of rsyslog. However, it currently is only available via anonymous CVS. It will be released as part of 1.20.0, hopefully later this week. We are doing some final cleanup on that release, but it looks very well. You may also want to read my blogpost on this topic:

A very great thanks to sur5r for creating this very useful piece of software!

rsyslog now does postgres sql

Rainer

Is the interface the same to the plugins regardless of database?
I guess what I'm asking is if this is rather abstracted in anyway?

Also, is queueing being planned?

quick reply, i am on the road with a pda.

the plugin interface is abstracted, but the config file sytax is different for different ds. by intension. want to support multiple dbs at the same time.

what exactky do you mean by queueing?

rainer