MonitorWare Knowledge Base

Ok, I'm logging into a database the various messages.

However, the MARK messages from various boxes sometimes look funny.
As an example

| oceanic | -- | MARK -|
| winggundam | | -- MARK -- |

In this case, the FromHost is either oceanic or winggundam, and the SysLogTag from the remote machine ( running rsyslog as well ) is being filled with "--" instead of being blank.

While I don't really care about the MARK messages, I like to know that a machine is still logging and checking every 20 minutes is nice to make sure, however, the way they are being parsed/inserted or even sent seems to be causing a minor bug.

--Edit

I'm running 1.12.0

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

This looks indeed oddly. I'll have a look at it and post the outcome/questions.

Rainer

OK, I could reproduce this. I am working on a fix. Looks indeed quite trivial, I think I can get you one later today. Thanks for alerting me

Rainer

OK, quick fix. open syslogd.c in editor, search for function "logmsgInternal". There, you need to add line 3 below:

Code: Select all

       if(MsgSetRawMsg(pMsg, msg) != 0) return;
        if(MsgSetHOSTNAME(pMsg, LocalHostName) != 0) return;


        if(MsgSetTAG(pMsg, "rsyslogd:") != 0) return;


        pMsg->iFacility = LOG_FAC(pri);
        pMsg->iSeverity = LOG_PRI(pri);


I will do some other cleanup and then upload that to CVS, but that will be later today or even tomorrow.

Rainer

The patch is now in CVS at sourceforge.net. I've done some cleanup (else the text "rsyslog:" appears twice in some messages. I'll release a new source tarball most probably next week.

Rainer

Cool, thanks for the quick turnaround. I'll apply the code changes and recompile and redeploy this.

Have you been able to load test this with a mysql database backend?

Things I'm looking for ( I have not had time to load test this yet either ), is if the database dies, do you queue the message until the database comes back up again?

Also, off the top of your head ( no need to go looking it up ) what is the difference between say the stock syslog format and the rsyslog format.

What I have noticed is that machines using syslog to send the message get their SyslogTag put into the message column.

If you don't know off the top of your head, no worries.

Thanks

Sorry, no load testing so far. I myself would be very interested in real-world feedback of a high volume environment. As of my understanding, the new code with dual threads should perform quite well, but with obvious room for (maybe dramatic improvement). Just a thought...

No queueing is done if the database dies.

The message difference depends on which stock syslogd you look at Rsyslog basically supports RFC 3164 format. The main difference to BSD syslogd is that the host name is included in rsyslog messages (and many others), but not in BSD. This leads to the problem you see. The good news is that I have tackled this just this week There now is code that in almost all cases correctly guesses if a host is present or not. Get the current cvs snapshot via anonymous cvs from sourceforge.net, it has these fixes. I'll post an updated source tarball most probably next week.

Rainer

Have you thought about or implemented a queing thread to handle the messages?

I have not looked at the code, but it might be a possible solution for what I personally need.

Right now we have we have 20 servers that are at some point going to go into production to replace the existing 20. These new 20 are set to remote log to the dual rsyslog cluster ( each 20 is split in half into 2 racks, and each rack will log to it's own rsyslog daemon remotely ). Then the former 20 will be added into the mix once they have been formatted and reconfigured.

I will end up doing some load testing, also we will start having some Cisco routers log to the rsyslog daemons as well, though they will end up passing through a load balancer in front of them.

Sadly, my big concern is losing logs because mysql shutsdown or does something for an unknown reason.

I'll take a look at your codebase and maybe I can add a queue thread to this and let it keep the messages if they couldn't be written to mysql. Then again, what happens if the rsyslog daemon dies?

Threading is a brand-new feature. It currently does the bare essentials. So there is no queing thread, although I, too, have thought about that. I think it should not be too hard to add one. Of course, you've also found the culpit: if rsyslog terminates for whatever reason (the machine itself might go offline), the queue is lost. Out of my head, one solution might be to write to text files and then feed these text files in parallel to the database. Of course, this is not trivial and no code at all currently exists for such.

Rainer

I'll have to have a look at your codebase in the coming few days.
Maybe a solution might be a collector of sorts, on both ends, that puts it in a flatfile db, and a transmitter that pulls it out of the db file and removes them when it receives an acknowledgement of sorts. However, this wouldn't be overly effective over UDP, but could be usefull for TCP based traffic instead. If the messages for syslog are that important, you wouldn't use UDP anyway .
I'm just shooting in the dark, probably better solutions, but I'll take a look and see what is workable for my needs, and bounce off of you if you want.

Yes, please let me know with whatever you come up. My long-term goal is to make rsyslog a very viable ressource for enterprise-class logging on *nix, and I am obviously interested in anything that helps toward that goal.

Let me also elaborate a little on the current threading approach. It actually is dual-threading. There is one thread which is accepting the messages (via a select server). This thread puts the messages into a fixed-size queue (size configurable via pre-processor #define, default 10,000). The second thread pull messages from that queue and processes them. It is the second thread that carries out the actions specified in the config file, e.g. the database writing. As such, there is some queing, but with only a single queue. The primary purpose of the current threading logic is that it should de-couple receiving and processing messages so that message bursts can be handled better. Then, it shall offer better performance on multi-processors including the variety of upcoming multi-core machines. Finally, I excpect some performance improvement even on single processors because the lenghty actions (database writing specifically) do not block the receiver. This is just so that you better understand what you see in code.

When it comes to message loss, please note that rsyslogs prime goal is to keep running itself and not hurt the system. This goal is more important than the goal to capture all messages. If in doubt, message loss is accepted in favour of server health. Please note that a blocking syslogd can lead to a defunctional system. This is because local logging is blocking too, so when syslogd blocks, all processes using syslog also block shortly thereafter. With the large number of important daemons using syslog, this means the whole system would become defunctional. Just like Cisco PIX stalls if a configured TCP syslogd can not be contacted.

I hope this information is useful for understanding the code base.

Rainer