MonitorWare Knowledge Base

Ok, it's been a couple of months, but we went through a mail migration that didn't go as smoothly as I had hoped for.

One of the problems we encountered, was MailScanner would crash when syslog went down.

Ok, well, I started looking into it, and it seems that rsyslog would crash after running for awhile. As these were production servers and this started occuring regularly, I couldn't fully load test. However, I'd like to know what everyone else is running for load on their systems. As far as ours go, we have the following ( I can't give the number of messages generated in a day, sorry ).

Mail Servers running Postfix + MailScanner.
Mail Server Load - At the time, about 300 emails per minute, peak of 4000 emails per minute.

We reject in the order of about 4 million emails per 24 hour period at the door.
We deliver about 500,000 emails localy or to remote servers.

We have 12 servers handling email, the above stats are per server.

The MySQL server is hit in the order of 3,000 queries per second ( postfix is mysql driven in this case, all virtual users are in the MySQL database, as well as virtual domains, the transport table, etc ). The database also maintains around 7000 connections active.

Our maillog reaches almost a gig in size per 24 hour rotation.

So, what is everyone else running on and any issues?

I plan on running rsyslog again on a different system with a different MySQL database for performance reasons.

• Custom written rsyslog.conf?
• Maintenance Contract?
• Installation support?

I forgot to mention, that we were running rsyslog logging to a mysql database at the time, however, rsyslog on the servers would crash. At some point, well after the rsyslog on the servers would crash a couple of times, the receiver rsyslog would collapse. Not sure what the reasons were, nothing appeared in the logs, and rsyslog just disappeared.

Hi Omniplex,

while I obviously can not contribute to the load question, I am of course very interested in your findings and hope that others will post, too. Can you let me know which version of rsyslogd you were using on your machines. There have been some fixes over time, maybe I can correlate some of your experiences to a fix. It also looks like there is a issue with multi-threading on FreeBSD, even though I could not fully confirm that yet (and thus not address...).

Thanks,
Rainer

We were running 1.12.0 compiled on x86_64 Linux.
When I revisit this issue, which should be coming up in the next month or so ( We are also doing a site move as well ), I will use the latest code at that time.

If you find any fixes that the problems might be related to, let me know.

I'm also trying to find out what kind of load and environments other users are running rsyslog in as well to see if I'm over extending what it's capable of doing at the moment, or if I need to change some settings.

Thanks.

Ok, looks like I will be revisiting this issue in around the middle of this month.
Do you happen to have some sort of syslog load generator ( I could probably write one, but we are preparing for a physical relocation that is eating up time )?

I will have about 10 systems available to me so i can generate alot of messages and test rsyslog on each system as well as a single receiver.

Not sure if it will be stressful enough, but for starters you could probably try something like:

while true; do echo $RANDOM | logger; done

omniplex wrote:Ok, it's been a couple of months, but we went through a mail migration that didn't go as smoothly as I had hoped for.

One of the problems we encountered, was MailScanner would crash when syslog went down.

Ok, well, I started looking into it, and it seems that rsyslog would crash after running for awhile. As these were production servers and this started occuring regularly, I couldn't fully load test. However, I'd like to know what everyone else is running for load on their systems. As far as ours go, we have the following ( I can't give the number of messages generated in a day, sorry ).

Mail Servers running Postfix + MailScanner.
Mail Server Load - At the time, about 300 emails per minute, peak of 4000 emails per minute.

We reject in the order of about 4 million emails per 24 hour period at the door.
We deliver about 500,000 emails localy or to remote servers.

We have 12 servers handling email, the above stats are per server.

The MySQL server is hit in the order of 3,000 queries per second ( postfix is mysql driven in this case, all virtual users are in the MySQL database, as well as virtual domains, the transport table, etc ). The database also maintains around 7000 connections active.

Our maillog reaches almost a gig in size per 24 hour rotation.

So, what is everyone else running on and any issues?

I plan on running rsyslog again on a different system with a different MySQL database for performance reasons.

I am running rsyslog 1.12.2 on two machines, doing like 3000 inserts (combined), and logging over 25 G of data per day.

Machine #1:
hw.machine: i386
hw.model: Intel(R) Xeon(TM) CPU 2.80GHz
hw.ncpu: 2
hw.physmem: 3211718656

Machine #2:
hw.machine: amd64
hw.model: AMD Opteron(tm) Processor 244
hw.ncpu: 2
hw.physmem: 3209990144

Well i can give a few numbers in case this issue interests anyone still.
I have moved on to another company and will be looking at other items and possibly using rsyslog there.
On average we were getting about 4500 queries per second for logging.
However, that could spike to about 15,000 inserts per second.

rsyslog would bog down or crash for an unknown reason just shortly after peak load.
The systems were dual Xeon 3.4Ghz with HT running 64bit Fedora. ( I think it was 3.4Ghz anyway )
FC listed 4 processors ( HT was turned on )
Each machine had 4 gigs of RAM and the hard drive was a raid 5 configuration.
MySQL would be available the entire time ( users could pop and receive emails ), so it wasn't a mysql error.


In my new environment, syslog-ng is handling about 250k messages a minute on a dual 1ghz processor system with a Raid 5 and 2 gigs of RAM.

I will see how rsyslog handles this.

EDIT: Eash sending system can send at peak about 250k messages a minute and currently only one machine is sending.

Hi Omniplex,

glad to hear again from you. And, yes, I still would be most interested in real performance data from the field.

omniplex wrote:rsyslog would bog down or crash for an unknown reason just shortly after peak load.

That was probably related to a bug we are hunting since summer, but that seems to have been present all time since multithreading was introduced. It causes a segfault and is extremely hard to hunt down, as it does not show up in the lab. HOWEVER, it looks like we finally got hold of it (but it is too soon to say for sure). If so, it was a race condition and traffic spikes could definitely trigger it.

Rsyslog has been pretty much evolved since the 1.12.x days. I am right now working on v3, and a very early first preview is available at

http://download.rsyslog.com/rsyslog/rsy ... pre.tar.gz

The file changes from time to time and it always is just a snapshot with me continuing development without thinking about the snapshot (and its bug...) for more than a minute.

I would really be interested in performance feedback about it. The doc, however, is ...ummm... sparse But you can always ask.

So far, I'd not recommend v3 on a production box.

Rainer