TLS limited at 17800 messages per second

This is the place for you, if you got rsyslog up and running but wonder how to make it do what you want.

Moderator: rgerhards

Google Ads


TLS limited at 17800 messages per second

Postby johannreinhard » Tue Nov 09, 2010 7:39 pm

Using RSyslog v5.7.1, RHEL, 8core Intel Servers

If I listen on 514 UDP using SERVER1 then forward via TLS to SERVER2 and then measure the UDP packet rate outbound from SERVER2, I'm only seeing 17800 messages per second when I know SERVER1 is receiving much higher. If I remove the TLS and just use UDP 514 for everything, I can send well over 20000 messages per second.
-UDP-> SERVER1 -TLS-> SERVER2 -UDP->

Is anyone else seeing this limitation? Could it be because the TLS thread can only use a single core?

Thanks,
J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Fri Nov 12, 2010 7:25 pm

Looks like it's depending on a single thread. A single CPU on the receiving server is getting pegged.

-J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Mon Nov 15, 2010 7:19 pm

I tried again with the same messages & config, but a different group of servers and only got 11,000 messages per second as the highest stable number. It seems to be highly dependent on a single core.

-J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Re: TLS limited at 17800 messages per second

Postby rgerhards » Mon Nov 15, 2010 8:49 pm

The current input uses a single thread. No way around. However, I am short of writing a new TLS input module, as the current one's design involves a number of other performance issues. Plus we have some stability issue that I can not reproduce and thus fix. It begins to look like rewriting is less effort than fixing.
rgerhards
Site Admin
 
Posts: 3795
Joined: Thu Feb 13, 2003 11:57 am

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Thu Nov 18, 2010 3:46 pm

rgerhards wrote:The current input uses a single thread. No way around. However, I am short of writing a new TLS input module, as the current one's design involves a number of other performance issues. Plus we have some stability issue that I can not reproduce and thus fix. It begins to look like rewriting is less effort than fixing.


Thanks for the reply. It seems like the server receiving the messages has a higher CPU load than the server sending the messages. Would there be a way for the receiving server to spawn a new thread for each new connection to decrypt it and then pump the data unencrypted to a single thread that merges the data with the rest of RSyslog?

-J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Thu Nov 18, 2010 11:44 pm

One more thing to consider regarding TLS in v5.7.1. Why does the receiving server require much more processing power than the sending server? Using identical servers the receiving server has a core that's 99% used while the sending server's top utilized core is only at 70%. These are brand new Opteron 6136 processors.

Thanks,
-J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Re: TLS limited at 17800 messages per second

Postby rgerhards » Wed Nov 24, 2010 2:51 pm

johannreinhard wrote:One more thing to consider regarding TLS in v5.7.1. Why does the receiving server require much more processing power than the sending server? Using identical servers the receiving server has a core that's 99% used while the sending server's top utilized core is only at 70%. These are brand new Opteron 6136 processors.


That's an interesting question, to which I have no answer right now.

However,I have thought about the situation in general. Do you expect only a single or multiple sender? With a single sender, there is obviously not so much I can do, but if we have multiple senders, I could see if I modify the input module to utilize multiple threads, but a connection will always need to be handled by one thread at a time.

Feedback is appreciated.

Rainer
rgerhards
Site Admin
 
Posts: 3795
Joined: Thu Feb 13, 2003 11:57 am

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Wed Nov 24, 2010 11:03 pm

rgerhards wrote:That's an interesting question, to which I have no answer right now.

However,I have thought about the situation in general. Do you expect only a single or multiple sender? With a single sender, there is obviously not so much I can do, but if we have multiple senders, I could see if I modify the input module to utilize multiple threads, but a connection will always need to be handled by one thread at a time.

Feedback is appreciated.

Rainer


Thanks Rainer.

We typically have multiple senders and a single receiver, so the multiple thread modification you described would work wonderfully. If you have some way to let rsyslog do some sort of round-robin load balancing, you could even improve server to server throughput by using multiple connections.

-J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Re: TLS limited at 17800 messages per second

Postby rgerhards » Thu Nov 25, 2010 7:22 am

johannreinhard wrote:We typically have multiple senders and a single receiver, so the multiple thread modification you described would work wonderfully. If you have some way to let rsyslog do some sort of round-robin load balancing, you could even improve server to server throughput by using multiple connections.


That kind of round-robin thing is what I have on my mind. In essence, I'd try to get event notifications on one thread and pass the actual read of the descriptor of to a worker pool. In detail, I'd try some other algorighm I have on my mind, which would work a bit differently in order to prevent the context switch (cache thrashing!) that would otherwise be inevitable.

But I currently do not have the time to setup a good enough test environment, so it would be very useful if you could try it out when I have reached a decent stage. If that's OK with you, I would probably be able to implement it within a week or so.

Rainer
rgerhards
Site Admin
 
Posts: 3795
Joined: Thu Feb 13, 2003 11:57 am

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Mon Nov 29, 2010 5:39 pm

rgerhards wrote:But I currently do not have the time to setup a good enough test environment, so it would be very useful if you could try it out when I have reached a decent stage. If that's OK with you, I would probably be able to implement it within a week or so.
Rainer


Absolutely! I'd be honored to test it.

-J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Fri Dec 10, 2010 3:14 pm

rgerhards wrote:I would probably be able to implement it within a week or so.
Rainer


Hi Rainer,

Do you have anything available for me to test?

Thanks,
J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Re: TLS limited at 17800 messages per second

Postby rgerhards » Mon Dec 13, 2010 9:49 am

sorry, John, I seem to have overlooked your initial reply :( I will try to get something done soon, but I am unfortunately not sure if I actually manage to do that before my xmas-break (as usual, a lot of late minute stuff came in...). I am really sorry for that. Please let me know if you would still be interested if the work extends into the second half of jan 2011...

Rainer
rgerhards
Site Admin
 
Posts: 3795
Joined: Thu Feb 13, 2003 11:57 am

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Wed Dec 15, 2010 11:44 pm

I'm currently using a hacked system of multiple daemons which results in data loss, so yes I will still be interested in Jan 2011! A load-balanced single daemon system with TLS support and multiple threads will be key to getting this running well.

-J

rgerhards wrote:sorry, John, I seem to have overlooked your initial reply :( I will try to get something done soon, but I am unfortunately not sure if I actually manage to do that before my xmas-break (as usual, a lot of late minute stuff came in...). I am really sorry for that. Please let me know if you would still be interested if the work extends into the second half of jan 2011...
Rainer
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Re: TLS limited at 17800 messages per second

Postby rgerhards » Mon Jan 24, 2011 5:28 pm

status update: I just found a method on how NOT to do it, see: ;)

http://blog.gerhards.net/2011/01/multi- ... input.html

But I am finally working on this issue (nicely fits into my current peek at performance) and will see that I create a working solution as well ;) The problem is that I probably need to refactor the old and quite complex existing code a bit.

Rainer
rgerhards
Site Admin
 
Posts: 3795
Joined: Thu Feb 13, 2003 11:57 am

Re: TLS limited at 17800 messages per second

Postby johannreinhard » Fri Jan 28, 2011 4:49 pm

Thanks Rainer. Seems like a complex problem to solve.

We might try throwing better hardware at the issue while you're researching. What would I need to do to get rsyslog running on Itanium?

-J
johannreinhard
Advanced
 
Posts: 33
Joined: Thu Oct 21, 2010 3:07 pm

Google Ads


Next

Return to Configuration

Who is online

Users browsing this forum: No registered users and 4 guests

cron