Performance and SMP Kernel support

Questions around SyslogAppliance, the virtual logging appliance.

Google Ads


Performance and SMP Kernel support

Postby hypyke » Sat Nov 15, 2008 6:35 pm

I have 2 or 3 systems pointing to the appliance and they log pretty lightly. But it seems if I filter more than one system (IE. exclude it from the results) it pegs apache at 100% cpu and most times, times out. I can increase the timeout but it still has performance issues. I have the vmware package installed on a server with 2 quad cores and 8 gigs of ram. Memory does not seem to be the issue so I was wondering about assigning 2 cores to the appliance but it seems that the kernel was not compiled for SMP.

Is there another reason for the performance problems and if not can the next version support SMP?

Thanks.

Hy Pyke
hypyke
New
 
Posts: 1
Joined: Sat Nov 15, 2008 6:30 pm

Re: Performance and SMP Kernel support

Postby thrushwolf » Tue Nov 18, 2008 8:00 pm

I would be in the same boat as Hypyke with the exact same system (2 quad cores, 8GB of RAM) and i am having the same issues - except the logging is much more intense. I have a few mail gateway servers shooting logs off to the server plus dns and web servers so it's getting hit a lot, and whenever i try to search or seemingly at random times apache2 gets pegged @ 100%CPU Usage - which is frustrating as the page always times out.
SMP Kernel Support would be nice, and perhaps a more efficient http daemon (lighttpd IMO)
thrushwolf
New
 
Posts: 7
Joined: Tue Nov 18, 2008 7:56 pm

Re: Performance and SMP Kernel support

Postby rgerhards » Wed Nov 19, 2008 9:50 am

Sorry, I did overlook the initial post. SMP support is a good idea, but I would first like to isolate the root cause of the performance issue. It looks like this is somewhere in phpLogCon, so I will talk to Andre, the lead developer there.

Did you increase the memory assigned to the VM? I kept it low, maybe too low. Even if it is too low, it should not account for 100% CPU, but maybe it is worth a try.

Many thanks for your feedback, I will keep you posted.

Rainer
rgerhards
Site Admin
 
Posts: 3807
Joined: Thu Feb 13, 2003 11:57 am

Re: Performance and SMP Kernel support

Postby alorbach » Wed Nov 19, 2008 1:36 pm

I am currently reproducing this problem with some filters, and analysing the process.
I see the apache service assuming lots of CPU, and something is going wrong the SQL queries.

This seems to be a filtering bug in phplogcon introduced in one of the last versions. I am looking into this right now.

--
regards,
Andre
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Re: Performance and SMP Kernel support

Postby alorbach » Wed Nov 19, 2008 1:53 pm

I found and fixed the bug in phplogcon, to fix it manually please edit the file "classes/logstreamdb.php"

Search for this code:
Code: Select all
      // Copy rows into the buffer!
      $iBegin = $this->_currentRecordNum;
      while ($myRow = mysql_fetch_array($myquery,  MYSQL_ASSOC))
      {
         $this->bufferedRecords[$iBegin] = $myRow;
         $iBegin++;
      }

Directly after, add this code:
Code: Select all
      // --- Check if results were found
      if ( $iBegin == $this->_currentRecordNum )
         return ERROR_NOMORERECORDS;
      // ---


This bug will be fixed in the next phpLogCon release (most likely this will be v2.5.20)

best regards,
Andre Lorbach
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Re: Performance and SMP Kernel support

Postby rgerhards » Wed Nov 19, 2008 1:55 pm

I'll update the appliance as soon as some additonal testing is conducated. Feedback would be appreciated.

Rainer
rgerhards
Site Admin
 
Posts: 3807
Joined: Thu Feb 13, 2003 11:57 am

Re: Performance and SMP Kernel support

Postby alorbach » Wed Nov 19, 2008 2:32 pm

Little update, fixed the issue and release phpLogCon 2.5.20, download here:
http://www.phplogcon.org/Downloads-req- ... d-47.phtml

You can manually update your syslogappliance installations with this release, just copy all files over your existing installation in /var/www/

best regards,
Andre Lorbach
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Re: Performance and SMP Kernel support

Postby thrushwolf » Wed Nov 19, 2008 2:41 pm

Thanks Guys!
That little piece of code seems to have fixed my issue! I had in fact increased the RAM to 512MB just for fun but the issue was persistent.
As soon as i added that snippet of code i can now login (i couldnt even login before without it timing out) and searching is actually working.
Thanks for the fast response - much appreciated!
*edit*
As an edit, my CPU Load Averages are still pretty high, and when searching for a string with the page refresh enabled it does time out when trying to refresh - but at least it works i guess. My script timeout is set to 45 seconds yet it still times out, just an FYI.
thrushwolf
New
 
Posts: 7
Joined: Tue Nov 18, 2008 7:56 pm

Re: Performance and SMP Kernel support

Postby thrushwolf » Wed Nov 19, 2008 3:12 pm

And as an afterthought - phplogcon still times out when im on the default view trying to refresh @ 60 second intervals, and CPU is still pegged at 100 percent when this is happening (apache again)
thrushwolf
New
 
Posts: 7
Joined: Tue Nov 18, 2008 7:56 pm

Re: Performance and SMP Kernel support

Postby alorbach » Wed Nov 19, 2008 3:43 pm

When it doesn't timeout, can you tell me how many "DB queries" it shows at the bottom of the page in phplogcon?
Edit: Another question, do you have the option "Enable Row Counting" in your logstream source configuration enabled? This option can cause performance problems on larger databases
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Re: Performance and SMP Kernel support

Postby thrushwolf » Wed Nov 19, 2008 5:20 pm

When it does not time out (IE when it is normally working) it uses 11 DB Queries.
I am using the Appliance Version and Enable Row Counting is set to 'No'.
I can definitely re-create the issue when i attempt to use the search function. When i attempt to filter by anything, Apache goes to 100% and the script times out.
It has stopped happening (quite as often) when sitting on the index page watching the logs come in, but it still happens almost every time that i search or filter.
Not sure how many queries it attempts to open when it times out as i cant see the number being displayed.
thrushwolf
New
 
Posts: 7
Joined: Tue Nov 18, 2008 7:56 pm

Re: Performance and SMP Kernel support

Postby alorbach » Wed Nov 19, 2008 5:32 pm

Just to make sure, have you updated your phplogcon installation to Version 2.5.20 already?
I have about 500000 records now in my vmware syslogappliance, and not expiriencing problems when using the filters - yet.
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Re: Performance and SMP Kernel support

Postby rgerhards » Wed Nov 19, 2008 5:32 pm

Andre,

is the full update required or just the fix you posted?

Rainer
rgerhards
Site Admin
 
Posts: 3807
Joined: Thu Feb 13, 2003 11:57 am

Re: Performance and SMP Kernel support

Postby alorbach » Wed Nov 19, 2008 5:35 pm

I fixed this in 2 logstream classes, logstreamdb and logstreampdo, so I recommend to use the full update.

regards,
Andre Lorbach
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Re: Performance and SMP Kernel support

Postby thrushwolf » Wed Nov 19, 2008 6:23 pm

Ahh - i was using just the fix that you posted above.
Will replace with the new version and let you know if i run into any more issues.
Thanks!
thrushwolf
New
 
Posts: 7
Joined: Tue Nov 18, 2008 7:56 pm

Google Ads


Next

Return to SyslogAppliance

Who is online

Users browsing this forum: No registered users and 0 guests

cron