Information: Forum is in read-only mode
For details and other support options see https://www.adiscon.com/news/support-forum-set-to-read-only-mode/

Events being throttled?

General discussions here

Moderator: alorbach

Google Ads


Events being throttled?

Postby jems » Wed Apr 09, 2014 8:16 pm

I'm running LogAnalyzer 3.6.5, on CentOS.
I got everything working well, but I notice I'm missing a significant number of events in LogAnalyzer.

I have a couple config lines to help me compare my results. In my /etc/rsyslog.conf I have a file capturing everything:

*.* /var/log/all.log

I then have my loganalyzer entry for mysql:

*.* :ommysql:127.0.0.1,Syslog,rsyslog,myPass

I have a few messages per second showing up in my /var/log/all.log file, but only 1 message per minute showing up in my LogAnalyzer

Is there a "1 minute" setting somewhere for data insertions into mysql? I couldn't find it in the web configuration pages, and it doesn't appear to be in the rsyslog.conf

Any help would be great.

Thanks!
jems
Avarage
 
Posts: 10
Joined: Sat Jul 23, 2011 12:35 am

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: Events being throttled?

Postby alorbach » Thu Apr 10, 2014 8:31 am

Are the messages you are missing repeated messages?
Loganalyzer has a setting to hide duplicated messages which is on by default.

best regards,
Andre Lorbach
alorbach
Site Admin
 
Posts: 1627
Joined: Thu Feb 13, 2003 11:55 am

Re: Events being throttled?

Postby jems » Thu Apr 10, 2014 4:24 pm

Hello, thanks for the reply!
Not necessarily duplicates.
But like clockwork, the messages-- whatever they are-- seem to be showing up in 1 minute intervals.
Also, if I log into mysql and check the Syslog/SystemEvents table, I see the same events-- 1 minute apart:

| 1907 | NULL | 2014-04-10 15:21:44 | 2014-04-10 15:21:44 | 17 | 4 |
| 1906 | NULL | 2014-04-10 15:20:43 | 2014-04-10 15:20:43 | 17 | 4 |
| 1905 | NULL | 2014-04-10 15:19:42 | 2014-04-10 08:19:39 | 0 | 4 |
| 1904 | NULL | 2014-04-10 15:18:41 | 2014-04-10 15:18:41 | 17 | 4 |
| 1903 | NULL | 2014-04-10 15:17:33 | 2014-04-10 15:17:33 | 17 | 4 |

But my /var/log/all.log file has tons more messages.
Could this be a mysql throttle? Seems like an odd option for mysql.. but that's what seems to be happening.

Thanks!
jems
Avarage
 
Posts: 10
Joined: Sat Jul 23, 2011 12:35 am

Re: Events being throttled?

Postby jems » Thu Apr 10, 2014 5:00 pm

After a little analysis, I found the problem. As I suspected, it was a rsyslog issue. I actually caused this problem myself when setting up email notifications from the rsyslog itself.

My goal was to have entries with regex matching, which would email when certain things occurred.
However, when setting this up, it was advised to throttle messages so thousands of emails aren't sent out. The rsyslog.conf file entries for this look like this:

# email sending
#$ActionMailSMTPServer 10.8.7.25
#$ActionMailFrom loganalyzer@foo.com
#$ActionMailTo jems@foo.com
#$template mailSubject,"Rsyslog Alert for %hostname%"
#$template mailBody,"%msg%"
#$ActionMailSubject mailSubject
#$ActionExecOnlyOnceEveryInterval 60
#:msg, regex, "bad thing . alert with this text match" :ommail:;mailBody

The line:
$ActionExecOnlyOnceEveryInterval 60

Was the culprit.

However, since then, I've found this which should help in that scenario:
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command), rate limited to 50 messages in 2 seconds
$SystemLogRateLimitInterval 2
$SystemLogRateLimitBurst 50

I'll go with this for now and see how it goes.

If anyone has ideas on how they've done this same sort of thing (emailing via rsyslog), please post.

Thanks!
jems
Avarage
 
Posts: 10
Joined: Sat Jul 23, 2011 12:35 am

Re: Events being throttled?

Postby rgerhards » Fri Apr 11, 2014 7:27 am

You were almost right. The problem is, that $ActionExecOnlyOnceInterval must be re-set. If not, it affects all following actions as well. So this would be the code to use:

# email sending
#$ActionMailSMTPServer 10.8.7.25
$ActionMailFrom loganalyzer@example.net
$ActionMailTo jems@example.net
$template mailSubject,"Rsyslog Alert for %hostname%"
$template mailBody,"%msg%"
$ActionMailSubject mailSubject
$ActionExecOnlyOnceEveryInterval 60
:msg, regex, "bad thing . alert with this text match" :ommail:;mailBody
$ActionExecOnlyOnceEveryInterval 0 # disable this feature

Everything else should work with this config. I have also updated the doc to contain this information and the samples to contain that re-set line. Thanks for pointing this out!

Rainer
rgerhards
Site Admin
 
Posts: 3807
Joined: Thu Feb 13, 2003 11:57 am

Re: Events being throttled?

Postby jems » Fri Apr 11, 2014 4:57 pm

Thanks for the tip!
I guess that makes sense, since it uses a "top down" processing.
jems
Avarage
 
Posts: 10
Joined: Sat Jul 23, 2011 12:35 am

Google Ads



Return to General

Who is online

Users browsing this forum: No registered users and 0 guests

cron