rsyslogd (relay) message queues

This is the place for you, if you got rsyslog up and running but wonder how to make it do what you want.

Moderator: rgerhards

Google Ads


rsyslogd (relay) message queues

Postby cybersec » Wed Oct 18, 2017 10:22 pm

Hi,

Newbie question...

So I have stood-up a rsyslogd (relay) server (7.4.7). Been reading about message queues and was after some pointers on how the configuration should look likes based on the following setup:

1) We will be sending syslog events from around 200 devices (mainly CentOS servers but also network devices)
2) We will be sending mostly using TLS (where supported by network devices of course)
3) We will be receiving events and forwarding to multiple (possibly 5 destinations). No requirement to write logs locally on the rsyslogd (relay) server.
4) We will be expecting (based on legacy logging setup statistics) event received by the rsyslogd (relay) server to be in the tune of 3 million events in a 10 minute period (600 seconds). Not sure the percentage of these logs that will forwarded to each of the possible 5 destinations.
5) MaxMessageSize is set to default but requirement to increase this to 10K (I know, against the spirit of the RFC!)
6) I do not want to loose any message if the destinations are not available.

I'm particularly interested in how I should configure the main message and action queues (one action queue for each of the 5 possible destinations) and any (from experience) design considerations that I may need to be aware of with using queues. Thoughts please.


Thanks!
cybersec
New
 
Posts: 8
Joined: Thu Oct 12, 2017 11:01 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: rsyslogd (relay) message queues

Postby dlang » Wed Oct 18, 2017 10:37 pm

yes you will want one queue per remote destination, that queue needs to be large enough to handle the flow of logs until you get the destination back up (not that large if it's highly available, larger if it's not and you want to survive a holiday weekend), you will probably want disk assisted queues.

3m logs/600 seconds is 5k logs/sec. If this is your peak, rsyslog will be loafing along, if this is your average and your peak is 100x this, it will require careful attention to work well at peak :-)

There's nothing wrong with large messages, the RFC is suggesting the minimum size, not the max.

try configuring things and ask for help (ideally on the mailing list) as you run into problems or get confused. If you are looking for someone to write your configs for you, Adiscon offers professional services contracts to do this sort of work, the community support is from volunteers who are willing to help you, but not do your job for you :-)
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslogd (relay) message queues

Postby cybersec » Thu Oct 19, 2017 9:59 pm

Thanks for the design pointers dlang. I have enough information for now to continue the configuration myself. No doubt I will be back here with configs when/if I hit any challenges :-)
cybersec
New
 
Posts: 8
Joined: Thu Oct 12, 2017 11:01 pm

Google Ads



Return to Configuration

Who is online

Users browsing this forum: No registered users and 1 guest

cron