rsyslog stops logging on remote logserver service reload

General discussions here

Moderator: rgerhards

Google Ads


rsyslog stops logging on remote logserver service reload

Postby jjasen » Mon Jul 10, 2017 3:24 am

I have multiple servers running stock CentOS 7 rsyslog 7.4.7-16.el7,
which are configured to log locally and over TCP to a remote logserver,
also running stock CentOS 7 rsyslog. The remote server uses imptcp to
receive, and pretty basic rules to parse and commit to disk.

I have several systems that log prolifically, but periodically, they
stop soon after the remote log server HUPs (daily logrotate). Very soon
after they stop logging (completely, even to local files), the services
on these systems block, and our monitoring system starts alerting.
Restarting rsyslog on the clients proves ineffectual.

The situation may clear itself without intervention after 90 minutes to
several hours.

However, this does not happen on all client systems in a similar
situation (CentOS 7, large volume of constant log data); nor does it
happen daily.

Any ideas as to what's going on?

Thanks in advance.
jjasen
New
 
Posts: 3
Joined: Mon Jul 10, 2017 3:19 am

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: rsyslog stops logging on remote logserver service reload

Postby dlang » Mon Jul 10, 2017 3:56 am

it's really hard to be sure about what's happening without any details, but if the receiving system has a problem, other systems sending to it will queue messages. If those systems don't have an action queue on their network delivery, they will not be able to write locally either. Once the queues all fill up, rsyslog will not accept any additional messages from the system, and all sorts of bad things happen.

The key is to figure out what's going on on each of the different systems

the impstats module lets you get reports of each queue, but the big question is what's different about the network paths between the systems that are able to keep delivering logs and those that block.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog stops logging on remote logserver service reload

Postby jjasen » Mon Jul 10, 2017 2:46 pm

Afflicted and non-afflicted systems are running identical versions of rsyslog.

The central log server accepts TCP syslog streams (and UDP) from the majority of the clients after a HUP, perfectly fine. Just these particular systems, every couple of days, stop sending and eventually block on logging.

I didn't see anything immediately revealing from impstats on the central log server, that indicated failure -- or even, really, much of a drop in traffic.

I'm happy to explore details.
jjasen
New
 
Posts: 3
Joined: Mon Jul 10, 2017 3:19 am

Re: rsyslog stops logging on remote logserver service reload

Postby dlang » Mon Jul 10, 2017 8:52 pm

are the good and bad clients on the same subnet? or are they on different subnets?

you may also want to look at impstats on the clients.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog stops logging on remote logserver service reload

Postby jjasen » Tue Jul 11, 2017 3:50 am

There are good, low volume clients, on the same subnet as the troublesome (and the central log server), and there are high volume clients on the other side of a firewall (as was as low volume), who have had no issue.
jjasen
New
 
Posts: 3
Joined: Mon Jul 10, 2017 3:19 am

Google Ads



Return to General

Who is online

Users browsing this forum: No registered users and 2 guests

cron