rsyslog not logging

General discussions here

Moderator: rgerhards

Google Ads


rsyslog not logging

Postby hokie1999 » Fri Jul 28, 2017 6:19 pm

Hello

I am running rsyslog 7.4.7 on CentOS7 on bare metal Dell which also has service Salt 2016.11.3. For a year rsyslog has worked by logging entries to /var/log/messages

A day ago, ran this command:

sudo salt "Servername.com" cmd.run "python /home/xyz/scriptname.py --proto 17 --spt 47428 --sip 10.101.10.10" --dip 10.10.10.11 start 1500659026009 -sensor XYZ --stats --outfile filename runas=somename

rsyslog stopped logging. Running a 'logger "HI"' command did not log to /var/log/messages -- have set permissions to 600 and 777

Restarts of rsyslog service has failed. Have removed and touched messages file.

Ran salt command seen above on another similar server and rsyslog failed.

Have tried removing /var/lib/rsyslog/imjournal.state and restarting rsyslog and systemd-journald

There is a small amount of logs on rsyslog service start in messages after rsyslog restart:

Jul 28 16:48:18 CRPdgMUSsalt02 rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="18451" x-info="http://www.rsyslog.com"] start
Jul 28 16:48:18 CRPdgMUSsalt02 rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
Jul


Nothing appears after this. /var/log/messages should be filling continuously.

No disk utilization issues. Everything else working on servers.

Any thoughts on what is causing this failure? Many thanks!

:)
hokie1999
New
 
Posts: 4
Joined: Fri Jul 28, 2017 6:07 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: rsyslog not logging

Postby dlang » Sat Jul 29, 2017 10:37 am

does rsyslog write anything out to stdout/stderr when you try to start it?

you can also start it in debug mode and see if that shows anything interesting.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog not logging

Postby hokie1999 » Mon Jul 31, 2017 2:23 pm

Tried debug mode rsyslog -dn, lots of output, no errors, warnings, failures. Checked rsyslog -N 1 for a check of the config file, returned nothing. Restarting rsyslog produces no stdout. Perhaps we need to reboot server. Thanks for your suggestions, tho. :)
hokie1999
New
 
Posts: 4
Joined: Fri Jul 28, 2017 6:07 pm

Re: rsyslog not logging

Postby dlang » Tue Aug 01, 2017 1:39 am

does logging still stop when you are in debug mode? (we have had bugs in the past that do not show up in debug mode, indicating threading related problems)

do you have SELinux of AppArmor running on these systems? If so, it's possible that their permissions will not allow you to write to a file, even though the filesystem permissions look good (IIRC, -Z is the flag on ls to show you SELinux tags)

rsyslog creates output files if they don't exist, you don't need to touch them (and if you do touch them with the wrong permissions, it can cause problems), try just removing the files and restarting rsyslog.

enabling the impstats module is always a good thing to do, it gives a lot of information about the state of rsyslog.
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: rsyslog not logging

Postby hokie1999 » Tue Aug 01, 2017 5:14 pm

does logging still stop when you are in debug mode? (we have had bugs in the past that do not show up in debug mode, indicating threading related problems)

debug mode has no effect on logging; with or without debugging, nothing appears in /var/log/messages


do you have SELinux of AppArmor running on these systems? If so, it's possible that their permissions will not allow you to write to a file, even though the filesystem permissions look good (IIRC, -Z is the flag on ls to show you SELinux tags)

SELinux is disabled

rsyslog creates output files if they don't exist, you don't need to touch them (and if you do touch them with the wrong permissions, it can cause problems), try just removing the files and restarting rsyslog.

Removed /var/log/messages, did stop then start on rsyslog. Problem persists. Notice that permissions on /var/log/messages is 644; used to be 600 before this broke

enabling the impstats module is always a good thing to do, it gives a lot of information about the state of rsyslog.

Can look into this.
hokie1999
New
 
Posts: 4
Joined: Fri Jul 28, 2017 6:07 pm

Re: rsyslog not logging

Postby hokie1999 » Wed Aug 02, 2017 3:45 pm

When I replicated error on test device -- ran pcap related command then ran a stderr command via a python script -- saw this at prompt: '/var/log/messages Killed'
hokie1999
New
 
Posts: 4
Joined: Fri Jul 28, 2017 6:07 pm

Google Ads



Return to General

Who is online

Users browsing this forum: No registered users and 3 guests

cron