app using rsyslog for local file then send to remote server?

General discussions here

Moderator: rgerhards

Google Ads


app using rsyslog for local file then send to remote server?

Postby GhostRider2110 » Tue Mar 28, 2017 1:58 pm

I have a problem I'm trying to solve.

RHEL 7 rsyslogd 7.4.7

I have a configuration where an app is logging to a file via rsyslog with a .conf file as follows:

Code: Select all
$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
if ($msg contains "[ jupiter ]") then /var/opt/lrms/log/jupiter.log;JupiterFormat
if ($msg contains "[ jupiter ]") then ~

I have a Nagios Log server setup where I am trying to send the logs also to it. I modified the .conf to this:

Code: Select all
$template JupiterFormat,"%TIMESTAMP:::date-rfc3339% %msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n"
if ($msg contains "[ jupiter ]") then /var/opt/lrms/log/jupiter.log;JupiterFormat
#if ($msg contains "[ jupiter ]") then ~

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/opt/lrms/log/jupiter.log
$InputFileTag jupiter:
#$InputFileStateFile nls-state-var_opt_lrms_log_jupiter_log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == "jupiter" then @@iganagioslog:5583
if $programname == "jupiter" then ~


I cleared out the log file before restarting rsyslog and everything starts off fine. If rsyslog is restarted though, I get duplicate entries in the log file with another timestamp added on. If the app logs the original is logged, and then multiples start getting logged. I see this in the logs as each time a new timestamp is added and the log fills up. Example of log below starting correct and then progressively getting repeats.


Code: Select all
2017-03-27T14:27:26.464641-04:00   - [ jupiter ] - 0.0010020733 - 1d7de0e1-85c4-482d-99e3-4ec83b22bce1 - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2017-03-27T14:27:26.464833-04:00   - [ jupiter ] - 0.0010859966 - 1d7de0e1-85c4-482d-99e3-4ec83b22bce1 - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2017-03-27T14:27:33.538057-04:00   - [ jupiter ] - 7.0751628876 - 1d7de0e1-85c4-482d-99e3-4ec83b22bce1 - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75
2017-03-27T14:27:33.596202-04:00   - [ jupiter ] - 0.0001859665 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2017-03-27T14:27:33.596532-04:00   - [ jupiter ] - 0.0003421307 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2017-03-27T14:27:33.596727-04:00   - [ jupiter ] - 0.0004539490 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- Path: /dev/admin/jsi18n/ - jupiter.lib.middleware:63
2017-03-27T14:27:33.596950-04:00   - [ jupiter ] - 0.0005919933 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- User: admin - jupiter.lib.middleware:65
2017-03-27T14:27:33.597182-04:00   - [ jupiter ] - 0.0007100105 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- IP: 10.100.52.117 - jupiter.lib.middleware:67
2017-03-27T14:27:33.597399-04:00   - [ jupiter ] - 0.0007979870 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 - jupiter.lib.middleware:69
2017-03-27T14:27:33.597609-04:00   - [ jupiter ] - 0.0010039806 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2017-03-27T14:27:33.597829-04:00   - [ jupiter ] - 0.0010919571 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2017-03-27T14:27:33.600533-04:00   - [ jupiter ] - 0.0051729679 - a3e2da2f-ef16-4b4a-bf5e-c2bdd199448f - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75


### Added in forward to NLS and restart of rsyslog ####

2017-03-27T14:28:16.867303-04:00   2017-03-27T14:25:29.095371-04:00   - [ jupiter ] - 0.0002100468 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2017-03-27T14:28:16.867310-04:00   2017-03-27T14:25:29.095647-04:00   - [ jupiter ] - 0.0003750324 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2017-03-27T14:28:16.867314-04:00   2017-03-27T14:25:29.095836-04:00   - [ jupiter ] - 0.0004770756 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- Path: /dev/admin/jupiter/lrms_revision/1948166/ - jupiter.lib.middleware:63
2017-03-27T14:28:16.867317-04:00   2017-03-27T14:25:29.096052-04:00   - [ jupiter ] - 0.0006041527 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- User: admin - jupiter.lib.middleware:65
2017-03-27T14:28:16.867320-04:00   2017-03-27T14:25:29.096256-04:00   - [ jupiter ] - 0.0007221699 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- IP: 10.100.52.117 - jupiter.lib.middleware:67
2017-03-27T14:28:16.867324-04:00   2017-03-27T14:25:29.096453-04:00   - [ jupiter ] - 0.0008101463 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 - jupiter.lib.middleware:69
2017-03-27T14:28:16.867328-04:00   2017-03-27T14:25:29.096651-04:00   - [ jupiter ] - 0.0009069443 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2017-03-27T14:28:16.867331-04:00   2017-03-27T14:25:29.096847-04:00   - [ jupiter ] - 0.0010361671 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2017-03-27T14:28:16.867335-04:00   2017-03-27T14:25:36.133069-04:00   - [ jupiter ] - 7.0384359360 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75
2017-03-27T14:28:16.867580-04:00   2017-03-27T14:25:36.199134-04:00   - [ jupiter ] - 0.0002019405 - 17cee959-df92-44b5-9942-8d69ab48294a - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
.....
017-03-27T14:28:26.879920-04:00   2017-03-27T14:28:16.867303-04:00   2017-03-27T14:25:29.095371-04:00   - [ jupiter ] - 0.0002100468 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2017-03-27T14:28:26.879939-04:00   2017-03-27T14:28:16.867310-04:00   2017-03-27T14:25:29.095647-04:00   - [ jupiter ] - 0.0003750324 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2017-03-27T14:28:26.879945-04:00   2017-03-27T14:28:16.867314-04:00   2017-03-27T14:25:29.095836-04:00   - [ jupiter ] - 0.0004770756 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- Path: /dev/admin/jupiter/lrms_revision/1948166/ - jupiter.lib.middleware:63
2017-03-27T14:28:26.879949-04:00   2017-03-27T14:28:16.867317-04:00   2017-03-27T14:25:29.096052-04:00   - [ jupiter ] - 0.0006041527 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- User: admin - jupiter.lib.middleware:65
2017-03-27T14:28:26.879954-04:00   2017-03-27T14:28:16.867320-04:00   2017-03-27T14:25:29.096256-04:00   - [ jupiter ] - 0.0007221699 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- IP: 10.100.52.117 - jupiter.lib.middleware:67
2017-03-27T14:28:26.879960-04:00   2017-03-27T14:28:16.867324-04:00   2017-03-27T14:25:29.096453-04:00   - [ jupiter ] - 0.0008101463 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 - jupiter.lib.middleware:69
2017-03-27T14:28:26.879964-04:00   2017-03-27T14:28:16.867328-04:00   2017-03-27T14:25:29.096651-04:00   - [ jupiter ] - 0.0009069443 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- Parameters: <QueryDict: {}> - jupiter.lib.middleware:71
2017-03-27T14:28:26.879969-04:00   2017-03-27T14:28:16.867331-04:00   2017-03-27T14:25:29.096847-04:00   - [ jupiter ] - 0.0010361671 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- ------------------------------------------------------------ - jupiter.lib.middleware:72
2017-03-27T14:28:26.879974-04:00   2017-03-27T14:28:16.867335-04:00   2017-03-27T14:25:36.133069-04:00   - [ jupiter ] - 7.0384359360 - 0757e896-dc05-4bd9-9c18-c42ab125dafc - site:dev -     INFO -- Finished processing request - jupiter.lib.middleware:75
2017-03-27T14:28:26.879979-04:00   2017-03-27T14:28:16.867580-04:00   2017-03-27T14:25:36.199134-04:00   - [ jupiter ] - 0.0002019405 - 17cee959-df92-44b5-9942-8d69ab48294a - site:dev -     INFO -- ============================================================ - jupiter.lib.middleware:59
2017-03-27T14:28:26.879984-04:00   2017-03-27T14:28:16.867590-04:00   2017-03-27T14:25:36.199435-04:00   - [ jupiter ] - 0.0003769398 - 17cee959-df92-44b5-9942-8d69ab48294a - site:dev -     INFO -- Received GET request. - jupiter.lib.middleware:61
2017-03-27T14:28:26.879987-04:00   2017-03-27T14:28:16.867593-04:00   2017-03-27T14:25:36.199631-04:00   - [ jupiter ] - 0.0004949570 - 17cee959-df92-44b5-9942-8d69ab48294a - site:dev -     INFO -- Path: /dev/admin/jsi18n/ - jupiter.lib.middleware:63
2017-03-27T14:28:26.879991-04:00   2017-03-27T14:28:16.867596-04:00   2017-03-27T14:25:36.199877-04:00   - [ jupiter ] - 0.0006239414 - 17cee959-df92-44b5-9942-8d69ab48294a - site:dev -     INFO -- User: admin - jupiter.lib.middleware:65
2017-03-27T14:28:26.879996-04:00   2017-03-27T14:28:16.867599-04:00   2017-03-27T14:25:36.200089-04:00   - [ jupiter ] - 0.0007510185 - 17cee959-df92-44b5-9942-8d69ab48294a - site:dev -     INFO -- IP: 10.100.52.117 - jupiter.lib.middleware:67
2017-03-27T14:28:26.880001-04:00   2017-03-27T14:28:16.867613-04:00   2017-03-27T14:25:36.200275-04:00   - [ jupiter ] - 0.0008380413 - 17cee959-df92-44b5-9942-8d69ab48294a - site:dev -     INFO -- User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36 - jupiter.lib.middleware:69


Am I just trying to do something that can't be done? Because of the first entry in the config when rsyslog processes the lines form the file to sent to the log server, they get reprocessed into the local file?

Thanks
Mitch
GhostRider2110
New
 
Posts: 2
Joined: Tue Mar 28, 2017 1:43 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: app using rsyslog for local file then send to remote ser

Postby dlang » Wed Mar 29, 2017 3:56 am

having rsyslog write logs to a file and at the same time read logs from that file is a crazy thing to do. At best you have the potential of an endless loop.

Can you back up a step and explain what you are trying to do?
dlang
Frequent Poster
 
Posts: 1001
Joined: Mon Sep 15, 2008 7:44 am

Re: app using rsyslog for local file then send to remote ser

Postby GhostRider2110 » Wed Mar 29, 2017 1:11 pm

dlang wrote:having rsyslog write logs to a file and at the same time read logs from that file is a crazy thing to do. At best you have the potential of an endless loop.

Can you back up a step and explain what you are trying to do?


Thanks. We have a python app that uses syslog for it's logging. Why the developers didn't just send it directly to a file, I don't know since all the other apps from this vendor do send logs direct to files.

I'm trying to get all our app logs sent to our nagios log server, but I still need to have the logs sent to a file for quick debugging.

So I need to figure out how to get them to a file and sent to our NLS.

Thanks again for the reply.

Mitch
GhostRider2110
New
 
Posts: 2
Joined: Tue Mar 28, 2017 1:43 pm

Google Ads



Return to General

Who is online

Users browsing this forum: No registered users and 2 guests

cron