omelasticsearch plugin rsyslog dies?

This is the place for developers to discuss bugs, new features and everything else about code changes.

Google Ads


omelasticsearch plugin rsyslog dies?

Postby johwes » Tue Sep 03, 2013 2:09 pm

Hi there, i'm running :
rsyslogd 7.4.3, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: No
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
uuid support: Yes

and using the omelasticsearch plugin. However i've run into a problem when i restart elasticsearch.
My elasticsearch.conf looks like this

## BEGIN CONFIG FILE ##
module(load="omelasticsearch")
$PreserveFQDN on
#
# the template below will output a JSON like this:
# {"message":"test","host":"rgheorghe","severity":"6","date":"2012-05-10T10:17:38.045","tag":"test:"}
#$template customSchema,"{\"@message\":\"%msg:::json%\",\"@source_host\":\"%HOSTNAME:::json%\",\"@severity\":\"%syslogseverity%\",\"@timestamp\":\"%timereported:1:19:date-rfc3339%.%timereported:1:3:date-subseconds%+01:00\",\"@tags\":\"%syslogtag:::json%\"}"
$template customSchema,"{\"@message\":\"%msg:::json%\",\"@source_host\":\"%HOSTNAME:::json%\",\"@severity\":\"%syslogseverity%\",\"@timestamp\":\"%timereported:::date-rfc3339%\",\"@tags\":\"%syslogtag:::json%\"}"
$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"

#
#the template below outputs something like "2012-05-10" to have our variable index names
#$template srchidx,"logstash-%timereported:1:10:date-rfc3339%"
$template srchidx,"logstash-%timereported:1:4:date-rfc3339%.%timereported:6:7:date-rfc3339%.%timereported:9:10:date-rfc3339%"

#
#now we put everything together
# "template" is for storing the syslog fields we want
# dynSearchIndex="on" is for having variable index names
# searchIndex is for letting rsyslog know where to get these names
*.* action(type="omelasticsearch"
searchType="syslog"
template="customSchema"
searchIndex="srchidx"
dynSearchIndex="on"
server="127.0.0.1"
bulkmode="on"
queue.dequeuebatchsize="300"
queue.size="50000"
action.resumeretrycount="-1"
action.resumeinterval="60")

## END CONFIG FILE ##

Starting rsyslog while elasticsearch is running works fine. However if i restart the elasticsearch service i get this in the debug file:

## BEGIN DEBUG LOG
2668.696509931:7f9c6258d700: processBatch: batch of 1 elements must be processed
2668.696514124:7f9c6258d700: scriptExec: batch of 1 elements, active (nil), active[0]:1
2668.696517492:7f9c6258d700: ACTION 0x7f9c6b05fc40 [omelasticsearch:action(type="omelasticsearch" ...)]
2668.696525264:7f9c6258d700: RRRR: execAct [omelasticsearch]: batch of 1 elements, active (nil)
2668.696529812:7f9c6258d700: Called action(NotAllMark), processing batch[0] via 'omelasticsearch'
2668.696532975:7f9c6258d700: Called action(Batch), logging to omelasticsearch
2668.696546831:7f9c6258d700: submitBatch: enter, nElem 1
2668.696550173:7f9c6258d700: tryDoAction 0x7f9c6b05fc40, pnElem 1, nElem 1
2668.696553940:7f9c6258d700: omelasticsearch: beginTransaction
2668.696556936:7f9c6258d700: Action 0x7f9c6b05fc40 transitioned to state: itx
2668.696560009:7f9c6258d700: entering actionCalldoAction(), state: itx
2668.696565364:7f9c6258d700: omelasticsearch: result doAction: -2121 (bulkmode 1)
2668.696568203:7f9c6258d700: action 0x7f9c6b05fc40 call returned -2121
2668.696571111:7f9c6258d700: omelasticsearch: endTransaction init
2668.696575146:7f9c6258d700: omelasticsearch: endTransaction, batch: '{"index":{"_index": "logstash-2013.09.03","_type":"syslog"}}
{"@message":" pam_unix(runuser:session): session opened for user elasticsearch by root(uid=0)","@source_host":"test1.example.com","@severity":"6","@timestamp":"2013-09-03T14:51:08.696450+02:00","@tags":"runuser:"}
'
2668.696585152:7f9c6258d700: omelasticsearch: using REST URL: 'http://127.0.0.1:9200/_bulk?'
2668.696799512:7f9c6258d700: omelasticsearch: we are suspending ourselfs due to failure 7 of curl_easy_perform()
2668.696806807:7f9c6258d700: omelasticsearch: endTransaction done with -2007
2668.696809941:7f9c6258d700: Action 0x7f9c6b05fc40 transitioned to state: rtry
2668.696812513:7f9c6258d700: action ret RS_RET_SUSPENDED - retry full batch
2668.696815299:7f9c6258d700: tryDoAction 0x7f9c6b05fc40, pnElem 1, nElem 1
2668.696818336:7f9c6258d700: actionDoRetry: enter loop, iRetries=0
2668.696821149:7f9c6258d700: omelasticsearch: tryResume called
2668.696902046:7f9c6258d700: omelasticsearch: checkConn() curl_easy_perform() failed: Couldn't connect to server
2668.696917274:7f9c6258d700: actionDoRetry: action->tryResume returned -2007
2668.696920657:7f9c6258d700: actionDoRetry: check for max retries, iResumeRetryCount -1, iRetries 0
2668.709354729:7f9c6398f700: Message from UNIX socket: #0
2668.709401271:7f9c6398f700: main Q: qqueueAdd: entry added, size now log 1, phys 2 entries
2668.709407614:7f9c6398f700: main Q: EnqueueMsg advised worker start
2668.709412962:7f9c6398f700: --------imuxsock calling select, active file descriptors (max 0): 0
2728.752594225:7f9c6258d700: actionDoRetry: enter loop, iRetries=1
2728.752659907:7f9c6258d700: omelasticsearch: tryResume called

## END DEBUG LOG

If i now check for the rsyslog service using ps aux | grep rsys. it is no longer running.

Anything i've done wrong or can anyone else reproduce this?
Elasticsearch version: 0.90.3
cat /etc/system-release
CentOS release 6.4 (Final)
[root@test1 log]# uname -a
Linux test1.example.com 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28 17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
johwes
New
 
Posts: 3
Joined: Tue Sep 03, 2013 1:56 pm

Urgent Question?

  • Pulling out your Hair?
  • Wasting Time and Money?
  • Deadline Approaching?

Re: omelasticsearch plugin rsyslog dies?

Postby rgerhards » Tue Sep 03, 2013 3:41 pm

That's most probably a bug fixed in 7.4.4 (currently being released).
rgerhards
Site Admin
 
Posts: 3806
Joined: Thu Feb 13, 2003 11:57 am

Re: omelasticsearch plugin rsyslog dies?

Postby johwes » Wed Sep 04, 2013 3:30 pm

Full of awsome!
Thanks for awsome response time!! :)

I'll try new version once released and get back here.

Thanks again!

/J
johwes
New
 
Posts: 3
Joined: Tue Sep 03, 2013 1:56 pm

Re: omelasticsearch plugin rsyslog dies?

Postby johwes » Mon Sep 09, 2013 12:15 pm

Well i can't reproduce this anymore when running on 7.4.4 so i'm thinking it is fixed :)

Thanks again.
And rsyslog rocks btw :)
Keep up the awsome work!

/J
johwes
New
 
Posts: 3
Joined: Tue Sep 03, 2013 1:56 pm

Google Ads



Return to Developer's Corner

Who is online

Users browsing this forum: No registered users and 0 guests

cron